1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-26 08:42:41 +01:00
Commit graph

3104 commits

Author SHA1 Message Date
epriestley
ab0ac7f61b Remove very old "owners-default-path" code from Owners
Summary: Ref T12590. This is ancient code which was used to prefill `/trunk/tfb/www/` or similar at Facebook. I don't think it ever had a UI and no install has asked for this feature since 2011.

Test Plan: Grepped for affected symbols, edited paths in Owners.

Maniphest Tasks: T12590

Differential Revision: https://secure.phabricator.com/D19189
2018-03-07 18:25:27 -08:00
epriestley
229d467770 Restore lightbox behavior for thumbnailed images
Summary: Ref T13099. See <https://discourse.phabricator-community.org/t/lightbox-not-working/1201>. The Content-Security-Policy changes rewrote some of this code and the handling for "Download" links is incorrectly catching clicks on thumbnailed images.

Test Plan: Clicked a thumbnailed image, got a lightbox. Command-clicked a download link, still got link behavior instead of a lightbox.

Maniphest Tasks: T13099

Differential Revision: https://secure.phabricator.com/D19188
2018-03-07 07:33:43 -08:00
epriestley
df1e9ce646 Treat Owners paths like "/src/backend" and "/src/backend/" identically
Summary:
Depends on D19183. Ref T11015. Currently, adding a trailing slash works great and omitting it mysteriously doesn't work.

Store a normalized version with an unconditional trailing slash for the lookup logic to operate on, and a separate display version which tracks what the user actually typed.

Test Plan:
  - Entered "/src/main.c", "/src/main.c/", saw them de-duplicate.
  - Entered "/src/main.c", saw it stay that way in the UI but appear as "/src/main.c/" internally.
  - Added a rule for "/src/applications/owners" (no slash), created a revision touching paths in that directory, saw Owners fire for it.
  - Changed the display value of a path only ("/src/main.c" to "/src/main.c/"), saw the update reflected in the UI without any beahvioral change.

Maniphest Tasks: T11015

Differential Revision: https://secure.phabricator.com/D19184
2018-03-06 20:31:46 -08:00
epriestley
adde4089b4 Allow owners paths to be arbitrarily long and add storage for display paths
Summary:
Depends on D19182. Ref T11015. This changes `path` from `text255` to `longtext` because paths may be arbitrarily long.

It adds `pathDisplay` to prepare for display paths and storage paths having different values. For now, `pathDisplay` is copied from `path` and always has the same value.

Test Plan:
  - Ran migration, checked database for sanity (all `pathDisplay` and `path` values identical).
  - Added new paths, saw `pathDisplay` and `path` get the same values.
  - Added an unreasonably enormous path with far more than 255 characters.

Maniphest Tasks: T11015

Differential Revision: https://secure.phabricator.com/D19183
2018-03-06 20:31:22 -08:00
epriestley
8cb273a053 Add a unique key to OwnersPath on "<packageID, repositoryPHID, pathIndex>"
Summary:
Depends on D19181. Ref T11015. This nukes duplicates from the table if they exist, then adds a unique key.

(Duplicates should not exist and can not be added with any recent version of the web UI.)

Test Plan:
  - Tried to add duplicates with web UI, didn't have any luck.
  - Explicitly added duplicates with manual `INSERT`s.
  - Viewed packages in web UI and saw duplicates.
  - Ran migrations, got a clean purge and a nice unique key.
  - There's still no way to actually hit a duplicate key error in the UI (unless you can collide hashes, I suppose), this is purely a correctness/robustness change.

Maniphest Tasks: T11015

Differential Revision: https://secure.phabricator.com/D19182
2018-03-06 20:30:59 -08:00
epriestley
1bf4422c74 Add and populate a pathIndex column for OwnersPath
Summary: Ref T11015. This supports making path names arbitrarily long and putting a proper unique key on the table.

Test Plan:
  - Migrated, checked database, saw nice digested indexes.
  - Edited a package, saw new rows update with digested indexes.

Maniphest Tasks: T11015

Differential Revision: https://secure.phabricator.com/D19181
2018-03-06 20:30:33 -08:00
epriestley
dbccfb234f Perform a client-side redirect after OAuth server authorization
Summary:
Ref T13099. See that task for discussion. Chrome is unhappy with an MFA form submitting to an endpoint which redirects you to an OAuth URI.

Instead, do the redirect entirely on the client.

Chrome's rationale here isn't obvious, so we may be able to revert this at some point.

Test Plan: Went through the OAuth flow locally, was redirected on the client. Will verify in production.

Maniphest Tasks: T13099

Differential Revision: https://secure.phabricator.com/D19177
2018-03-06 12:18:27 -08:00
epriestley
743d1ac426 Mostly modularize the Differential "update" transaction
Summary: Ref T13099. Move most of the "Update" logic to modular transactions

Test Plan: Created and updated revisions. Flushed the task queue. Grepped for `TYPE_UPDATE`. Reviewed update transactions in the timeline and feed.

Maniphest Tasks: T13099

Differential Revision: https://secure.phabricator.com/D19175
2018-03-06 09:10:32 -08:00
epriestley
44f0664d2c Add a "lock log" for debugging where locks are being held
Summary: Depends on D19173. Ref T13096. Adds an optional, disabled-by-default lock log to make it easier to figure out what is acquiring and holding locks.

Test Plan: Ran `bin/lock log --enable`, `--disable`, `--name`, etc. Saw sensible-looking output with log enabled and daemons restarted. Saw no additional output with log disabled and daemons restarted.

Maniphest Tasks: T13096

Differential Revision: https://secure.phabricator.com/D19174
2018-03-05 17:55:34 -08:00
epriestley
1f40e50f7e Improve live Harbormaster log follow behaviors
Summary:
Depends on D19166. Ref T13088. When the user scrolls away from a followed log, break the focus lock.

Let users stop following a live log.

Show when lines are added more clearly.

Don't refresh quite as quickly give users a better shot at clicking the stop button.

These behaviors can probably be refined but are at least more plausible and less actively user-hostile than the first version of this behavior was.

Test Plan: Used `write-log --rate` to write a large log slowly. Clicked "Follow Log", followed for a bit. Scrolled away, still got live updates but no more scroll lock. Clicked stop, no more updates.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19167
2018-03-01 13:11:22 -08:00
epriestley
4e91ad276d Prevent copying Harbormaster build log line numbers with CSS psuedocontent instead of ZWS
Summary:
Depends on D19165. Ref T13088. Currently, in other applications, we use Zero Width Spaces and Javascript "copy" listeners to prevent line numbers from being copied. This isn't terribly elegant.

Modern browsers support a second approach: using psuedo-elements with `content`. Try this in Harbormaster since it's conceptually cleaner, at least. One immediate drawback is that Command-F can't find this text either.

Test Plan: In Safari, Chrome and Firefox, highlighted ranges of lines and copy/pasted text. Got just text (no line numbers) in all cases.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19166
2018-03-01 13:03:40 -08:00
epriestley
73619c4643 Share the Paste line highlighting behavior for Harbormaster build logs
Summary: Depends on D19164. Ref T13088. Now that the JS behaviors are generic, use them on the Harbormaster standalone page.

Test Plan: Clicked lines and dragged across line ranges. Reloaded pages. Saw expected highlighting behavior in the client and on the server across reloads.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19165
2018-03-01 12:57:30 -08:00
epriestley
fe3de5dd58 Make Paste source code line highlighting behavior more generic
Summary: Depends on D19163. Ref T13088. Increase the generality of this code so it can be shared with Harbormaster.

Test Plan: Clicked individual lines, clicked-and-dragged, etc., in Paste. Got sensible URI and highlight behaviors.

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19164
2018-03-01 12:46:36 -08:00
epriestley
49af4165bc Support rendering arbitrary sections in the middle of a Harbormaster build log so links to line 3500 work
Summary:
Depends on D19162. Ref T13088. When a user links to `$1234`, we need to render a default view of the log with a piece at the head, a piece at the end, and a piece in the middle.

We also need to figure out the offset for line 1234, or multiple offsets for "1234-2345".

Since the logic views/reads mostly anticipated this it isn't too much of a mess, although there are a couple of bugs this exposes with view specifications that use combinations of parameters which were previously impossible.

Test Plan: Viewed a large log with no line marker. Viewed `$1`. Viewed `$end`. Viewed `$35-40`, etc. Expanded context around logs.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19163
2018-03-01 11:18:21 -08:00
epriestley
a2fdf14275 Stop using forms to download files in file embed and lightbox elements
Summary: Depends on D19156. Ref T13094. This replaces the remaining forms in the file embed view and lightbox with normal download links.

Test Plan: Clicked "Download" and lightbox -> download for embedded files.

Maniphest Tasks: T13094

Differential Revision: https://secure.phabricator.com/D19157
2018-02-28 17:21:07 -08:00
epriestley
ab579f2511 Never generate file download forms which point to the CDN domain, tighten "form-action" CSP
Summary:
Depends on D19155. Ref T13094. Ref T4340.

We can't currently implement a strict `form-action 'self'` content security policy because some file downloads rely on a `<form />` which sometimes POSTs to the CDN domain.

Broadly, stop generating these forms. We just redirect instead, and show an interstitial confirm dialog if no CDN domain is configured. This makes the UX for installs with no CDN domain a little worse and the UX for everyone else better.

Then, implement the stricter Content-Security-Policy.

This also removes extra confirm dialogs for downloading Harbormaster build logs and data exports.

Test Plan:
  - Went through the plain data export, data export with bulk jobs, ssh key generation, calendar ICS download, Diffusion data, Paste data, Harbormaster log data, and normal file data download workflows with a CDN domain.
  - Went through all those workflows again without a CDN domain.
  - Grepped for affected symbols (`getCDNURI()`, `getDownloadURI()`).
  - Added an evil form to a page, tried to submit it, was rejected.
  - Went through the ReCaptcha and Stripe flows again to see if they're submitting any forms.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13094, T4340

Differential Revision: https://secure.phabricator.com/D19156
2018-02-28 17:20:12 -08:00
epriestley
f114b2dd7d When viewing a live build log, trap users in a small personal hell where nothing but slavish devotion to the log exists
Summary: Depends on D19152. Ref T13088. This adds live log tailing. It is probably not the final version of this feature because it prevents escape once you begin tailing a log.

Test Plan: Used `bin/harbormaster write-log --rate ...` to write a log slowly. Viewed it in the web UI. Clicked "Follow Log". Followed the log until the write finished, a lifetime later.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19153
2018-02-28 12:38:41 -08:00
epriestley
143033dc1f When showing a small piece of a Harbormaster build log, load a small piece of data instead of the entire log
Summary: Depends on D19148. Ref T13088. The new rendering always executes range requests for data it needs, and we can satisfy these requests by loading the smallest number of chunks which span that range.

Test Plan: Piped 50,000 lines of Apache log into Harbormaster, viewed it in the new UI, got sensible rendering times and a reasonable amount of data actually going over the wire.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19149
2018-02-28 12:32:26 -08:00
epriestley
dba4c4bdf6 Emit a "Content-Security-Policy" HTTP header
Summary:
See PHI399. Ref T4340. This header provides an additional layer of protection against various attacks, including XSS attacks which embed inline `<script ...>` or `onhover="..."` content into the document.

**style-src**: The "unsafe-inline" directive affects both `style="..."` and `<style>`. We use a lot of `style="..."`, some very legitimately, so we can't realistically get away from this any time soon. We only use one `<style>` (for monospaced font preferences) but can't disable `<style>` without disabling `style="..."`.

**img-src**: We use "data:" URIs to inline small images into CSS, and there's a significant performance benefit from doing this. There doesn't seem to be a way to allow "data" URIs in CSS without allowing them in the document itself.

**script-src** and **frame-src**: For a small number of flows (Recaptcha, Stripe) we embed external javascript, some of which embeds child elements (or additional resources) into the document. We now whitelist these narrowly on the respective pages.

This won't work with Quicksand, so I've blacklisted it for now.

**connect-src**: We need to include `'self'` for AJAX to work, and any websocket URIs.

**Clickjacking**: We now have three layers of protection:

  - X-Frame-Options: works in older browsers.
  - `frame-ancestors 'none'`: does the same thing.
  - Explicit framebust in JX.Stratcom after initialization: works in ancient IE.

We could probably drop the explicit framebust but it wasn't difficult to retain.

**script tags**: We previously used an inline `<script>` tag to start Javelin. I've moved this to `<data data-javelin-init ...>` tags, which seems to work properly.

**`__DEV__`**: We previously used an inline `<script>` tag to set the `__DEV__` mode flag. I tried using the "initialization" tags for this, but they fire too late. I moved it to `<html data-developer-mode="1">`, which seems OK everywhere.

**CSP Scope**: Only the CSP header on the original request appears to matter -- you can't refine the scope by emitting headers on CSS/JS. To reduce confusion, I disabled the headers on those response types. More headers could be disabled, although we're likely already deep in the land of diminishing returns.

**Initialization**: The initialization sequence has changed slightly. Previously, we waited for the <script> in bottom of the document to evaluate. Now, we go fishing for tags when domcontentready fires.

Test Plan:
  - Browsed around in Firefox, Safari and Chrome looking for console warnings. Interacted with various Javascript behaviors. Enabled Quicksand.
  - Disabled all the framebusting, launched a clickjacking attack, verified that each layer of protection is individually effective.
  - Verified that the XHProf iframe in Darkconsole and the PHPAST frame layout work properly.
  - Enabled notifications, verified no complaints about connecting to Aphlict.
  - Hit `__DEV__` mode warnings based on the new data attribute.
  - Tried to do sketchy stuff with `data:` URIs and SVGs. This works but doesn't seem to be able to do anything dangerous.
  - Went through the Stripe and Recaptcha workflows.
  - Dumped and examined the CSP headers with `curl`, etc.
  - Added a raw <script> tag to a page (as though I'd found an XSS attack), verified it was no longer executed.

Maniphest Tasks: T4340

Differential Revision: https://secure.phabricator.com/D19143
2018-02-27 10:17:30 -08:00
epriestley
f450c6c55b Fix some of the most egregious errors in Harbormaster log paging
Summary:
Depends on D19141. Ref T13088. Some of the fundamental log behaviors like "loading the correct rows" are now a bit better behaved.

The UI is a little less garbage, too.

Test Plan: Viewed some logs and loaded more context by clicking the buttons.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19142
2018-02-26 17:59:13 -08:00
epriestley
11d1dc484b Sort of make Harbormaster build logs page properly
Summary: Depends on D19139. Ref T13088. This doesn't actually work, but is close enough that a skilled attacker might be able to briefly deceive a small child.

Test Plan:
  - Viewed some very small logs under very controlled conditions, saw content.
  - Larger logs vaguely do something resembling working correctly.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19141
2018-02-26 17:58:33 -08:00
epriestley
6dc341be87 As Harbormaster logs are processed, build a sparse map of byte offsets to line numbers
Summary:
Depends on D19138. Ref T13088. When we want to read the last part of a logfile //and show accurate line numbers//, we need to be able to get from byte offsets to line numbers somehow.

Our fundamental unit must remain byte offsets, because a test can emit an arbitrarily long line, and we should accommodate it cleanly if a test emits 2GB of the letter "A".

To support going from byte offsets to line numbers, compute a map with periodic line markers throughout the offsets of the file. From here, we can figure out the line numbers for arbitrary positions in the file with only a constant amount of work.

Test Plan: Added unit tests; ran unit tests.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19139
2018-02-26 17:56:52 -08:00
epriestley
d6311044bb Store the Harbormaster log chunk format on the log record
Summary: Depends on D19137. Ref T13088. This allows `rebuild-log` to skip work if the chunks are already compressed. It also prepares for a future GC which is looking for "text" or "gzip" chunks to throw away in favor of archival into Files; such a GC can use this column to find collectable logs and then write "file" to it, meaning "chunks are gone, this data is only available in Files".

Test Plan: Ran migration, saw logs populate as "text". Ran `rebuild-log`, saw logs rebuild as "gzip".

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19138
2018-02-26 17:56:14 -08:00
epriestley
57e3d607f5 In Harbormaster, record byte length on the build logs
Summary: Depends on D19135. Ref T13088. Denormalize the total log size onto the log itself. This makes reasoning about the log at display time easier, and we don't need to fish around in the database as much to figure out what we're dealing with.

Test Plan: Ran `bin/harbormaster rebuild-log`, saw an existing log populate. Ran `bin/harbormaster write-log`, saw new log write with proper length information.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19136
2018-02-26 17:54:47 -08:00
epriestley
8a2604cf06 Add a "filePHID" to HarbormasterBuildLog and copy logs into Files during finalization
Summary: Depends on D19131. Ref T13088. During log finalization, stream the log into Files to support "Download Log", archive to Files, and API access.

Test Plan: Ran `write-log` and `rebuild-log`, saw Files objects generate with log content and appropriate permissions.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19132
2018-02-26 17:52:39 -08:00
epriestley
4c7370a1a3 Make the filetree view width sticky across show/hide and reload
Summary:
Ref T13090. The default width changed recently to become much wider, but the behavior on this control isn't great. Instead:

  - Pick a default width somewhere between the two.
  - Make the width sticky across show/hide (pressing "f" twice remembers your width instead of resetting it).
  - Make the width sticky across reloads (dragging the bar, then reloading the page keeps the bar in the same place).

Test Plan:
  - Without settings, loaded page: got medium-width bar.
  - Dragged bar wide/narrow, toggled on/off with "f", got persistent width.
  - Dragged bar wide/narrow, reloaded page, got persistent width.
  - Dragged bar wide/narrow, toggled it off, reloaded page, toggled it on, got persistent width.

Maniphest Tasks: T13090

Differential Revision: https://secure.phabricator.com/D19129
2018-02-22 13:47:41 -08:00
epriestley
0dee34b3fa Make Facts more modern, DRY, and dimensional
Summary:
Ref T13083. Facts has a fair amount of weird hardcoding and duplication of responsibilities. Reduce this somewhat: no more hard-coded fact aggregates, no more database-driven list of available facts, etc. Generally, derive all objective truth from FactEngines. This is more similar to how most other modern applications work.

For clarity, hopefully: rename "FactSpec" to "Fact". Rename "RawFact" to "Datapoint".

Split the fairly optimistic "RawFact" table into an "IntDatapoint" table with less stuff in it, then dimension tables for the object PHIDs and key names. This is primarily aimed at reducing the row size of each datapoint. At the time I originally wrote this code we hadn't experimented much with storing similar data in multiple tables, but this is now more common and has worked well elsewhere (CustomFields, Edges, Ferret) so I don't anticipate this causing issues. If we need more complex or multidimension/multivalue tables later we can accommodate them. The queries a single table supports (like "all facts of all kinds in some time window") don't make any sense as far as I can tell and could likely be UNION ALL'd anyway.

Remove all the aggregation stuff for now, it's not really clear to me what this should look like.

Test Plan: Ran `bin/fact analyze` and viewed web UI. Nothing exploded too violently.

Subscribers: yelirekim

Maniphest Tasks: T13083

Differential Revision: https://secure.phabricator.com/D19119
2018-02-19 12:05:19 -08:00
epriestley
eb3fd2b7f5 Fix an issue with marking aborted buildables failed when more than one build is aborted
Summary: See <https://discourse.phabricator-community.org/t/upgrade-issue-2018-week-7-mid-february/1139>.

Test Plan: Used `bin/storage upgrade -f --apply ...` to re-apply the migration.

Differential Revision: https://secure.phabricator.com/D19116
2018-02-17 04:36:25 -08:00
epriestley
8796a6036e Let users escape more easily from the autosuggester after typing "[" or "("
Summary:
Ref T13077. The autosuggester is a little too eager right now, and will eat carriage returns after typing `[` if you never activate the tokenizer.

To fix this, try just canceling sooner. If that doesn't work, we might need to cancel more eagerly by testing to see if the tokenizer is actually open.

Test Plan: Typed `[x]<return>`, got my return instead of getting trapped by the autosuggester.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19110
2018-02-16 11:02:48 -08:00
epriestley
0202c36b62 Suggest Phurl URLs on "((..." in Remarkup text areas
Summary: Depends on D19108. Ref T12241. Ref T13077. See D19108. This extends the `[[ ...` autocompleter to `((...` for Phurl URLs.

Test Plan: Typed `((th`, got `((thing))` suggested.

Reviewers: avivey

Reviewed By: avivey

Maniphest Tasks: T13077, T12241

Differential Revision: https://secure.phabricator.com/D19109
2018-02-16 09:56:39 -08:00
epriestley
8771b7d5c4 Add autocomplete for Phriction documents on "[[ ..." in Remarkup
Summary: Depends on D19107. Ref T13077. The underlying datasource may need some adjustment but this appears to work properly locally.

Test Plan: Typed `[[ por` locally, was suggested "Porcupine Facts". Typed `[[ / ]]`, saw it render as a reference to the wiki root instead of the install root.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19108
2018-02-16 09:56:18 -08:00
epriestley
f82206a4d1 Add a rough Quick Search datasource for Phriction documents
Summary:
Depends on D19106. Fixes T5941. Ref T13077. Allows you to find Phriction documents as suggestions from global quick search.

Also supports `w` to jump to Phriction and `w query` to query Phriction.

The actual query logic for the datasource may need some tweaking after it collides with reality, but seems to produce fairly reasonable results in local testing against synthetic data.

Test Plan: Searched for "Porcupine Facts", "Travel Companions", and other useful local pages. Searched for `w`. Searched for `w zebra facts`.

Maniphest Tasks: T13077, T5941

Differential Revision: https://secure.phabricator.com/D19107
2018-02-16 09:55:54 -08:00
epriestley
143350fdba Give Phriction documents modern string status constants instead of numeric constants
Summary:
Depends on D19099. Ref T13077. Updates Phriction documents to string constants to make API interactions cleaner and statuses more practical to extend.

This does not seem to require any transaction migrations because none of the Phriction transactions actually store status values: status is always a side effect of other edits.

Test Plan: Created, edited, deleted, moved documents. Saw appropriate UI cues. Browsed and filtered documents by status in the index.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19100
2018-02-15 18:23:41 -08:00
epriestley
a965d8d6ae Make PhrictionContent "description" non-nullable
Summary:
Depends on D19095. Ref T6203. Ref T13077. This column is nullable in an inconsistent way. Make it non-nullable.

Also clean up one more content query on the history view.

Test Plan: Ran migration, then created and edited documents without providing a descriptino or hitting `NULL` exceptions.

Maniphest Tasks: T13077, T6203

Differential Revision: https://secure.phabricator.com/D19096
2018-02-15 17:55:11 -08:00
epriestley
e492c717c6 Give PhrictionContent objects (older versions of wiki pages) legitimate PHIDs
Summary: Ref T13077. Prepares for modern API access to document history using standard "v3" APIs.

Test Plan: Ran migration, verified PHIDs appeared in the database. Created/edited a document, got even more PHIDs in the database.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19092
2018-02-15 17:39:07 -08:00
epriestley
2b0f98900b Fail outstanding buildables with aborted builds
Summary:
Ref T13072. See PHI361. The bug in T10746 where aborting builds didn't propagate properly to the buildable was fixed, but existing builds are still stuck "Building".

Since it doesn't look like anything will moot this before these changes promote to `stable`, just migrate these builds into "failed".

Test Plan: Ran migration, saw it affect only relevant builds and correctly fail them.

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D19091
2018-02-15 03:56:58 -08:00
epriestley
a4053bb580 When a ChangesetList sleeps after a Quicksand navigation, also hide any visible banner
Summary: Fixes T13080. The banner wasn't properly included in the sleep/wake logic.

Test Plan:
Mentioned `Dxxx` on a task. Enabled persistent chat to activate Quicksand. Reloaded page. Clicked `Dxxx`. Scrolled down until a changeset header appeared. Pressed back button.

  - Before patch: ended up on task, with header still around.
  - After patch: ended up on task, with header properly vanquished.

Pressed "forward", ended up back on the revision with the header again.

Maniphest Tasks: T13080

Differential Revision: https://secure.phabricator.com/D19086
2018-02-14 15:25:25 -08:00
epriestley
c42bbd6f5c Rename HarbormasterBuildMessage "buildTargetPHID" to "receiverPHID"
Summary: Ref T13054. Companion storage change for D19062.

Test Plan: Applied migration and adjustments. Viewed messages in Harbormaster; created them with `harbormaster.sendmessage`; processed them with `bin/phd debug task`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13054

Differential Revision: https://secure.phabricator.com/D19063
2018-02-12 12:17:44 -08:00
epriestley
f43d08c2bb Completely remove the legacy hunk table
Summary: Depends on D19056. Fixes T8475. Ref T13054. Merges "ModernHunk" back into "Hunk".

Test Plan: Grepped for `modernhunk`. Reviewed revisions. Created a new revision. Used `bin/differential migrate-hunk` to migrate hunks between storage formats and back.

Maniphest Tasks: T13054, T8475

Differential Revision: https://secure.phabricator.com/D19057
2018-02-10 16:12:50 -08:00
epriestley
b0d1d46a73 Drop the legacy hunk table
Summary: Ref T13054. Ref T8475. This table has had no readers or writers for more than a year after it was migrated to the modern table.

Test Plan: Ran migration, verified that all the data was still around.

Maniphest Tasks: T13054, T8475

Differential Revision: https://secure.phabricator.com/D19056
2018-02-10 16:09:31 -08:00
epriestley
ffc5c95c2f Correct flipped transaction constants in "Closed Date" migration
Summary: These transaction constants are flipped, which can produce the wrong result in some cases.

Test Plan: `./bin/storage upgrade -f --apply phabricator:20180208.maniphest.02.populate.php`

Differential Revision: https://secure.phabricator.com/D19054
2018-02-10 06:10:55 -08:00
epriestley
0470125d9e Add skeleton code for webhooks
Summary: Ref T11330. Adds general support for webhooks. This is still rough and missing a lot of pieces -- and not yet useful for anything -- but can make HTTP requests.

Test Plan: Used `bin/webhook call ...` to complete requests to a test endpoint.

Maniphest Tasks: T11330

Differential Revision: https://secure.phabricator.com/D19045
2018-02-09 13:55:04 -08:00
epriestley
261a4a0e51 Add inline comment counts to the filetree view
Summary: See PHI356. Adds inline comment and done counts to the filetree. Also makes the filetree wider by default.

Test Plan: Fiddled with filetrees in different browsers on different revisions. Added inlines, marked them done/undone.

Differential Revision: https://secure.phabricator.com/D19041
2018-02-08 17:15:36 -08:00
epriestley
6ea1b8df9b Colorize filetree for adds, moves, and deletes
Summary: See PHI356. Makes it easier to pick out change types in the filetree view in Differential.

Test Plan: Created a diff with adds, copies, moves, deletions, and binary files. Viewed in Differential, had an easier time picking stuff out.

Differential Revision: https://secure.phabricator.com/D19040
2018-02-08 16:11:35 -08:00
epriestley
f028aa6f60 Track closed date and closing user for tasks explicitly
Summary:
Ref T4434. Although some of the use cases for this data are better fits for Facts, this data is reasonable to track separately.

I have an approximate view of it already ("closed, ordered by date modified") that's useful to review things that were fixed recently. This lets us make that view more effective.

This just adds (and populates) the storage. Followups will add Conduit, Export, Search, and UI support.

This is slightly tricky because merges work oddly (see T13020).

Test Plan:
  - Ran migration, checked database for sensible results.
  - Created a task in open/closed status, got the right database values.
  - Modified a task to close/open it, got the right values.
  - Merged an open task, got updates.

Maniphest Tasks: T4434

Differential Revision: https://secure.phabricator.com/D19037
2018-02-08 15:40:49 -08:00
epriestley
ab04d2179b Add "Mute/Unmute" for subscribable objects
Summary: Ref T13053. See PHI126. Add an explicit "Mute" action to kill mail and notifications for a particular object.

Test Plan: Muted and umuted an object while interacting with it. Saw mail route appropriately.

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19033
2018-02-08 11:06:22 -08:00
epriestley
aa74af1983 Remove all "originalTitle"/"originalName" fields from objects
Summary:
Depends on D19012. Ref T13053. In D19012, I've changed "Thread-Topic" to always use PHIDs.

This change drops the selective on-object storage we have to track the original, human-readable title for objects.

Even if we end up backing out the "Thread-Topic" change, we'd be better off storing this in a table in the Mail app which just has `<objectPHID, first subject we used when sending mail for that object>`, since then we get the right behavior without needing every object to have this separate field.

Test Plan: Grepped for `original`, `originalName`, `originalTitle`, etc.

Reviewers: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19013
2018-02-08 06:22:03 -08:00
epriestley
b3880975e5 Add aliases for "party" emoji (🎉)
Summary:
This is currently `🎉`, which I'd never have guessed.

(This isn't a super scalable approach, but this emoji is in particularly common use. See also T12644.)

Test Plan: Typed `:party`, `:confet`, etc.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18993
2018-02-05 12:23:26 -08:00
epriestley
c9df8f77c8 Fix transcription of single-value bulk edit fields ("Assign to")
Summary: See PHI333. Some of the cleanup at the tail end of the bulk edit changes made "Assign To" stop working properly, since we don't strip the `array(...)` off the `array(PHID)` value we receive.

Test Plan:
  - Used bulk editor to assign and unassign tasks (single value datasource).
  - Used bulk editor to change projects (multi-value datasource).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18975
2018-01-31 10:56:16 -08:00
epriestley
40e9806e3c Remove the caret dropdown from transaction lists when no actions are available
Summary:
See PHI325. When a transaction group in Differential (or Pholio) only has an inline comment, it renders with a "V" caret but no actual dropdown menu.

This caret renders in a "disabled" color, but the color is "kinda grey". The "active" color is "kinda grey with a dab of blue". Here's what they look like today:

{F5401581}

Just remove it.

Test Plan: Viewed one of these, no longer saw the inactive caret.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18963
2018-01-29 15:14:59 -08:00
epriestley
fd49acd033 Fix Herald repetition policy migration for NULL
When we change a nullable column to a non-nullable column, we can get a
data truncation error if any value was "NULL".

This is exceptionally unusual, but our two very oldest Herald rules have
a "NULL" policy on `secure`.
2018-01-26 13:17:15 -08:00
epriestley
ad7755d9a9 Fix an issue with symbol lookup identifying path names in Diffusion
Summary:
Depends on D18939. Ref T13047. Symbol lookup can be activated from a diff (in Differential or Diffusion) or from the static view of a file at a particular commit.

In the latter case, we need to figure out the path a little differently. The character and line number approaches still work as written.

Test Plan:
  - Command-clicked symbols in the Diffusion browse view with blame on and off; saw path, line and char populate properly.
  - Command-clicked symbols in Differential diff view to check I didn't break anything.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13047

Differential Revision: https://secure.phabricator.com/D18940
2018-01-26 13:02:20 -08:00
epriestley
fdc36677ba Provide character position information to symbol queries
Summary: Depends on D18937. Ref T13047. When available, provide character positions so external indexers can return more accurate results.

Test Plan: Clicked symbols in Safari, Firefox and Chrome, got sensible-looking character positions.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13047

Differential Revision: https://secure.phabricator.com/D18939
2018-01-26 13:01:57 -08:00
epriestley
c37b6c6633 When users click a symbol in Differential to jump to the definition, include path/line context
Summary:
Ref T13047. In some reasonable cases, knowing the path and line number where a symbol appears is useful in ranking or filtering the set of matching symbols.

Giving symbol sources more information can't hurt, and it's generally free for us to include this context since we just need to grab it out of the document and pass it along.

We can't always get this data (for example, if a user types `s idx` into global search, we have no clue) but this is similar to other types of context which are only available sometimes (like which repository a symbol appears in).

Test Plan: Command-clicked some symbols in 1-up (unified) and 2-up (side-by-side) diff views with symbol indexes configured. Got accurate path and line information in the URI I was redirected to.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13047

Differential Revision: https://secure.phabricator.com/D18936
2018-01-26 11:59:48 -08:00
epriestley
204d1de683 Convert storage for Herald repetition policy to "text32"
Summary:
Depends on D18926. Ref T6203. Ref T13048. Herald rule repetition policies are stored as integers but treated as strings in most contexts.

After D18926, the integer stuff is almost totally hidden inside `HeraldRule` and getting rid of it completely isn't too tricky.

Do so now.

Test Plan:
  - Created "only the first time" and "every time" rules. Did a SELECT on their rows in the database.
  - Ran migrations, got a clean bill of health from `storage adjust`.
  - Did another SELECT on the rows, saw a faithful conversion to strings "every" and "first".
  - Edited and reviewed rules, swapping them between "every" and "first".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048, T6203

Differential Revision: https://secure.phabricator.com/D18927
2018-01-26 11:05:37 -08:00
epriestley
042c43d6d8 Remove a very old Herald garbage collection migration
Summary:
Ref T13048. This migration is from January 2012 and probably only impacted Facebook.

It references `HeraldRepetitionPolicyConfig`, which I'd like to change significantly. I initially just replaced the constant with a literal `0`, but I don't think there's any actual value in retaining this migration nowadays.

The cost of removing this migration is: if you installed Phabricator before January 2012 and haven't upgraded since then, you'll have a few more rows in the `APPLIED` table than necessary. Herald will still work correctly.

Test Plan: Reading.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048

Differential Revision: https://secure.phabricator.com/D18924
2018-01-26 10:54:37 -08:00
epriestley
21e415299f Mark all existing password hashes as "legacy" and start upgrading digest formats
Summary:
Depends on D18907. Ref T13043. Ref T12509. We have some weird old password digest behavior that isn't terribly concerning, but also isn't great.

Specifically, old passwords were digested in weird ways before being hashed. Notably, account passwords were digested with usernames, so your password stops working if your username is chagned. Not the end of the world, but silly.

Mark all existing hashes as "v1", and automatically upgrade then when they're used or changed. Some day, far in the future, we could stop supporting these legacy digests and delete the code and passwords and just issue upgrade advice ("Passwords which haven't been used in more than two years no longer work."). But at least get things on a path toward sane, modern behavior.

Test Plan: Ran migration. Spot-checked that everthing in the database got marked as "v1". Used an existing password to login successfully. Verified that it was upgraded to a `null` (modern) digest. Logged in with it again.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043, T12509

Differential Revision: https://secure.phabricator.com/D18908
2018-01-23 14:01:09 -08:00
epriestley
cab2bba6f2 Remove "passwordHash" and "passwordSalt" from User objects
Summary:
Ref T13043. After D18903, this data has migrated to shared infrastructure and has no remaining readers or writers.

Just delete it now, since the cost of a mistake here is very small (users need to "Forgot Password?" and pick a new password).

Test Plan: Grepped for `passwordHash`, `passwordSalt`, and variations.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18904
2018-01-23 13:44:26 -08:00
epriestley
abc030fa00 Move account passwords to shared infrastructure
Summary:
Ref T13043. This moves user account passwords to the new shared infrastructure.

There's a lot of code changes here, but essentially all of it is the same as the VCS password logic in D18898.

Test Plan:
- Ran migration.
- Spot checked table for general sanity.
- Logged in with an existing password.
- Hit all error conditions on "change password", "set password", "register new account" flows.
- Verified that changing password logs out other sessions.
- Verified that revoked passwords of a different type can't be selected.
- Changed passwords a bunch.
- Verified that salt regenerates properly after password change.
- Tried to login with the wrong password, which didn't work.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18903
2018-01-23 13:43:07 -08:00
epriestley
5a8a56f414 Prepare the new AuthPassword infrastructure for storing account passwords
Summary:
Ref T13043. In D18898 I moved VCS passwords to the new shared infrastructure.

Before account passwords can move, we need to make two changes:

  - For legacy reasons, VCS passwords and Account passwords have different "digest" algorithms. Both are more complicated than they should be, but we can't easily fix it without breaking existing passwords. Add a `PasswordHashInterface` so that objects which can have passwords hashes can implement custom digest logic for each password type.
  - Account passwords have a dedicated external salt (`PhabricatorUser->passwordSalt`). This is a generally reasonable thing to support (since not all hashers are self-salting) and we need to keep it around so existing passwords still work. Add salt support to `AuthPassword` and make it generate/regenerate when passwords are updated.

Then add a nice story about password digestion.

Test Plan: Ran migrations. Used an existing VCS password; changed VCS password. Tried to use a revoked password. Unit tests still pass. Grepped for callers to legacy `PhabricatorHash::digestPassword()`, found none.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18900
2018-01-23 10:57:40 -08:00
epriestley
753c4c5ff1 Remove the "PhabricatorRepositoryVCSPassword" class and table
Summary:
Ref T13043. After D18898, this has been migrated to new, more modern storage and no longer has any readers or writers.

One migration from long ago (early 2014) is affected. Since this is ancient and the cost of dropping this is small (see inline), I just dropped it.

I'll note this in the changelog.

Test Plan: Ran migrations, got a clean bill of health from `storage status`. Grepped for removed symbol.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18899
2018-01-23 10:56:37 -08:00
epriestley
dd8f588ac5 Migrate VCS passwords to new shared password infrastructure
Summary:
Ref T13043. Migrate VCS passwords away from their dedicated table to new the new shared infrastructure.

Future changes will migrate account passwords and remove the old table.

Test Plan:
- Ran migrations.
  - Cloned with the same password that was configured before the migrations (worked).
  - Cloned with a different, invalid password (failed).
- Changed password.
  - Cloned with old password (failed).
  - Cloned with new password (worked).
- Deleted password in web UI.
  - Cloned with old password (failed).
- Set password to the same password as it currently is set to (worked, no "unique" collision).
- Set password to account password. !!This (incorrectly) works for now until account passwords migrate, since the uniqueness check can't see them yet.!!
- Set password to a new unique password.
  - Cloned (worked).
  - Revoked the password with `bin/auth revoke`.
  - Verified web UI shows "no password set".
  - Verified that pull no longer works.
  - Verified that I can no longer select the revoked password.
- Verified that accounts do not interact:
  - Tried to set account B to account A's password (worked).
  - Tried to set account B to a password revoked on account A (worked).
- Spot checked the `password` and `passwordtransaction` tables for saniity.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18898
2018-01-23 10:56:13 -08:00
epriestley
9c00a43784 Add a more modern object for storing password hashes
Summary:
Ref T13043. Currently:

  - Passwords are stored separately in the "VCS Passwords" and "User" tables and don't share as much code as they could.
  - Because User objects are all over the place in the code, password hashes are all over the place too (i.e., often somewhere in process memory). This is a very low-severity, theoretical sort of issue, but it could make leaving a stray `var_dump()` in the code somewhere a lot more dangerous than it otherwise is. Even if we never do this, third-party developers might. So it "feels nice" to imagine separating this data into a different table that we rarely load.
  - Passwords can not be //revoked//. They can be //deleted//, but users can set the same password again. If you believe or suspect that a password may have been compromised, you might reasonably prefer to revoke it and force the user to select a //different// password.

This change prepares to remedy these issues by adding a new, more modern dedicated password storage table which supports storing multiple password types (account vs VCS), gives passwords real PHIDs and transactions, supports DestructionEngine, supports revocation, and supports `bin/auth revoke`.

It doesn't actually make anything use this new table yet. Future changes will migrate VCS passwords and account passwords to this table.

(This also gives third party applications a reasonable place to store password hashes in a consistent way if they have some need for it.)

Test Plan: Added some basic unit tests to cover general behavior. This is just skeleton code for now and will get more thorough testing when applications move.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18894
2018-01-22 15:35:28 -08:00
epriestley
fbfcc37531 Respect token limits for "Assign to" and custom datasource fields in Herald
Summary:
See PHI173. Currently, Herald has an "Assign to" action for tasks, and you can specify custom fields with datasource values (like users or projects) that have a limit (like 1 "Owner", or 12 "Jury Members").

Herald doesn't support these limits right now, so you can write `[ Assign to ][ X, Y, Z ]`. This just means "Assign to X", but make it more clear by actually enforcing the limit in the UI.

Test Plan:
  - Created a "projects" custom field with limit 1.
  - Tried to create actions that 'assign to' or 'set custom field to' more than one thing, got helpfully rebuffed by the UI.
  - Created an "add subscribers" action with more than one value.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18887
2018-01-22 11:54:12 -08:00
epriestley
3038d564a6 Allow bulk edits to be made silently if you have CLI access
Summary:
Fixes T13042. This hooks up the new "silent" mode from D18882 and makes it actually work.

The UI (where we tell you to go run some command and then reload the page) is pretty clumsy, but should solve some problems for now and can be cleaned up eventually. The actual mechanics (timeline aggregation, Herald interaction,  etc.) are on firmer ground.

Test Plan:
  - Made a normal bulk edit, got mail and feed stories.
  - Made a silent bulk edit, no mail and no feed.
  - Saw "Silent Edit" marker in timeline for silent edits:

{F5386245}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13042

Differential Revision: https://secure.phabricator.com/D18883
2018-01-19 13:24:54 -08:00
epriestley
7a43181337 Organize bulk edit actions into nice groups
Summary: Ref T13025. We're getting kind of a lot of actions, so put them in nice groups so they're easier to work with.

Test Plan: {F5386038}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18880
2018-01-19 13:22:25 -08:00
epriestley
687fada5af Restore bulk edit support for remarkup fields (description, add comment)
Summary:
Depends on D18866. Ref T13025. Fixes T12415. This makes the old "Add Comment" action work, and adds support for a new "Set description to" action (possibly, I could imagine "append description" being useful some day, maybe).

The implementation is just a `<textarea />`, not a whole fancy remarkup box with `[Bold] [Italic] ...` buttons, preview, typeaheads, etc. It would be nice to enrich this eventually but doing the rendering in pure JS is currently very involved.

This requires a little bit of gymnastics to get the transaction populated properly, and adds some extra validation since we need some code there anyway.

Test Plan:
  - Changed the description of a task via bulk editor.
  - Added a comment to a task via bulk editor.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T12415

Differential Revision: https://secure.phabricator.com/D18867
2018-01-19 12:45:34 -08:00
epriestley
09e71a4082 Define bulk edits in terms of EditEngine, not hard-coded ad-hoc definitions
Summary:
Depends on D18862. See PHI173. Ref T13025. Fixes T10005. This redefines bulk edits in terms of EditEngine fields, rather than hard-coding the whole thing.

Only text fields -- and, specifically, only the "Title" field -- are supported after this change. Followup changes will add more bulk edit parameter types and broader field support.

However, the title field now works without any Maniphest-specific code, outside of the small amount of binding code in the `ManiphestBulkEditor` subclass.

Test Plan: Used the bulk edit workflow to change the titles of tasks.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T10005

Differential Revision: https://secure.phabricator.com/D18863
2018-01-19 12:43:47 -08:00
epriestley
7f91c8c4ac Rebuild the bulk editor on SearchEngine
Summary:
Depends on D18805. Ref T13025. Fixes T10268.

Instead of using a list of IDs for the bulk editor, power it with SearchEngine queries. This gives us the full power of SearchEngine and lets us use a query key instead of a list of 20,000 IDs to avoid issues with URL lengths.

Also, split it into a base `BulkEngine` and per-application subclasses. This moves us toward T10005 and universal support for bulk operations.

Also:

  - Renames most of "batch" to "bulk": we're curently inconsitent about this, I like "bulk" better since I think it's more clear if you don't regularly interact with `.bat` files, and newer stuff mostly uses "bulk".
  - When objects in the result set can't be edited because you don't have permission, show the status more clearly.

This probably breaks some stuff a bit since I refactored so heavily, but it seems mostly OK from poking around. I'll clean up anything I missed in followups to deal with remaining items on T13025.

Test Plan:
{F5302300}

  - Bulk edited from Maniphest.
  - Bulk edited from a workboard (no more giant `?ids=....` in the URL).
  - Hit most of the error conditions, I think?
  - Clicked the "Cancel" button.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T10268

Differential Revision: https://secure.phabricator.com/D18806
2018-01-19 12:40:08 -08:00
epriestley
ad659627b3 Make bulk editor working set editable and more homogenous
Summary:
Ref T13025. See PHI50. Fixes T11286. Ref T10005. Begin modernizing the bulk editor.

For T10005 ("move the bulk editor to modern infrastructure"), rewrite the rendering of the editable set so that it is application-agnostic and can work with any kind of object.

For T11286 ("let users de-select items in the working set"), make the working set editable.

Test Plan:
{F5302158}

  - Deselected some objects, applied an edit, saw the edit apply to only selected objects.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T11286, T10005

Differential Revision: https://secure.phabricator.com/D18805
2018-01-19 12:39:27 -08:00
epriestley
c9a0d68340 Allow Herald rules to add comments
Summary:
See PHI242. All use cases for this that I know of are pretty hacky, but they don't seem perilous, and it's easier than webhooks.

See P1895, T10183, and T9853 for me previously refusing to implement this since all those use cases were also pretty bad.

Test Plan:
  - Wrote a rule to add comments, saw it add comments.
  - Reviewed summary, re-edited rule, reviewed transcript to check that all the strings worked OK.
  - Wrote a new rule for a non-commentable object (a blog) to make sure I wasn't offered the "Add a comment" action.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18823
2017-12-18 09:10:57 -08:00
epriestley
0807b70ea1 Add an explicit warning in the Differential transaction log when users skip review
Summary:
Ref T10233. See PHI231. When users ignore the `arc land` prompt about bad revision states, make it explicitly clear in the transaction log that they broke the rules.

You can currently figure this out by noticing that there's no "This revision is accepted and ready to land." message, but it's unrealistic to expect non-expert users to look for the //absence// of a message to indicate something, and this state change is often relevant.

Test Plan: {F5302351}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T10233

Differential Revision: https://secure.phabricator.com/D18808
2017-11-30 11:03:55 -08:00
epriestley
49b57eae7d Revert partial/nonfunctional OpenGraph support
Summary:
Ref T13018. See that task and the Discourse thread for discussion.

This doesn't work as-is and we need to `og:description` everything to make it work. I don't want to sink any more time into this so just back all the changes out for now.

(The `<html>` change is unnecessary anyway.)

Test Plan: Strict revert.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13018

Differential Revision: https://secure.phabricator.com/D18782
2017-11-22 15:21:10 -08:00
epriestley
c3d6c4b0ee Include OpenGraph prefix material in <html> tag if OpenGraph is enabled
Summary: Ref T13018. Discourse doesn't seem to be picking this up yet (see <https://discourse.phabricator-community.org/t/opengraph-test-topic/762>) so maybe it really needs this meta-meta-XML stuff?

Test Plan: Will push.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13018

Differential Revision: https://secure.phabricator.com/D18781
2017-11-22 11:37:08 -08:00
epriestley
3700bcb638 Warn and prevent 1-up/2-up switch in Differential if the user is editing an inline
Summary:
See PHI180. Currently, if you begin creating or editing an inline and then swap display modes (for example, with "View Unified"), your edit is lost.

Persisting the editor state is complicated and this is very rare, so just prevent the action and warn the user instead.

Also make the warning persist for a little longer since a few of the messages, including this one, take a couple seconds to read now.

Test Plan:
  - Edited a comment, tried to swap display modes, got a warning.
  - Swapped display modes normally with no comment being edited.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18774
2017-11-15 10:02:48 -08:00
epriestley
6d36eb9113 Denormalize Diff PHIDs onto Revisions
Summary:
Ref T12539. See PHI190. Currently, each Diff has a `revisionID`, but Revisions do not point at the current active diff. To find the active diff for a given revision, we need to issue a separate query.

Furthermore, this query is inefficient for bulk loads: if we have a lot of revisions, we end up querying for all diff IDs for all those revisions first, then selecting the largest ones and querying again to get the actual diff objects. This strategy could likely be optimized but the query is a mess in any case.

In several cases, it's useful to have the active diff PHID without needing to do a second query -- sometimes for convenience, and sometimes for performance.

T12539 is an example of such a case: it would be nice to refine the bucketing logic (which only depends on active diff PHIDs), but it feels bad to make the page heavier to do it.

For now, this is unused. I'll start using it to fix the bucketing issue, and then we can expand it gradually to address other performance/convenience issues.

Test Plan:
  - Ran migrations, inspected database, saw sensible values.
  - Created a new revision, saw a sensible database value.
  - Updated an existing revision, saw database update properly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12539

Differential Revision: https://secure.phabricator.com/D18756
2017-11-01 17:19:38 -07:00
epriestley
2a7cdcf740 Fix an issue where the repository symbol index would incorrectly activate inside inline comments
Summary:
See PHI185. When looking at a revision, you can Command-Click (Mac) symbols to jump to their definitions (provided the symbol index has been built).

Currently, the code works on any node inside the changeset list, so it activates when clicking links inside inline comments and opening them in a new window.

To avoid this, don't activate if we're inside an inline comment. This technically prevents you from doing a symbol lookup on a symbol inside a codeblock inside an inline, but that seems fine/reasonable.

Test Plan: Wrote `Dxxx` in an inline, command-clicked it. Before: got a symbol lookup. After: just a new tab with the revision.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18753
2017-10-31 15:13:40 -07:00
epriestley
80ebe401e5 Tweak padding/spacing on Diffusion blame view for profile pictures
Summary: Give profile images a little more space, fix "/" spacing, add a tooltip.

Test Plan: {F5251205}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18749
2017-10-31 12:56:36 -07:00
epriestley
bde71324f8 Show small author portraits in Diffusion blame view
Summary: Depends on D18746. See PHI174. Adds small author portraits next to each blame line (this is similar to GitHub).

Test Plan:
My local test data isn't that great since I don't have commits from a lot of accounts, but looks functional:

{F5251056}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18747
2017-10-31 12:10:45 -07:00
epriestley
90d0f8ac6c Revert changes to Diffusion blame view
Summary:
Ref PHI174. This reverts most of these changes:

- 37843127e9 / D18481
- 94cad30ac3 / D18474
- 12ae08b6b1 / D18473
- 0a01334172 / D18462
- ac91ab1ef9 / D18452

These changes made the Diffusion blame view very similar to GitHub's blame view. See D18452 for a before/after of the bulk of these changes; the other revisions are bugfixes.

I think this was generally a step backward, and not motivated by solving a specific problem. I've found the new UI less usable than the old one, and at least one install (see PHI174) also has.

In particular, the revision/commit titles are very bulky and not terribly useful; the date column also isn't terribly useful; the "age" color actually IS pretty useful and was heavily de-emphasized.

I've kept one bugfix here (missing `'a'` tag type) and kept the upgraded icon for "Skip Past This Commit".

I'm going to follow this up with some additional changes:

  - Show a small author profile icon, similar to GitHub, to address PHI174 more directly.
  - Try a zebra-stripe on blocks of rows to make it more clear where changes affected by a particular commit begin and end.
  - Try a hue shift, not just a brightness/saturation shift, to make the "age" color more distinct.
  - Try computing colors as even steps, not based purely on age. Currently, if a file has one long-distant commit and several recent commits, all the recent ones show up as very bright green. I think this would probably be more useful if they were distributed more evenly across the available color bands.

Test Plan:
Viewed blame views in Diffusion, saw a more compact UI similar to the old UI.

{F5251019}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18746
2017-10-31 11:54:47 -07:00
epriestley
f1204c8c45 Convert Ponder Questions to Ferret engine
Summary: See PHI177. Ref T12974. PonderQuestion was overlooked during the Ferret engine conversions.

Test Plan:
Ran migrations, searched for questions, got results:

{F5241185}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12974

Differential Revision: https://secure.phabricator.com/D18736
2017-10-26 18:18:04 -07:00
epriestley
b583093186 Fix Celerity map definition after spelling corrections
See D18693.
2017-10-09 10:52:27 -07:00
epriestley
1de130c9f5 Allow the Ferret engine to remove "common" ngrams from the index
Summary:
Ref T13000. This adds support for tracking "common" ngrams, which occur in too many documents to be useful as part of the ngram index.

If an ngram is listed in the "common" table, it won't be written when indexing documents, or queried for when searching for them.

In this change, nothing actually writes to the "common" table. I'll start writing to the table in a followup change.

Specifically, I plan to do this:

  - A new GC process updates the "common" table periodically, by writing ngrams which appear in more than X% of documents to it, for some value of X, if there are at least a minimum number of documents (maybe like 4,000).
  - A new GC process deletes ngrams that have been added to the common table from the existing indexes.

Hopefully, this will pare down the ngrams index to something reasonable over time without requiring any manual tuning.

Test Plan:
  - Ran some queries and indexes.
  - Manually inserted ngrams `xxx` and `yyy` into the ngrams table, searched and indexed, saw them ignored as viable ngrams for search/index.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18672
2017-10-03 13:27:42 -07:00
Aviv Eyal
9f11f310f8 Make PHUITwoColumnView a little more printable
Summary: Hide navbar, and make curtain behave like on a phone, when printing.

Test Plan: {F5197340}

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18583
2017-09-25 19:56:22 +00:00
epriestley
3ad727ba78 Guarantee the key_position key is created properly
Summary:
Ref T12987. I was focused on the RefCursor table and overlooked that we need some care on this key.

It's currently possible to run `bin/storage upgrade --no-adjust`, then start Phabricator, and end up with duplicate records in this table. If you try to run `bin/storage adjust` later, it will try to add the unique key but fail. This is unusual for normal installs (they usually do not use `--no-adjust`) but we do it in the cluster and I did this exact thing on `secure`.

Normally, to avoid this, when a new table with a unique key is introduced, we also add a migration to explicitly add that key.

This is mostly harmless in this case. Fix this mistake (force the table to contain only unique rows; add the key) and try using `LOCK TABLES` to make this atomic. If this doesn't cause problems we can use this in similar situations in the future.

The "alter table may unlock things" warning comes from here:

https://dev.mysql.com/doc/refman/5.7/en/lock-tables.html

It seems like it's fine to issue `UNLOCK TABLES` even if you don't have any locks, so I think this script should always do the right thing now, regardless of ALTER TABLE unlocking or not unlocking tables.

Test Plan: Ran `bin/storage upgrade -f`, saw table end up in the right state. I'll also check this on `secure`, where the starting state is a little messier.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12987

Differential Revision: https://secure.phabricator.com/D18623
2017-09-18 14:00:22 -07:00
epriestley
5cf62f86d7 Remove obsolete columns from RefCursor table
Summary:
Ref T11823. This change isn't standalone, but prepares for the more involved code change by dropping obsolete columns from the RefCursor table and adding the unique key we need to prevent the ambiguous/duplicate refs issue.

This data was moved to the RefPosition table in D18612.

Test Plan: Ran storage upgrade. See next revision for more substantial testing of this change series.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11823

Differential Revision: https://secure.phabricator.com/D18613
2017-09-15 10:21:12 -07:00
epriestley
782b18e7e2 Migrate RefCursor data to RefPosition table
Summary:
Ref T11823. This populates the new RefPosition table based on the existing RefCursor table, and deletes now-duplicate rows in the RefCursor table so the next change can add a unique key.

This change is not standalone, and there need to be separate code updates. I have a rough version of that written, but this migration needs to happen first to test it.

I'll hold this whole series of changes until after the release cut and until the code is updated.

Test Plan: Ran migration, spot-checked database tables. Saw redundant rows remove and correct-looking rows populated into the new RefPosition table.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11823

Differential Revision: https://secure.phabricator.com/D18612
2017-09-15 10:19:32 -07:00
epriestley
9d5a2b3b4f Add a RefPosition table to hold branch/tag positions once the RefCursor table is split
Summary:
Ref T11823. Currently, we have a "RefCursor" table which stores rows like `<branch or tag name, commit it is pointing at>` with some more data.

Because Mercurial can have a single branch pointing at several different places, this table must allow multiple rows with the same branch or tag name.

Among other things, this means there isn't a single PHID which can be used to identify a branch name in a stable way. However, we have several UIs where we want to be able to do this.

Some specific examples where we run into trouble: in Mercurial, if there are 5 heads for "default", that means there are 5 phids. And currently, if someone deletes a branch, we lose the PHID for it. Instead, we'd rather retain it so the whole world doesn't break if you accidentally delete a branch and then fix it a little later.

(I'll likely hold this until the rest of the logic is fleshed out a little more in followup changes.)

Test Plan: Ran `bin/storage upgrade`, saw the table get created without warnings.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11823

Differential Revision: https://secure.phabricator.com/D18602
2017-09-15 10:19:17 -07:00
epriestley
29f625ef68 Make "No Notifications" setting less broad, and fix a bug with default display behavior
Summary:
Fixes T12979. In D18457, we added a "No Notifications" setting to let users disable the blue and yellow pop-up notifications that alert you when an object has been updated, since some users found them distracting.

However, the change made "do nothing" the default, so all other `JX.Notification` callsites -- which never pass a preference -- were effectively turned off no matter what your setting was set to. This includes the "Read-Only" mode warning (grey), the "High Security" mode warning (purple), the "timezone" warning, and a few others.

Tweak things a little bit so the setting applies to ONLY blue and yellow ("object you're following was updated" / "this object was updated") notifications, not other types of popup notifications.

Test Plan:
  - With notifications on in settings, got blue notifications and "Read-only".
  - With notifications off in settings, got "Read-only" but no blue notifications.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12979

Differential Revision: https://secure.phabricator.com/D18600
2017-09-13 15:32:46 -07:00
epriestley
6cedd4a95c Revert quickstart for tables with native FULLTEXT indexes to MyISAM
See D18594.
2017-09-12 12:24:23 -07:00
epriestley
124e580f6e Issue upgrade guidance to rebuild indexes for the Ferret engine
Summary:
Ref T12819. This is shipping, so issue upgrade guidance to instruct installs to rebuild the index.

Also generate a new `quickstart.sql` since we haven't regenerated in a bit and there's been a large amount of table churn fairly recently.

Test Plan: Ran `bin/storage upgrade`, saw guidance notification in UI.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18594
2017-09-12 12:21:20 -07:00
Chad Little
f9b109dc0d Fix info view error layout in config boxes
Summary: Adds some side margin here.

Test Plan: error out a form field in a white box

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18571
2017-09-07 14:09:03 -07:00
epriestley
b1b638bd14 Support the Ferret engine in Diffusion
Summary: Ref T12819. More ferret engine support.

Test Plan: Indexed and searched commits and repositories.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18572
2017-09-07 13:41:04 -07:00
epriestley
d8132db75b Support Ferret engine in Pholio
Summary: Ref T12819. Support for Pholio.

Test Plan: Indexed and searched mocks.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18569
2017-09-07 13:25:29 -07:00
epriestley
e0f3de9c64 Support Ferret engine in Calendar
Summary: Ref T12819. Adds ferret engine support for Calendar events.

Test Plan: Indexed and queried calendar events.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18568
2017-09-07 13:25:12 -07:00
epriestley
a25bbc1dca Support Ferret engine in Phriction
Summary: Ref T12819. Adds Ferret engine support.

Test Plan: Indexed and searched for documents.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18567
2017-09-07 13:24:40 -07:00
epriestley
184f201ce2 Support Ferret engine in Projects
Summary: Ref T12819. Adds support for projects.

Test Plan: Indexed and searched for projects.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18566
2017-09-07 13:24:23 -07:00
epriestley
b1703c8801 Support Ferret engine in Phame
Summary: Ref T12819. Mostly straightforward, with a couple of minor query modernization things.

Test Plan: Indexed and searched for posts and blogs.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18565
2017-09-07 13:24:07 -07:00
epriestley
c9152b586b Support Ferret engine in Owners
Summary: Ref T12819. Same deal as before, but smaller diffs after D18559.

Test Plan: Indexed and searched for packages.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18564
2017-09-07 13:23:46 -07:00
epriestley
2020c1e7bd Support Ferret engine for Passphrase credentials
Summary: Ref T12819. Adds Ferret support to Passphrase.

Test Plan: Indexed credentials, searched for credentials.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18556
2017-09-07 13:23:13 -07:00