1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-09-21 09:48:47 +02:00
Commit graph

12746 commits

Author SHA1 Message Date
epriestley
cb49acc2ca Update Phabricator to use intermediate tokens from the query compiler
Summary:
Depends on D17669. Ref T12137. Ref T12003. Ref T2632. Ref T7860.

Converts Phabricator to the new parse + compile workflow with intermediate tokens.

Also fixes a bug where searches for `cat"` or similar (unmatched quotes) wouldn't produce a nice exception.

Test Plan:
  - Fulltext searched.
  - Fulltext searched in Conpherence.
  - Fulltext searched with bad syntax.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12137, T12003, T7860, T2632

Differential Revision: https://secure.phabricator.com/D17670
2017-04-12 19:07:33 -07:00
epriestley
4bf968148c Fix pagination of fulltext search results
Summary:
Fixes T8285. Fulltext search relies on an underlying engine which can not realistically use cursor paging. This is unusual and creates some oddness.

Tweak a few numbers -- and how offsets are handled -- to separate the filtered offset and unfiltered offset.

Test Plan:
  - Set page size to 2.
  - Ran a query.
  - Paged forward and backward through results sensibly, seeing the full result set.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8285

Differential Revision: https://secure.phabricator.com/D17667
2017-04-12 17:57:46 -07:00
Chad Little
a7ebfc12c0 Modernize Conpherence with Modular Transactions
Summary: Begin converting Conpherence to ModularTransactions, this converts title, topic, and picture to use modular transactions. Participants seems hairy so I'll do that in another diff

Test Plan: Create a room with a topic, change room name, topic. Add people, remove people. Set a room image. Unset topic.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17668
2017-04-12 16:33:57 -07:00
Chad Little
a9845b0b1d Remove picture crop transaction from Conpherence
Summary: Fixes T11730. Removes an old transaction that hasn't been used in a year.

Test Plan: Run sql, check various rooms.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11730

Differential Revision: https://secure.phabricator.com/D17666
2017-04-12 14:18:54 -07:00
Austin McKinley
c6c25b055b Cleanup Countdown manual construction of monograms/uris
Summary: looked for places where Countdown monograms/uris were being constructed by hand, and updated with modern versions

Test Plan: clicked around the Countdown UI, looking for broken links

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: chad, Korvin

Maniphest Tasks: T12524

Differential Revision: https://secure.phabricator.com/D17665
2017-04-12 13:33:19 -07:00
Chad Little
6bf595b951 Check is viewer is a participant before showing count
Summary: In Conpherence ProfileMenuItem we show an unread count if you're a participant, but all message count if you're not. Just remove that.

Test Plan: Log out of room in Conpherence, leave messages on second account, check menu item on both accounts.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17664
2017-04-12 13:27:07 -07:00
Chad Little
75303567b3 Add a Conpherence Profile Menu Item
Summary: Builds a Conpherence Profile Menu Item, complete with counts for the unreads. This allows pinning to home as well as swapping out thread list in Conpherence for pinning eventually.

Test Plan: Add a menu item, chat in room, log into other account, see room count. Room count disappears after viewing.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17662
2017-04-12 13:07:44 -07:00
Chad Little
099c90e7ec Remove "First Message" from New Conpherence Room workflow
Summary: Removes this feature, makes creating a room simpler and less confusing.

Test Plan: Create a room on Conpherence.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17661
2017-04-12 09:15:50 -07:00
Chad Little
cd7547dc57 Update UI for PhortuneAccount
Summary: Primarily, this splits individual sections of the single account page into a more managable and robust sidenav for subscriptions, billing, and managers. The functionality on the subpages is light, but I expect to build on then in coming diffs. This also starts building out a more effective "status" area on the lead page.

Test Plan:
- Load up default account
- Make some edits
- Click on each of the new navigation items
- Verify links to "see all" work
- Test overdue and no payment states for status

{F4337317}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17589
2017-04-11 16:54:58 -07:00
Austin McKinley
6886e9c12d Remove "Destroy" action for Countdown objects
Summary: fixes T12523

Test Plan:
- view Countdown edit screen, Destroy action missing
- checked that `./bin/remove destroy <some-countdown-phid>` removes the DB rows as expected

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12523

Differential Revision: https://secure.phabricator.com/D17659
2017-04-11 16:52:35 -07:00
Chad Little
149c1a6de7 Correctly initialize new PhortuneAccount automatically
Summary: There is currently a validation error triggered if you initialize a new account without a member set. I think this is the correct fix, but let me know.

Test Plan: truncate phortune_account database, navigate to phortune, see account automatically created to "Default Account".

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17657
2017-04-11 22:46:40 +00:00
epriestley
eeef60a678 Update "bin/policy show" to use PolicyCodex
Summary: Fixes T12541. `describeAutomaticCapability()` is no longer required to implement `PolicyInterface`. Use PolicyCodex instead.

Test Plan: {F4889642}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12541

Differential Revision: https://secure.phabricator.com/D17658
2017-04-11 15:21:18 -07:00
Chad Little
58a127f2f9 Update Phortune Merchant UI
Summary: Builds out Phortune Merchant pages to have a sidenav and sub-pages for further expansion. For now this links Orders and Subscriptions to the query engine pages, but could be split out to be more informative (unpaid, upcoming, etc).

Test Plan:
Create a new merchant, edit some information, add a manager in new UI, edit logo, click through to subscriptions, orders.

{F4883013}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17655
2017-04-11 14:36:49 -07:00
Chad Little
5dd18a7ec1 Modernize PhortuneAccount with EditEngine/Modular Transactions
Summary: This updates the backend of PhortuneAccount to use EditEngine and Modular Transactions and updates language to "account manager" for clarity of role.

Test Plan:
- Wiped `phortune_account` table
- Visit Phortune, see new account automatically created.
- Edit name and managers
- Try to set no name or remove myself as a manager, get error messages
- Visit `/phortune/` and create another new account

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17585
2017-04-11 12:33:15 -07:00
epriestley
21709a2bbc Remove 'isPartial' parameter with no effect
Summary: Fixes T12536. Nothing reads this parameter; `PhabricatorFile::newChunkedFile` sets the `isPartial` flag automatically.

Test Plan: Grepped for `isPartial`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12536

Differential Revision: https://secure.phabricator.com/D17654
2017-04-11 11:29:23 -07:00
epriestley
af1d494d66 Fix an issue where rejecting reviewers weren't powerful enough
Summary:
Previously, "reject" and "reject older" were separate statuses. Now, they're both shades of "reject".

Set the "older reject" flag properly when we find a non-current reject.

Test Plan:
  - User A accepts a revision.
  - User B rejects it.
  - Author updates it.
  - Before patch: incorrectly transitions to "accepted" ("older" reject is ignored).
  - After patch: correctly transitions to "needs review".

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17653
2017-04-11 09:54:34 -07:00
Chad Little
28941b3105 Update PhortuneMerchant to Modular Transactions
Summary: Modernize PhortuneMerchant for Modular Transactions. Also changed the language of "Members" to "Managers", which I think fits better given the power/capability.

Test Plan:
- Create a new Merchant
- Test not filling in a name, see error
- Test removing myself, see error
- Edit an existing Merchant
- Add new managers
- Test removing myself, see error
- Replace Picture
- Update various fields, contact info, email, footer
- Verify transactions are now nice and pretty

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17584
2017-04-11 09:32:12 -07:00
epriestley
26d6096e0a When reviewing, always show "Accept" checkboxes for packages/projects, even if there's only one checkbox
Summary: Fixes T12533.

Test Plan: {F4853371}

Reviewers: chad, lvital

Reviewed By: lvital

Maniphest Tasks: T12533

Differential Revision: https://secure.phabricator.com/D17652
2017-04-10 17:28:02 -07:00
epriestley
a7a068f84c Correct two parameter strictness issues with file uploads
Summary:
Fixes T12531. Strictness fallout from adding typechecking in D17616.

  - `chunkedHash` is not a real parameter, so the new typechecking was unhappy about it.
  - `mime-type` no longer allows `null`.

Test Plan:
  - Ran `arc upload --conduit-uri ... 12MB.zero` on a 12MB file full of zeroes.
  - Before patch: badness, failure, fallback to one-shot uploads.
  - After patch: success and glory.

Reviewers: chad

Subscribers: joshuaspence

Maniphest Tasks: T12531

Differential Revision: https://secure.phabricator.com/D17651
2017-04-10 16:01:15 -07:00
epriestley
49132b884b Sell Yellow! Buy Indigo!
Summary: Fixes T12504. Replaces all tags with indigo.

Test Plan: {F4849487}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12504

Differential Revision: https://secure.phabricator.com/D17649
2017-04-10 15:01:10 -07:00
Chad Little
4a84954957 Prevent Send on Enter in Fullscreen Remarkup Mode
Summary: Fixes T12138. Test for the presence of being in fullscreen mode, and disable send on enter if present. Side note, I'd love a first class "hasClass" type Javelin function.

Test Plan:
- Go to Conpherence
- Type some smack, see it send on enter
- Go fullscreen like a boss
- Let the words flow
- Close fullscreen, then send on enter.
- (might be nice someday to add a "submit" button to fullscreen editor)

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12138

Differential Revision: https://secure.phabricator.com/D17590
2017-04-10 14:39:50 -07:00
epriestley
dee9c33be2 Suggest use of "usermod" rather than manually editing critical files in /etc
Summary: Fixes T12529.

Test Plan: O_O

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12529

Differential Revision: https://secure.phabricator.com/D17648
2017-04-10 14:15:38 -07:00
epriestley
00a1dec7a6 Render timezones in event reminder mail, and render them more nicely
Summary:
Fixes T12356.

  - In this mail, we currently render "6:00 AM". Instead, render "6:00 AM (PDT)" or similar. This is consistent with times in other modern Transaction mail.
  - Previously, we would render "UTC-7". Render "PDT" instead. For obscure zones with no known timezone abbreviation, fall back to "UTC-7".

Test Plan:
  - Used `bin/calendar notify --minutes X` to trigger notifications, read email bodies.
  - Used this script to list all `T` values and checked them for sanity:

```lang=php
<?php

$now = new DateTime();

$locales = DateTimeZone::listIdentifiers();
foreach ($locales as $locale) {
  $zone = new DateTimeZone($locale);
  $now->setTimeZone($zone);

  printf(
    "%s (%s)\n",
    $locale,
    $now->format('T'));
}
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12356

Differential Revision: https://secure.phabricator.com/D17646
2017-04-10 08:48:37 -07:00
epriestley
50e809e06f Fix an issue where recurring ghost events could go missing if queried with a limit
Summary:
Ref T11816. Depends on D17644. When you executed a query like "upcoming, limit 5 events" you might match some recurring events starting from, say, a year ago and repeating every month.

We'd then generate the first 5 ghosts for these events (say, last January, February, ... May) and later throw them out, so the correct events in the query window (say, this April) would never get generated.

Instead, generate ghosts beginning with the start of the window. The fix in D17644 to number results correctly allows us to do this.

Test Plan:
  - Made a query panel showing 5 events, scheduled an event long in the past, did not visit any of the instances of it so they didn't generate concrete objects.
  - Before the patch, near-future instances failed to show; after the patch, they show.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D17645
2017-04-10 08:48:21 -07:00
epriestley
ab06a9681c Fix two issues with user Calendar event availability cache display
Summary:
Ref T11816. Two minor issues:

  - We used `$event`, not `$next_event`, as the event providing the PHID for "Busy at <event name>". This rendered "Busy at <most future event>" on the profile instead of "Busy at <next upcoming event".
  - The TTL computation used the event start, not the event end, so we could end up rebuilding the cache too often for users busy at an event.

Test Plan:
  - Attended an event in the near future and one later on.
  - Saw profile now say "busy at <near future event>" correctly.
  - In DarkConsole "Services" tab, no longer saw unnecessary cache refills while attending an event.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D17643
2017-04-10 08:47:27 -07:00
epriestley
c9f51fd405 Write a "Developer Setup" guide for onboarding
Summary: Fixes T11561. Collect guidance about local configuration which hasn't been obvious in the past.

Test Plan:
  - Read document carefully.
  - Used `./bin/diviner generate` to generate documentation.
  - Previewed in Diviner locally:

{F4795021}

Reviewers: amckinley, chad

Reviewed By: chad

Subscribers: cspeckmim

Maniphest Tasks: T11561

Differential Revision: https://secure.phabricator.com/D17641
2017-04-10 08:36:49 -07:00
epriestley
d1421bc3a1 Add "bin/storage optimize" to run OPTIMIZE TABLE on everything
Summary:
Even with `innodb_file_per_table` enabled, individual table files on disk don't normally shrink.

For most tables, like `maniphest_task`, this is fine, since the data in the table normally never shrinks, or only shinks a tiny amount.

However, some tables (like the "worker" and "daemon" tables) grow very large during a huge import but most of the data is later deleted by garbage collection. In these cases, this lost space can be reclaimed by running `OPTIMIZE TABLE` on the tables.

Add a script to `OPTIMIZE TABLE` every table.

My primary goal here is just to reduce storage pressure on `db001` since there are a couple of "import the linux kernel" installs on that host wasting a bunch of space. We're not in any trouble, but this should buy us a good chunk of headroom.

Test Plan: Ran `bin/storage optimize` locally and manually ran `OPTIMIZE TABLE` in production, saw tables get optimized.

Reviewers: chad

Reviewed By: chad

Subscribers: cspeckmim

Differential Revision: https://secure.phabricator.com/D17640
2017-04-08 15:15:49 -07:00
epriestley
7707685733 Fix two strings with missing pht()
Summary: Fixes T12517.

Test Plan: Viewed Config application; viewed repository list.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12517

Differential Revision: https://secure.phabricator.com/D17639
2017-04-07 10:07:01 -07:00
Rabah Meradi
0bf106eeea Fix 4 typos in code
Summary: Fixes T12516

Test Plan: grep for those typos

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12516

Differential Revision: https://secure.phabricator.com/D17638
2017-04-07 04:09:56 -07:00
epriestley
9856802ba2 Disallow /source/ in robots.txt
Summary: Ref T4245. We disallow `/diffusion/` in robots.txt already because indexers tend to get lost blaming every line of every file throughout history, but didn't update the list for the `/source/` alias. Update it.

Test Plan: Visited `/robots.txt` locally.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D17637
2017-04-06 16:28:09 -07:00
epriestley
3d816e94df Rename "PhabricatorHash::digest()" to "weakDigest()"
Summary: Ref T12509. This encourages code to move away from HMAC+SHA1 by making the method name more obviously undesirable.

Test Plan: `grep`, browsed around.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17632
2017-04-06 15:43:33 -07:00
epriestley
3a3626834e Replace Remarkup calls to PhabricatorHash::digest() with SHA256
Summary:
Ref T12509. Many of the calls to HMAC+SHA1 are just to compute cachekeys for remarkup objects.

Make these use HMAC+SHA256 instead. There is no downside to swapping these since they just cause a cache miss in the worst case.

I also plan to get rid of `PhabricatorMarkupInterface` eventually, but this doesn't go that far.

Test Plan: Browsed some different types of documents (tasks, legalpad documents, phame blogs / posts, pholio mocks, etc).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17631
2017-04-06 15:43:18 -07:00
epriestley
d450a08890 Support HMAC+SHA256 with automatic key generation and management
Summary:
Ref T12509. This adds support for HMAC+SHA256 (instead of HMAC+SHA1). Although HMAC+SHA1 is not currently broken in any sense, SHA1 has a well-known collision and it's good to look at moving away from HMAC+SHA1.

The new mechanism also automatically generates and stores HMAC keys.

Currently, HMAC keys largely use a per-install constant defined in `security.hmac-key`. In theory this can be changed, but in practice essentially no install changes it.

We generally (in fact, always, I think?) don't use HMAC digests in a way where it matters that this key is well-known, but it's slightly better if this key is unique per class of use cases. Principally, if use cases have unique HMAC keys they are generally less vulnerable to precomputation attacks where an attacker might generate a large number of HMAC hashes of well-known values and use them in a nefarious way. The actual threat here is probably close to nonexistent, but we can harden against it without much extra effort.

Beyond that, this isn't something users should really have to think about or bother configuring.

Test Plan:
  - Added unit tests.
  - Used `bin/files integrity` to verify, strip, and recompute hashes.
  - Tampered with a generated HMAC key, verified it invalidated hashes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17630
2017-04-06 15:42:59 -07:00
epriestley
08a4225437 Provide "bin/files integrity" for debugging, maintaining and backfilling integrity hashes
Summary:
Ref T12470. Provides an "integrity" utility which runs in these modes:

  - Verify: check that hashes match.
  - Compute: backfill missing hashes.
  - Strip: remove hashes. Useful for upgrading across a hash change.
  - Corrupt: intentionally corrupt hashes. Useful for debugging.
  - Overwrite: force hash recomputation.

Users normally shouldn't need to run any of this stuff, but this provides a reasonable toolkit for managing integrity hashes.

I'll recommend existing installs use `bin/files integrity --compute all` in the upgrade guidance to backfill hashes for existing files.

Test Plan:
  - Ran the script in many modes against various files, saw expected operation, including:
  - Verified a file, corrupted it, saw it fail.
  - Verified a file, stripped it, saw it have no hash.
  - Stripped a file, computed it, got a clean verify.
  - Stripped a file, overwrote it, got a clean verify.
  - Corrupted a file, overwrote it, got a clean verify.
  - Overwrote a file, overwrote again, got a no-op.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12470

Differential Revision: https://secure.phabricator.com/D17629
2017-04-06 15:42:43 -07:00
epriestley
845a7d8716 Allow the PullLocal daemon to actually hibernate
Summary:
Ref T12298. The PullLocal daemon has had hibernation code for a little while, but it never actually activated because we don't sleep for more than 15 seconds in any case.

Add a maximum sleep instead and use that to control the longest sleep we'll do for hibernation purposes.

Also, when a repository or repository URI is edited, write a NEEDS_UPDATE event into the message table to make sure the daemons de-hibernate.

Test Plan: Used `bin/phd debug pull`, saw the daemon actually hibernate instead of just sleeping for 15 seconds.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12298

Differential Revision: https://secure.phabricator.com/D17635
2017-04-06 15:41:19 -07:00
epriestley
f1eeaaf59f Fix scope of "Accept" when you don't check all the "Force Accept" boxes
Summary:
Ref T12272. I wrote this correctly, then broke it by adding the simplification which treats "accept the defaults" as "accept everything".

This simplification lets us render "epriestley accepted this revision." instead of "epriestley accepted this revision onbehalf of: long, list, of, every, default, reviewer, they, have, authority, over." so it's a good thing, but make it only affect the reviewers it's supposed to affect.

Test Plan:
  - Did an accept with a force-accept available but unchecked.
  - Before patch: incorrectly accepted all possible reviewers.
  - After patch: accepted only checked reviewers.
  - Also checked the force-accept box, accepted, got a proper force-accept.

Reviewers: chad, lvital

Reviewed By: lvital

Maniphest Tasks: T12272

Differential Revision: https://secure.phabricator.com/D17634
2017-04-06 15:03:32 -07:00
epriestley
cefbdbcffe Provide a "Reviewers" attachment to "differential.revision.search"
Summary: Allow API callers to retrieve reviewer information via a new "reviewers" attachment.

Test Plan: {F4675784}

Reviewers: chad, lvital

Reviewed By: lvital

Subscribers: lvital

Differential Revision: https://secure.phabricator.com/D17633
2017-04-06 14:46:39 -07:00
epriestley
2f4ff6a850 Fix bad "editPolicy" key in Paste
Summary: Fixes T12508. Files don't have an `editPolicy`, and we started actually checking that the keys are real things in D17616.

Test Plan:
  - Before patch: created a paste, got an "editPolicy" exception.
  - After patch: created a paste that worked properly.

Reviewers: avivey, chad

Reviewed By: avivey

Maniphest Tasks: T12508

Differential Revision: https://secure.phabricator.com/D17628
2017-04-05 13:09:51 -07:00
epriestley
f2a26b2601 When requesting file data, make "Range: bytes=0-" work correctly
Summary: Ref T12219. Chrome can send requests with a "Range: bytes=0-" header, which just means "the whole file", but we don't respond correctly because of a `null` vs `0` issue.

Test Plan: Sent a raw `bytes=0-` request, saw a proper resonse.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12219

Differential Revision: https://secure.phabricator.com/D17627
2017-04-05 11:48:33 -07:00
epriestley
d1a971e221 Support "Range: bytes=123-" requests
Summary:
Ref T12219. We currently only support Range requests like "bytes=123-456", but "bytes=123-", meaning "until end of file", is valid, and Chrome can send these requests.

I suspect this is the issue with T12219.

Test Plan: Used `nc local.phacility.com 80` to pipe raw requests, saw both "bytes=123-456" and "bytes=123-" requests satisfied correctly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12219

Differential Revision: https://secure.phabricator.com/D17626
2017-04-05 11:25:44 -07:00
epriestley
63828f5806 Store and verify content integrity checksums for files
Summary:
Ref T12470. This helps defuse attacks where an adversary can directly take control of whatever storage engine files are being stored in and change data there. These attacks would require a significant level of access.

Such attackers could potentially attack ranges of AES-256-CBC encrypted files by using Phabricator as a decryption oracle if they were also able to compromise a Phabricator account with read access to the files.

By storing a hash of the data (and, in the case of AES-256-CBC files, the IV) when we write files, and verifying it before we decrypt or read them, we can detect and prevent this kind of tampering.

This also helps detect mundane corruption and integrity issues.

Test Plan:
  - Added unit tests.
  - Uploaded new files, saw them get integrity hashes.
  - Manually corrupted file data, saw it fail. Used `bin/files cat --salvage` to read it anyway.
  - Tampered with IVs, saw integrity failures.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12470

Differential Revision: https://secure.phabricator.com/D17625
2017-04-05 11:12:31 -07:00
epriestley
c4c3de7bfa Correct an issue where the wrong "Content-Length" was set for partial content responses
Summary: Ref T8266. Although we compute this correctly above, we ignored it when actually setting the header. Use the computed value to set the "Content-Length" header. This is consistent with the spec/documentation.

Test Plan: Before, some audio (like `rain.mp3`) was pretty spotty about loading in Safari. It now loads consistently for me locally.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8266

Differential Revision: https://secure.phabricator.com/D17624
2017-04-05 10:03:48 -07:00
epriestley
45fc4f6f64 Iterate over ranges correctly for encryped files
Summary:
Fixes T12079. Currently, when a file is encrypted and a request has "Content-Range", we apply the range first, //then// decrypt the result. This doesn't work since you can't start decrypting something from somewhere in the middle (at least, not with our cipher selection).

Instead: decrypt the result, //then// apply the range.

Test Plan: Added failing unit tests, made them pass

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12079

Differential Revision: https://secure.phabricator.com/D17623
2017-04-05 09:56:30 -07:00
epriestley
f70ff34d97 Fix a copy/paste typo with sticky accept
The root issue here is actually just that I cherry-picked stable locally
but did not push it. However, this is a minor issue I also caught while
double-checking things.

Auditors: chad
2017-04-04 18:33:59 -07:00
epriestley
58011a4e8e Upgrade File content hashing to SHA256
Summary:
Ref T12464. This defuses any possible SHA1-collision attacks by using SHA256, for which there is no known collision.

(SHA256 hashes are larger -- 256 bits -- so expand the storage column to 64 bytes to hold them.)

Test Plan:
  - Uploaded the same file twice, saw the two files generate the same SHA256 content hash and use the same underlying data.
  - Tried with a fake hash algorihtm ("quackxyz") to make sure the failure mode worked/degraded correctly if we don't have SHA256 for some reason. Got two valid files with two copies of the same data, as expected.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12464

Differential Revision: https://secure.phabricator.com/D17620
2017-04-04 16:23:08 -07:00
epriestley
440ef5b7a7 Remove SHA1 file content hashing and make Files work without any hashing
Summary:
Ref T12464. We currently use SHA1 to detect when two files have the same content so we don't have to store two copies of the data.

Now that a SHA1 collision is known, this is theoretically dangerous. T12464 describes the shape of a possible attack.

Before replacing this with something more robust, shore things up so things work correctly if we don't hash at all. This mechanism is entirely optional; it only helps us store less data if some files are duplicates.

(This mechanism is also less important now than it once was, before we added temporary files.)

Test Plan: Uploaded multiple identical files, saw the uploads work and the files store separate copies of the same data.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12464

Differential Revision: https://secure.phabricator.com/D17619
2017-04-04 16:22:10 -07:00
epriestley
1e181f0781 Deprecate "file.uploadhash"
Summary:
Ref T12464. This is a very old method which let you create a file on the server by referring to data which already existed in another file.

Basically, long ago, `arc` could say "Do you already have a file with hash X?" and just skip some work if the server did.

`arc` has not called this method since D13017, in May 2015.

Since it's easy to do so, just make this method pretend that it never has the file. Very old clients will continue to work, since they would expect this response in the common case and continue by uploading data.

Test Plan:
  - Grepped for `uploadhash` in Phabricator and Arcanist.
  - Called the method with the console, verified it returned `null`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12464

Differential Revision: https://secure.phabricator.com/D17618
2017-04-04 16:18:26 -07:00
epriestley
873b39be82 Remove PhabricatorFile::buildFromFileDataOrHash()
Summary:
Ref T12464. This is a very old method which can return an existing file instead of creating a new one, if there's some existing file with the same content.

In the best case this is a bad idea. This being somewhat reasonable predates policies, temporary files, etc. Modern methods like `newFromFileData()` do this right: they share underlying data in storage, but not the actual `File` records.

Specifically, this is the case where we get into trouble:

  - I upload a private file with content "X".
  - You somehow generate a file with the same content by, say, viewing a raw diff in Differential.
  - If the diff had the same content, you get my file, but you don't have permission to see it or whatever so everything breaks and is terrible.

Just get rid of this.

Test Plan:
  - Generated an SSH key.
  - Viewed a raw diff in Differential.
  - (Did not test Phragment.)

Reviewers: chad

Reviewed By: chad

Subscribers: hach-que

Maniphest Tasks: T12464

Differential Revision: https://secure.phabricator.com/D17617
2017-04-04 16:18:00 -07:00
epriestley
45b386596e Make the Files "TTL" API more structured
Summary:
Ref T11357. When creating a file, callers can currently specify a `ttl`. However, it isn't unambiguous what you're supposed to pass, and some callers get it wrong.

For example, to mean "this file expires in 60 minutes", you might pass either of these:

  - `time() + phutil_units('60 minutes in seconds')`
  - `phutil_units('60 minutes in seconds')`

The former means "60 minutes from now". The latter means "1 AM, January 1, 1970". In practice, because the GC normally runs only once every four hours (at least, until recently), and all the bad TTLs are cases where files are normally accessed immediately, these 1970 TTLs didn't cause any real problems.

Split `ttl` into `ttl.relative` and `ttl.absolute`, and make sure the values are sane. Then correct all callers, and simplify out the `time()` calls where possible to make switching to `PhabricatorTime` easier.

Test Plan:
- Generated an SSH keypair.
- Viewed a changeset.
- Viewed a raw diff.
- Viewed a commit's file data.
- Viewed a temporary file's details, saw expiration date and relative time.
- Ran unit tests.
- (Didn't really test Phragment.)

Reviewers: chad

Reviewed By: chad

Subscribers: hach-que

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17616
2017-04-04 16:16:28 -07:00
epriestley
2896da384c Only require POST to fetch file data if the viewer is logged in
Summary:
Ref T11357. In D17611, I added `file.search`, which includes a `"dataURI"`. Partly, this is building toward resolving T8348.

However, in some cases you can't GET this URI because of a security measure:

  - You have not configured `security.alternate-file-domain`.
  - The file isn't web-viewable.
  - (The request isn't an LFS request.)

The goal of this security mechanism is just to protect against session hijacking, so it's also safe to disable it if the viewer didn't present any credentials (since that means there's nothing to hijack). Add that exception, and reorganize the code a little bit.

Test Plan:
  - From the browser (with a session), tried to GET a binary data file. Got redirected.
  - Got a download with POST.
  - From the CLI (without a session), tried to GET a binary data file. Go a download.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17613
2017-04-04 16:16:01 -07:00
epriestley
2369fa38e1 Provide a modern ("v3") API for querying files ("file.search")
Summary: Ref T11357. Implements a modern `file.search` for files, and freezes `file.info`.

Test Plan: Ran `file.search` from the Conduit console.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17612
2017-04-04 16:15:36 -07:00
epriestley
260a08a128 Move Files editing and commenting to EditEngine
Summary:
Ref T11357. This moves editing and commenting (but not creation) to EditEngine.

Since only the name is really editable, this is pretty straightforward.

Test Plan: Renamed files; commented on files.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17611
2017-04-04 16:15:11 -07:00
epriestley
8500f78e45 Move Files to ModularTransactions
Summary: Ref T11357. A lot of file creation doesn't go through transactions, so we only actually have one real transaction type: editing a file name.

Test Plan:
Created and edited files.

{F4559287}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17610
2017-04-04 10:25:05 -07:00
epriestley
5e44711218 Provide a missing feed transaction string for space creation
Summary:
Fixes T12502. This transaction probably should not be getting picked for feed rendering, but it currently does get selected in some cases.

This should probably be revisited eventually (e.g., when Maniphest moves to ModularTransactions) but just fix the brokenness for now.

Test Plan:
  - Created a task in a space.
  - Viewed feed.
  - Saw the story render with readable text.

{F4555747}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12502

Differential Revision: https://secure.phabricator.com/D17609
2017-04-04 10:24:11 -07:00
epriestley
9ebb5f8cda Don't downgrade accepts on update (fix "sticky accept")
Summary:
Fixes T12496. Sticky accept was accidentally impacted by the "void" changes in D17566.

Instead, don't always downgrade all accepts/rejects: on update, we only want to downgrade accepts.

Test Plan:
  - With sticky accept off, updated an accepted revision: new state is "needs review".
  - With sticky accept on, updated an accepted revision: new state is "accepted" (sticky accept working correctly).
  - Did "reject" + "request review" to make sure that still works, worked fine.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12496

Differential Revision: https://secure.phabricator.com/D17605
2017-04-03 09:55:22 -07:00
epriestley
163e1ec442 Expose the commit/task/revision relationship edges to "edge.search"
Summary: Fixes T12480.

Test Plan: {F4465908}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12480

Differential Revision: https://secure.phabricator.com/D17604
2017-04-02 19:49:55 -07:00
epriestley
009aff1a23 Return task descriptions from "maniphest.search"
Summary:
Fixes T12461. This returns the field as a dictionary with a `"raw"` value, so we could eventually do this if we want without breaking the API:

```
{
  "type": "remarkup",
  "raw": "**raw**",
  "html": "<strong>raw</strong>",
  "text": "raw"
}
```

Test Plan: Called `maniphest.search`, reviewed output.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12461

Differential Revision: https://secure.phabricator.com/D17603
2017-04-02 17:44:22 -07:00
epriestley
7e6f37fffb Rename "ElasticSearch" filenames to "Elasticsearch" (2/2)
Sometimes git does some odd magic on case-insensitive filesystems, try to
trick it.

Auditors: chad
2017-04-02 14:59:36 -07:00
epriestley
a9e2732a5c Spell "Elasticsearch" correctly, not "ElasticSearch"
Summary: Ref T12450. These are like 95% my fault, but Elastic appears to spell the name "Elasticsearch" consistently in their branding.

Test Plan: `grep ElasticSearch`

Reviewers: chad, 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17601
2017-04-02 14:58:59 -07:00
epriestley
304d19f92a After a fulltext write to a particular service fails, keep trying writes to other services
Summary:
Ref T12450. Currently, if a write fails, we stop and don't try to write to other index services. There's no technical reason not to keep trying writes, it makes some testing easier, and it would improve behavior in a scenario where engines are configured as "primary" and "backup" and the primary service is having some issues.

Also, make "no writable services are configured" acceptable, rather than an error. This state is probably goofy but if we want to detect it I think it should probably be a config-validation issue, not a write-time check. I also think it's not totally unreasonable to want to just turn off all writes for a while (maybe to reduce load while you're doing a background update).

Test Plan:
  - Configured a bad ElasticSearch engine and a good MySQL engine.
  - Ran `bin/search index ... --force`.
  - Saw MySQL get updated even though ElasticSearch failed.

Reviewers: chad, 20after4

Reviewed By: 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17599
2017-04-02 13:47:52 -07:00
epriestley
0f144d29e9 When "cluster.search" changes, don't trust the old index versions
Summary:
Ref T12450. We track a "document version" for updating search indexes, so that if a document is rapidly updated many times in a row we can skip most of the work.

However, this version doesn't consider "cluster.search" configuration, so if you add a new service (like a new ElasticSearch host) we still think that every document is up-to-date. When you run `bin/search index` to populate the index (without `--force`), we just do nothing.

This isn't necessarily very obvious. D17597 makes it more clear, by printing "everything was skipped and nothing happened" at the end.

Here, fix the issue by considering the content of "cluster.search" when computing fulltext document versions: if you change `cluster.search`, we throw away the version index and reindex everything.

This is slightly more work than we need to do, but changes to "cluster.search" are rare and this is much easier than trying to individually track which versions of which documents are in which services, which probably isn't very useful anyway.

Test Plan:
  - Ran `bin/search index --type project`, saw everything get skipped.
  - Changed `cluster.search`.
  - Ran `search index` again, saw everything get updated.
  - Ran a third time without changing `cluster.search`, everything was properly skipped.

Reviewers: chad, 20after4

Reviewed By: 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17598
2017-04-02 13:45:48 -07:00
epriestley
bd93978200 Count and report skipped documents from "bin/search index"
Summary:
Ref T12450. There's currently a bad behavior where inserting a document into one search service marks it as up to date everywhere.

This isn't nearly as obvious as it should be because `bin/search index` doesn't make it terribly clear when a document was skipped because the index version was already up to date.

When running `bin/seach index` without `--force` or `--background`, keep track of updated vs not-updated documents and print out some guidance. In other configurations, try to provide more help too.

Test Plan: {F4452134}

Reviewers: chad, 20after4

Reviewed By: 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17597
2017-04-02 13:45:30 -07:00
epriestley
6d81675032 Remove "url" from Elasticsearch index
Summary:
Ref T12450. This was added a very very long time ago (D2298).

I don't want to put this in the upstream index anymore because I don't want to encourage third parties to develop software which reads the index directly. Reading the index directly is a big skeleton key which bypasses policy checks.

This was added before much of the policy model existed, when that wasn't as much of a concern. On a tecnhnical note, this also doesn't update when `phabricator.base-uri` changes.

This can be written as a search index extension if an install relies on it for some bizarre reason, although none should and I'm unaware of any actual use cases in the wild for it, even at Facebook.

Test Plan: Indexed some random stuff into ElasticSearch.

Reviewers: chad, 20after4

Reviewed By: chad

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17600
2017-04-02 13:26:45 -07:00
epriestley
287e708c4d Adjust and wordsmith Search documentation
Summary:
Ref T12450. General adjustments:

  - Try to make "Cluster: Search" more about "stuff in common + types" instead of pretty much all being Elastic-specific, so we can add Solr or whatever later.
  - Provide guidance about rebuilding indexes after making a change.
  - Simplify the basic examples, then provide a more advanced example at the ed.
  - Really try to avoid suggesting anyone configure Elasticsearch ever for any reason.

Test Plan: Read documents, previewed in remarkup.

Reviewers: chad, 20after4

Reviewed By: 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17602
2017-04-02 13:09:07 -07:00
epriestley
64234535e3 Remove FIELD_KEYWORDS, index project slugs as body content
Summary:
D17384 added a "keywords" field but only partially implemented it.

  - Remove this field.
  - Index project slugs as part of the document body instead.

Test Plan:
  - Ran `bin/search index PHID-PROJ-... --force`.
  - Found project by searching for a unique slug.

Reviewers: chad, 20after4

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17596
2017-04-02 09:36:32 -07:00
epriestley
515cb98819 When running unit tests, ignore any custom task fields
Summary:
If you have `maniphest.custom-field-definitions` set to include "required" fields, a bunch of tests which create tasks can fail.

To avoid this, reset this config while running tests.

This mechanism should probably be more general (e.g., reset all config by default, only whitelist some config) but just fix this for now since it's a one-liner and doesn't make eventual cleanup any harder.

Test Plan: Ran `arc unit`, hitting tests that create tasks.

Reviewers: chad, 20after4

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17595
2017-04-02 09:36:17 -07:00
Daniel Stone
1c5503cb29 Custom fields: Render 'required' for tokenizer fields
Summary: When building a tokenizer-based edit control for a custom field (e.g. a datasource type), preserve a field validation error whilst building edit controls.

Test Plan:
  - Create custom datasource field, set it to required
  - Observe that 'Required' does not appear next to control
  - Apply patch
  - Observe 'Required' appears next to control

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17592
2017-04-02 15:26:26 +00:00
Chad Little
7ab4e7dbce Allow Owner Packages to be in a Dashboard Panel
Summary: Ref T12324. Add back Owners.

Test Plan: read carefully

Reviewers: epriestley, eadler

Reviewed By: eadler

Subscribers: Korvin

Maniphest Tasks: T12324

Differential Revision: https://secure.phabricator.com/D17588
2017-03-30 15:13:40 -07:00
Chad Little
eb6f4c4a28 Update PhortuneLanding page UI
Summary: Minor, uses 'user-circle' for account, and merchant logo for merchants in lists.

Test Plan: View the landing page, see updated logos and icons.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17586
2017-03-30 12:27:41 -07:00
Chad Little
86673486c0 Move Phortune Contollers into folders
Summary: Move individual controller files into cooresponding folders. Makes it easier to locate sections and expand without clutter. Also made "chargelist" part of account since it's tied to having an account specifically.

Test Plan: Vist charges, merchants, subscription, accounts, and other pages. No errors from file move.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17587
2017-03-30 12:26:15 -07:00
Mukunda Modell
cb1d904654 Make sure writes go to the right cluster
Summary:
Two little issues

1. there was an extra call to getHostForWrite,
2. The engine instance was shared between multiple service definitions so it
was overwriting the list of writable hosts from one service with hosts from another.

Test Plan:
tested in wikimedia production with multiple services defined like this:

```language=json
 [
        {
          "hosts": [
            {
              "host": "search.svc.codfw.wmnet",
              "protocol": "https",
              "roles": {
                "read": true,
                "write": true
              },
              "version": 5
            }
          ],
          "path": "/phabricator",
          "port": 9243,
          "type": "elasticsearch"
        },
        {
          "hosts": [
            {
              "host": "search.svc.eqiad.wmnet",
              "protocol": "https",
              "roles": {
                "read": true,
                "write": true
              },
              "version": 5
            }
          ],
          "path": "/phabricator",
          "port": 9243,
          "type": "elasticsearch"
        }
      ]
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17581
2017-03-30 18:08:05 +00:00
Mukunda Modell
67a1c40476 Set content-type to application/json
Summary:
Elasticsearch really wants a raw json body and it fails to accept
the request as of es version 5.3

Test Plan:
Tested with elasticsearch 5.2 and 5.3.

Before this change 5.2 worked but 5.3 failed with
`HTTP/406 "Content-Type header [application/x-www-form-urlencoded] is not supported"` [1]

After this change, both worked.

[1] https://phabricator.wikimedia.org/P5158

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17580
2017-03-30 18:07:47 +00:00
Mukunda Modell
654f0f6043 Make messages translatable and more sensible.
Summary:
These exception messages & comments didn't quite match reality.
Fixed and added pht() around a couple of them.

Test Plan: I didn't test this :P

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17578
2017-03-28 23:17:35 +00:00
epriestley
add1038109 Don't summon the emoji autocompleter for ":3"
Summary:
Fixes T12460. Also ":)", ":(", ":/", and oldschool ":-)" variants.

Not included are variants with actual letters (`:D`, `:O`, `:P`) and obscure variants (`:^)`, `:*)`).

Test Plan: Typed `:3` (no emoji summoned). Typed `:dog3` (emoji summoned). Typed `@3` (user autocomplete summoned).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12460

Differential Revision: https://secure.phabricator.com/D17577
2017-03-28 15:50:56 -07:00
epriestley
88798354e8 Soften a possible cluster search setup fatal
Summary:
Ref T12450. The way that config repair and setup issues interact is kind of complicated, and if `cluster.search` is invalid we may end up using `cluster.search` before we repair it.

I poked at things for a bit but wasn't confident I could get it to consistently repair before we use it without doing a big messy change.

The only thing that really matters is whether "type" is valid or not, so just put a slightly softer/more-tailored check in for that.

Test Plan:
  - With `"type": "elastic"`, loaded setup issues.
  - Before patch: hard fatal.
  - After patch: softer fatal with more useful messaging.

{F4321048}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17576
2017-03-28 15:28:16 -07:00
epriestley
5f939dcce0 Re-run config validation from bin/search
Summary:
Ref T12450. Normally, we validate config when:

  - You restart the webserver.
  - You edit it with `bin/config set ...`.
  - You edit it with the web UI.

However, you can also change config by editing `local.json`, `some_env.conf.php`, a `SiteConfig` class, etc. In these cases, you may miss config warnings.

Explicitly re-run search config checks from `bin/search`, similar to the additional database checks we run from `bin/storage`, to try to produce a better error message if the user has made a configuration error.

Test Plan:
```
$ ./bin/search init
Usage Exception: Setting "cluster.search" is misconfigured: Invalid search engine type: elastic. Valid types are: elasticsearch, mysql.
```

Reviewers: chad, 20after4

Reviewed By: 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17574
2017-03-28 14:53:26 -07:00
epriestley
8879118b69 Fix a mid-air collision around SearchService roles
My D17571 didn't interact nicely with D17564, which added callsites for one of
the methods I removed.

Auditors: 20after4
2017-03-28 14:01:45 -07:00
epriestley
c40be811ea Fix isReadable() and isWritable() in SearchService
Summary:
Ref T12450. Minor cleanup:

  - setRoles() has no callers.
  - getRoles() has no callers (these two methods are leftovers from an earlier iteration of the change).
  - The `hasRole()` logic doesn't work since nothing calls `setRole()`.
  - `hasRole()` has only `isreadable/iswritable` as callers.
  - The `isReadable()/isWritable()` logic doesn't work since `hasRole()` doesn't work.

Instead, just check if there are any readable/writable hosts. `Host` already inherits its config from `Service` so this gets the same answer without any fuss.

Also add some read/write constants to make grepping this stuff a little easier.

Test Plan:
  - Grepped for all removed symbols, saw only newer-generation calls in `Host`.
  - See next diff for use of `isWritable()`.

Reviewers: chad, 20after4

Reviewed By: 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17571
2017-03-28 13:58:46 -07:00
epriestley
c22693ff29 Remove PhabricatorSearchEngineTestCase
Summary:
Ref T12450. This is now pointless and just asserts that `cluster.search` has a default value.

We might restore a fancier version of this eventually, but get rid of this for now.

Test Plan: Scruitinized the test case.

Reviewers: chad, 20after4

Reviewed By: 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17573
2017-03-28 13:57:55 -07:00
epriestley
e7c76d92d5 Make bin/search init messaging a little more consistent
Summary:
Ref T12450. This mostly just smooths out the text a little to improve consistency. Also:

  - Use `isWritable()`.
  - Make the "skipping because not writable" message more clear and tailored.
  - Try not to use the word "index" too much to avoid confusion with `bin/search index` -- instead, talk about "initialize a service".

Test Plan: Ran `bin/search init` with a couple of different (writable / not writable) configs, saw slightly clearer messaging.

Reviewers: chad, 20after4

Reviewed By: 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17572
2017-03-28 13:57:37 -07:00
Mukunda Modell
699228c73b Address some New Search Configuration Errata
Summary:
  [ ] Write an "Upgrading: ..." guidance task with narrow instructions for installs that are upgrading.
  [ ] Do we need to add an indexing activity (T11932) for installs with ElasticSearch?
  [ ] We should more clearly detail exactly which versions of ElasticSearch are supported (for example, is ElasticSearch <2 no longer supported)? From T9893 it seems like we may //only// have supported ElasticSearch <2 before, so are the two regions of support totally nonoverlapping and all ElasticSearch users will need to upgrade?
  [ ] Documentation should provide stronger guidance toward MySQL and away from Elastic for the vast majority of installs, because we've historically seen users choosing Elastic when they aren't actually trying to solve any specific problem.
  [ ] When users search for fulltext results in Maniphest and hit too many documents, the current behavior is approximately silent failure (see T12443). D17384 has also lowered the ceiling for ElasticSearch, although previous changes lowered it for MySQL search. We should not fail silently, and ideally should build toward T12003.
  [ ] D17384 added a new "keywords" field, but MySQL does not search it (I think?). The behavior should be as consistent across MySQL and Elastic as we can make it. Likely cleaner is giving "Project" objects a body, with "slugs" and "description" separated by newlines?
  [ ] `PhabricatorSearchEngineTestCase` is now pointless and only detects local misconfigurations.
  [ ] It would be nice to build a practical test suite instead, where we put specific documents into the index and then search for them. The upstream test could run against MySQL, and some `bin/search test` could run against a configured engine like ElasticSearch. This would make it easier to make sure that behavior was as uniform as possible across engine implementations.
  [ ] Does every assigned task now match "user" in ElasticSearch?
  [x] `PhabricatorElasticFulltextStorageEngine` has a `json_encode()` which should be `phutil_json_encode()`.
  [ ] `PhabricatorSearchService` throws an untranslated exception.
  [ ] When a search cluster is down, we probably don't degrade with much grace (unhandled exception)?
  [ ] I haven't run bin/search init, but bin/search index doesn't warn me that I may want to. This might be worth adding. The UI does warn me.
  [ ] bin/search init warns me that the index is "incorrect". It might be more clear to distinguish between "missing" and "incorrect", since it's more comforting to users to see "everything is as we expect, doing normal first-time setup now" than "something is wrong, fixing it".
  [ ] CLI message "Initializing search service "ElasticSearch"" does not end with a period, which is inconsistent with other UI messages.
  [ ] It might be nice to let bin/search commands like init and index select a specific service (or even service + host) to act on, as bin/storage --ref ... now does. You can generally get the result you want by fiddling with config.
  [ ] When a service isn't writable, bin/search init reports "Search cluster has no hosts for role "write".". This is accurate but does not provide guidance: it might be more useful to the user to explain "This service is not writable, so we're skipping index check for it.".
  [x] Even with write off for MySQL, bin/search index --type task --trace still updates MySQL, I think? I may be misreading the trace output. But this behavior doesn't make sense if it is the actual behavior, and it seems like reindexAbstractDocument() uses "all services", not "writable services", and the MySQL engine doesn't make sure it's writable before indexing.
  [x] Searching or user fails to find task Grant users tokens when a mention is created, suggesting that stemming is not working.
  [x] Searching for users finds that task, but fails to find a task containing "per user per month" in a comment, also suggesting that stemming is not working.
  [x] Searching for maniphest fails to find task maniphest.query elephant, suggesting that tokenization in ElasticSearch is not as good as the MySQL tokenization for these words (see D17330).
  [x] The "index incorrect" warning UI uses inconsistent title case.
  [x] The "index incorrect" warning UI could format the command to be run more cleanly (with addCommand(), I think).

refs T12450

Test Plan:
* Stared blankly at the code.
* Disabled 'write' role on mysql fulltext service.
* Edited a task, ran search indexer, verified that the mysql index wasn't being updated.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17564
2017-03-28 20:19:38 +00:00
epriestley
2fbc9a52da Allow users to "Force accept" package reviews if they own a more general package
Summary:
Ref T12272. If you own a package which owns "/", this allows you to force-accept package reviews for packages which own sub-paths, like "/src/adventure/".

The default UI looks something like this:

```
[X] Accept as epriestley
[X] Accept as Root Package
[ ] Force accept as Adventure Package
```

By default, force-accepts are not selected.

(I may do some UI cleanup and/or annotate "because you own X" in the future and/or mark these accepts specially in some way, particularly if this proves confusing along whatever dimension.)

Test Plan: {F4314747}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12272

Differential Revision: https://secure.phabricator.com/D17569
2017-03-28 11:51:40 -07:00
epriestley
ddc02ce420 When voiding "Accept" reviews, also void "Reject" reviews
Summary: Ref T10967. This change is similar to D17566, but for rejects.

Test Plan:
  - Create a revision as A, with reviewer B.
  - Reject as B.
  - Request review as A.
  - Before patch: stuck in "rejected".
  - After patch: transitions back to "needs review".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17568
2017-03-28 11:51:06 -07:00
epriestley
415ad78484 Remove old code for "Request Review" action from Differential
Summary: Ref T10967. This moves all remaining "request review" pathways (just `differential.createcomment`) to the new code, and removes the old action.

Test Plan: Requested review on a revision, `grep`'d for the action constant.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17567
2017-03-28 11:50:40 -07:00
epriestley
aea46e55da Fix an issue where "Request Review" of a fully-accepted revision would transition to "Accepted"
Summary:
Ref T10967. This is explained in more detail in T10967#217125

When an author does "Request Review" on an accepted revision, void (in the sense of "cancel out", like a bank check) any "accepted" reviewers on the current diff.

Test Plan:
  - Create a revision with author A and reviewer B.
  - Accept as B.
  - "Request Review" as A.
  - (With sticky accepts enabled.)
  - Before patch: revision swithced back to "accepted".
  - After patch: the earlier review is "voided" by te "Request Review", and the revision switches to "Review Requested".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17566
2017-03-28 11:50:15 -07:00
epriestley
7d3956bec1 Correct spelling of "Dasbhoard"
Summary: Before the speling pollice lock us in prisun.

Test Plan: Used a dicationairey.

Reviewers: chad, jmeador

Reviewed By: jmeador

Differential Revision: https://secure.phabricator.com/D17570
2017-03-28 10:04:26 -07:00
Mukunda Modell
9e2f263bb4 Add repositories to fulltext search index.
Summary:
This implements a simplistic `PhabricatorRepositoryFulltextEngine`
Currently only the repository name, description, timestamps and
status are indexed.

Note: I had to change the `search index` workflow to disambiguate
PhabricatorRepository from PhabricatorRepositoryCommit

Test Plan:
* ran `./bin/search index --type PhabricatorRepository --force`
 * searched for some repositories. Saw reasonable results matching on either title or description.
* Edited a repository in the web ui
 * Added unique key words to the repo description.
 * I was then able to find that repo by searching for the new keywords.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin

Tags: #search, #diffusion

Differential Revision: https://secure.phabricator.com/D17300
2017-03-28 07:58:22 +00:00
Chad Little
76404c5fdb Cleaner fullscreen / preview states for Remarkup bar
Summary: General CSS and usability touchup of the Remarkup bar states for fullscreen and preview. Larger fonts, more spacing, some hint of the underlying page. Disable buttons that can't be used in preview mode.

Test Plan:
Formal test coming with mobile, browsers. This is a kick the tires upload.

{F4283448}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17563
2017-03-27 09:19:23 -07:00
Mukunda Modell
e41c25de50 Support multiple fulltext search clusters with 'cluster.search' config
Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.

When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.

Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.

These two roles make it possible to have any combination of:

* read-only
* write-only
* read-write
* disabled

This 'roles' mechanism is extensible to add new roles should that be needed in the future.

In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).

The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)

Hopefully this is useful in the upstream as well.

Remaining TODO:

* test cases
* documentation

Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.

Tested with elasticsearch versions 2.4 and 5.2 using the following config:

```lang=json
  "cluster.search": [
    {
      "type": "elasticsearch",
      "hosts": [
        {
          "host": "localhost",
          "roles": { "read": true, "write": true }
        }
      ],
      "port": 9200,
      "protocol": "http",
      "path": "/phabricator",
      "version": 5
    },
    {
      "type": "mysql",
      "roles": { "write": true }
     }
  ]

Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Tags: #elasticsearch, #clusters, #wikimedia

Differential Revision: https://secure.phabricator.com/D17384
2017-03-26 08:16:47 +00:00
epriestley
a41d158490 Only hibernate the Taskmaster after 15 seconds of inactivity
Under some workloads, the taskmaster may hibernate and launch more rapidly
than it should. Require 15 seconds of inactivity before hibernating. Also
hibernate for longer.

Auditors: chad
2017-03-25 05:01:32 -07:00
epriestley
2cda280cde Make the default Trigger hibernation 3 minutes instead of 5 seconds
The `min()` vs `max()` fix in D17560 meant that the Trigger daemon only
hibernates for 5 seconds, so we do a full GC sweep every 5 seconds. This ends
up eating a fair amount of CPU for no real benefit.

The GC cursors should move to persistent storage, but just bump this default
up in the meantime.

Auditors: chad
2017-03-25 04:14:32 -07:00
epriestley
b4effdf26c Fix a rendering fatal for unknown edge constants
If we try to render an edge transaction which uses unknown edge constants,
it turns out we fatal. Degrade instead. This happened when viewing very old
badges.

Auditors: chad
2017-03-24 16:58:48 -07:00
Chad Little
186460888d Funbeta Badges
Summary: Ships Badges. I can write up some basic docs too if needed.

Test Plan: /applications/

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12270

Differential Revision: https://secure.phabricator.com/D17360
2017-03-24 21:15:42 +00:00
epriestley
080bf064c4 Remove obsolete Badges edge types
Summary: Ref T12270. These no longer have any callsites.

Test Plan: Used `grep` to search for each edge class constant, found no hits.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12270

Differential Revision: https://secure.phabricator.com/D17562
2017-03-24 14:11:46 -07:00
epriestley
6f80a04699 Paginate the profile badges view
Summary: Ref T12270. Adds a pager, plus a few little cleanups from copy/paste and accumulated cruft.

Test Plan:
  - Paginated a user with 180 badges.
  - Viewed a user with 0 badges.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12270

Differential Revision: https://secure.phabricator.com/D17561
2017-03-24 14:10:59 -07:00
epriestley
8b553d2f18 Allow taskmaster daemons to hibernate
Summary: Ref T12298. Like PullLocal daemons, this allows the last daemon in the pool to hibernate if there's no work to be done, and awakens the pool when work arrives.

Test Plan:
  - Ran `bin/phd debug task --trace`.
  - Saw the pool hibernate and look for tasks.
  - Commented on an object.
  - Saw the pool wake up and process the queue.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12298

Differential Revision: https://secure.phabricator.com/D17559
2017-03-24 13:51:37 -07:00
epriestley
3cdabb9588 Provide a hint that submitting a Conduit call shows you how to encode particular parameters
Summary: Ref T12447.

Test Plan: {F4270003}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12447

Differential Revision: https://secure.phabricator.com/D17557
2017-03-24 13:15:03 -07:00
epriestley
24b6c7d718 Allow users to resign if they have authority over any reviewer
Summary:
Ref T11050. The old rule was "you can only resign if you're a reviewer".

With the new behavior of "resign", the rule should be "you can resign if you're a reviewer, or you have authority over any reviewer". Make it so.

Also fixes T12446. I don't know how to reproduce that but I'm pretty sure this'll fix it?

Test Plan:
  - Could not resign from a revision with no authority/reviewer.
  - Resigned from a revision with myself as a reviewer.
  - Resigned from a revision with a package I owned as a reviewer.
  - Could not resign from a revision I had already resigned from.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12446, T11050

Differential Revision: https://secure.phabricator.com/D17558
2017-03-24 13:14:47 -07:00
epriestley
daeb94561f When destroying Calendar events, destroy invitees and notifications
Summary: Fixes T12395.

Test Plan: Ran `bin/remove destroy E... --trace`, saw invitee and notification destruction.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12395

Differential Revision: https://secure.phabricator.com/D17555
2017-03-24 09:21:13 -07:00
epriestley
0ffde484e5 Give Daemons a mobile menu
Summary: Fixes T12422.

Test Plan: {F4269080}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12422

Differential Revision: https://secure.phabricator.com/D17554
2017-03-24 09:19:56 -07:00