1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-24 15:52:41 +01:00
Commit graph

14836 commits

Author SHA1 Message Date
epriestley
65f13b156f Improve "refengine" performance for testing large numbers of Mercurial branches
Summary:
See PHI158. In the RefEngine, we test if any old branch positions have been removed from the repository. This is uncommon (but not impossible) in Mercurial, and corresponds to users deleting branches in Git.

Currently, we end up running `hg log` for each position, in parallel. Because of Python's large startup overhead, this can be resource intensive for repositories with a large number of branches.

We have to do this in the general case because the caller may be asking us to resolve `tip`, `newfeature`, `tip~3`, `9`, etc. However, in the specific case where the refs are 40-digit hashes, we can bulk resolve them if they exist, like this:

```
hg log ... --rev (abcd or def0 or ab12 or ...)
```

In the general case, we could probably do less of this than we currently do (instead of testing all old heads, we could prune the list by removing commits which we know are still pointed to by current heads) but that's a slightly more involved change and the effect here is already dramatic.

Test Plan:
Verified that CPU usage drops from ~110s -> ~0.9s:

Before:

```
epriestley@orbital ~/dev/phabricator $ time ./bin/repository refs nss
Updating refs in "nss"...
Done.

real	0m14.676s
user	1m24.714s
sys	0m21.645s
```

After:

```
epriestley@orbital ~/dev/phabricator $ time ./bin/repository refs nss
Updating refs in "nss"...
Done.

real	0m0.861s
user	0m0.882s
sys	0m0.213s
```

  - Manually resolved `blue`, `tip`, `9`, etc., got expected results.
  - Tried to resolve invalid hashes, got expected result (no resolution).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18717
2017-10-20 11:09:14 -07:00
epriestley
cbcab60fbb Remove obsolete instructions from information on prototype applications
Summary: Most of this document is no longer relevant, since we're happy to work on prototypes if you're paying us and no longer have any meaningful free support.

Test Plan: Read document.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18719
2017-10-20 11:00:00 -07:00
epriestley
01b1854fb4 Fix an issue with attempting to index comments on packages
Summary: See rPcd14194a329788d5fff6365bcade278fd18f3612 for a similar change. Implement `getApplicationTransactionCommentObject()` to return `null` explicitly.

Test Plan: Ran `bin/search index --type ownerspackage`, got indexing after change.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18715
2017-10-20 09:38:45 -07:00
epriestley
1755ec2429 Show more detailed hints about draft revisions in the UI
Summary: Ref T2543. When revisions are in the draft state, tell the user what we're waiting for or why they aren't moving forward.

Test Plan: {F5228840}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18714
2017-10-20 08:40:17 -07:00
epriestley
bfabe49c5a Start revisions in "Draft" if prototypes are enabled
Summary: Ref T2543. This is a less ambitious version of the rule in D18628, which I backed off from, since I think this probably still has a fair number of loose ends to tie up.

Test Plan: Created a revision locally.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18713
2017-10-20 08:39:58 -07:00
epriestley
d36f98a15a Clarify acceptable values for --threshold in search ngrams
Summary: See D18710.

Test Plan: o_O

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18712
2017-10-17 14:32:25 -07:00
epriestley
63d1230ade Parameterize the common ngrams threshold
Summary:
Ref T13000. Since other changes have generally made the ngrams table manageable, I'm not planning to enable common ngrams by default at this time.

Instead, make the threshold configurable with "--threshold" so we can guide installs through tuning this if they want (e.g. PHI110), and tune hosted instances.

(This might eventually become automatic, but just smoothing this bit off for now feels reasonable to me.)

Test Plan: Ran with `--reset`, and with various invalid and valid `--threshold` arguments.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18710
2017-10-17 14:13:49 -07:00
epriestley
819b833607 Tweak rate limiting point counts for omnipotent users
Summary:
Ref T13008. We haven't hit any issues with this, but I can imagine we might in the future.

When one host makes an intracluster request to another host, the `$viewer` ends up as the omnipotent viewer. This viewer isn't logged in, so they'll currently accumulate rate limit points at a high rate.

Instead, don't give them any points. These requests are always legitimate, and if they originated from a user request, that request should be the one getting rate limited.

Test Plan: Browsed around.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13008

Differential Revision: https://secure.phabricator.com/D18708
2017-10-16 06:43:54 -07:00
epriestley
0e645b8f11 Disconnect rate limits in the PhabricatorStartup shutdown handler
This makes counts more accurate, particularly for connection limits.
2017-10-14 07:14:31 -07:00
epriestley
c5e8de9450 Make bin/storage dump insert CREATE DATABASE and USE statements
Summary:
Ref T13000. The new approach for dumping database-by-database means that we don't get CREATE DATABASE or USE statements, which makes importing the dump again inconvenient.

Manually stitch these into the dump.

Test Plan:
  - Used `bin/storage dump --namespace ...` to dump a smaller local instance.
  - Used `bin/storage destroy --namespace ...`, to destroy the namespace, then inported the dump cleanly.
  - Verified that each CREATE DATABASE statement appears only once.
  - Verified that `bin/storage renamespace --live` can correctly process this file.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18707
2017-10-13 14:35:18 -07:00
epriestley
acb145b3d7 Allow duplicates and merged-in tasks to be queried with edge.search
Summary: See PHI147.

Test Plan: Called the method from the web UI, got sensible results.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18706
2017-10-13 13:12:50 -07:00
epriestley
3f53718d10 Modularize rate/connection limits in Phabricator
Summary:
Depends on D18702. Ref T13008. This replaces the old hard-coded single rate limit with multiple flexible limits, and defines two types of limits:

  - Rate: reject requests if a client has completed too many requests recently.
  - Connection: reject requests if a client has too many more connections than disconnections recently.

The connection limit adds +1 to the score for each connection, then adds -1 for each disconnection. So the overall number is how many open connections they have, at least approximately.

Supporting multiple limits will let us do limiting by Hostname and by remote address (e.g., a specific IP can't exceed a low limit, and all requests to a hostname can't exceed a higher limit).

Configuring the new limits looks something like this:

```
PhabricatorStartup::addRateLimit(new PhabricatorClientRateLimit())
  ->setLimitKey('rate')
  ->setClientKey($_SERVER['REMOTE_ADDR'])
  ->setLimit(5);

PhabricatorStartup::addRateLimit(new PhabricatorClientConnectionLimit())
  ->setLimitKey('conn')
  ->setClientKey($_SERVER['REMOTE_ADDR'])
  ->setLimit(2);
```

Test Plan:
  - Configured limits as above.
  - Made a lot of requests, got cut off by the rate limit.
  - Used `curl --limit-rate -F 'data=@the_letter_m.txt' ...` to upload files really slowly. Got cut off by the connection limit. With `enable_post_data_reading` off, this correctly killed the connections //before// the uploads finished.
  - I'll send this stuff to `secure` before production to give it more of a chance.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13008

Differential Revision: https://secure.phabricator.com/D18703
2017-10-13 13:12:05 -07:00
epriestley
9777c66576 Allow Phabricator to run with "enable_post_data_reading" disabled
Summary:
Ref T13008. Depends on D18701. The overall goal here is to make turning `enable_post_data_reading` off not break things, so we can run rate limiting checks before we read file uploads.

The biggest blocker for this is that turning it off stops `$_FILES` from coming into existence.

This //appears// to mostly work. Specifically:

  - Skip the `max_post_size` check when POST is off, since it's meaningless.
  - Don't read or scrub $_POST at startup when POST is off.
  - When we rebuild REQUEST and POST before processing requests, do multipart parsing if we need to and rebuild FILES.
  - Skip the `is_uploaded_file()` check if we built FILES ourselves.

This probably breaks a couple of small things, like maybe `__profile__` and other DarkConsole triggers over POST, and probably some other weird stuff. The parsers may also need more work than they've received so far.

I also need to verify that this actually works (i.e., lets us run code without reading the request body) but I'll include that in the change where I update the actual rate limiting.

Test Plan:
  - Disabled `enable_post_data_reading`.
  - Uploaded a file with a vanilla upload form (project profile image).
  - Uploaded a file with drag and drop.
  - Used DarkConsole.
  - Submitted comments.
  - Created a task.
  - Browsed around.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13008

Differential Revision: https://secure.phabricator.com/D18702
2017-10-13 13:11:11 -07:00
epriestley
821c7ac833 For backup persitsence, mark the "common ngrams" table as a data table, not an index table
Summary:
Ref T13000. Garbage collecting common ngrams is slow because MySQL isn't all that great at deleting rows quickly. See PHI96, where it looks like it's going to take a week to GC ngrams for a ~million objects at a relatively conservative 0.15 threshold.

In the event of a restore, we can reduce the impact by persisting this table so the ngrams just don't get built when the reindex happens.

Test Plan: Viewed schema in Config, saw common ngrams tables marked as "Data" instead of "Index".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18696
2017-10-11 11:07:43 -07:00
Dmitri Iouchtchenko
5897294fa9 Add spelling TODOs
Summary: Ref T13005. Added reminders not to copy/paste.

Test Plan: None.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13005

Differential Revision: https://secure.phabricator.com/D18695
2017-10-09 11:56:53 -07:00
Dmitri Iouchtchenko
cf3e198b9f Change 'tempate' to 'template'
Summary: Fixed typo. See D18693.

Test Plan: None.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D18694
2017-10-09 11:56:06 -07:00
epriestley
b583093186 Fix Celerity map definition after spelling corrections
See D18693.
2017-10-09 10:52:27 -07:00
Dmitri Iouchtchenko
9bd6a37055 Fix spelling
Summary: Noticed a couple of typos in the docs, and then things got out of hand.

Test Plan:
  - Stared at the words until my eyes watered and the letters began to swim on the screen.
  - Consulted a dictionary.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18693
2017-10-09 10:48:04 -07:00
epriestley
4fd9d2d4bb Fix "bin/storage dump" with no "--output"
Ref T13004. (I distinctly remember testing this, but must have tweaked things afterward.)
2017-10-07 13:23:18 -07:00
epriestley
1ee7b3ab8c Correct "bin/storage dump" command construction with passwords
Fixes T13004. This should mirror the other branch.
2017-10-07 04:59:29 -07:00
epriestley
85011a46d0 Bail out of PhabricatorRepositoryGraphCache more aggressively after cache fills
Summary:
Ref PHI109. Ref T11786. We currently test elapsed time every 64 iterations (since iterations are normally very fast), but at least one install is seeing the page timeout after 30 seconds.

One reason could be that cache fills may occur, and are likely to be much slower than normal iterations. In an extreme case, we could do 64 cache fills before checking the time. Tweak thing so that we always check the time after doing a cache fill, regardless of how many iterations have elapsed since the last attempt.

Additionally, this API method currently accepts an arbitrary number of paths, but implicitly limits each cache query to 500ms. If more than 60 paths are passed, this may exceed 30s. Only let the cache churn for a maximum of 10s across all paths.

If this is more the latter issue than the former, this might replace the GraphCache timeouts with `git` timeouts, but at least our understanding of what's going on here will improve.

Test Plan: This is difficult to test convincingly locally, since I can't reproduce the original issue. It still works after these changes, but it worked fine before these changes too.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11786

Differential Revision: https://secure.phabricator.com/D18692
2017-10-06 14:12:58 -07:00
epriestley
17e83b53d5 Add "bin/search query" for debugging query execution
Summary:
Ref T13000. Currently, queries can only be executed from the web UI, which requires logging in as a user. I really want to avoid doing that wherever we can, but being able to execute queries on an instance (and, particularly, see the ngrams and timings on the underlying lookups) would have been helpful in several cases.

Improve tooling a bit in advance of the "common ngrams" stuff going out since it seems likely that it will be useful if issues arise.

Test Plan: Ran `bin/search query --query ...`, got useful minimal output. Ran with `--trace` to get internals.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18690
2017-10-06 08:50:34 -07:00
epriestley
66df5b1493 Add a garbage collector for common ngrams
Summary:
Ref T13000. After an ngram is marked as "common", we can delete it from the storage table.

Currently, the only way to get ngrams marked as "common" is to manually run `bin/search ngrams`, so this has no impact on normal installs.

Test Plan: Ran `bin/garbage collect`, saw it start chewing through my local Maniphest ngrams table and removing common ngrams.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18687
2017-10-05 11:41:18 -07:00
epriestley
c767c971ca Add "persistence" types (data, cache, or index) to tables, and tweak what "storage dump" dumps
Summary:
Ref T13000. This marks each table as either "data" (normal data), "cache" (automatically rebuilt, no need to ever dump) or "index" (can be manually rebuilt).

By default, `bin/storage dump` dumps data and index tables, but not cache tables.

With `--no-indexes`, it dumps only data tables. Indexes can be rebuilt after a restore with `bin/search index --all ...`.

Test Plan:
  - Ran `--no-indexes` and normal dumps with `--trace`, verified that cache and index (former case) or cache only (latter case) tables were dumped with `--no-data`.
  - Verified dump has the same number of `CREATE TABLE` statements as before the changes.
  - Reviewed persistence tags in the web UI (note Ferret engine tables are "Index"):

{F5210886}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18682
2017-10-04 12:09:33 -07:00
epriestley
02e1440ef2 Dump tables one at a time, rather than all at once
Summary:
Ref T13000. This allows us to be more selective about which tables we dump data for, to reduce the size of backups and exports. The immediate goal is to make large `ngrams` tables more manageable in the cluster, but this generally makes all backups and exports faster and easier.

Here, tables are dumped one at a time. A followup change will sometimes add the `--no-data` flag, to skip dumping readthrough caches and (optionally) rebuildable indexes.

Test Plan: Compared a dump from `master` and from this branch, found them to be essentially identical. The new dump has a little more header information in each section. Verified each contains the same number of `CREATE TABLE` statements.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18679
2017-10-04 12:08:52 -07:00
epriestley
0ea5d668d1 Enable hovercards for the "Task Graph" UI in Maniphest
Summary: See PHI118. Enables hovercards to support peeking at tags and other details if you, e.g., create numerous identical subtasks of each task.

Test Plan: {F5210816}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18681
2017-10-04 11:12:01 -07:00
epriestley
b9fd526250 Fix a fatal on user email settings when account.editable is disabled
Summary:
If `account.editable` is set to false, we try to add a `null` button and fatal:

> Argument 1 passed to PHUIHeaderView::addActionLink() must be an instance of PHUIButtonView, null given, called in /srv/phabricator/phabricator/src/applications/settings/panel/PhabricatorSettingsPanel.php on line 290

Instead, don't try to render `null` as a button.

Test Plan:
  - Configured `account.editable` false.
  - Viewed email address settings.
  - Before: fatal.
  - After: page works, no button is provided.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18677
2017-10-04 10:16:30 -07:00
Austin McKinley
bc9de7ecee Typo fix
Summary: vive la resistance

Test Plan: doitlive

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18674
2017-10-03 15:10:13 -07:00
epriestley
3e589cdd73 Add a workflow for populating (or depopulating) the common ngrams table
Summary:
Depends on D18672. Ref T13000. This does an on-demand build of the common ngrams table.

Plan here is:

  - Push to `secure`.
  - Build the common ngrams table here.
  - See if stuff breaks?

If it looks okay on this dataset, we can build out the GC support and try it in production.

Test Plan:
  - Locally, my dataset has a bunch of `bin/lipsum` tasks with similar, common words.
  - Verified that ipsum terms now skip ngrams. For "lorem ipsum" search performance actually IMPROVED by skipping the ngrams table (12s to 9s).
  - Queried for normal terms, got very fast results using the ngram table, as normal.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18673
2017-10-03 13:28:19 -07:00
epriestley
1de130c9f5 Allow the Ferret engine to remove "common" ngrams from the index
Summary:
Ref T13000. This adds support for tracking "common" ngrams, which occur in too many documents to be useful as part of the ngram index.

If an ngram is listed in the "common" table, it won't be written when indexing documents, or queried for when searching for them.

In this change, nothing actually writes to the "common" table. I'll start writing to the table in a followup change.

Specifically, I plan to do this:

  - A new GC process updates the "common" table periodically, by writing ngrams which appear in more than X% of documents to it, for some value of X, if there are at least a minimum number of documents (maybe like 4,000).
  - A new GC process deletes ngrams that have been added to the common table from the existing indexes.

Hopefully, this will pare down the ngrams index to something reasonable over time without requiring any manual tuning.

Test Plan:
  - Ran some queries and indexes.
  - Manually inserted ngrams `xxx` and `yyy` into the ngrams table, searched and indexed, saw them ignored as viable ngrams for search/index.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18672
2017-10-03 13:27:42 -07:00
epriestley
89fe84f978 Add a "/source/..." URI for Diffusion commits which redirects
Summary:
See PHI112. The install presumably wants to generate links to Diffusion commits from an external tool, but only knows the short name of the repository.

Provide a `/source/phabricator/commit/abcdef908273` URI which redirects to the canonical URI for the commit.

Test Plan:
  - Visited `/source/` URI for a commit, got a redirect.
  - Visited normal URI for a commit, got a commit page.
  - Visited `/branches/` and `/tags/` for a `/source/` repository, got proper pages.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18676
2017-10-03 13:27:03 -07:00
epriestley
f9110b87ab In "Move Tasks to Column...", show only visible columns
Summary:
See PHI94. I considered this initially but wasn't sure about it. However, PHI94 brings up the good point that we already use a similar rule in Maniphest.

For consistency, only show visible columns here too.

Test Plan: Used "Move tasks to column..." on a board with visible and hidden columns, only saw visbile columns offered in the dropdown.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18668
2017-10-02 11:41:02 -07:00
epriestley
cd14194a32 Fix transaction queries using withComments() for transactions with no comments
Summary:
See <https://discourse.phabricator-community.org/t/daemons-tasks-crashing-in-a-loop-during-reindex/506/1>. Some object types (for example, Passphrase Credentials) support indexing but not commenting.

Make `withComments(...)` work properly if the transaction type does not support comments.

Test Plan:
Indexed a credential (no comments) and a revision (comments) with `bin/search index --trace ...`.

Before, credential fataled.

After, credetial succeeds, and skips the transaction query.

Before and after, the revision queries the transaction table.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18667
2017-10-02 09:09:53 -07:00
epriestley
5c73c169fd Rough cut of "Move tasks to column..."
Summary:
Ref T5523. See PHI50. See PHI40. This isn't perfect, but should improve things.

Add a "Move tasks to column..." action to workboards which moves all visible tasks in a column to another column, either on the same board or on a different board.

This is a two-step process so that I don't have to write Javascript, and because I'm not 100% sure this is what users actually want/need. If it sticks, the UI could be refined later.

  - The first dialog asks you to choose a project, defaulting ot the current project.
  - The second dialog asks you to choose a column on that project's board.

Test Plan:
  - Moved tasks on the same board.
  - Moved tasks to a different board.
  - Tried to move tasks to the starting column, got a sensible error.
  - Tried to move tasks to no project, got a sensible error.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T5523

Differential Revision: https://secure.phabricator.com/D18665
2017-09-29 17:38:52 -07:00
epriestley
fe646ec328 Mark Owners package reviewers which own nothing in the current diff
Summary:
Ref PHI91. When Owners (or Herald, or manual user action) adds package reviewers to a revision, later updates to the revision make some of them less relevant or irrelevant.

Provide a hint when a package reviewer doesn't own any of the paths that a diff changes. Humans can then decide if the reviewer is obsolete/irrelevant or not.

This is a rough cut to get the feature working, design could probably use some tweaking if it sticks.

Test Plan: {F5204309}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: jboning

Differential Revision: https://secure.phabricator.com/D18663
2017-09-29 15:06:00 -07:00
epriestley
a39f5e1113 Correct bad context path when doing pattern search inside a repository
Summary:
Ref PHI101. It looks like this was maybe copy/pasted by mistake in recent design refactoring.

We need to pass the full path, not the `basename()` of the path, to the search form.

Test Plan: Searched inside `scripts/test/`, found results inside `scripts/test/`.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18664
2017-09-29 14:51:49 -07:00
epriestley
a3a6c4ed2e Fix fatal when searching for "r matey prepare to be boarded"
Summary:
See <https://discourse.phabricator-community.org/t/unrecoverable-fatal-error-on-repository-search-in-top-search-bar/503/2>.

The Ferret engine replaced `withNameContains()`, but I missed this obscure callsite.

Test Plan:
  - Searched for `r matey prepare to be boarded`.
  - Before: fatal.
  - After: no fatal.
  - Also searched for `r <actual repository name>`, got repository.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18661
2017-09-29 09:45:39 -07:00
epriestley
b75a4151c8 Allow the fulltext index to select only transactions with comments
Summary:
Ref T12997. Although we can't query by transaction type (since we can't easily enumerate all possible types which may have comments -- inline types may also have comments), we //can// just check if there's a comment row or not.

This reduces the amount of garbage we need to load to rebuild indexes for unusual objects with hundreds and hundreds of mentions.

Test Plan:
  - Used batch editor to mention a task 700 times.
  - Indexed it before and after this change, saw index time drop from {nav 1600ms > 160ms}.
  - Made some new comments on it, verified that they still indexed/queried properly.
  - Browsed around, made normal transactions, made inline comments.
  - Added a unique word to an inline comment, indexed revision, searched for word, found revision.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12997

Differential Revision: https://secure.phabricator.com/D18660
2017-09-28 12:55:23 -07:00
epriestley
9875af739f When we purge the request cache, also force PHP to collect cycles
Summary:
Ref T12997. See that task for more details. Briefly, an unusual dataset (where commits are mentioned hundreds of times by other commits) is causing some weird memory behavior in the daemons.

Forcing PHP to GC cycles explicitly after each task completes seems to help with this, by cleaning up some of the memory between tasks. A more thorough fix might be to untangle the `$xactions` structure, but that's significantly more involved.

Test Plan:
  - Did this locally in a controlled environment, saw an immediate collection of a 500MB `$xactions` cycle.
  - Put a similar change in production, memory usage seemed less to improve. It's hard to tell for sure that this does anything, but it shouldn't hurt.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12997

Differential Revision: https://secure.phabricator.com/D18657
2017-09-28 12:37:22 -07:00
epriestley
086a125ad5 Improve performance of Ferret engine ngram extraction, particularly for large input strings
Summary:
See PHI87. Ref T12974. The `array_slice()` method of splitting the string apart can perform poorly for large input strings. I think this is mostly just the large number of calls plus building and returning an array being not entirely trivial.

We can just use `substr()` instead, as long as we're a little bit careful about keeping track of where we're slicing the string if it has UTF8 characters.

Test Plan:
  - Created a task with a single, unbroken blob of base64 encoded data as the description, roughly 100KB long.
  - Saw indexing performance improve from ~6s to ~1.5s after patch.
  - Before: https://secure.phabricator.com/xhprof/profile/PHID-FILE-nrxs4lwdvupbve5lhl6u/
  - After: https://secure.phabricator.com/xhprof/profile/PHID-FILE-6vs2akgjj5nbqt7yo7ul/

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12974

Differential Revision: https://secure.phabricator.com/D18649
2017-09-27 10:41:39 -07:00
epriestley
a1d9a2389d Improve Ferret engine indexing performance for large blocks of text
Summary:
See PHI87. Ref T12974. Currently, we do a lot more work here than we need to: we call `phutil_utf8_strtolower()` on each token, but can do it once at the beginning on the whole block.

Additionally, since ngrams don't care about order, we only need to convert unique tokens into ngrams. This saves us some `phutil_utf8v()`. These calls can be slow for large inputs.

Test Plan:
  - Created a ~4MB task description.
  - Ran `bin/search index Txxx --profile ...` to profile indexing performance before and after the change.
  - Saw total runtime drop form 38s to 9s.
  - Before: <https://secure.phabricator.com/xhprof/profile/PHID-FILE-wiht5d7lkyazaywwxovw/>
  - After: <https://secure.phabricator.com/xhprof/profile/PHID-FILE-efxv56q2hulr6kjrxbx6/>

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12974

Differential Revision: https://secure.phabricator.com/D18647
2017-09-27 08:15:40 -07:00
Aviv Eyal
9f11f310f8 Make PHUITwoColumnView a little more printable
Summary: Hide navbar, and make curtain behave like on a phone, when printing.

Test Plan: {F5197340}

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18583
2017-09-25 19:56:22 +00:00
epriestley
1ac52c09e7 Improve search highlighting for CJK and substring queries
Summary:
Fixes T12995. Currently, the result highlighter (which shows //where// terms matched) only works in "term" mode, not in "substring" mode.

Provide better feedback and behvaior:

  - When a term is a substring term, color it a little differently and add a tooltip. (This is partly to make it easier to debug/diagnose things, probably not enormously valuable to users.)
  - When a term is a substring term, highlight it anywhere in the results.

Test Plan:
Queried for latin and CJK terms.

Here is CJK being highlighted:

{F5192195}

Here is substring vs non-substring implicit behavior:

{F5192196}

Here's ONLY terms being highlighted:

{F5192198}

Here's terms and substrings, since the query now has a substring:

{F5192201}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12995

Differential Revision: https://secure.phabricator.com/D18635
2017-09-22 11:34:46 -07:00
epriestley
36df39761e Create revisions into "Draft", publish them when builds finish
Summary:
Ref T2543. This doesn't stand alone since mail still goes out normally, but gets this piece working: new revisions start as "Draft", then after updates if there are no builds they go into "Needs Review".

This should work in general because builds update revisions when they complete, to publish a "Harbormaster finished build yada yada" transaction. So either we'll un-draft immediately, or un-draft after the last build finishes.

I'll hold this until the mail and some other stuff (like UI hints) are in slightly better shape since I think it's probably too rough on its own.

Test Plan: Created revisions locally, saw them un-draft after builds.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18628
2017-09-21 07:21:21 -07:00
epriestley
fca553f142 Prepare revision mail for the "Draft" status
Summary:
Ref T2543. Currently, we always do some special things when a revision is created, mostly adding more stuff to the mail.

With drafts, we want to suppress initial mail and send this big, rich mail only when the revision actually moves out of "draft".

Prepare the code for this, with the actual methods hard-coded to the current behavior. This will probably take some tweaking but I think I got most of it.

Test Plan: Banged around in Differential so it sent some mail, saw normal mail without anything new.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18627
2017-09-21 07:21:07 -07:00
epriestley
c7af663523 Align most revision actions to the new "Draft" state
Summary:
Ref T2543. Most actions are not available for drafts.

Authors can "Request Review" (move out of draft to become a normal revision) or "Abandon".

Non-authors can't do anything (maybe we'll let them do something later -- like "Commandeer"? -- if there's a good reason).

Test Plan: Viewed a draft revision as an author and non-author, saw fewer actions available.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18626
2017-09-21 07:20:48 -07:00
epriestley
5112dac491 Update an old SSH redirect URI when editing a bot's SSH keys
Summary: See PHI79. When you edit another user's SSH keys (normally, for a bot account) we currently redirect you to an older URI.

Test Plan:
  - Viewed a bot's profile page.
  - Clicked "Edit Settings" on the Manage page.
  - Went to "SSH Keys".
  - Uploaded an SSH key.
  - Before: redirected to a 404 after finishing the workflow.
  - After: redirected to the same page after the workflow.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18633
2017-09-20 14:46:31 -07:00
epriestley
3fbad684c1 More completely explain why we're refusing to send reset mail to an unverified address
Summary:
See PHI78. The user was getting this message and (reasonably) interpreted it to mean "reset mail can never be sent to unverified addresses".

Reword it to be more clear, albeit an entire paragraph long. I don't really have a good solution in these cases where we'd need a whole page to explain what's happening (this, plus "we can't tell you which address you should use because an attacker could get information if we did" and "this rule defuses the risk that an opportunistic attacker may try to compromise your account after you add an email you don't own by mistake"). We could write it up separately and link to it, but I feel like that stuff tends to get out of date.

Just land somewhere in the middle.

Test Plan: {F5189105}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18630
2017-09-20 10:46:22 -07:00
epriestley
03e5d69817 Fix an error in Diffusion when the Owners application is uninstalled
Summary:
See <https://discourse.phabricator-community.org/t/undefined-view-when-owners-is-uninstalled/451>.

When Owners is not installed, Diffusion can fatal with a bad `$view`.

Test Plan:
  - Uninstall Owners.
  - View the content of any file in Diffusion.
  - Before: fatal on `$view` undefined.
  - After: Valid page with no owners information.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18629
2017-09-19 09:42:42 -07:00
epriestley
23867c1487 Add a "Draft" state for revisions, and action bucket support
Summary:
Ref T2543. There's no way to put revisions into this state yet, but start adding support for when there is.

Adds the status constant, plus support for bucketing them.

Test Plan:
  - Manually put a revision in "Draft" state by updating the database directly.
  - Verified my drafts showed up in a "Drafts" section on the bucket view.
  - Verified others' drafts did not appear on the action bucket view.
  - Viewed revisions, queried for "Draft" revisions, etc (stuff we get for free).

{F5186781}

{F5186782}

{F5186783}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18625
2017-09-18 14:01:00 -07:00