1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-25 08:12:40 +01:00
Commit graph

15542 commits

Author SHA1 Message Date
epriestley
f94cee8628 Fix querying for transactions over "transaction.search" when the object does not support comments
Summary: See PHI725. Ref T13151. We currently try to load comments unconditionally, but not all objects (like projects) have comments. Only try to load comments if an object actually has comments.

Test Plan: Queried for an object with no comments, like project `#masonry`, via `transaction.search`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19507
2018-06-26 07:59:01 -07:00
epriestley
cac3dc4983 Give "create" transactions a readable type in "transaction.search"
Summary:
Ref T13151. See PHI725. By default, "transaction.search" doesn't provide details about transactions because many have bad/weird/policy-violating internal types or fields.

The "create" transaction is simple and straightforward, so label it to allow callers to distinguish it.

Test Plan:
  - Created a new task.
  - Called `transaction.search` on it.
  - Saw the labelled "create" transaction.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: swisspol

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19505
2018-06-22 17:42:35 -07:00
epriestley
d84f866ca0 When search indexers contend for a lock, just yield
Summary:
Depends on D19503. Ref T13151. See PHI719. If you have something like a script which updates an object in a loop, we can end up queueing many search reindex tasks.

These tasks may reasonably contend for the lock, especially if the object is larger (lots of text and/or lots of comments) and indexing takes a few seconds.

This isn't concerning, and the indexers should converge to good behavior quickly once the updates stop.

Today, they'll spew a bunch of serious-looking lock exceptions into the log. Instead, just yield so it's more clear that there's (normally) no cause for concern here.

Test Plan: Ran `bin/search index Txxx --force` on a large object in multiple windows with a 0 second lock, saw an explicit yield instead of a lock exception.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19504
2018-06-22 17:41:45 -07:00
epriestley
14e911a0d8 Index only the first 1,000 comments on any object
Summary:
Depends on D19502. Ref T13151. See PHI719. An install ended up with an object with 111,000+ comments on it because someone wrote a script to treat it like a logfile.

Although we seem to do mostly okay with this (locally, it only takes about 30s to index a similar object) we'll hit a wall somewhere (since we need to hold everything in memory), and it's hard to imagine a legitimate object with more than 1,000 comments. Just ignore comments past the first thousand.

(Conpherence threads may legitimately have more than 1,000 comments, but go through a different indexer.)

Test Plan:
  - Piped some comments into `maniphest.edit` in a loop to create a task with 100K comments.
  - Ran `bin/search index Txxx --force` to reindex it, with `--trace`.
    - Before: task indexed in about 30s.
    - After: script loaded comments with LIMIT 1000 and indexed in a couple seconds.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19503
2018-06-22 17:41:05 -07:00
epriestley
cbc71e75fa When queueing search index tasks, include the "objectPHID" in the task metadata
Summary:
Ref T13151. See PHI719. One minor hiccup in debugging the issue (which ended up being "revision has 100K comments") was that the `SearchWorker` did not show which object it was indexing.

Add `'objectPHID'` to the queue call so you can see which object is affected from the web UI.

Test Plan:
  - Stopped daemons.
  - Used `bin/search index D123 --background` to queue a search task.
  - Viewed task details in web UI from `/daemon/`.
    - Before change: no indication of which object was being indexed.
    - After change: page helpfully shows that the task is indexing D123.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19502
2018-06-22 17:40:32 -07:00
epriestley
b1f4a974fe Fix two minor breadcrumb issues in Config
Summary:
Fixes T13159. Two issues here:

  - When viewing a particular config setting, there's an extra "Config" crumb.
  - On the page for a config group, the link to the parent group has an extra "/config/" in it.

Test Plan:
  - Viewed a page for a particular setting, no longer saw an extra "Config" crumb.
  - Viewed a page for a setting group, clicked parent crumb, got taken to a real page.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13159

Differential Revision: https://secure.phabricator.com/D19501
2018-06-22 17:38:50 -07:00
epriestley
8ab8c390b7 If "branch" is provided to "diffusion.branchquery", use it as the "<pattern>" argument to "git branch --contains ..."
Summary:
Ref T13151. See PHI720. If you want to test if commit X appears on specific branch Y, `git branch --contains X -- Y` is faster than (effectively) `git branch --contains X | grep Y`.

Since this call has a "branch" parameter anyway, use it as the pattern argument if provided.

Test Plan:
  - Called the API method with no parameters, got all branches.
  - Called the API method with `master`, got just master.
  - Called the API method with `maste*`, got master. This behavior is not officially supported and may change in the future.
  - Viewed a commit, still saw all branches.
    - Grepped for `diffusion.branchquery` and verified that no remaining callsites pass a default "branch" parameter.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19499
2018-06-22 17:38:19 -07:00
epriestley
6136b83275 Fix changeset construction special case for empty commits in pre-commit hooks
Summary: Fixes T13155. Ref T13151. A recent change (D19455) changed the return format here, but I missed this special case for empty commits.

Test Plan:
  - T13155 has a good set of reproduction instructions.
  - Pushed an empty commit.
    - Before: bunch of warning log spew.
    - After: clean logs.

Reviewers: amckinley, avivey

Reviewed By: avivey

Maniphest Tasks: T13155, T13151

Differential Revision: https://secure.phabricator.com/D19500
2018-06-21 16:43:20 -07:00
Austin McKinley
9db5ad3476 Allow null identities to be attached to commit objects
Summary: I landed D19491 a little aggressively, so allow this field to be null until after the migration goes out.

Test Plan: Loaded commits without identity objects; did not get any errors.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19496
2018-06-20 08:35:36 -07:00
Austin McKinley
05f333dfba Attach identities to commits and users to identities
Summary: Ref T12164. Make it easier to work with identity objects by attaching them to commits and attaching users to identities.

Test Plan: Loaded some commits with `->needIdentities(true)` and checked the resulting objects.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19491
2018-06-18 15:31:41 -07:00
Austin McKinley
787c59744b Correctly attach users to identities
Summary: This never worked.

Test Plan: Ran `bin/repository rebuild-identities` and viewed identity objects with `currentEffectiveUserID`s and no longer got errors about attempting to attach `null` objects instead of `PhabricatorUser` objects.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19495
2018-06-18 15:21:11 -07:00
epriestley
a7c681b549 Don't set mail HTML bodies if there's no actual HTML body
Summary:
See <https://discourse.phabricator-community.org/t/commit-6011085b0fcd-breaks-sending-certain-email/1571>. Some mailers get upset if we `setHTMLBody(...)` with an empty string.

There's some possible argument they should be more graceful about this, but it's reasonably pretty ambiguous.

Only try to set the HTML body if we actually have a nonempty HTML body.

Test Plan:
  - Configured an "smtp" mailer.
  - Ran `echo hi | ./bin/mail send-test --to someone@somewhere.com --subject test`.
  - Before: error about empty message body.
  - After: no more message body error.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19494
2018-06-15 14:01:40 -07:00
epriestley
1459fb3037 Make re-running rebuild-identities a bit faster and add a little progress information
Summary:
Ref T13151. Ref T12164. Two small tweaks:

  - If we aren't actually going to change anything, just skip the writes. This makes re-running/resuming a lot faster (~20x, locally).
  - Print when we touch a commit so there's some kind of visible status.

This is just a small quality-of-life tweak that I wrote anyway while investigating T13152, and will make finishing off db024, db025 and db010 manually a little easier.

Test Plan:
  - Set `authorIdentityPHID` + `committerIdentityPHID` to `NULL`.
  - Ran `rebuild-identities`, saw status information.
  - Ran `rebuild-identiites` again, saw it go faster with status information.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151, T12164

Differential Revision: https://secure.phabricator.com/D19484
2018-06-12 13:18:54 -07:00
epriestley
6011085b0f Respect "metamta.email-body-limit" when building mail HTML bodies
Summary:
Ref T13151. See T11767. See PHI686. Although we limit outbound mail text bodies, the limit doesn't currently apply to attachments, HTML bodies, or headers. T11767 discusses improving this in the general case.

In the wild, an install hit an issue (see PHI686) where edits to Phriction pages generate very large HTML bodies. Check and respect the limit when building HTML bodies.

If we don't have enough room for the HTML body, we just drop it. We have the text body to fall back to, and HTML is difficult to truncate safely.

Test Plan: Added unit tests and made them pass.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19489
2018-06-12 12:02:15 -07:00
epriestley
c5b13a6be3 Allow object subtypes to be changed via bulk editor
Summary:
Ref T13151. See PHI683. Ref T12314.

You can currently change object subtypes via Conduit (`maniphest.edit`) but not via the web UI.

Changing object subtypes is inherently a somewhat-perilous operation that likely has a lot of rough edges we'll need to smooth over eventually, mostly around changing an object from subtype X to subtype Y, where some field exists on one but not the other. This isn't a huge issue, just not entirely intuitive.

It should also, in theory, be fairly rare.

As a reasonable middle ground, provide web UI access via the bulk editor. This makes it possible, but doesn't clutter the UI up with a rarely-used option with rough edges.

Test Plan:
  - With subtypes not configured, saw a normal bulk editor with no new option.
  - With subtypes configured, swapped tasks subtypes via bulk editor.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151, T12314

Differential Revision: https://secure.phabricator.com/D19490
2018-06-12 11:58:44 -07:00
epriestley
62a402491a Allow encrypted mail to be more specific about which object is affected
Summary:
Depends on D19487. Ref T13151. See PHI647. For some objects, like revisions, we can build slightly more useful secure email without actually disclosing anything.

In the general case, the object monogram may disclose information (`#acquire-competitor`) but most do not, so applications can whitelist an acceptable nondisclosing subject and link.

Support doing this, and make Differential do it. When we don't have a whitelisted URI but do know the object the mail is about, include a generic PHID-based URI; these are always nondisclosing.

Test Plan:
  - Without the Differential changes, sent normal mail (no changes) and secure mail (new generic PHID-based link).
  - With the Differential changes, sent secure mail; got richer subject and body link.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19488
2018-06-12 11:55:18 -07:00
epriestley
94752278f4 Add a generic PHID-based object redirection controller
Summary:
Ref T13151. See PHI647. This allows us to link to any object by PHID, without disclosing information in the monogram (like `#fire-steve`).

This capability is relevant when building "secure mail", to provide a link to the object regardless of whether the monogram discloses information or not.

Test Plan: Visited `/object/D123/` (redirect), `/object/xyz/` (404), `/object/PHID-DREV-.../` (redirect).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19487
2018-06-12 11:54:59 -07:00
epriestley
cbff913432 Add a "members of all projects" (vs "...any project") custom policy rule to the upstream
Summary:
Ref T13151. See PHI702. An install is interested in a "members of all projects" (vs "members of any project", which is currently implemented) rule.

Although this is fairly niche, I think it's reasonable and doesn't have much of a maintenance cost.

This could already be implemented as an extension, but it would have to copy/paste a bunch of code.

Test Plan:
  - Ran unit tests.
  - Used the UI to select this policy for a task, with various values. Joined/left projects to satisfy/fail the rule. Behavior seemed correct.
  - Used the UI to select the existing policy rule ("any project"), joined/left projects to satisfy/fail the rule. Doesn't look like I broke anything.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19486
2018-06-12 11:51:51 -07:00
Alex Vandiver
59b95f9397 Fix typo in "button"
Test Plan: Observation.

Reviewers: #blessed_reviewers, amckinley, epriestley

Reviewed By: #blessed_reviewers, amckinley, epriestley

Subscribers: Korvin, amckinley, epriestley

Differential Revision: https://secure.phabricator.com/D19483
2018-06-08 15:09:07 -07:00
epriestley
f375427177 Use more consistent diff coloration in unified diffs
Summary:
Ref T13151. See PHI701. Unified diffs are currently missing the logic to apply the "old-full" and "new-full" classes, which results in a too-light coloration for fully added or removed lines.

Make this logic consistent with the two-up renderer so we use the same colors in both.

Test Plan: Viewed diffs and swapped between 1-up and 2-up renderers, now saw the same coloration.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19482
2018-06-08 09:39:34 -07:00
epriestley
4e84d4d458 Allow the haunted comment panel ("Z") to take up more vertical room
Summary:
Ref T13151. See PHI685. When you haunt the panel, we only let it take up part of the screen. Let it take up slightly more of the screen so that it's more likely to fit completely on-screen without needing to scroll.

The behavior when it does scroll is fine (you get a scrollbar if your OS/browser is set up to show them) so this is a bit trivial/silly, but seems fine and doesn't have a big JS maintenance cost or anything.

Test Plan: Pressed "Z", resized my window to a weird tiny useless size, got slightly better (I guess) behavior.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19480
2018-06-07 13:19:35 -07:00
epriestley
13dd3014a7 Fix loop in QueryIterator when row count is an exact multiple of page size
Summary: Ref T13152. The pager does a bit of magic here and doesn't populate `nextPageID` when it knows it got an exact final page. The logic misfired in this case and sent us back to the start.

Test Plan:
  - Set page size to 1 to guarantee rows were an exact multiple of page size.
  - Ran `rebuild-identities` (I no-op'd the actual logic to make it faster).
  - Before: looped forever.
  - After: clean exit after processing everything.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13152

Differential Revision: https://secure.phabricator.com/D19479
2018-06-07 13:17:34 -07:00
epriestley
7acda7e94e Truncate package names in diff table of contents views
Summary:
Ref T13151. See PHI654. Depends on D19477. If you have long package names, the table of contents (e.g., in Differential) can end up expanding to be gigantic.

Getting tables to behave nicely is hard (or, at least, I can't figure it out after spending a decent amount of time on it; see also `AphrontTableView::renderSingleDisplayLine()`). I tried a bunch of things and Googled for a bit but didn't make any progress on finding a CSS solution. Just truncate the package names to get reasonable behavior without falling down any kind of CSS rabbit hole.

Test Plan:
  - Created a package named "Very long package name...".
  - Created a package named "MMMMMMMMMMMMMMMMMMMMMM...".
  - Had them own a file in a Differential revision, viewed that revision.
  - Before: table is pushed out to several times the browser window width and everything is kind of a mess.
  - After: package names get truncated to something reasonable.

{F5652953}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19478
2018-06-07 13:17:01 -07:00
epriestley
2951e0c86b Include owners packages in the MailableFunction datasource
Summary:
Ref T13151. See PHI684. Currently, the `MailableFunction` datasource does not include Owners packages, but they are valid subscribers and the `Mailable` datasource includes them.

Include them in the `MailableFunction` datasource, too.

Test Plan: Searched for revisions with particular package subscribers, got expected results in the UI (tokenizer knew about packages) and response.

Reviewers: amckinley, jmeador

Reviewed By: jmeador

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19476
2018-06-07 12:02:50 -07:00
epriestley
7729c51cc4 Fix an issue where scrolling down, then up, then down fails to show changeset header in Differential
Summary: Ref T13151. See PHI616. There's a bug where the current banner changeset isn't cleared correctly when we hide the banner.

Test Plan:
  - View revision with several changesets.
  - Scroll down slowly through first changeset until banner appears.
  - Scroll up until banner disappears.
  - Scroll back down.
  - Before: banner fails to reappear (code still thinks it's visible and we don't want to update it).
  - After: banner reappears correctly.

Reviewers: amckinley, jmeador

Reviewed By: jmeador

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19474
2018-06-07 12:02:18 -07:00
Austin McKinley
b8b2d1672d Prevent creation of empty repository identities
Summary: Fixes issue reported in https://secure.phabricator.com/rPf191a66490b194785fae28c062b71be99bb14584#43240

Test Plan: Imported an SVN repo, observed clean import instead of daemon exception.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19466
2018-06-05 16:13:59 -07:00
Aviv Eyal
dbe72df557 minor: fix translation error in exception
Test Plan: look hard at code.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D19463
2018-06-04 17:46:13 +00:00
epriestley
376ea1ddf5 Support logged-out access to more Harbormaster controllers
Summary:
Fixes T13145. The list controllers properly support public access already, but some of the view/detail controllers did not.

Allow logged-out users to browse builds, buildables, plans, etc., provided they can see the corresponding objects.

Test Plan: As a logged-out user, browsed around builds, build plans, logs, etc., without hitting any login pages.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13145

Differential Revision: https://secure.phabricator.com/D19459
2018-06-04 10:11:25 -07:00
epriestley
5bcca675e1 Add card expiration information to Phortune cart screen
Summary:
See PHI689. It can be difficult to distinguish between cards with the same number but different expiration dates (common when the bank sends you a new card).

For now, show the expiration date on the cart checkout screen.

Test Plan: Viewed a cart checkout screen with multiple cards, saw expiration dates.

Reviewers: amckinley

Differential Revision: https://secure.phabricator.com/D19462
2018-06-02 18:23:44 -07:00
epriestley
a894c99935 Add "max-width: 100%;" to stop large images from overflowing the new rendering engine UI
Summary:
Fixes T13148. Ref T13105. The new document rendering engine for images let them overflow the UI bounds.

Add `max-width: 100%;` to keep them contained.

Test Plan:
  - Viewed a very wide image in Safari, Firefox and Chrome. Saw sensible rendering.
  - Also viewed a normal image, saw normal behavior.

Reviewers: amckinley, avivey

Reviewed By: avivey

Maniphest Tasks: T13148, T13105

Differential Revision: https://secure.phabricator.com/D19457
2018-06-01 14:53:10 -07:00
epriestley
31ee49b14d Fix Javascript busy loop when trying to delete tokens from an empty tokenizer
Summary:
Fixes T13147. In D19437, I changed this logic to support deleting the `""` (empty string) token, but `[].pop()` returns `undefined`, not `null`, if the list is empty and I didn't think to try deleting an empty input.

Fix the logic so we don't end up in a loop if the input is empty.

Test Plan:
  - In any browser, deleted all tokens in a tokenizer; then pressed delete again.
  - Before: tab hangs in an infinte loop.
  - After: smooth sailing.

Reviewers: amckinley, avivey

Reviewed By: avivey

Maniphest Tasks: T13147

Differential Revision: https://secure.phabricator.com/D19456
2018-06-01 14:51:06 -07:00
Austin McKinley
2f6784ee1c Add workflow to create repository identities
Summary:
Depends on D19443. Creates a workflow for populating the new identity table by iterating over commits, either one repo at a time or all at once. Locally caches identities to avoid fetching them `inf` times. An actual migration that invokes this workflow will come in another revision that won't land until at least next week.

Performance is ~2k commits in 4.9s on my local machine.

Test Plan: Ran locally a few times with a few different states of the `repository_identity` table.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: jcox, Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19446
2018-05-31 07:29:57 -07:00
Austin McKinley
fe5fde5910 Assign RepositoryIdentity objects to commits
Summary: Depends on D19429. Depends on D19423. Ref T12164. This creates new columns `authorIdentityPHID` and `committerIdentityPHID` on commit objects and starts populating them. Also adds the ability to explicitly set an Identity's assignee to "unassigned()" to null out an incorrect auto-assign. Adds more search functionality to identities. Also creates a daemon task for handling users adding new email address and attempts to associate unclaimed identities.

Test Plan: Imported some repos, watched new columns get populated. Added a new email address for a previous commit, saw daemon job run and assign the identity to the new user. Searched for identities in various and sundry ways.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19443
2018-05-31 07:28:23 -07:00
Austin McKinley
f191a66490 Add controllers/search/edit engine functionality to RepositoryIdentity
Summary: Depends on D19423. Ref T12164. Adds controllers capable of listing and editing `PhabricatorRepositoryIdentity` objects. Starts creating those objects when commits are parsed.

Test Plan: Reparsed some revisions, observed objects getting created in the database. Altered some `Identity` objects using the controllers and observed effects in the database. No attempts made to validate behavior under "challenging" author/committer strings.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19429
2018-05-31 07:03:25 -07:00
Austin McKinley
cd84e53c44 Begin building out RepositoryIdentity indirection layer
Summary: Ref T12164. Start building initial objects for managing `RepositoryIdentity` objects. This won't land until much more of the infrastructure is in place.

Test Plan: Ran `bin/storage upgrade` and observed expected table.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19423
2018-05-31 07:01:16 -07:00
epriestley
de999af614 Improve some behaviors around memory pressure when pushing many and/or large changes
Summary:
Ref T13142. When commits are pushed, we try to handle them on one of two pathways:

  - Normal changes: we load these into memory and potentially apply Herald content rules to them.
  - "Enormous" changes: we don't load these into memory and skip content rules for them.

The goal is to degrade gracefully when users push huge changes: they should work, just not support all the features.

However, some changes can slip through the cracks right now:

  - If you push a lot of commits at once, we'll try to cache all of the changes smaller than 1GB in memory. This can require an arbitrarily large amount of RAM.
  - We calculate sizes by just looking at the `strlen()` of the diff, but a changeset takes more RAM in PHP than the raw diff does. So even if a diff is "only" 500MB, it can take much more memory than that. On systems with relatively little memory available, this may result in OOM while processing changes that are close to the "enormous" limit.

This change makes two improvements:

  - Instead of caching everything, cache only 64MB of things.
    - For most pushes, this is the same, since they have less than 64MB of diffs.
    - For pushes of single very large changes, this is a bit slower (more CPU) since we have to do some work twice.
    - For pushes of many changes, this is slower (more CPU) since we have to do some work twice, but, critically, doesn't require unlimited memory.
  - Instead of flagging changes as "enormous" at 1GB, flag them as "enormous" at 256MB.
    - This reduces how much memory is required to process the largest "non-enormous" changes.
    - This also gets us under Git's hard-coded 512MB "always binary" cutoff; see T13143.
    - This is still completely gigantic and way larger than any normal change should be.

An additional improvement would be to try to reduce the amount of memory we need to use to hold a change in process memory. I think the other changes here alone will fix the immediate issue in PHI657, but it would be nice if the "largest non-enormous change" required only a couple gigs of RAM.

Test Plan:
- Used `ini_set('memory_limit', '1G')` to artificially limit memory to 1GB.
- Pushed a series of two commits which add two 550MB text files (Temporarily, I added a `--binary` flag to trick Git into showing real diffs for these, see T13143.)
- Got a memory limit error.
- Applied the "cache only 64MB of stuff" and "consider 256MB, not 1GB, to be enormous" changes.
- Pushed again, got properly rejected as enormous.
- Added `memory_get_usage()` calls to measure how actual memory size and reported "size" estimate compare. For these changes, saw a 639MB diff require 31,479MB of memory, i.e. a factor of about 50x. This is, uh, pretty not great.
- Allowed enormous changes, pushed again, push went through.

Reviewers: amckinley

Maniphest Tasks: T13142

Differential Revision: https://secure.phabricator.com/D19455
2018-05-18 17:15:34 -07:00
epriestley
3c5668b4a5 When database connection exceptions occur, raise them to the setup layer
Summary:
Ref T13141. Currently, during first-time setup we don't surface all the details about connection exceptions that we could: the underlying exception is discarded inside cluster connection management.

This isn't a huge issue since the reason for connection problems is usually fairly obvious, but in at least one case (see T13141) we hit a less-than-obvious exception.

Instead, store the original exception and propagate the message up the stack so users have more information about the problem.

Test Plan:
  - Configured an intentionally bad MySQL username.
  - Restarted Apache and loaded Phabricator.
  - Got a more helpful exception with a specific authentication error message.

{F5622361}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13141

Differential Revision: https://secure.phabricator.com/D19454
2018-05-18 09:02:36 -07:00
epriestley
8f9b948447 When showing a diff-of-diffs, hide files which didn't get any more changes and have no inlines
Summary:
Ref T13137. See that task for discussion.

When we show a diff-of-diffs, we often render stubs for files which didn't change between the diffs. These stubs usually aren't a big deal, but for certain types of changes (like refactors) they can create a lot of clutter.

Instead, hide these stubs and show a notice that we hid them.

Test Plan:
  - Created a revision affecting 4 files.
  - Updated it with a diff that changed only 1 of the 4 files.
  - Added an inline comment to a different file.
  - Viewed the diff of diffs.
    - Before: 4 changesets with two "nothing changed" stubs.
    - After: 2 changesets with the stubs hidden.

{F5621083}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19453
2018-05-16 17:18:53 -07:00
epriestley
79fdf5c127 Separate changeset analysis code from DifferentialDiff and provide a standalone rebuild-changesets workflow
Summary:
Ref T13137. The "analyze/cache data about changesets" step is becoming more involved. We recently added detection for generated code to support "Ignore generated changes" in Owners, and I now plan to hash the new file content so we can hide changes which have no effect.

Before adding this new hashing step, pull the "detect copied code" and "detect generated code" stuff out and move them to a separate `ChangesetEngine`. Then support doing a changeset rebuild directly with `bin/differential rebuild-changesets`.

This simplifies things a bit and makes testing easier since you don't need to keep creating new revisions to re-run copy/generated/hash logic.

Test Plan: Ran `bin/differential rebuild-changesets --revision Dxxx`, saw changesets rebuild. See also next change.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19452
2018-05-16 17:17:28 -07:00
epriestley
3544620209 Parse unusual Subversion protocol frames which contain extra whitespace
Summary:
Fixes T13140. See PHI660.

Recent versions of Subversion can send a `(get-file true false  false )` protocol frame with extra space between "false" and "false". This is allowed by the protocol spec, but never normally happens, and we do not parse it correctly.

Instead, parse it correctly.

Test Plan:
  - Added unit tests.
  - Ran `svn proplist svn+ssh://.../diffusion/X/file.c` under SVN 1.10 before and after the change.
    - Before: indefinite hang.
    - After: completed in finite time.

Reviewers: amckinley, asherkin

Reviewed By: amckinley, asherkin

Maniphest Tasks: T13140

Differential Revision: https://secure.phabricator.com/D19451
2018-05-16 17:12:41 -07:00
epriestley
7aa12f192a Add PhabricatorQueryIterator, for buffered iteration over a CursorPagedPolicyAwareQuery
Summary:
See D19446. This should make it easier to process larger, more complex result sets in constant memory.

Today, `LiskMigrationIterator` takes constant memory but can't apply `needX()` reqeusts or `withY(...)` constraints.

Using a raw `Query` can handle this stuff, but requires memory proportional to the size of the result set.

Offer the best of both worlds: constant memory and full access to the power of `Query` classes.

Test Plan:
Used this script to iterate over every commit, saw sensible behavior:

```name=list-commits.php
<?php

require_once 'scripts/init/init-script.php';

$viewer = PhabricatorUser::getOmnipotentUser();

$query = id(new DiffusionCommitQuery())
  ->setViewer($viewer);

$iterator = new PhabricatorQueryIterator($query);
foreach ($iterator as $commit) {
  echo $commit->getID()."\n";
}
```

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19450
2018-05-14 12:13:51 -07:00
epriestley
29df80b48f Fix a fatal during breadcrumb construction when viewing a dashboard you don't have permission to view
Summary: Ref PHI662. Viewing a dashboard you don't have permission to view (in the Dashboard application) currently fatals while building crumbs, since we fail to build the ` ... > Dashboard 123 > ...` crumb.

Test Plan:
  - Viewed a dashboard I didn't have permission to view in the Dashboards application.
  - Before patch, fatal when calling `getID()` on a non-object.
  - After patch, sensible policy error page.
  - Viewed a dashboard I can view, saw sensible crumbs.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19449
2018-05-14 12:06:56 -07:00
epriestley
28ee6b8080 Consistently require MFA on the actual user creation flow
Summary:
See <https://hackerone.com/reports/351361>. We currently require MFA on the screen leading into the user create flow, but not the actual create flow.

That is, `/people/create/` (which is just a "choose a type of account" page) requires MFA, but `/people/new/<type>/` does not, even though this is the actual creation page.

Requiring MFA to create users isn't especially critical: creating users isn't really a dangerous action. The major threat is probably just that an attacker can extend their access to an install by creating an account which they have credentials for.

It also isn't consistently enforced: you can invite users or approve users without an MFA check.

So there's an argument for just removing the check. However, I think the check is probably reasonable and that we'd likely prefer to add some more checks eventually (e.g., require MFA to approve or invite) since these actions are rare and could represent useful tools for an attacker even if they are not especially dangerous on their own. This is also the only way to create bot or mailing list accounts, so this check does //something// on its own, at least.

Test Plan:
  - Visited `/people/new/standard/` as an admin with MFA configured.
  - Before patch: no MFA prompt.
  - After patch: MFA prompt.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19448
2018-05-14 12:03:07 -07:00
epriestley
26d0862f4f Apply the new patch byte size limit to mail patch generation in Differential
Summary: Ref T13137. See PHI592. Depends on D19444. Apply a limit up front to stop patches which are way too big (e.g., 600MB of videos) from generating in the first place.

Test Plan:
  - Configured inline patches in git format.
  - Created a normal revision, got an inline git patch.
  - Created a revision with a 10MB video file, got no inline patch.
  - (Added a bunch of debugging stuff to make sure the internal pathway was working.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19445
2018-05-14 09:10:47 -07:00
Aviv Eyal
7281300446 Allow number in generated clone uri
Summary:
See https://discourse.phabricator-community.org/t/numerical-characters-are-stripped-from-diffusion-git-repository-name-in-the-uri/

Digits are often considered reasonable characters.

Test Plan: Looked at an ascii table.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, Sam2304, epriestley

Differential Revision: https://secure.phabricator.com/D19447
2018-05-11 16:18:06 +00:00
epriestley
10a4b05ecb Fix "Any Owner" and "No Owners" searches in Maniphest
Summary:
See <https://discourse.phabricator-community.org/t/maniphest-home-page-crash-after-d19417/1445/3>. These special-token-only searches currently end up populating an empty `ownerPHIDs`, which fatals after the stricter check in D19417.

Make the fatal on `withConstraint(array())` explicit and only set the PHID constraint if we have some PHIDs left.

Test Plan: Searched for "No Owner", "Any Owner", an actual owner, "No Owner + actual user".

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19440
2018-05-09 13:24:23 -07:00
epriestley
d280b24239 Fix "arc paste" to stop creating pastes with an empty string ("") as the "language"
Summary:
See PHI652. When you `echo x | arc paste` today, you end up with a Paste object that has the empty string as its "language".

This is normally not valid. Pastes where the language should be autodetected should have the value `null`, not the empty string.

This behavior likely changed when `paste.create` got rewritten in terms of `paste.edit`. Adjust the implementation so it only adds the LANGUAGE transaction if there's an actual language.

Also, fix an issue where you can't use the "delete" key to delete tokens with the empty string as their value.

Test Plan:
  - Created a paste with `echo x | arc paste`, got a paste in autodetect mode instead of with a bogus language value.
  - Created a paste with `echo x | arc paste --lang rainbow`, got a rainbow paste.
  - Deleted an empty string token with the keyboard.
  - Deleted normal tokens with the keyboard.
  - Edited subscribers/etc normally with the keyboard and mouse to make sure I didn't ruin anything.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19437
2018-05-09 13:22:58 -07:00
epriestley
26c0db8dd7 Allow navigation breadcrumbs to be marked as "always visible" so they show up on phones
Summary:
See PHI624. Some of the mobile navigation and breadcrumbs in support pacts aren't as good as they could be.

In particular, we generally collapse crumbs on mobile to just the first and last crumbs. The first crumb is the application; the last is the current page.

On `/PHIxxx` pages, the first crumb isn't very useful since the Support landing page is two levels up: you usually want to go back to the pact, not all the way back to the Support landing page.

We also don't need the space since the last crumb (`PHIxxx`) is always small.

Allow Support and other similar applications to tailor the crumb behavior more narrowly if they end up in situations like this.

Test Plan:
  - With an additional change to instances (see next diff), viewed a support issue page (`/PHI123`) on mobile and desktop.
  - Saw a link directly back to the pact on both mobile and desktop.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19438
2018-05-09 13:21:47 -07:00
epriestley
5b640a434c Support an "Ancestors Of: ..." constraint in commit queries
Summary:
Ref T13137. See PHI609. An install would like to filter audit requests on a particular branch, e.g. "master".

This is difficult in the general case because we can not apply this constraint efficiently under every conceivable data shape, but we can do a reasonable job in most practical cases.

See T13137#238822 for more detailed discussion on the approach here.

This is a bit rough, but should do the job for now.

Test Plan:
- Filtered commits by various branches, e.g. "master"; "lfs". Saw correct-seeming results.
- Stubbed out the "just list everything" path to hit the `diffusion.internal.ancestors` path, saw the same correct-seeming results.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19431
2018-05-08 15:51:42 -07:00
epriestley
397645b273 Export task point values as double, not int
Summary:
See <https://discourse.phabricator-community.org/t/maniphest-non-integer-point-values-in-csv-export/1443>.

We currently export the Maniphest "points" field as an integer, but allow it to accept decimal values (e.g. "6.25").

Also fix a bug where we wouldn't roll over from "..., X, Y, Z, AA, AB, ..." correctly for Excel column names if sheet had more than 26 columns.

Test Plan:
  - Set a task point value to 6.25.
  - Exported to text, JSON, XLS.
  - Saw 6.25 represented accurately in exports.
  - Exported an excel sheet with 27+ columns.
  - Manually printed the first 200 column names to check that the algorithm looks correct.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19434
2018-05-08 15:49:40 -07:00