Summary:
Fixes T9808.
An instance imported a very large repository, generating approximately 4 million tasks over the course of a few days. A week later, these tasks started expiring and became candidates for garbage collection.
The GC works by deleting 100 rows at at time over and over again. It finds the rows it's going to delete by querying for old rows.
Currently, this query generates a `WHERE dateCreated < X ORDER BY id DESC` query. This query can not efficiently execute using a single key, because it relies on `dateCreated` order to find the rows, then on `id` order to sort them. With a table with 4M rows, this is slow.
This would still be OK, except that the query has to execute a lot of times since it only deletes 100 rows each time. Particularly, it needs to execute a total of ~40K times.
Instead, generate `WHERE dateCreated < X ORDER BY dateCreated DESC, id DESC`. This should have the same effect in general and the GC definitely doesn't care about the difference, but it should be more efficient at large scales.
Test Plan:
I had to `TRUNCATE` the problem table so I don't have a perfect repro to completely convincingly test this anymore. Both queries behave fine at small scales, which is why we haven't seen this before.
I was able to run the newer query in production before I nuked the table and have it complete in a reasonable amount of time, while the old query hung longer than I wanted to wait (several minutes?). The query plan for the new query was also a good one, while the query plan for the old query was terrible.
I loaded the daemon console and ran `bin/garbage collect --collector worker.tasks --trace`. I verified the queries looked reasonable and produced reasonable results in production.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9808
Differential Revision: https://secure.phabricator.com/D14505
Summary: Ref T6049, remarkup links to use short URLs and make commenting on Phurl's actually work
Test Plan:
- Create Phurl `U123`
- Comment on that Phurl `((123))`
Comment should link to `/u/123`
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Maniphest Tasks: T6049
Differential Revision: https://secure.phabricator.com/D14477
Summary:
Fixes T9787. Currently, file PHID extraction logic happens very early, before we normalize/merge/etc the transactions.
In D14390, I changed how the CONTENT transaction works: before, callers would pass in a file PHID. Afterward, they just pass in the content.
Passing in the content is generaly easier and feels more correct, but inadvertenly broke PHID extraction because converting the content into a file PHID now happened after we extracted the PHID. So we'd extract the entire text of the paste as a "file PHID", which wouldn't work.
Instead, extract file PHIDs later. This impacts a couple of other applications (Conpherence, Pholio) which receive an object or have an unusual file-oriented transaction.
Test Plan:
- Made a new paste, verified the raw file attached to it properly.
- Made and updated a mock, verified all the files attached properly.
- Updated a Conpherence room image, verified the files attached properly.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9787
Differential Revision: https://secure.phabricator.com/D14494
Summary:
Ref T9787. To fix this, I want to change how file PHIDs are extracted slightly: specifically, I'm going to extract them later in the editing process.
Before doing this, clean up a couple of bad implementations:
- Owners extracts its description as a file PHID. This is an error.
- Extract the description as a remarkup block instead.
- Add an edge table so stuff like file attachment works properly.
- Differential has a no-op extract method. This is presumably just a copy/paste issue from long ago.
Test Plan:
- Edited a revision in Differential.
- Dropped a file into the description of an Owners package.
- Before change: this did not attach the file.
- After change: the file now attaches properly and shows up as "Attached" in the file details.
Reviewers: chad, joshuaspence
Reviewed By: joshuaspence
Subscribers: joshuaspence
Maniphest Tasks: T9787
Differential Revision: https://secure.phabricator.com/D14493
Summary: Fixes T9757.
Test Plan: Created a Herald rule and then subscribed to it with a different account.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin
Maniphest Tasks: T9757
Differential Revision: https://secure.phabricator.com/D14468
Summary: Move some `PhabricatorPolicyRule` implementations to a subdirectory of the parent application.
Test Plan: N/A
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14478
Summary:
Fixes T9799. Currently, if you can't see an application like Paste, we fatal when trying to generate a result for `conduit.query`, because the new EditEngine-based `paste.edit` method doesn't "know" that it's a "Paste" method.
Straighten this out, and use policies and queries a little more correctly/consistently.
Test Plan:
- Called `conduit.query` as a user who does not have permission to use Paste.
- Before change: fatal.
- After change: results, excluding "paste.*" methods.
Reviewers: chad
Reviewed By: chad
Subscribers: cburroughs
Maniphest Tasks: T9799
Differential Revision: https://secure.phabricator.com/D14492
Summary:
Fixes T9798. That task has good repro instructions.
In sub-views, we don't link the "History" icon correctly -- we only link it to `history/README` instead of `history/path/to/README`. Add the full path.
Also canonicalize the paths in a slightly prettier and more consistenty way.
Test Plan: Viewed root and non-root browse tables, saw links show up properly.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9798
Differential Revision: https://secure.phabricator.com/D14491
Summary: As suggested in D14461.
Test Plan: Used `./bin/remove destroy` on an Almanac service with properties attached, saw entries removed from the `phabricator_almanac.almanac_property` table.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14490
Summary: Ref T9762. Currently it is not possible to destroy an Alamanac device because any associate bindings cannot be destroyed.
Test Plan: Destroyed an Almanac device.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin
Maniphest Tasks: T9762
Differential Revision: https://secure.phabricator.com/D14461
Summary: Some linter messages, such as those produced by `ArcanistPHPCompatibilityXHPASTLinterRule`, contain backticks but are currently rendered as Remarkup literals. I think that it is generally desirable to allow lint messages to be rendered as Remarkup, although we should ideally have a way to render Remarkup for use on the command line (I actually think that this already exists, but I don't think that `arc lint` does this when rendering linter messages).
Test Plan: Resubmitted D14481 to my dev install and saw Remarkuped lint messages.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14485
Summary: Rename the XHPAST database from `{$NAMESPACE}_xpastview` to `{$NAMESPACE}_xhpast`.
Test Plan: Ran `./bin/storage --namespace test upgrade --no-quickstart`.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14442
Summary: This logic is inverted. Re-vert it.
Test Plan: Write and publish a new post, see publish time.
Reviewers: epriestley, joshuaspence
Reviewed By: joshuaspence
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14464
Summary: I think `HeraldRule`s are the only objects which have monograms but are not accesible via `/{$monogram}`. This diff changes the `/herald/rule/{$id}` URI to `/{$monogram}`.
Test Plan: Clicked a bunch of links in Herald to ensure there were no dead links.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14469
Summary:
Current JIRA integration is quite noisy in terms of email, and makes users hunt and peck for the related revisions.
Teach it to create an Issue Link on the JIRA side, and allow to disable commenting.
Test Plan: comment on revision in each of the 4 settings, check JIRA end for expected result.
Reviewers: btrahan, eMxyzptlk, epriestley, #blessed_reviewers, avivey
Reviewed By: epriestley, #blessed_reviewers
Subscribers: avivey, vhbit, jra3, eMxyzptlk, frenchs, aik099, svemir, rmuslimov, cpa199, waynea, epriestley, Korvin, hach-que
Projects: #doorkeeper
Maniphest Tasks: T5422
Differential Revision: https://secure.phabricator.com/D9858
Summary: Fixes T9772. We now need an EditEngineConfiguration to do interesting things with EditEngine, but this public API wasn't properly making sure we have one.
Test Plan: Called `conduit.query` from web console. Fatal prior to patch; success afterward.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9772
Differential Revision: https://secure.phabricator.com/D14475
Summary: Ref T7053. Remove the `envHash` and `envInfo` fields, which are no longer used now that the daemons restart automagically. Depends on D14458.
Test Plan: Saw no more setup issues.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: tycho.tatitscheff, epriestley
Maniphest Tasks: T7053
Differential Revision: https://secure.phabricator.com/D14446
Summary: Fixes T7053. Depends on D14452.
Test Plan:
Created a custom daemon which dumps out the config hash (by querying `PhabricatorEnv::calculateEnvironmentHash()`). Ran this daemon with `./bin/phd debug PhabricatorDebugDaemon` and saw the config hash update within 30 seconds.
{P1886}
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin
Maniphest Tasks: T7053
Differential Revision: https://secure.phabricator.com/D14458
Summary: Right now we're attaching the body of every Phame post on each comment, at least restrict it to newly created objects only.
Test Plan: Write a new post, get full email, leave a comment, get less email.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14459
Summary: We currently orphan posts when you delete a blog. Fixes some visibility and permission errors when that happens. Also... should allow you to archive posts.
Test Plan: Delete a blog, visit a post I made, still can see it.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T9360
Differential Revision: https://secure.phabricator.com/D14457
Summary:
Ref T9132. This diff doesn't do anything interesting, it just lays the groundwork for more interesting future diffs.
Broadly, the idea here is to let you create multiple views of each edit form. For example, we might create several different "Create Task" forms, like:
- "New Bug Report"
- "New Feature Request"
These would be views of the "Create Task" form, but with various adjustments:
- A form might have additional instructions ("how to file a good bug report").
- A form might have prefilled values for some fields (like particular projects, subscribers, or policies).
- A form might have some fields locked (so they can not be edited) or hidden.
- A form might have a different field order.
- A form might have a limited visibility policy, so only some users can access it.
This diff adds a new storage object (`EditEngineConfiguration`) to keep track of all those customizations and represent "a form which has been configured to look and work a certain way".
This doesn't let these configurations do anything useful/interesting, and you can't access them directly yet, it's just all the boring plumbing to enable more interesting behavior in the future.
Test Plan:
ApplicationEditor forms now let you manage available forms and edit the current form:
{F959025}
There's a new (bare bones) list of all available engines:
{F959030}
And if you jump into an engine, you can see all the forms for it:
{F959038}
The actual form configurations have standard detail/edit pages. The edit pages are themselves driven by ApplicationEditor, of course, so you can edit the form for editing forms.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9132
Differential Revision: https://secure.phabricator.com/D14453
Summary: Via HackerOne. See D14025. I missed this comparison when making the original change.
Test Plan:
- Used `cat mail.txt | scripts/mail/mail_handler.php --process-duplicates` to pipe mail in a whole lot of times.
- Tried bad hashes, saw rejections.
- Tried good hash, saw mail accepted.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14455
Summary: Adds mail reply support to Phame Posts.
Test Plan: Comment on a post, get mail.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T9746
Differential Revision: https://secure.phabricator.com/D14454
Summary: When accessing an invalid URL on the short Phurl domain, users should see informative message
Test Plan: Open URL in the previously configured Phurl short domain such as `https://www.zz.us` and see dialog with message. Open `https://www.zz.us/u/123` for a valid `U123` Phurl and access destination URL.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14450
Summary: Adds commenting to Phame Posts, also testing a new "document comment style". Unsure about it but Phame is a prototype so good place to explore.
Test Plan: Leave some comments, see some comments, test show/hide.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T9746
Differential Revision: https://secure.phabricator.com/D14451
Summary:
This diff adds a new mode argument to the Diffusion Conduit API with two options:
- "overwrite": the default, maintains the current behavior of deleting all coverage
in the specified branch before uploading the new coverage
- "update": does not delete old coverage, but will overwrite previous
coverage information if it's for the same file and commit
`DiffusionRequest::loadCoverage` already loads a file's coverage from the
latest available commit, so uploading coverage for different files in different
commits with "update" will result in seeing the latest uploaded coverage in
Diffusion.
Test Plan: manual local verification
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14428
Summary: Crumbies
Test Plan: View post, see blog link, click on crumb, see blog
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14449
Summary: Cleaning up house, may revisit in a v2. Removes ability to set Disqus or Facebook comments as comment system on Phame Posts.
Test Plan: Create blog, create post, edit blog, view live pages.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: btrahan, Korvin
Maniphest Tasks: T9746
Differential Revision: https://secure.phabricator.com/D14448
Summary: Ref T8995, config option for Phurl short domain to share shortened URL's
Test Plan:
- Configure Phurl short domain to something like "zz.us"
- Navigate to `zz.us`; get 404
- Navigate to `zz.us/u/3` or `zz.us/u/alias` where `U3` is an existing Phurl; redirect to correct destination
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin
Maniphest Tasks: T8995
Differential Revision: https://secure.phabricator.com/D14447
Summary: Drops Join Policy, uses Edit Policy where needed. Allows anyone with Blog Edit permissions to post and edit any post on that blog. Fixes T5371
Test Plan: Draft Post as chad, see post, log in with notchad, edit that post and publish it.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T5371
Differential Revision: https://secure.phabricator.com/D14444
Summary: Currently, a bunch of developers are using #xhpast for writing custom linter rules. As such, we end up with a fair few `XHPASTSyntaxErrorException` in our PHP error logs. I think that throwing an exception is not quite correct in this case because it is somewhat expected that invalid PHP may be entered. Instead, catch the exception and show the user a helpful message.
Test Plan: This doesn't quite work yet... the stream and tree views render as blank but the exceptions still propogate to the error logs. Mostly, I'm not sure how the exception should be rendered for display.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14028
Summary: Add some mailkeys, allow feed stories to be published.
Test Plan: New Blog, Edit Blog
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14434
Summary:
Fixes T9735. I changed how the TYPE_LANGUAGE transction works a little but that accidentally tripped an error condition in `paste.create`.
- Don't bail on no-effect transactions to `paste.create` (like not setting a language).
- When a transaction type has no tailored UI message, make it easier to figure out which transaction is problematic.
Test Plan: Ran `arc paste ...` locally. Got an error before the patch, clean paste creation afterward.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9735
Differential Revision: https://secure.phabricator.com/D14440
Summary: Fixes T9051, adds ability to edit blogs and posts and manually add subscribers. Also fixed bug granting tokens to posts.
Test Plan: Create a new blog, subcribe chad and notchad. Write a post, both are notified. Award token for hard work.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T9051
Differential Revision: https://secure.phabricator.com/D14432
Summary:
Fixes T9732. We currently tokenize strings (like user realnames) in the default non-unicode mode, which can cause patterns like `\s` to work incorrectly.
Use `/u` to use unicode-aware tokenization instead.
Test Plan:
The behavior of "\s" depends upon environmental settings like LC_ALL.
With LC_ALL set to "C", `\xA0` is not considered a whitespace character.
With LC_ALL set to "en_US", it is:
```
$ php -r 'setlocale(LC_ALL, "C"); echo count(preg_split("/\s/", "\xE5\xBF\xA0")) . "\n";'
1
$ php -r 'setlocale(LC_ALL, "en_US"); echo count(preg_split("/\s/", "\xE5\xBF\xA0")) . "\n";'
2
```
To reproduce the original issue, I added an explicit:
```
setlocale(LC_ALL, "en_US");
```
...call before the `preg_split()` call. This caused "忠" to be improperly split.
I then added "/u", and observed proper tokenization.
Reviewers: chad
Reviewed By: chad
Subscribers: qiu8310
Maniphest Tasks: T9732
Differential Revision: https://secure.phabricator.com/D14441
Summary: Larger (open) installs may want to restrict Blog to formal entities, like with Phriction.
Test Plan: Set policy to administrators, have notchad try to create a blog. See error.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14433
Summary: fix T9718.
Test Plan: view project page when maniphest is and isn't. Look for Workboards.
Reviewers: #blessed_reviewers, chad
Reviewed By: #blessed_reviewers, chad
Subscribers: epriestley
Maniphest Tasks: T9718
Differential Revision: https://secure.phabricator.com/D14438
Summary: Sends out the body of the post along with the details.
Test Plan: Write a new post, see body in email.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T9360
Differential Revision: https://secure.phabricator.com/D14431
Summary: Ref T7951, Starting the Calendar user guide
Test Plan: Go to {nav Diviner > Phabricator User Docs > Calendar User Guide}, read about how fabulous the Calendar application is.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Maniphest Tasks: T7951
Differential Revision: https://secure.phabricator.com/D13496
Summary: Adds ability to set visibility when authoring a Post. New default is "Visible". If you write a post and save it as a Draft, and later click publish, a feed story and mail will go out.
Test Plan: Write a new Post, see feed story and get email. Write a new Draft, get nothing. Click Publish, see story and email.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T9360
Differential Revision: https://secure.phabricator.com/D14429
Summary: Ref T9722, Add Phurl Remarkup as `((id))` or `((alias))`
Test Plan: Add a comment to any object as `((id))` or `((alias))`. Make sure comment renders as a link.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Maniphest Tasks: T9722
Differential Revision: https://secure.phabricator.com/D14427
Summary: Allows feed stories and mail for new Phame Posts.
Test Plan: Write Post, Get Mail
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14426
Summary: Ref T9546. I only got the title to always show the blog title (better than nothing) -- showing the post title properly isn't trivial and is more work than I want to do right now.
Test Plan:
- Description now has remarkup.
- Title now shows blog title (better than nothing).
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9546
Differential Revision: https://secure.phabricator.com/D14423
Summary: Adds Remarkup rules and CSS, cleans up some spacing a color. Ref T9546
Test Plan: Review a blog post list, and a blog
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T9546
Differential Revision: https://secure.phabricator.com/D14421
