Summary:
Ref T13042. This is a very, very old policy-violating option from yesteryear which supported build systems publishing updates by adding comments to revisions, without sending email about it.
Harbormaster has served this role for a long time and this is policy-violating in the general case (it allows attackers to act in secret).
Test Plan: Grepped for affected symbols.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13042
Differential Revision: https://secure.phabricator.com/D18881
Summary: Ref T13035. See that task for a description of the issue.
Test Plan:
- Enabled prototypes.
- Disabled all Herald rules that trigger Harbormaster builds.
- Created a new revision.
- Before patch: initial review request email was dropped.
- After patch: initial review request email is sent.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13035
Differential Revision: https://secure.phabricator.com/D18851
Summary:
See PHI230. Currently, we denormalize raw line counts onto diffs and revisions, but not added/removed line counts.
I'd like to try a `[---+ ]` sort of size hint element (see D16322 for more) as a general approach to conveying size information at a glance and see how it feels, since I think the raw size number isn't very scannable/useful and it may be a significant improvement to hint about how much of a change is throwing stuff out vs adding new stuff.
This just makes the data available without any subquerying and doesn't actually change the UI.
Test Plan:
Created a revision, saw detailed change information populate in the database.
```
mysql> select * from differential_revision where id = 292\G
*************************** 1. row ***************************
id: 292
title: WIP
originalTitle: WIP
phid: PHID-DREV-ux3cxptibn3l5pxsug3z
status: draft
summary: asdf
testPlan: asdf
authorPHID: PHID-USER-cvfydnwadpdj7vdon36z
lastReviewerPHID: NULL
lineCount: 41
dateCreated: 1513179418
dateModified: 1513179418
attached: []
mailKey: h4mn6perdio47o4beomyvu75zezwvredx3mbrlgz
branchName: NULL
viewPolicy: users
editPolicy: users
repositoryPHID: PHID-REPO-wif5lutk5gn3y6ursk4p
properties: {"lines.added":40,"lines.removed":1}
activeDiffPHID: PHID-DIFF-ixjphpunpkenqgukpmce
1 row in set (0.00 sec)
```
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18832
Summary:
Fixes T13027. Ref T2543. When revisions promote from "Draft" because builds finish or no builds are configured, the status currently switches from "Draft" to "Needs Review" without re-running Herald.
This means that some rules -- notably, "Send me an email" rules -- don't fire as soon as they should.
Instead of applying this promotion in a hacky way inline, queue it and apply it normally in a second edit, after the current group finishes.
Test Plan:
- Created a revision, reviewed Herald transcripts.
- Saw three Herald passes:
- First pass (revision creation) triggered builds and no email.
- Second pass (builds finished) did not trigger builds (no update) and did not trigger email (revision still a draft).
- Third pass (after promotion out of 'draft') did not trigger builds (no update) but did trigger email (revision no longer a draft).
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13027, T2543
Differential Revision: https://secure.phabricator.com/D18819
Summary:
Ref T10233. See PHI231. When users ignore the `arc land` prompt about bad revision states, make it explicitly clear in the transaction log that they broke the rules.
You can currently figure this out by noticing that there's no "This revision is accepted and ready to land." message, but it's unrealistic to expect non-expert users to look for the //absence// of a message to indicate something, and this state change is often relevant.
Test Plan: {F5302351}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T10233
Differential Revision: https://secure.phabricator.com/D18808
Summary:
Depends on D18801. Ref T2543. See PHI229. I missed "Accept" before, but intended to disallow it (like "Reject") since I don't want drafts to be reviewable.
However, "Resign" seems fine to allow? So let's allow that for now.
Test Plan: Was no longer offered "Accept" on draft revisions.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18802
Summary:
See PHI228. Ref T2543. The current logic gets this slightly wrong: prototypes are off, you create a draft with `--draft`, then promote it with "Request Review". This misses both branches.
Instead, test these conditions a little more broadly. We also need to store broadcast state since `getIsNewObject()` isn't good enough with this workflow.
Test Plan:
- With prototypes on and autopromotion, got a rich email after builds finished.
- With prototypes off, got a rich email immediately.
- With prototypes off and `--draft`, got a rich email after "Request Review".
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18801
Summary: Whitespace has semantic meaning for yaml files, so we shouldn't suppress whitespace-only lines of diff by default.
Test Plan: Edited local config to include yaml files, saw expected whitespace changes.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D18775
Summary:
See PHI180. Currently, if you begin creating or editing an inline and then swap display modes (for example, with "View Unified"), your edit is lost.
Persisting the editor state is complicated and this is very rare, so just prevent the action and warn the user instead.
Also make the warning persist for a little longer since a few of the messages, including this one, take a couple seconds to read now.
Test Plan:
- Edited a comment, tried to swap display modes, got a warning.
- Swapped display modes normally with no comment being edited.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18774
Summary:
Depends on D18771. See PHI206. Currently, `arc diff --draft` only holds revisions in draft mode: it doesn't put them into draft mode if the install isn't configured to use draft mode.
Instead, make it a bit more forceful so that `arc diff --draft` can create into draft mode explicitly even if protoypes are off. This aligns with expection a little more clearly.
Test Plan: Ran `arc diff --draft` with prototypes off, got a revision held in draft mode.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18772
Summary: See PHI210. Ref T2543. Currently, we don't set this flag if you have prototypes off and don't get any of the new draft stuff, so the mail drops some of the details it is supposed to have.
Test Plan: Disabled prototypes, created a revision, saw summary / test plan in the initial mail.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18771
Summary:
See PHI199. Ref T2543. When you run a RevisionQuery with a legacy status constraint (via `differential.query`), we currently don't match "Draft" revisions.
Use the actual complete map from `DifferentialRevisionStatus` instead of hard coding the status list so "Draft" is included.
Test Plan:
- Ran `differential.query` with `ids` and `status` for a draft revision.
- Before patch: revision not returned in results.
- After patch: revision returned in results.
(Note that it returns as "Needs Review", for compatibility.)
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18765
Summary:
See PHI190. This clarifies the ruleset a bit:
- If you accepted, then the author used "Request Review" explicitly, we now show "Accepted Earlier" instead of "Accepted" in the "Reviewers" list on the main revision page. This makes it sligthly more clear why the revision is back in your review queue without picking through the transaction log.
- Instead of moving all non-current accepts into "Ready to Review", move only voided accepts into "Ready to Review". This stops us from pulling older accepts which haven't been voided (which could have been incorrectly pulled) and correctly pulls older, voided accepts from before an update (for example: accept, then request review, then update) and generally aligns better with intent/expectation.
Test Plan:
- Accepted, requested review.
- Saw reviewer as "Accepted Earlier".
- Saw review in "Ready to Review" bucket.
- Accepted, updated (with sticky accept).
- Saw reviewer as "Accepted Prior Diff".
- Saw review as "Waiting on Authors".
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18764
Summary: See PHI195. This bulks out these API methods since all the requests are pretty straightforward.
Test Plan: Ran `edge.search` and `differential.revision.search`.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18762
Summary:
See PHI197. Populates "status" transactions and a few other obvious types where there's no security/performance/payload/formatting issue I can come up with.
The names here are the same as the names for editing with `maniphest.edit`.
Test Plan: Used `transaction.search` to retrieve transactions of all new types.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18761
Summary: Ref T12689. See PHI178. This isn't a complete solution (you may still get mailed via packages/projects) but should fix the obvious issue, where "Resigned" reviewers are incorrectly always sent mail directly.
Test Plan: Had Alice resign, interacted as Bailey, no mail to Alice.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T12689
Differential Revision: https://secure.phabricator.com/D18758
Summary:
Depends on D18756. Fixes T12539. See PHI190. Currently, when this occurs:
- Alice accepts.
- Bailey requests review.
- Alice views her dashboard.
...the revision appears in "Waiting on Other Reviewers" (regardless of whether other reviewers actually exist or not).
Instead, ignore these voided/non-current accepts and let the revisions appear in "Ready to Review", which is more natural.
Test Plan: Went through the steps above. On `master`, saw revision in "Waiting on Other Reviewers". After patch, saw it in "Ready to Review".
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T12539
Differential Revision: https://secure.phabricator.com/D18757
Summary:
Ref T12539. See PHI190. Currently, each Diff has a `revisionID`, but Revisions do not point at the current active diff. To find the active diff for a given revision, we need to issue a separate query.
Furthermore, this query is inefficient for bulk loads: if we have a lot of revisions, we end up querying for all diff IDs for all those revisions first, then selecting the largest ones and querying again to get the actual diff objects. This strategy could likely be optimized but the query is a mess in any case.
In several cases, it's useful to have the active diff PHID without needing to do a second query -- sometimes for convenience, and sometimes for performance.
T12539 is an example of such a case: it would be nice to refine the bucketing logic (which only depends on active diff PHIDs), but it feels bad to make the page heavier to do it.
For now, this is unused. I'll start using it to fix the bucketing issue, and then we can expand it gradually to address other performance/convenience issues.
Test Plan:
- Ran migrations, inspected database, saw sensible values.
- Created a new revision, saw a sensible database value.
- Updated an existing revision, saw database update properly.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T12539
Differential Revision: https://secure.phabricator.com/D18756
Summary:
Ref T2543. Fixes two relatively minor things:
- When builds finish in Harbormaster, send mail "From" the author.
- Set the `firstBroadcast` flag so that initial mail picks up earlier history (notably, the "reviewers" line).
For now, I'm not setting `firstBroadcast` on explicit "Request Review" (but maybe we should), and not trying to deal with weird cases where you leave a bunch of comments on a draft. Those might be fine as-is or may get tweaked later.
Test Plan: Created a revision with Harbormaster builds, ran builds, saw initial email come "From" the right user with more metadata.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18748
Summary: Ref T2543. After D18731, Herald build rules run more often, but now incorrectly try to run builds when Diffusion closes a revision because a commit landed.
Test Plan: Made some mundane updates locally; this is tricky to test comprehensively locally so I'm mostly planning to just push it to `secure`.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18745
Summary:
Ref T2543. Instead of autosubmitting revisions to "Needs Review" when builds finish, allow them to be held in "Draft" indefinitely.
There's currently no UI for this. I plan to just expose it as `arc diff --draft` for now, in a followup change.
Test Plan:
- Created a revision (via Conduit) with "hold as draft", saw it hold as draft after builds finished.
- Created a revision (normally), saw it autosubmit after builds finished.
- Requested review of a "hold as draft" revision to kick it out of draft state.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18737
Summary:
Depends on D18740. Prepares `arc` to receive a `--draft` flag by letting us switch to "differential.revision.edit" instead of "differential.createrevision".
To "differential.revision.edit", we need a transaction list, but we can't automatically construct this list from a field map. Return the transaction list alongside the field map.
The next change uses this list (if available) to switch us to the modern API method.
Test Plan: Ran `arc diff` on the experiemntal branch with followup changes, got a new revision.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18741
Summary: See PHI90. For now, this only provides a limited amount of information, but should satisfy the use case in PHI90 and build toward a more complete version in the future.
Test Plan: Used new Conduit method to retrieve information about diffs.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18744
Summary:
Ref T2543. Fixes T10109.
Currently, Herald only runs in Differential when a change updates the diff. This is partly for historical reasons, and partly because we don't want to restart builds every time someone makes a comment. However, this behavior is inconsistent with other applications (which always trigger on any change), and occasionally confusing to users (in T10109, for example) or otherwise undesirable.
A similar issue is that T2543 has introduced a "Draft" state, where revisions don't send normal mail until builds finish. This interacts poorly with "Send me an email" rules (which shouldn't do anything here) and particularly with "Send me an email + only run these actions the first time the rule matches", since that might have an effect like "do nothing when the revision is created, then never anything again since you already did nothing once".
To navigate both of these issues, let objects tell Herald that certain actions (like mail or builds) are currently forbidden. If a rule uses a field or action which is currently forbidden, the whole rule automatically fails before it executes, but doesn't count toward "only the first time" as far as Herald's tracking of rule execution is concerned.
Then, forbid mail for draft revisions, and forbid builds for revisions which didn't just get updated. Forbidding mail fixes the issues with "Send me an email" that were created by the introduction of the draft state.
Finally, make Herald run on every revision update, not just substantive updates to the diff. This resolves T10109.
Test Plan:
Created revisions via the draft -> submit workflow, saw different transcripts. Here's a mail action being forbidden for a draft:
{F5237324}
Here's a build action being forbidden for a "mundane" update:
{F5237326}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T10109, T2543
Differential Revision: https://secure.phabricator.com/D18731
Summary:
Ref T2543. When Harbormaster finishes builds and promotes a draft revision to review, we currently publish "Harbormaster requested review of...".
Instead, attribute this action to the author, since that's more natural and more useful.
Test Plan: Promoted a diff locally, saw it attributed to me rather than Harbormaster.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18730
Summary: Ref T12680. Use the slightly sleeker construction from D18722 in Differential.
Test Plan: Viewed revision list, reordered by date modified.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T12680
Differential Revision: https://secure.phabricator.com/D18727
Summary:
Ref T2543. When a revision is created, we check if any builds are waiting/failed, and submit it for review immediately if we aren't waiting for anything.
In doing this, we ignore builds with only autotargets, since these are client-side and failures from local `arc lint` / `arc unit` should not count (the user has already chosen to ignore/skip them).
The way we do this has some issues:
- Herald may have started builds, but they may still be PENDING and not have any targets yet. In this case, we'll see "no non-autotargets" and ignore the build, which is wrong.
- We have to load targets but don't really care about them, which is more work than we really need to do.
- And it's kind of complex, too.
Instead, just let `BuildQuery` filter out "autobuilds" (builds generated from autoplans) with a JOIN.
Test Plan: Ran `arc diff` with builds configured, got a clean "Draft" state instead of an incorrect promotion directly to "Needs Review".
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18721
Summary: Ref T2543. When revisions are in the draft state, tell the user what we're waiting for or why they aren't moving forward.
Test Plan: {F5228840}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18714
Summary: Ref T2543. This is a less ambitious version of the rule in D18628, which I backed off from, since I think this probably still has a fair number of loose ends to tie up.
Test Plan: Created a revision locally.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18713
Summary: Noticed a couple of typos in the docs, and then things got out of hand.
Test Plan:
- Stared at the words until my eyes watered and the letters began to swim on the screen.
- Consulted a dictionary.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
Differential Revision: https://secure.phabricator.com/D18693
Summary:
Ref T13000. This marks each table as either "data" (normal data), "cache" (automatically rebuilt, no need to ever dump) or "index" (can be manually rebuilt).
By default, `bin/storage dump` dumps data and index tables, but not cache tables.
With `--no-indexes`, it dumps only data tables. Indexes can be rebuilt after a restore with `bin/search index --all ...`.
Test Plan:
- Ran `--no-indexes` and normal dumps with `--trace`, verified that cache and index (former case) or cache only (latter case) tables were dumped with `--no-data`.
- Verified dump has the same number of `CREATE TABLE` statements as before the changes.
- Reviewed persistence tags in the web UI (note Ferret engine tables are "Index"):
{F5210886}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13000
Differential Revision: https://secure.phabricator.com/D18682
Summary:
Ref PHI91. When Owners (or Herald, or manual user action) adds package reviewers to a revision, later updates to the revision make some of them less relevant or irrelevant.
Provide a hint when a package reviewer doesn't own any of the paths that a diff changes. Humans can then decide if the reviewer is obsolete/irrelevant or not.
This is a rough cut to get the feature working, design could probably use some tweaking if it sticks.
Test Plan: {F5204309}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: jboning
Differential Revision: https://secure.phabricator.com/D18663
Summary:
Ref T2543. This doesn't stand alone since mail still goes out normally, but gets this piece working: new revisions start as "Draft", then after updates if there are no builds they go into "Needs Review".
This should work in general because builds update revisions when they complete, to publish a "Harbormaster finished build yada yada" transaction. So either we'll un-draft immediately, or un-draft after the last build finishes.
I'll hold this until the mail and some other stuff (like UI hints) are in slightly better shape since I think it's probably too rough on its own.
Test Plan: Created revisions locally, saw them un-draft after builds.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18628
Summary:
Ref T2543. Currently, we always do some special things when a revision is created, mostly adding more stuff to the mail.
With drafts, we want to suppress initial mail and send this big, rich mail only when the revision actually moves out of "draft".
Prepare the code for this, with the actual methods hard-coded to the current behavior. This will probably take some tweaking but I think I got most of it.
Test Plan: Banged around in Differential so it sent some mail, saw normal mail without anything new.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18627
Summary:
Ref T2543. Most actions are not available for drafts.
Authors can "Request Review" (move out of draft to become a normal revision) or "Abandon".
Non-authors can't do anything (maybe we'll let them do something later -- like "Commandeer"? -- if there's a good reason).
Test Plan: Viewed a draft revision as an author and non-author, saw fewer actions available.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18626
Summary:
Ref T2543. There's no way to put revisions into this state yet, but start adding support for when there is.
Adds the status constant, plus support for bucketing them.
Test Plan:
- Manually put a revision in "Draft" state by updating the database directly.
- Verified my drafts showed up in a "Drafts" section on the bucket view.
- Verified others' drafts did not appear on the action bucket view.
- Viewed revisions, queried for "Draft" revisions, etc (stuff we get for free).
{F5186781}
{F5186782}
{F5186783}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18625
Summary:
Fixes T12986. I caught this bug in the changes from D18584: when we moved a large hunk to file storage, we would decompress it but keep the "deflated" flag. This could cause confusion when loading it later. I missed this in testing since I wasn't exhaustive enough in checking hunks and didn't run into a compressed one.
Instead of compressing on `save()`, compress during the normal workflow.
We currently never advise users to run this workflow so I didn't bother trying to clean up possible existing migrations.
Test Plan:
- Ran `bin/differential migrate-hunk` on compressed hunks, moving them to and from file storage. Saw them work correctly and remain compressed.
- Created new small (uncompressed) and large (compressed) hunks, verified they work properly and get compressed (if applicable).
- Used `bin/cache purge --caches changeset` to clear changeset caches and make sure the actual table was being hit.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T12986
Differential Revision: https://secure.phabricator.com/D18624
Summary:
See PHI71. These didn't get properly updated when we wrote Subprojects and Milestones, and should use materialized members, not raw members. Swap the query so projects you are an indirect member of (e.g., milestones you are a member of the parent for, and parent projects you are a member of a subproject of) are included in the result list.
Also fix a bad typeahead datasource.
Test Plan:
- Ran a dry run with the test console, saw project PHIDs for milestones and parent projects in the raw field value.
- Tried to set "Author's projects" to a user, no longer could.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18619
Summary:
Ref T11823. This is the meaty part of the change, and updates `RefEngine` to use separate RefCursor (for names) and RefPosition (for actual commit positions) tables.
I'll hold this whole series until after the release cut so it has some time to bake on `secure` to look for issues. It's also not a huge problem if there are bugs here since these tables are just caches anyway, although they do feed into some other things, and obviously it's never good to have bugs.
Test Plan:
- This logic can be invoked directly with `bin/repository refs <repository> --trace --verbose`.
- Ran that on unchanged repositories, new branches, removed branches, and modified branches. Saw appropriate output and cursor positions.
- Ran on a mercurial repository to test the close/open logic, saw it correct open/closed state of incorrect positions.
- Browed around Diffusion in various repositories.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T11823
Differential Revision: https://secure.phabricator.com/D18614
Summary:
Ref T11823. See PHI68. T11823 has a full description of this issue and a plan to fix it, but the full plan is relatively complicated.
Until that can happen, provide a workaround for the biggest immediate issue, where multiple copies of a ref cursor can cause `executeOne()` to throw, since it expects a single result. In practice, these copies are always identical so we can just pick the first one.
This will get cleaned up once T11823 is fixed properly.
Test Plan:
Forced the table into a duplicate/ambiguous state, reproduced a similar-looking error:
{F5180999}
Applied the patch, got the "Land" to work as expected:
{F5181000}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T11823
Differential Revision: https://secure.phabricator.com/D18599
Summary:
Ref T12033. This is a very narrow fix for this issue, but it should fix the major error: don't attach patches if they're bigger than the mail body limit (by default, 512KB).
Specifically, the logs from an install in T12033 show a 112MB patch being attached, and that's the biggest practical problem here.
I'll follow up on the tasks with more nuanced future work.
Test Plan: Enabled `differential.attach-patches`, saw a patch attached to email. Set the byte limit very low, saw patches get thrown away.
Reviewers: chad, amckinley
Reviewed By: amckinley
Maniphest Tasks: T12033
Differential Revision: https://secure.phabricator.com/D18598
Summary:
Ref T12932. For long-lived installs, one of the largest tables tends to be the hunk data table. Although it doesn't grow tremendously fast, it's also well suited to storage in Files instead of the database (infrequent access, relatively large blobs of data, mostly one-at-a-time access), and earlier work anticipated eventually adding support for Files storage.
Make Files storage work, and provide `bin/differential migrate-hunk` to manually test/migrate hunks. This is currently the only way hunks get moved to file storage, but I expect to add a GC step which moves them to File storage after 30 days shortly.
The immediate motivation for this is to relieve storage pressure on db001/db002 so we have more headroom for deploying the Ferret engine and its larger indexes (see also T12819).
Test Plan:
- Used `bin/differential migrate-hunk` to move a hunk to and from file storage, verified it survived intact.
- Downloaded the actual stored file, sanity-checked it. Verified permissions.
- Destroyed a diff with `bin/remove destroy`, saw the hunk and file storage destroyed.
- Verified that going from file -> text destroys the old file properly with `migrate-hunk --trace ...`.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12932
Differential Revision: https://secure.phabricator.com/D18584
Summary:
See brief discussion in D18554. All the index tables are the same for every application (and, at this point, seem unlikely to change) and we never actually pass these objects around (they're only used internally).
In some other cases (like Transactions) not every application has the same tables (for example, Differential has extra field for inline comments), and/or we pass the objects around (lots of stuff uses `$xactions` directly).
However, in this case, and in Edges, we don't interact with any representation of the database state directly in much of the code, and it doesn't change from application to application.
Just automatically define document, field, and ngram tables for anything which implements `FerretInterface`. This makes the query and index logic a tiny bit messier but lets us delete a ton of boilerplate classes.
Test Plan: Indexed objects, searched for objects. Same results as before with much less code. Ran `bin/storage upgrade`, got a clean bill of health.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819
Differential Revision: https://secure.phabricator.com/D18559
Summary:
Ref T12819. Currently, under the Ferret engine, we query each application's index separately and then aggregate the results.
At the moment, results are aggregated by type first, then by actual rank. For example, all the revisions appear first, then all the tasks.
Instead, surface the internal ranking data from the underlying query and sort by it.
Test Plan: Searched for "A B" with a task named "A B" and a revision named "A". Saw task first. Broadly, saw mixed task and revision order in result sets.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819
Differential Revision: https://secure.phabricator.com/D18551
Summary:
Ref T12819. I started trying to get individual engines to drive these constraints (e.g., `ManiphestTaskQuery` can do most of the work) but this is a big pain, especially since most engines don't support "any owner" or "no owner", and not everything has an owner, and so on and so on. Going down this path would have meant a huge pile of stub functions everywhere, I think.
Instead, drive these through the main engine using the fulltext document table, which already has everything we need to apply these constraints in a uniform way.
Also tweak some parts of query construction and result ordering.
Test Plan: Searched for documents by author, owner, unowned, any owner, tags, subscribers, fulltext in global search. Got sensible results without any application-specific code.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819
Differential Revision: https://secure.phabricator.com/D18550
Summary:
Ref T12819. Provides a Ferret-engine-based fulltext engine to ultimately replace the InnoDB fulltext engine.
This is still pretty basic (hard-coded and buggy) but technically sort of works.
To activate this, you must explicitly configure it, so it isn't visible to users yet.
Test Plan: Searched for objects with global fulltext search, got a mixture of matching revisions and tasks back.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819
Differential Revision: https://secure.phabricator.com/D18548
Summary:
Fixes T12118. See PHI54. This adds a special case for the initial "reviewers" transactions, similar to the existing special case for "projects" transactions.
Although these transactions are redudnant in the web view since you can see the information clearly on the page, they're more reasonably useful in mail.
Test Plan: {F5168838}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12118
Differential Revision: https://secure.phabricator.com/D18542
Summary: Create a diff page, new UI
Test Plan: Create a diff from page
Reviewers: epriestley
Reviewed By: epriestley
Spies: Korvin
Differential Revision: https://secure.phabricator.com/D18529
