Summary: See discussion in D19415.
Test Plan: Searched for some owners, found tasks as expected.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D19417
Summary:
Ref T13120. See PHI571. Fixes T5024. This adds a "View as Query" action to workboard columns, which builds a query in Maniphest that has the current query constraints plus an additional constraint to select only tasks in the specified column.
This is a normal query and can be turned into a dashboard panel, added to a menu, edited, saved as a link, etc.
Much of the complexity here is that finding tasks in a given column isn't entirely straightforward because of how board layout works: when you create a task, it isn't immediately placed in columns. It's only actually added to the "Backlog" column on any boards when someone looks at the board.
To get the right behavior, we must do "board layout" for any queried columns before we can constrain results. This isn't enormously efficient, but should be OK for reasonable boards.
Test Plan:
- Used "View as Query" for normal columns and milestome columns, got appropriate queries in Maniphest.
- Applied filters to the board (e.g., "Priorities: wishlist"), then used "View As Query" and had my custom filters respected.
- Queried some large boards/columns with more than a thousand tasks, got results back within a second or so.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13120, T5024
Differential Revision: https://secure.phabricator.com/D19366
Summary: Depends on D19038. Fixes T4434. Updates the SearchEngine and Query to handle these fields.
Test Plan: Filtered and ordered by date and closer.
Maniphest Tasks: T4434
Differential Revision: https://secure.phabricator.com/D19039
Summary:
Depends on D19037. Ref T4434. Adds closed date to `maniphest.search` and "Export Data".
When a task has been closed, show the closed date with a checkmark in the UI instead of the modified date.
Test Plan:
- Exported data to CSV, saw close information.
- Saw close information in `/maniphest/`.
- Queried for close information via `maniphest.search`.
Maniphest Tasks: T4434
Differential Revision: https://secure.phabricator.com/D19038
Summary:
Depends on D18959. Ref T13049. Provide tags, subscribers, spaces, and created/modified as automatic extensions for all objects which support them.
(Also, for JSON export, be a little more consistent about exporting `null` instead of empty string when there's no value in a text field.)
Test Plan: Exported users and tasks, saw relevant fields in the export.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13049
Differential Revision: https://secure.phabricator.com/D18960
Summary: Depends on D18958. Ref T13049. Support the new stuff. There are a couple more fields this needs to strictly improve on the old export, but I'll add them as extensions shortly.
Test Plan: Exported tasks to Excel, saw reasonble-looking data in the export.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13049
Differential Revision: https://secure.phabricator.com/D18959
Summary:
Ref T12819. Obsoleted by the Ferret engine, which is unprototyping shortly.
This breaks compatibility in two ways:
- `maniphest.query` no longer supports "fullText" (now throws an explicit exception).
- Existing saved searches with a "Contains Words" constraint will no longer have that constraint.
It seems unlikely (?) that either of these are seeing too much use, and they should be easy to fix. I'll note them in the changelog.
Test Plan: Viewed Maniphest, no more "Contains Words" field. Called `maniphest.query` with "fullText", got explicit exception.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819
Differential Revision: https://secure.phabricator.com/D18586
Summary:
Ref T12819. Currently, under the Ferret engine, we query each application's index separately and then aggregate the results.
At the moment, results are aggregated by type first, then by actual rank. For example, all the revisions appear first, then all the tasks.
Instead, surface the internal ranking data from the underlying query and sort by it.
Test Plan: Searched for "A B" with a task named "A B" and a revision named "A". Saw task first. Broadly, saw mixed task and revision order in result sets.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819
Differential Revision: https://secure.phabricator.com/D18551
Summary: Ref T12819. Uses an extension rather than hard-coding support into Maniphest.
Test Plan: Saw "Query" field appear in Differential, which also implements the interface and has support. Used field in both applications.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819
Differential Revision: https://secure.phabricator.com/D18547
Summary:
Ref T12819. Distinguishes between "term" queries and "substring" queries, and tries to match them correctly most of the time. For example:
- `example` matches "example", obviously.
- `~amp` matches "example", but `amp` does not.
- `examples` matches "example" through stemming.
- `"examples"` does not match "example" (quoted text does not stem).
- `"an examp"` does not match "an example" (quoted text is still term text).
- `~"an examp"` matches "an example" (quoted, substring-operator text uses substring search).
Test Plan: Ran searches similar to the above, they seemed to do what they should.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819
Differential Revision: https://secure.phabricator.com/D18500
Summary:
Ref T12819. Depends on D18492. Instead of passing a raw query into the Query layer, parse it first.
This allows the query layer to figure out which parts should be substring vs term match, and would allow the SearchEngine layer to do `author:...` eventually by picking it out before sending it to the Ferret engine.
Test Plan: Ran some Ferret queries. They work like before, except that nonsense like `-+"quack"` raises an exception now.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819
Differential Revision: https://secure.phabricator.com/D18499
Summary:
Ref T12819. I gave this stuff a sweet code name because all the terms related to "fulltext" and "search" already mean 5 different things. It, uh, ferrets out documents for you?
I'm building this to work a lot like the existing ngram index, which seems to work pretty well. If this sticks, it will auto-resolve the join issue (in T12443) by letting us do the entire thing locally in a JOIN and thus dodge a lot of mess.
This index gets built alongside other indexes, but only shows up in the UI if you have prototypes enabled. If you do, it appears under the existing fulltext field in Maniphest. No existing functionality is affected or disrupted.
NOTE: The query engine half of this is still EXTREMELY primitive, and this probably performs worse than the existing field for now. If this doesn't show obvious signs of being awful on `secure` I'll improve that in followup changes.
Test Plan:
Indexed my tasks, ran some simple queries, got the results I wanted, even for queries "ko", "k", "v0.1".
{F5147746}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12819, T12443
Differential Revision: https://secure.phabricator.com/D18484
Summary:
See PHI42. Currently, `maniphest.search` incorrectly applies this default (group by priority) to all queries via Conduit.
The correct behavior is to apply no grouping constraint.
I think this is also a reasonable general behavior, and the current code seems to date from D6960 in 2013 and didn't seem particularly carefully considered.
This is a minor compatibility break -- saved queries which are more than 4 years old might change their group behavior. I'll note this in the change logs but expect essentially no one to be affected.
Test Plan: Ran a `maniphest.search` Conduit call and observed the underlying query. Before this change, it executed `ORDER BY priority, id`. After this change, it correctly executed `ORDER BY id` only.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D18459
Summary:
Ref T7664. The current algorithm for moving task subpriorities can end up stuck in a real sticky swamp in some unusual situations.
Instead, use an algorithm which works like this:
- When we notice two tasks are too close together, look at the area around those tasks (just a few paces).
- If things look pretty empty, we can just spread the tasks out a little bit.
- But, if things are still real crowded, take another look further.
- Keep doing that until we're looking at a real nice big spot which doesn't have too many tasks in it in total, even if they're all in one place right now.
- Then, move 'em out!
Also:
- Just swallow our pride and do the gross `INSERT INTO ... "", "", "", "", "", "", ... ON DUPLICATE KEY UPDATE` to bulk update.
- Fix an issue where a single move could cause two different subpriority recalculations.
Test Plan:
- Changed `ManiphesTaskTestCase->testTaskAdjacentBlocks()` to insert 1,000 tasks with identical subpriorities, saw them spread out in 11 queries instead of >1,000.
- Dragged tons of tasks around on workboards.
- Ran unit tests.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T7664
Differential Revision: https://secure.phabricator.com/D17959
Summary: Ref T12450. These are like 95% my fault, but Elastic appears to spell the name "Elasticsearch" consistently in their branding.
Test Plan: `grep ElasticSearch`
Reviewers: chad, 20after4
Maniphest Tasks: T12450
Differential Revision: https://secure.phabricator.com/D17601
Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.
When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.
Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.
These two roles make it possible to have any combination of:
* read-only
* write-only
* read-write
* disabled
This 'roles' mechanism is extensible to add new roles should that be needed in the future.
In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).
The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)
Hopefully this is useful in the upstream as well.
Remaining TODO:
* test cases
* documentation
Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.
Tested with elasticsearch versions 2.4 and 5.2 using the following config:
```lang=json
"cluster.search": [
{
"type": "elasticsearch",
"hosts": [
{
"host": "localhost",
"roles": { "read": true, "write": true }
}
],
"port": 9200,
"protocol": "http",
"path": "/phabricator",
"version": 5
},
{
"type": "mysql",
"roles": { "write": true }
}
]
Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Tags: #elasticsearch, #clusters, #wikimedia
Differential Revision: https://secure.phabricator.com/D17384
Summary:
Ref T12314. Allow tasks to be queried by subtype using a typeahead.
Open to a better default icon. I'll probably let you configure them later.
Just hide this constraint if there's only one subtype.
Test Plan:
- Searched for subtypes.
- Verified that the control hides if there is only one subtype.
{F3492293}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12314
Differential Revision: https://secure.phabricator.com/D17444
Summary: Ref T12314. Provides a field on tasks for storing subtypes. Does nothing interesting yet.
Test Plan:
- Ran storage upgrade.
- Created some tasks.
- Looked in the database.
- Used Conduit to query some tasks.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12314
Differential Revision: https://secure.phabricator.com/D17441
Summary: Ref T10390. Simplifies dropdown by rolling out canUseInPanel in useless panels
Test Plan: Add a query panel, see less options.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T10390
Differential Revision: https://secure.phabricator.com/D17341
Summary:
Fixes T11812.
- Pull the logic for building the "Create Whatever" dropdown out.
- Use it to generate NUX buttons, too.
- Use the new logic in Paste and Maniphest.
Test Plan:
- Viewed Paste NUX, button worked.
- Viewed Maniphest NUX with multiple create forms, button worked.
Reviewers: chad, avivey
Reviewed By: avivey
Subscribers: avivey
Maniphest Tasks: T11812
Differential Revision: https://secure.phabricator.com/D16797
Summary:
Ref T10747. Rough flow is:
- Run a query.
- Select a new "Export Events..." action.
- This lets you define an "Export", which has a unique URL you can paste into Google Calendar or Calendar.app or whatever.
Most of this does nothing yet but here's the boilerplate.
Test Plan: Doesn't do anything yet.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10747
Differential Revision: https://secure.phabricator.com/D16675
Summary:
Ref T8126. Ref T4788. This adds a way to query by parent or subtask.
I plan to link to this from the task graph (e.g., {nav View > Search Subtasks} or similar, in a dropdown on the "Task Graph" element) as a way to let us bail out if tasks have 300 subtasks and send the user to a big query result list. That'll give us more flexibility to tailor the UI for reasonable numbers of tasks.
There's no UI for this unless you specify a query yourself, so the only ways to get to it are:
- Manually put `?parentIDs=...` into the URI.
- Use the API.
- Future link from task graphs.
It doesn't seem too useful to me on its own, outside of the context of links from tasks.
Test Plan:
- Manually put `?parentIDs=...` and `?subtaskIDs=...` into Maniphest query UI, got expected results.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4788, T8126
Differential Revision: https://secure.phabricator.com/D16341
Summary:
Ref T8126. See that task for discussion. This change:
- Updates language to be more consistent ("Parents", "Subtasks") since I moved us away from the often-confusing "Block" language in T4788.
- Fixes bugs with finding the wrong set of tasks if tasks have a mixture of open and closed subtasks or parents.
Test Plan:
- Created four tasks: no subtasks, one closed subtask, one open subtask, mixture of open and closed subtasks.
- Created four more tasks: no parents, one closed parent, one open parent, mixture of open and closed parents.
- Searched for all this stuff, got the proper results:
{F1740683}
{F1740684}
{F1740685}
{F1740686}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T8126
Differential Revision: https://secure.phabricator.com/D16340
Summary:
Ref T10939. Ref T4144. This moves the revision SearchEngine to modern code so I can add some kind of bucketing layer on top of it.
This seems to have worked pretty cleanly. One thing is that I removed the ability to search for "pending drafts":
- This was added in D1927 from a bootcamp task, was an indirect solution to a questionable problem, and almost certainly would not meet the bar today.
- Later, in D3324, we added the icons to the list. I think this is a better solution in general. In particular, it specifically addressed the query being kind of junky.
- At the time, Differential had a prebuilt "Drafts" filter. This was removed in D6347 with the move to ApplicationSearch, which simplified the large number of prebuilt filters. Although we got a lot of feedback about that, none requested that the drafts filter be restored.
Test Plan: Searched for responsible users, subscribers, orders, projects, repositories.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4144, T10939
Differential Revision: https://secure.phabricator.com/D15921
Summary:
Ref T10537. These are objects which are bound to some external object, like a Maniphest task which is a representation of a GitHub issue.
This doesn't do much yet and may change, but my thinking is:
- I'm putting these on-object instead of on edges because I think we want to actively change the UI for them (e.g., clearly call out that the object is bridged) but don't want every page to need to do extra queries in the common case where zero bridged objects exist anywhere in the system.
- I'm making these one-to-one, more or less: an issue can't be bridged to a bunch of tasks, nor can a bunch of tasks be bridged to a single issue. Pretty sure this makes sense? I can't come up with any reasonable, realistic cases where you want a single GitHub issue to publish to multiple different tasks in Maniphest.
- Technically, one type of each bridgable object could be bridged, but I expect this to never actually occur. Hopefully.
Test Plan: Ran storage upgrade, loaded some pages.
Reviewers: chad
Reviewed By: chad
Subscribers: Luke081515.2
Maniphest Tasks: T10537
Differential Revision: https://secure.phabricator.com/D15502
Summary:
Ref T10144. This isn't comprehensive, but we can give it a try and see how it feels?
- EditEngine forms now say "Tags" instead of "Projects".
- Modern SearchEngine forms now say "Tags" instead of "Projects".
- For clarity, replaced as much "in project" language as I could find with "tagged with project" language.
Test Plan: reading / grepping + used "not tagged with any project" token
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10144
Differential Revision: https://secure.phabricator.com/D15108
Summary:
Ref T9979. I picked this name long before the advent of modern "Engine" architecture and it ended up being pretty confusing.
Rename "SearchEngine" (currently: mysql or elasticsearch, used to store and query fulltext indexes) to "FulltextStorageEngine" to make it more clear what it does and disambituate it from ApplicationSearch, which also has a bunch of stuff called "SearchEngine", "SearchEngineExtension", etc.
Test Plan: Grepped for `phabricatorsearchengine`.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9979
Differential Revision: https://secure.phabricator.com/D14843
Summary: Adds basic NUX to Dashboards, Herald, Repositories, Maniphest. Note Herald and Dashboard Panels don't fine the nux for some reason, assume they will when modernized?
Test Plan: Read text, click buttons.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14844
Summary:
Ref T9964. I added several hacks to get these working. Clean them up and pull this into a proper extension.
The behavior in the web UI is:
- they work in all applications; but
- they only show up in the UI if a value is specified.
So if you visit `/view/?ids=1,2` you get the field, but normally it's not present. We could refine this later. I'm going to add documentation about how to prefill these forms regardless, which should make this discoverable by reading the documentation.
There's one teensey weensey hack: in the API, I push these fields to the top of the table. That one feels OK, since it's purely a convenience/display adjustment.
Test Plan: Queried by IDs, reviewed docs.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9964
Differential Revision: https://secure.phabricator.com/D14769
Summary: Ref T9964. I left a couple of these unsupported for now since they're weird in some way.
Test Plan: {F1024031}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9964
Differential Revision: https://secure.phabricator.com/D14767
Summary: Move some `PhabricatorSearchField` subclasses to be adjacent to the application to which they belong. This seems generally better to me than lumping them all together in the `src/applications/search/field/` directory. I was also wondering if it makes sense to rename these subclasses as `PhabricatorXSearchField` rather than `PhabricatorSearchXField` (as per T5655), but wasn't really sure if these objects are meant to be search-fields, or just fields belonging to the #search application.
Test Plan: N/A.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D13374
Summary: Ref T8637. This class has some really old parameter handling which can send `withIDs(array())` down a "fetch everything" pathway. Clean up most of it.
Test Plan: Issued every ApplicationSearch query.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8637
Differential Revision: https://secure.phabricator.com/D13390
Summary: Ref T8099. This adds a new class which all search engines return for layout. I thought about this a number of ways, and I think this is the cleanest path. Each Engine can return whatever UI bits they needs, and AppSearch or Dashboard picks and lays the bits out as needed. In the AppSearch case, interfaces like Notifications, Calendar, Legalpad all need more custom layouts. I think this also leaves a resonable path forward for NUX as well. Also, not sure I implemented the class correctly, but assume thats easy to fix?
Test Plan: Review and do a search in each application changed. Grep for all call sites.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T8099
Differential Revision: https://secure.phabricator.com/D13332
Summary: This should (hopefully) be the last one of these since D13185 has landed.
Test Plan: `arc unit`
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D13284
Summary:
Ref T8441. Ref T7715. Move Maniphest to SearchFields.
The only new tech here is hiding fields, which we use to hide some fields on the dashboard query UI.
Test Plan:
- Queried by each field, including custom fields.
- Used some standrad queries.
- Used dashboards, used standard + custom queries.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7715, T8441
Differential Revision: https://secure.phabricator.com/D13225
Summary: Ref T8441. Ref T7715. Ref T7909. Clean up all the ordering and grouping hacks in Maniphest so we can drive it through normal infrastructure, move it to SearchField, introduce Spaces, and eventually modernize the Conduit API.
Test Plan:
- Executed all grouping/ordering queries, including custom queries.
- Forced execution with old aliases; got modern results.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7909, T7715, T8441
Differential Revision: https://secure.phabricator.com/D13197
Summary:
Ref T8441. Ref T7715. Automatically generate a modern "Order" control in ApplicationSearch for engines which fully support SearchField.
Notably, this allows the standard "Order" control to automatically support custom field orders. We do this in Maniphest today, but in an ad-hoc way.
Test Plan: Performed order-by queries in Almanac (Services), Pholio, Files, People, Projects, and Paste.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7715, T8441
Differential Revision: https://secure.phabricator.com/D13193
Summary: Remove the `PhabricatorDefaultSearchEngineSelector` class. This is quite similar to D12053.
Test Plan: Went to `/view/PhabricatorSearchApplication/` and saw the storage engine configuration. Set `search.elastic.host` and saw the highlighted storage engine change.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12670
Summary: Use `__CLASS__` instead of hard-coding class names. Depends on D12605.
Test Plan: Eyeball it.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: hach-que, Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12806
Summary:
Parameter `type` is not used anywhere. Instead param `types` is used.
Due this bug search was performed over whole index instead on tasks only.
Test Plan:
- Setup phabricator to run with Elasticsearch.
- Open http://yourphab.com/conduit/method/maniphest.query/
- Fill `fullText` field only.
- Expect get results. Query should be performed on `/phabricator/TASK/` only (not on whole `/phabricator/` index).
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12542
Summary: Fixes T7904. Builtin queries won't set these to anything.
Test Plan: "Authored" builtin works again.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Maniphest Tasks: T7904
Differential Revision: https://secure.phabricator.com/D12539
Summary:
Ref T4100.
- Make it easy to choose all open or closed tasks.
- Make "special" tokenizers composable.
- Get `viewer()` generating documentation properly.
Test Plan:
- Ran queries with new tokens.
- Browsed new tokens.
- Viewed docs on new tokens.
- Used plain status tokens.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: chad, epriestley
Maniphest Tasks: T4100
Differential Revision: https://secure.phabricator.com/D12530
Summary:
Ref T4100.
This makes it slightly harder to choose, say, all priorities above X or all priorities except Y. We could add `open()`, `closed()`, `min()`, `max()`, and `not()` functions if there's a meaningful demand for them. I suspect some of these are maybe worthwhile while others aren't as worthwhile.
Test Plan: {F380058}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4100
Differential Revision: https://secure.phabricator.com/D12528
Summary: Ref T4100. Update these controls to allow functions like `viewer()`.
Test Plan: Used the new controls.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4100
Differential Revision: https://secure.phabricator.com/D12527
