Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.
When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.
Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.
These two roles make it possible to have any combination of:
* read-only
* write-only
* read-write
* disabled
This 'roles' mechanism is extensible to add new roles should that be needed in the future.
In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).
The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)
Hopefully this is useful in the upstream as well.
Remaining TODO:
* test cases
* documentation
Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.
Tested with elasticsearch versions 2.4 and 5.2 using the following config:
```lang=json
"cluster.search": [
{
"type": "elasticsearch",
"hosts": [
{
"host": "localhost",
"roles": { "read": true, "write": true }
}
],
"port": 9200,
"protocol": "http",
"path": "/phabricator",
"version": 5
},
{
"type": "mysql",
"roles": { "write": true }
}
]
Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Tags: #elasticsearch, #clusters, #wikimedia
Differential Revision: https://secure.phabricator.com/D17384
Summary:
Ref T12271. Don't do anything with this yet, but store who accepted/rejected/whatever on behalf of reviewers.
In the future, we could use this to render stuff like "Blessed Committers (accepted by epriestley)" or whatever. I don't know that this is necessarily super useful, but it's easy to track, seems likely to be useful, and would be a gigantic pain to backfill later if we decide we want it.
Test Plan: Accepted/rejected a revision, saw reviewers update appropriately.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12271
Differential Revision: https://secure.phabricator.com/D17537
Summary:
Ref T12271. Currenty, when you "Accept" a revision, you always accept it for all reviewers you have authority over.
There are some situations where communication can be more clear if users can accept as only themselves, or for only some packages, etc. T12271 discusses some of these use cases in more depth.
Instead of making "Accept" a blanket action, default it to doing what it does now but let the user uncheck reviewers.
In cases where project/package reviewers aren't in use, this doesn't change anything.
For now, "reject" still acts the old way (reject everything). We could make that use checkboxes too, but I'm not sure there's as much of a use case for it, and I generally want users who are blocking stuff to have more direct accountability in a product sense.
Test Plan:
- Accepted normally.
- Accepted a subset.
- Tried to accept none.
- Tried to accept bogus reviewers.
- Accepted with myself not a reviewer
- Accepted with only one reviewer (just got normal "this will be accepted" text).
{F4251255}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12271
Differential Revision: https://secure.phabricator.com/D17533
Summary: Fixes T11865. Part of a 'clean up remarkup' pass, removing Aleo helps simplify coding, is lighter on the wire, and gives a more consistent, clean look.
Test Plan: run celerity, grep for 'aleo' and 'Aleo', test Phriction, tasks
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T11865
Differential Revision: https://secure.phabricator.com/D17535
Summary:
Ref T10967. We still have double writes, so all reviewers are being written to both old and new storage. This migrates all the data in the old storage to the new storage, so both storage tables should have a complete set of data and be getting identical updates as we move forward.
After this, I can move readers over one at a time and eventually get rid of the old writes and old storage.
This loads all of the edge data into memory in a big chunk. I reached out to one install to get some more information about their data size. Ours is quite manageable and I think even large installs will probably fit into memory, but we can do this in chunks if not.
However, because the Edge table doesn't have an `id` column, we can't use either the `RawMigrationIterator` or the `MigrationIterator`, and would need to write a new `EdgeMigrationIterator`. This isn't tons of work but might not be necessary.
Test Plan: Ran the migration locally, spot-checked the results in the database for sanity and correctness.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10967
Differential Revision: https://secure.phabricator.com/D17515
Summary:
Ref T10967. We have a "commented" state to help reviewers get a better sense of who is part of a discussion, and a "last action" state to help distinguish between "accept" and "accepted an older version", for the purposes of sticky accepts and as a UI hint.
Currently, these are first-class states, partly beacuse we were somewhat limited in what we could do with edges. However, a more flexible way to represent them is as flags separate from the primary state flag.
In the new storage, write them as separate state information: `lastActionDiffPHID` stores the Diff PHID of the last review action (accept, reject, etc). `lastCommentDiffPHID` stores the Diff PHID of the last comment (top-level or inline).
Test Plan: Applied storage changes, commented and acted on a revision. Saw appropriate state reflected in the database.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10967
Differential Revision: https://secure.phabricator.com/D17514
Summary: Extends PHUIListItemView to take an icon, link as an "Action Item" that displays on the right side of the menu link. Does not display on Favorites. This allows for adding edit, external, or other links (documentation?) to any menu item. Right now the secondary link is only visible when the item is selected. This feels right, but if we offer it in other ways, users may always want it visible. We could look at making it onhover.
Test Plan:
Add a bunch of random global and personal dashboards to my menu. Add a menu to Favorites, see no link. Test mobile, link works.
{F4136699}
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17505
Summary:
Via HackerOne. When you view a raw file in Differential, we currently generate a permanent file with default permissions. This may be incorrect: default permissions may be broader than the diff's permissions.
The other three methods of downloading/viewing raw files ("Download" in Diffusion and Differential, "View Raw" in Diffusion and Differential) already apply policies correctly and generate temporary files. However, this workflow was missed when other workflows were updated.
Beyond updating the workflow, delete any files we've generated in the past. This wipes the slate clean on any security issues and frees up a little disk space.
Test Plan:
- Ran migration script, saw existing files get purged.
- Did "View Raw File", got a new file.
- Verified that the file was temporary and properly attached to the diff, with "NO ONE" permissions.
- Double-checked that Diffusion already runs policy logic correctly and applies appropriate policies.
- Double-checked that "Download Raw Diff" in Differential already runs policy logic correctly.
- Double-chekced that "Download Raw Diff" in Diffusion already runs policy logic correctly.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D17504
Summary: Ref T12264. This allows users to install a dashboard they are viewing to their personal home menu or as a global home menu item. Has some basic ability to be extended later for maybe projects.
Test Plan:
Build a dashboard, click "Install Dashboard".
- As user only get personal option
- As HomeApp edit person, see both options
- Try installation as either, with and without label set
- Fake "global" form as user, get error
- Don't set anything, get error
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12264
Differential Revision: https://secure.phabricator.com/D17492
Summary:
Ref T10967. This is an incremental step toward removing "reviewers" back to a dedicated storage table so we can handle changes like T11050.
This adds the storage table, and starts doing double writes to it (so new or updated reviewers write to both the old edge table and the new "reviewers" table).
Then we can do a migration, swap readers over one at a time, and eventually remove the old write and old storage and then implement new features.
This change has no user-facing impact, it just causes us to write new data to two places instead of one.
This is not completely exhaustive: the Herald "Add Reviewers" action is still doing a manual EDGE transaction. I'll clean that up next and do another pass to look for anything else I missed.
This is also a bit copy/pastey for now but the logic around "RESIGN" is a little different in the two cases until T11050. I'll unify it in future changes.
Test Plan:
- Did a no-op edit.
- Did a no-op comment.
- Added reviewers.
- Removed reviewers.
- Accepted and rejected revisions.
After all of these edits, did a `SELECT * FROM differential_reviewer` manually and saw consistent-looking rows in the database.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10967
Differential Revision: https://secure.phabricator.com/D17495
Summary: Fixes T12391. Adds better scoping to these rules to contain changes to just Conpherence.
Test Plan: Test Conpherence, Task comment, persistent chat on mobile / desktop.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12391
Differential Revision: https://secure.phabricator.com/D17496
Summary: This alignment is off on edit forms, from the new overflow rules. Let's re-align everything for forms too.
Test Plan:
Add 2 new spaces, go to Maniphest, edit a task, see proper alignment of [Space] [Policy] view dropdowns. Check mobile alignment and Safari/FF.
{F3942187}
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17491
Summary: Fix tag alignment on project cards when there are multiple tags. Also fixes T12381.
Test Plan: Review a project and people hovercard in sandbox, ensure multiple tags look as expected.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12381
Differential Revision: https://secure.phabricator.com/D17488
Summary: Fixes T12326. Users can make really log tag titles, this forces ellipsis if it is too long.
Test Plan: Write a super long tag, see ellipisis. Test a small tag, see normal layout.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12326
Differential Revision: https://secure.phabricator.com/D17486
Summary: Fixes T12370. Moves this to be absolutely positions so the float doesn't mess up text-overflow layouts.
Test Plan: Chrome, Safari, Firefox, mobile and desktop layouts of Maniphest submenus.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12370
Differential Revision: https://secure.phabricator.com/D17484
Summary: The Safari hack in place casued a truncation issue in Firefox, so that hack is now gone. Instead the bug appears to be the creative inclusion of "space". In fiddling with this adding one space inside the span and one space outside the span seems to resolve all cases.
Test Plan: Chrome, Safari, Firefox. Test "hector" and copy paste of a Task ID.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17483
Summary: Fixes T12367. CSS here already truncates (or should have been) and is generally more effective. Remove the unneeded server side truncation. Any other UI place these render?
Test Plan: Set Policy to a group name of "Stanford University: Alumni Association and Friends" and see better truncation.
Reviewers: epriestley, eliaspro
Reviewed By: epriestley, eliaspro
Subscribers: eliaspro, Korvin
Maniphest Tasks: T12367
Differential Revision: https://secure.phabricator.com/D17479
Summary: Ref D17361. This sets a reasonable width on crumbs just in case a title is super long. Also fixes a weird Safari issue.
Test Plan:
Set a username to "hector" and check Safari. Create a badge named "MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM" and test length.
{F3771744}
{F3771747}
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17472
Summary: Fixes T12357. Adds some color to highlighted text in headers.
Test Plan: == Header with **strong** copy ==
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12357
Differential Revision: https://secure.phabricator.com/D17469
Summary: Ref T10319. Adds in database columns for upcoming default generated avatar support.
Test Plan: Ran storage upgrade, log into local site to verify it didn't blow up.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T10319
Differential Revision: https://secure.phabricator.com/D17459
Summary: Ref T10319. This swaps the default in the Picture Chooser to allow picking of the custom unique avatar. We're currently going with 100k unique possibilities. The logic roughly hashes a user name and picks an image pack, color, and border. Based on that, we select the first character of their username, or fall back to Psyduck if not [a-z][0-9].
Test Plan:
Set the following usernames from ProfilePicture as a test: chad, epriestley, sally, 007, _cat_, -doggie-.
{F3453979}
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T10319
Differential Revision: https://secure.phabricator.com/D17430
Summary: Doesn't seem popular, will rethink dashboard editing again in the future at some point.
Test Plan: Review a dashboard, edit, install.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17450
Summary:
Ref T12314. Ref T6064. Ref T11580. If an install defines several different task create forms (like "Create Plant" and "Create Animal"), allow any of them to be created directly onto a workboard column.
This is just a general consistency improvement that makes Custom Forms and Workboards work together a bit better. We might do something fancier eventually for T6064 (which wants fewer clicks) and/or T11580 (which wants per-workboard control over forms or defaults).
Test Plan:
- Created several different types of tasks directly onto a workboard.
- Faked just one create form, saw the UI unchanged (except that it respects any renaming).
{F3492928}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12314, T11580, T6064
Differential Revision: https://secure.phabricator.com/D17446
Summary: Ref T12314. Provides a field on tasks for storing subtypes. Does nothing interesting yet.
Test Plan:
- Ran storage upgrade.
- Created some tasks.
- Looked in the database.
- Used Conduit to query some tasks.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12314
Differential Revision: https://secure.phabricator.com/D17441
Summary:
Ref T12314. This adds storage so EditEngine forms can later be marked as edit fields for particular types of objects (like an "animal edit form" vs a "plant edit form").
We'll take you to the right edit form when you click "Edit" by selecting among forms with the same subtype as the task.
This doesn't do anything very interesting on its own.
Test Plan:
- Ran `bin/storage upgrade`.
- Verified database got the field with proper values.
- Created a new form, checked the database.
- Ran unit tests.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12314
Differential Revision: https://secure.phabricator.com/D17439
Summary: Ref T10319. This builds out a reasonably decent avatar generator. 256 colors x 74 images x 2 borders, 38k options. Not completely sure though how names disburse though, so likely half that number. I can add lowercase lettering to double the footprint if needed though.
Test Plan:
UIExamples. Color generator here: http://tools.medialab.sciences-po.fr/iwanthue/
{F3416622}
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T10319
Differential Revision: https://secure.phabricator.com/D17418
Summary: Looks nicer on profiles, cards. Added some additional colors.
Test Plan: change my avatar a few times
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: avivey, Korvin
Differential Revision: https://secure.phabricator.com/D17416
Summary: Ref T12270. Moves badges into their own page and menu item. Capable of displaying hundreds of useful tokens of appreciation and dedication.
Test Plan:
Test blank state, mobile, awards badges.
{F3284139}
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12270
Differential Revision: https://secure.phabricator.com/D17410
Summary:
Fixes T12172. Fixes T12060. This allows runtime code building CSS for mail to read CSS variables, then makes all the code do that.
It reverts the non-colorblind red/green to the colors in use before T12060, which seem better for non-colorblind users since no one really complained?
Test Plan:
- Viewed code diffs in Web UI.
- Viewed prose diffs in Web UI.
- Viewed code diffs in email.
- Viewed prose diffs in email.
All modes respected the accessibility color scheme.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12172, T12060
Differential Revision: https://secure.phabricator.com/D17269
Summary:
See downstream <https://phabricator.kde.org/T5404>. This code was doing some `.firstChild` shenanigans which didn't survive some UI refactoring.
This whole UI is a little iffy but just unbreak it for now.
Test Plan: Allowed and rejected desktop notifications, got largely reasonable UI rendering.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D17388
Summary:
Ref T12287. See D17361. That fixed a mostly-theoretical bug with crumbs named things like "MMMMMMMMMMMMMMMMMMMM", but caused a less-theoretical buggy side effect in Safari.
For now, just keep the "MMMMMMMM" crumbs around since that's the easiest/least-bad/safest fix prior to the release cut. We can fix this more broadly when we have more time to look at it.
Test Plan: Looked at profiles, saw the entire name for "hector" and the too-long-crumb for "MMMMM".
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12287
Differential Revision: https://secure.phabricator.com/D17382
Summary: Ref T12270. Any project, badge, dashboard, etc, that uses names in crumbs can over generate a long title. Restrict to a sane but generous width.
Test Plan: Make a project with a really long name, test various crumb layouts, boards, tasks, desktop, mobile.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12270
Differential Revision: https://secure.phabricator.com/D17361
Summary: Fixes T12248. Adds a flag for movable panels, and only allows those to be moved. Also cleaned up some CSS rules missing once a panel was drug into a new position.
Test Plan: Try to drag a tab panel content pane, cannot. Drag normal pane, see CSS, grab and drag same panel back, CSS looks the same.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12248
Differential Revision: https://secure.phabricator.com/D17356
Summary: Bump up the CSS scope, since we altered the normal rule for device-desktop
Test Plan: /diviner/ in sandbox
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17352
Summary: Aligns the search result list to the right of the input box, adds a little space, removes a double border.
Test Plan: Search on desktop, mobile, tablet breakpoints. Double check normal typeahead results.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17345
Summary: Fixes T12213. Removes truncation and allows titles to be full width if needed.
Test Plan:
Chrome / Firefox / Safari on Mac, mobile and desktop widths.
{F2754679}
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12213
Differential Revision: https://secure.phabricator.com/D17336
Summary: Ref T12232. I can't reproduce the original issue, but this should probably fix it without side effects?
Test Plan: Added a card with Stripe, but I could do that before too.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12232
Differential Revision: https://secure.phabricator.com/D17333
Summary: Ref T10390. This mostly shuffles layout into "View" and keepts "Manage" around for Edit/Copy/History. This feels better to me overall. Also tweaked some spacing and color.
Test Plan:
New Dashboard, edit Dashboard, shuffle panels. Create new panels.
{F2684043}
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T10390
Differential Revision: https://secure.phabricator.com/D17326
Summary: These colors are also off from the icon change.
Test Plan: Project -> Manage -> Edit Picture -> Choose Icon
Reviewers: epriestley, 20after4
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17328
Summary: Moves profile/project to use more standard colored boxes. Reverts dashboard border colors. Ensures better High-Contrast application more consistently across these projects. Also fix T12211.
Test Plan: Home, People, Projects in High Contrast / Standard
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12211
Differential Revision: https://secure.phabricator.com/D17321
Summary: These just cut off currently if the menu item name is too wide (like mobile). Add some ellipsis
Test Plan: Check long name on mobile action list menu
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17319
Summary: Lots of little details, fix workboard bg colors, darken up global backgrounds just a hair, add more "widgety" look to dashboard panels, remove underline on anchors on mobile. Also Fixes T12210
Test Plan: Use lots of pages on mobile and desktop.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12210
Differential Revision: https://secure.phabricator.com/D17315
Summary: Fixes T12205. These got over-ridden at a global scale (correctly) and need to adjust local scopes better. Also make it more bluer.
Test Plan: Go to Edit Dashboard, and pick a new icon for a Dashboard.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12205
Differential Revision: https://secure.phabricator.com/D17312
