1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-11 01:12:41 +01:00
phorge-phorge/src/applications
Mukunda Modell e41c25de50 Support multiple fulltext search clusters with 'cluster.search' config
Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.

When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.

Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.

These two roles make it possible to have any combination of:

* read-only
* write-only
* read-write
* disabled

This 'roles' mechanism is extensible to add new roles should that be needed in the future.

In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).

The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)

Hopefully this is useful in the upstream as well.

Remaining TODO:

* test cases
* documentation

Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.

Tested with elasticsearch versions 2.4 and 5.2 using the following config:

```lang=json
  "cluster.search": [
    {
      "type": "elasticsearch",
      "hosts": [
        {
          "host": "localhost",
          "roles": { "read": true, "write": true }
        }
      ],
      "port": 9200,
      "protocol": "http",
      "path": "/phabricator",
      "version": 5
    },
    {
      "type": "mysql",
      "roles": { "write": true }
     }
  ]

Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Tags: #elasticsearch, #clusters, #wikimedia

Differential Revision: https://secure.phabricator.com/D17384
2017-03-26 08:16:47 +00:00
..
almanac Cache Almanac URIs for repositories 2016-12-06 09:14:45 -08:00
aphlict Discard stdout/stderr from the aphlict subprocess when running in daemon (normal) mode 2016-11-13 16:43:42 -08:00
arcanist/conduit Remove remaining arcanist project code 2015-07-08 19:37:28 +10:00
audit Restrict Audit buckets to just ApplicationSearch views 2017-03-23 12:46:19 -07:00
auth Provide "bin/auth revoke" with a revoker for Conduit tokens 2017-03-03 14:38:55 -08:00
badges Funbeta Badges 2017-03-24 21:15:42 +00:00
base Fix errors found by PHPStan 2017-02-17 10:10:15 +00:00
cache With APCu 5+, use apcu_* function to examine cache state 2017-01-12 15:59:44 -08:00
calendar When destroying Calendar events, destroy invitees and notifications 2017-03-24 09:21:13 -07:00
celerity Add a "Red/Green Colorblind" accessibility mode, make all web UIs and email respect it 2017-02-23 10:57:39 -08:00
chatlog Remove empty implementations of describeAutomaticCapabilities() 2016-11-09 15:24:22 -08:00
conduit Provide a hint that submitting a Conduit call shows you how to encode particular parameters 2017-03-24 13:15:03 -07:00
config Support multiple fulltext search clusters with 'cluster.search' config 2017-03-26 08:16:47 +00:00
conpherence Reduce application search engine results list for Dashboards 2017-02-22 12:42:43 -08:00
console Remove "mysql.implementation" configuration 2016-11-22 04:15:34 -08:00
countdown Remove empty implementations of describeAutomaticCapabilities() 2016-11-09 15:24:22 -08:00
daemon Give Daemons a mobile menu 2017-03-24 09:19:56 -07:00
dashboard Provide a 'no dashboards' fallback state if you can't add any 2017-03-21 11:43:02 -07:00
differential Allow users to resign if they have authority over any reviewer 2017-03-24 13:14:47 -07:00
diffusion Rename "needReviewerStatus()" into "needReviewers()" 2017-03-20 16:46:16 -07:00
diviner Fix a constant typo in Diviner ("DECLARATAION" -> "TION") 2017-03-04 09:54:10 -08:00
doorkeeper Remove empty implementations of describeAutomaticCapabilities() 2016-11-09 15:24:22 -08:00
draft/storage Clean up some log spam caused by races in VersionedDraft 2016-09-05 13:01:53 -07:00
drydock Remove empty implementations of describeAutomaticCapabilities() 2016-11-09 15:24:22 -08:00
fact Update Facts for newPage 2016-04-03 15:07:52 -07:00
favorites Add some style to label in Favorites Menu 2017-02-01 07:20:31 -08:00
feed Prevent media from autoplaying when rendered as a feed story 2016-11-22 14:19:56 -08:00
files Fix some range issues and 32-bit issues with avatar generation 2017-03-23 10:51:33 -07:00
flag Remove counts from home navigation 2017-01-21 13:55:40 -08:00
fund Consistently refer to 'Projects' as 'Tags' 2016-04-19 16:48:21 +00:00
guides Add a bunch of Phacility-specific code to the upstream, thinly veiled as generic code 2016-11-15 09:11:22 -08:00
harbormaster Navigage Buildkite builds with more nuance 2017-01-31 17:26:45 -08:00
help Redesign header menus and search 2017-01-17 12:13:06 -08:00
herald Reduce application search engine results list for Dashboards 2017-02-22 12:42:43 -08:00
home Fix italics issue with nux state on homepage 2017-02-25 15:30:17 -08:00
legalpad Fix errors found by PHPStan 2017-02-17 10:10:15 +00:00
lipsum Add "--force" and "--quickly" flags to bin/lipsum 2017-02-27 09:09:41 -08:00
macro Fix display of current file on Macro creation 2017-02-03 17:10:02 -08:00
maniphest Support multiple fulltext search clusters with 'cluster.search' config 2017-03-26 08:16:47 +00:00
meta Redesign Home/Profile/Projects side navigation 2017-01-31 08:59:01 -08:00
metamta Allow "bin/mail send-test" to accept raw email addresses via "--to" 2017-03-10 14:52:33 -08:00
multimeter Add a cluster.read-only option 2016-04-09 13:40:47 -07:00
notification Add an icon to aphlict connection status 2016-10-02 08:17:21 -07:00
nuance Remove some no-op "canUninstall()" Application methods 2016-12-05 11:02:25 -08:00
oauthserver Reduce application search engine results list for Dashboards 2017-02-22 12:42:43 -08:00
owners Filter archived packages out of the "controlling packages" query earlier 2017-02-27 12:37:08 -08:00
packages Reduce application search engine results list for Dashboards 2017-02-22 12:42:43 -08:00
passphrase Make Passphrase "token" credentials accessible via the API 2016-11-15 09:12:35 -08:00
paste Get line count before truncating Paste snippets 2017-03-01 22:30:18 +00:00
people Paginate the profile badges view 2017-03-24 14:10:59 -07:00
phame Fix errors found by PHPStan 2017-02-17 10:10:15 +00:00
phid Fix errors found by PHPStan 2017-02-17 10:10:15 +00:00
phlux Remove empty implementations of describeAutomaticCapabilities() 2016-11-09 15:24:22 -08:00
pholio Make bin/lipsum generate hanldle generator keys and arguments more clearly 2017-02-27 09:09:28 -08:00
phortune Fix errors found by PHPStan 2017-02-18 09:24:56 +00:00
phpast Update phpast for new UI 2016-04-05 13:52:59 -07:00
phragment Fix errors found by PHPStan 2017-02-17 10:10:15 +00:00
phrequent Allow Phrequent to be used in dashboard panels 2017-03-06 11:00:55 -08:00
phriction Fix errors found by PHPStan 2017-02-17 10:10:15 +00:00
phurl Update Phurl for modular transactions 2017-02-24 08:30:47 -08:00
policy Make "Can Interact" and logged-out users interact more gracefully 2017-03-09 08:50:57 -08:00
ponder Fix errors found by PHPStan 2017-02-17 10:10:15 +00:00
project Support multiple fulltext search clusters with 'cluster.search' config 2017-03-26 08:16:47 +00:00
releeph Reduce application search engine results list for Dashboards 2017-02-22 12:42:43 -08:00
remarkup/conduit
repository Allow the PullLocal daemon to hibernate, and wake it when repositories need an update 2017-03-23 10:52:28 -07:00
search Support multiple fulltext search clusters with 'cluster.search' config 2017-03-26 08:16:47 +00:00
settings Fix bad JS rendering in "Allow Desktop Notifications" workflow 2017-02-20 12:55:34 -08:00
slowvote Fix errors found by PHPStan 2017-02-17 10:10:15 +00:00
spaces Fix errors found by PHPStan 2017-02-17 10:10:15 +00:00
subscriptions Allow task statuses to "lock" them, preventing additional comments and interactions 2017-03-02 16:57:10 -08:00
support/application
system Cache Almanac URIs for repositories 2016-12-06 09:14:45 -08:00
tokens Prevent awarding/revoking tokens when a task is locked 2017-03-04 09:55:35 -08:00
transactions Fix a rendering fatal for unknown edge constants 2017-03-24 16:58:48 -07:00
typeahead Drive CLI-based revision edits through "differential.revision.edit" API + EditEngine 2016-12-16 10:08:49 -08:00
uiexample Fix two typos ("Adminstrator", "Recipents") 2017-03-12 14:23:43 -07:00
xhprof Allow XHProf profiles to be drag-and-dropped to upload them 2017-02-23 11:16:19 -08:00