1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-23 05:50:55 +01:00
phorge-phorge/resources
Mukunda Modell e41c25de50 Support multiple fulltext search clusters with 'cluster.search' config
Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.

When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.

Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.

These two roles make it possible to have any combination of:

* read-only
* write-only
* read-write
* disabled

This 'roles' mechanism is extensible to add new roles should that be needed in the future.

In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).

The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)

Hopefully this is useful in the upstream as well.

Remaining TODO:

* test cases
* documentation

Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.

Tested with elasticsearch versions 2.4 and 5.2 using the following config:

```lang=json
  "cluster.search": [
    {
      "type": "elasticsearch",
      "hosts": [
        {
          "host": "localhost",
          "roles": { "read": true, "write": true }
        }
      ],
      "port": 9200,
      "protocol": "http",
      "path": "/phabricator",
      "version": 5
    },
    {
      "type": "mysql",
      "roles": { "write": true }
     }
  ]

Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Tags: #elasticsearch, #clusters, #wikimedia

Differential Revision: https://secure.phabricator.com/D17384
2017-03-26 08:16:47 +00:00
..
builtin Allow custom image generation when choosing a profile image 2017-03-03 20:21:31 -08:00
celerity When accepting revisions, allow users to accept on behalf of a subset of reviewers 2017-03-22 14:25:04 -07:00
chatbot Updated the docs so chatbots can use the Conduit API 2016-08-24 19:05:30 -04:00
cows Use PHP implementation of Cowsay for cowsay rule 2015-09-13 12:27:30 -07:00
emoji Add some sort of sort to Emoji Autocomplete 2017-01-24 20:21:06 -08:00
figlet/custom Include "Figlet" and PEAR "Text_Figlet" in externals 2015-09-13 12:30:48 -07:00
font Made Meme Generator 2013-01-19 18:43:43 -08:00
sprite Remove the Persona login method 2016-12-05 15:57:15 -08:00
sql Support multiple fulltext search clusters with 'cluster.search' config 2017-03-26 08:16:47 +00:00
sshd Improve Diffusion hosting setup instructions somewhat? 2016-04-26 10:06:59 -07:00