Add `cluster.addresses` and require membership before accepting cluster authentication tokens
Summary:
Ref T2783. Ref T6706.
- Add `cluster.addresses`. This is a whitelist of CIDR blocks which define cluster hosts.
- When we recieve a request that has a cluster-based authentication token, require the cluster to be configured and require the remote address to be a cluster member before we accept it.
- This provides a general layer of security for these mechanisms.
- In particular, it means they do not work by default on unconfigured hosts.
- When cluster addresses are configured, and we receive a request //to// an address not on the list, reject it.
- This provides a general layer of security for getting the Ops side of cluster configuration correct.
- If cluster nodes have public IPs and are listening on them, we'll reject requests.
- Basically, this means that any requests which bypass the LB get rejected.
Test Plan:
- With addresses not configured, tried to make requests; rejected for using a cluster auth mechanism.
- With addresses configred wrong, tried to make requests; rejected for sending from (or to) an address outside of the cluster.
- With addresses configured correctly, made valid requests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6706, T2783
Differential Revision: https://secure.phabricator.com/D11159
2015-01-03 00:13:41 +01:00
|
|
|
<?php
|
|
|
|
|
|
|
|
final class PhabricatorClusterConfigOptions
|
|
|
|
extends PhabricatorApplicationConfigOptions {
|
|
|
|
|
|
|
|
public function getName() {
|
|
|
|
return pht('Cluster Setup');
|
|
|
|
}
|
|
|
|
|
|
|
|
public function getDescription() {
|
|
|
|
return pht('Configure Phabricator to run on a cluster of hosts.');
|
|
|
|
}
|
|
|
|
|
2016-01-28 17:40:22 +01:00
|
|
|
public function getIcon() {
|
2015-02-02 19:17:25 +01:00
|
|
|
return 'fa-sitemap';
|
|
|
|
}
|
|
|
|
|
2015-02-09 22:10:56 +01:00
|
|
|
public function getGroup() {
|
|
|
|
return 'core';
|
|
|
|
}
|
|
|
|
|
Add `cluster.addresses` and require membership before accepting cluster authentication tokens
Summary:
Ref T2783. Ref T6706.
- Add `cluster.addresses`. This is a whitelist of CIDR blocks which define cluster hosts.
- When we recieve a request that has a cluster-based authentication token, require the cluster to be configured and require the remote address to be a cluster member before we accept it.
- This provides a general layer of security for these mechanisms.
- In particular, it means they do not work by default on unconfigured hosts.
- When cluster addresses are configured, and we receive a request //to// an address not on the list, reject it.
- This provides a general layer of security for getting the Ops side of cluster configuration correct.
- If cluster nodes have public IPs and are listening on them, we'll reject requests.
- Basically, this means that any requests which bypass the LB get rejected.
Test Plan:
- With addresses not configured, tried to make requests; rejected for using a cluster auth mechanism.
- With addresses configred wrong, tried to make requests; rejected for sending from (or to) an address outside of the cluster.
- With addresses configured correctly, made valid requests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6706, T2783
Differential Revision: https://secure.phabricator.com/D11159
2015-01-03 00:13:41 +01:00
|
|
|
public function getOptions() {
|
2016-04-09 14:41:08 +02:00
|
|
|
$databases_type = 'custom:PhabricatorClusterDatabasesConfigOptionType';
|
|
|
|
$databases_help = $this->deformat(pht(<<<EOTEXT
|
|
|
|
WARNING: This is a prototype option and the description below is currently pure
|
|
|
|
fantasy.
|
|
|
|
|
|
|
|
This option allows you to make Phabricator aware of database read replicas so
|
|
|
|
it can monitor database health, spread load, and degrade gracefully to
|
|
|
|
read-only mode in the event of a failure on the primary host. For help with
|
|
|
|
configuring cluster databases, see **[[ %s | %s ]]** in the documentation.
|
|
|
|
EOTEXT
|
|
|
|
,
|
|
|
|
PhabricatorEnv::getDoclink('Cluster: Databases'),
|
|
|
|
pht('Cluster: Databases')));
|
|
|
|
|
2016-04-13 14:52:15 +02:00
|
|
|
|
|
|
|
$intro_href = PhabricatorEnv::getDoclink('Clustering Introduction');
|
|
|
|
$intro_name = pht('Clustering Introduction');
|
|
|
|
|
Support multiple fulltext search clusters with 'cluster.search' config
Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.
When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.
Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.
These two roles make it possible to have any combination of:
* read-only
* write-only
* read-write
* disabled
This 'roles' mechanism is extensible to add new roles should that be needed in the future.
In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).
The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)
Hopefully this is useful in the upstream as well.
Remaining TODO:
* test cases
* documentation
Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.
Tested with elasticsearch versions 2.4 and 5.2 using the following config:
```lang=json
"cluster.search": [
{
"type": "elasticsearch",
"hosts": [
{
"host": "localhost",
"roles": { "read": true, "write": true }
}
],
"port": 9200,
"protocol": "http",
"path": "/phabricator",
"version": 5
},
{
"type": "mysql",
"roles": { "write": true }
}
]
Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Tags: #elasticsearch, #clusters, #wikimedia
Differential Revision: https://secure.phabricator.com/D17384
2017-03-26 10:16:47 +02:00
|
|
|
$search_type = 'custom:PhabricatorClusterSearchConfigOptionType';
|
|
|
|
$search_help = $this->deformat(pht(<<<EOTEXT
|
|
|
|
Define one or more fulltext storage services. Here you can configure which
|
|
|
|
hosts will handle fulltext search queries and indexing. For help with
|
|
|
|
configuring fulltext search clusters, see **[[ %s | %s ]]** in the
|
|
|
|
documentation.
|
|
|
|
EOTEXT
|
|
|
|
,
|
|
|
|
PhabricatorEnv::getDoclink('Cluster: Search'),
|
|
|
|
pht('Cluster: Search')));
|
|
|
|
|
Add `cluster.addresses` and require membership before accepting cluster authentication tokens
Summary:
Ref T2783. Ref T6706.
- Add `cluster.addresses`. This is a whitelist of CIDR blocks which define cluster hosts.
- When we recieve a request that has a cluster-based authentication token, require the cluster to be configured and require the remote address to be a cluster member before we accept it.
- This provides a general layer of security for these mechanisms.
- In particular, it means they do not work by default on unconfigured hosts.
- When cluster addresses are configured, and we receive a request //to// an address not on the list, reject it.
- This provides a general layer of security for getting the Ops side of cluster configuration correct.
- If cluster nodes have public IPs and are listening on them, we'll reject requests.
- Basically, this means that any requests which bypass the LB get rejected.
Test Plan:
- With addresses not configured, tried to make requests; rejected for using a cluster auth mechanism.
- With addresses configred wrong, tried to make requests; rejected for sending from (or to) an address outside of the cluster.
- With addresses configured correctly, made valid requests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6706, T2783
Differential Revision: https://secure.phabricator.com/D11159
2015-01-03 00:13:41 +01:00
|
|
|
return array(
|
|
|
|
$this->newOption('cluster.addresses', 'list<string>', array())
|
|
|
|
->setLocked(true)
|
|
|
|
->setSummary(pht('Address ranges of cluster hosts.'))
|
|
|
|
->setDescription(
|
|
|
|
pht(
|
2016-04-13 14:52:15 +02:00
|
|
|
'Define a Phabricator cluster by providing a whitelist of host '.
|
|
|
|
'addresses that are part of the cluster.'.
|
Add `cluster.addresses` and require membership before accepting cluster authentication tokens
Summary:
Ref T2783. Ref T6706.
- Add `cluster.addresses`. This is a whitelist of CIDR blocks which define cluster hosts.
- When we recieve a request that has a cluster-based authentication token, require the cluster to be configured and require the remote address to be a cluster member before we accept it.
- This provides a general layer of security for these mechanisms.
- In particular, it means they do not work by default on unconfigured hosts.
- When cluster addresses are configured, and we receive a request //to// an address not on the list, reject it.
- This provides a general layer of security for getting the Ops side of cluster configuration correct.
- If cluster nodes have public IPs and are listening on them, we'll reject requests.
- Basically, this means that any requests which bypass the LB get rejected.
Test Plan:
- With addresses not configured, tried to make requests; rejected for using a cluster auth mechanism.
- With addresses configred wrong, tried to make requests; rejected for sending from (or to) an address outside of the cluster.
- With addresses configured correctly, made valid requests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6706, T2783
Differential Revision: https://secure.phabricator.com/D11159
2015-01-03 00:13:41 +01:00
|
|
|
"\n\n".
|
2016-04-13 14:52:15 +02:00
|
|
|
'Hosts on this whitelist have special powers. These hosts are '.
|
|
|
|
'permitted to bend security rules, and misconfiguring this list '.
|
|
|
|
'can make your install less secure. For more information, '.
|
|
|
|
'see **[[ %s | %s ]]**.'.
|
Add `cluster.addresses` and require membership before accepting cluster authentication tokens
Summary:
Ref T2783. Ref T6706.
- Add `cluster.addresses`. This is a whitelist of CIDR blocks which define cluster hosts.
- When we recieve a request that has a cluster-based authentication token, require the cluster to be configured and require the remote address to be a cluster member before we accept it.
- This provides a general layer of security for these mechanisms.
- In particular, it means they do not work by default on unconfigured hosts.
- When cluster addresses are configured, and we receive a request //to// an address not on the list, reject it.
- This provides a general layer of security for getting the Ops side of cluster configuration correct.
- If cluster nodes have public IPs and are listening on them, we'll reject requests.
- Basically, this means that any requests which bypass the LB get rejected.
Test Plan:
- With addresses not configured, tried to make requests; rejected for using a cluster auth mechanism.
- With addresses configred wrong, tried to make requests; rejected for sending from (or to) an address outside of the cluster.
- With addresses configured correctly, made valid requests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6706, T2783
Differential Revision: https://secure.phabricator.com/D11159
2015-01-03 00:13:41 +01:00
|
|
|
"\n\n".
|
|
|
|
'Define a list of CIDR blocks which whitelist all hosts in the '.
|
2016-04-13 14:52:15 +02:00
|
|
|
'cluster and no additional hosts. See the examples below for '.
|
|
|
|
'details.'.
|
Add `cluster.addresses` and require membership before accepting cluster authentication tokens
Summary:
Ref T2783. Ref T6706.
- Add `cluster.addresses`. This is a whitelist of CIDR blocks which define cluster hosts.
- When we recieve a request that has a cluster-based authentication token, require the cluster to be configured and require the remote address to be a cluster member before we accept it.
- This provides a general layer of security for these mechanisms.
- In particular, it means they do not work by default on unconfigured hosts.
- When cluster addresses are configured, and we receive a request //to// an address not on the list, reject it.
- This provides a general layer of security for getting the Ops side of cluster configuration correct.
- If cluster nodes have public IPs and are listening on them, we'll reject requests.
- Basically, this means that any requests which bypass the LB get rejected.
Test Plan:
- With addresses not configured, tried to make requests; rejected for using a cluster auth mechanism.
- With addresses configred wrong, tried to make requests; rejected for sending from (or to) an address outside of the cluster.
- With addresses configured correctly, made valid requests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6706, T2783
Differential Revision: https://secure.phabricator.com/D11159
2015-01-03 00:13:41 +01:00
|
|
|
"\n\n".
|
|
|
|
'When cluster addresses are defined, Phabricator hosts will also '.
|
2016-04-13 14:52:15 +02:00
|
|
|
'reject requests to interfaces which are not whitelisted.',
|
|
|
|
$intro_href,
|
|
|
|
$intro_name))
|
Add `cluster.addresses` and require membership before accepting cluster authentication tokens
Summary:
Ref T2783. Ref T6706.
- Add `cluster.addresses`. This is a whitelist of CIDR blocks which define cluster hosts.
- When we recieve a request that has a cluster-based authentication token, require the cluster to be configured and require the remote address to be a cluster member before we accept it.
- This provides a general layer of security for these mechanisms.
- In particular, it means they do not work by default on unconfigured hosts.
- When cluster addresses are configured, and we receive a request //to// an address not on the list, reject it.
- This provides a general layer of security for getting the Ops side of cluster configuration correct.
- If cluster nodes have public IPs and are listening on them, we'll reject requests.
- Basically, this means that any requests which bypass the LB get rejected.
Test Plan:
- With addresses not configured, tried to make requests; rejected for using a cluster auth mechanism.
- With addresses configred wrong, tried to make requests; rejected for sending from (or to) an address outside of the cluster.
- With addresses configured correctly, made valid requests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6706, T2783
Differential Revision: https://secure.phabricator.com/D11159
2015-01-03 00:13:41 +01:00
|
|
|
->addExample(
|
|
|
|
array(
|
|
|
|
'23.24.25.80/32',
|
|
|
|
'23.24.25.81/32',
|
|
|
|
),
|
|
|
|
pht('Whitelist Specific Addresses'))
|
|
|
|
->addExample(
|
|
|
|
array(
|
|
|
|
'1.2.3.0/24',
|
|
|
|
),
|
|
|
|
pht('Whitelist 1.2.3.*'))
|
|
|
|
->addExample(
|
|
|
|
array(
|
|
|
|
'1.2.0.0/16',
|
|
|
|
),
|
|
|
|
pht('Whitelist 1.2.*.*'))
|
|
|
|
->addExample(
|
|
|
|
array(
|
|
|
|
'0.0.0.0/0',
|
|
|
|
),
|
|
|
|
pht('Allow Any Host (Insecure!)')),
|
2015-01-27 23:51:48 +01:00
|
|
|
$this->newOption('cluster.instance', 'string', null)
|
|
|
|
->setLocked(true)
|
|
|
|
->setSummary(pht('Instance identifier for multi-tenant clusters.'))
|
|
|
|
->setDescription(
|
|
|
|
pht(
|
|
|
|
'WARNING: This is a very advanced option, and only useful for '.
|
|
|
|
'hosting providers running multi-tenant clusters.'.
|
|
|
|
"\n\n".
|
|
|
|
'If you provide an instance identifier here (normally by '.
|
2015-05-22 09:27:56 +02:00
|
|
|
'injecting it with a `%s`), Phabricator will pass it to '.
|
|
|
|
'subprocesses and commit hooks in the `%s` environmental variable.',
|
|
|
|
'PhabricatorConfigSiteSource',
|
|
|
|
'PHABRICATOR_INSTANCE')),
|
2016-04-09 00:04:05 +02:00
|
|
|
$this->newOption('cluster.read-only', 'bool', false)
|
|
|
|
->setLocked(true)
|
|
|
|
->setSummary(
|
|
|
|
pht(
|
|
|
|
'Activate read-only mode for maintenance or disaster recovery.'))
|
|
|
|
->setDescription(
|
|
|
|
pht(
|
|
|
|
'WARNING: This is a prototype option and the description below '.
|
|
|
|
'is currently pure fantasy.'.
|
|
|
|
"\n\n".
|
|
|
|
'Switch Phabricator to read-only mode. In this mode, users will '.
|
|
|
|
'be unable to write new data. Normally, the cluster degrades '.
|
|
|
|
'into this mode automatically when it detects that the database '.
|
|
|
|
'master is unreachable, but you can activate it manually in '.
|
|
|
|
'order to perform maintenance or test configuration.')),
|
2016-04-09 14:41:08 +02:00
|
|
|
$this->newOption('cluster.databases', $databases_type, array())
|
|
|
|
->setHidden(true)
|
|
|
|
->setSummary(
|
|
|
|
pht('Configure database read replicas.'))
|
|
|
|
->setDescription($databases_help),
|
Support multiple fulltext search clusters with 'cluster.search' config
Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.
When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.
Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.
These two roles make it possible to have any combination of:
* read-only
* write-only
* read-write
* disabled
This 'roles' mechanism is extensible to add new roles should that be needed in the future.
In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).
The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)
Hopefully this is useful in the upstream as well.
Remaining TODO:
* test cases
* documentation
Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.
Tested with elasticsearch versions 2.4 and 5.2 using the following config:
```lang=json
"cluster.search": [
{
"type": "elasticsearch",
"hosts": [
{
"host": "localhost",
"roles": { "read": true, "write": true }
}
],
"port": 9200,
"protocol": "http",
"path": "/phabricator",
"version": 5
},
{
"type": "mysql",
"roles": { "write": true }
}
]
Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Tags: #elasticsearch, #clusters, #wikimedia
Differential Revision: https://secure.phabricator.com/D17384
2017-03-26 10:16:47 +02:00
|
|
|
$this->newOption('cluster.search', $search_type, array())
|
|
|
|
->setLocked(true)
|
|
|
|
->setSummary(
|
|
|
|
pht('Configure full-text search services.'))
|
|
|
|
->setDescription($search_help)
|
|
|
|
->setDefault(
|
|
|
|
array(
|
|
|
|
array(
|
|
|
|
'type' => 'mysql',
|
|
|
|
'roles' => array(
|
|
|
|
'read' => true,
|
|
|
|
'write' => true,
|
|
|
|
),
|
|
|
|
),
|
|
|
|
)),
|
Add `cluster.addresses` and require membership before accepting cluster authentication tokens
Summary:
Ref T2783. Ref T6706.
- Add `cluster.addresses`. This is a whitelist of CIDR blocks which define cluster hosts.
- When we recieve a request that has a cluster-based authentication token, require the cluster to be configured and require the remote address to be a cluster member before we accept it.
- This provides a general layer of security for these mechanisms.
- In particular, it means they do not work by default on unconfigured hosts.
- When cluster addresses are configured, and we receive a request //to// an address not on the list, reject it.
- This provides a general layer of security for getting the Ops side of cluster configuration correct.
- If cluster nodes have public IPs and are listening on them, we'll reject requests.
- Basically, this means that any requests which bypass the LB get rejected.
Test Plan:
- With addresses not configured, tried to make requests; rejected for using a cluster auth mechanism.
- With addresses configred wrong, tried to make requests; rejected for sending from (or to) an address outside of the cluster.
- With addresses configured correctly, made valid requests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6706, T2783
Differential Revision: https://secure.phabricator.com/D11159
2015-01-03 00:13:41 +01:00
|
|
|
);
|
|
|
|
}
|
|
|
|
|
|
|
|
}
|