Provide SSH host documentation, tweak/supplement cluster documentation

Summary: Ref T10751. I think this mostly brings us up to date with the state of the world.

Test Plan: Read documentation.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10751

Differential Revision: https://secure.phabricator.com/D15897

src/docs/user/cluster/cluster.diviner

@@ -4,11 +4,13 @@
 Guide to configuring Phabricator across multiple hosts for availability and
 performance.
 
+
 Overview
 ========
 
-WARNING: This feature is a very early prototype; the features this document
-describes are mostly speculative fantasy.
+WARNING: This feature is a prototype. Installs should expect a challening
+adventure when deploying clusters. In the best of times, configuring a
+cluster is complex and requires significant operations experience.
 
 Phabricator can be configured to run on multiple hosts with redundant services
 to improve its availability and scalability, and make disaster recovery much
@@ -19,9 +21,9 @@ single host, but greatly reduces the cost of recovering from hardware and
 network failures.
 
 Each Phabricator service has an array of clustering options that can be
-configured independently. Configuring a cluster is inherently complex, and this
-is an advanced feature aimed at installs with large userbases and experienced
-operations personnel who need this high degree of flexibility.
+configured somewhat independently. Configuring a cluster is inherently complex,
+and this is an advanced feature aimed at installs with large userbases and
+experienced operations personnel who need this high degree of flexibility.
 
 The remainder of this document summarizes how to add redundancy to each
 service and where your efforts are likely to have the greatest impact.
@@ -30,6 +32,25 @@ For additional guidance on setting up a cluster, see "Overlaying Services"
 and "Cluster Recipes" at the bottom of this document.
 
 
+Clusterable Services
+====================
+
+This table provides an overview of clusterable services, their setup
+complexity, and the rough impact that converting them to run on multiple hosts
+will have on availability, resistance to data loss, and scalability.
+
+| Service | Setup | Availability | Loss Resistance | Scalability
+|---------|-------|--------------|-----------|------------
+| **Databases** | Moderate | **High** | **High** | Low
+| **Repositories** | Complex | Moderate | **High** | Moderate
+| **Daemons** | Minimal | Low | No Risk | Low
+| **SSH Servers** | Minimal | Low | No Risk | Low
+| **Web Servers** | Minimal | **High** | No Risk | Moderate
+| **Notifications** | Minimal | Low | No Risk | Low
+
+See below for a walkthrough of these services in greater detail.
+
+
 Preparing for Clustering
 ========================
 
@@ -146,7 +167,7 @@ Cluster: Daemons
 Configuring multiple daemon hosts is straightforward, but you must configure
 repositories first.
 
-With daemons running on multiple hosts, you can transparently survive the loss
+With daemons running on multiple hosts you can transparently survive the loss
 of any subset of hosts without an interruption to daemon services, as long as
 at least one host remains alive. Daemons are stateless, so spreading daemons
 across multiple hosts provides no resistance to data loss.
@@ -161,14 +182,30 @@ in capacity.
 For details, see @{article:Cluster: Daemons}.
 
 
+Cluster: SSH Servers
+====================
+
+Configuring multiple SSH hosts is straightforward, but you must configure
+repositories first.
+
+With multiple SSH hosts you can transparently survive the loss of any subset
+of hosts without interruption to repository services, as long as at last one
+host remains alive. SSH services are stateless, so putting multiple hosts in
+service provides no resistance to data loss because no data is at risk.
+
+SSH hosts are very rarely a scalability bottleneck.
+
+For details, see @{article:Cluster: SSH Servers}.
+
+
 Cluster: Web Servers
 ====================
 
 Configuring multiple web hosts is straightforward, but you must configure
 repositories first.
 
-With multiple web hosts, you can transparently survive the loss of any subset
-of hosts as long as at least one host remains alive. Web hosts are stateless,
+With multiple web hosts you can transparently survive the loss of any subset
+of hosts as long as at least one host remains alive. Web services are stateless,
 so putting multiple hosts in service provides no resistance to data loss
 because no data is at risk.
 

src/docs/user/cluster/cluster_daemons.diviner

@@ -6,9 +6,6 @@ Configuring Phabricator to use multiple daemon hosts.
 Overview
 ========
 
-WARNING: This feature is a very early prototype; the features this document
-describes are mostly speculative fantasy.
-
 You can run daemons on multiple hosts. The advantages of doing this are:
 
   - you can completely survive the loss of multiple daemon hosts; and
@@ -18,7 +15,8 @@ This configuration is simple, but you must configure repositories first. For
 details, see @{article:Cluster: Repositories}.
 
 Since repository hosts must run daemons anyway, you usually do not need to do
-any additional work and can skip this entirely.
+any additional work and can skip this entirely if you have already configured
+multiple repository hosts.
 
 
 Adding Daemon Hosts

src/docs/user/cluster/cluster_devices.diviner

@@ -101,7 +101,7 @@ up with records that look like these:
   - Device: `repo001.mycompany.net`
     - Interface: `123.0.0.1:2222`
     - Interface: `123.0.0.1:80`
-  - Device: `repo002.mycopmany.net`
+  - Device: `repo002.mycompany.net`
     - Interface: `123.0.0.2:2222`
     - Interface: `123.0.0.2:80`
 

src/docs/user/cluster/cluster_notifications.diviner

@@ -6,9 +6,6 @@ Configuring Phabricator to use multiple notification servers.
 Overview
 ========
 
-WARNING: This feature is a very early prototype; the features this document
-describes are mostly speculative fantasy.
-
 You can run multiple notification servers. The advantages of doing this
 are:
 

src/docs/user/cluster/cluster_repositories.diviner

@@ -6,9 +6,6 @@ Configuring Phabricator to use multiple repository hosts.
 Overview
 ========
 
-WARNING: This feature is a very early prototype; the features this document
-describes are mostly speculative fantasy.
-
 If you use Git, you can deploy Phabricator with multiple repository hosts,
 configured so that each host is readable and writable. The advantages of doing
 this are:
@@ -296,8 +293,6 @@ the user whose change is holding the lock.
 currently held, this shows when the lock was acquired.
 
 
-
-
 Cluster Failure Modes
 =====================
 

src/docs/user/cluster/cluster_ssh.diviner

@@ -0,0 +1,47 @@
+@title Cluster: SSH Servers
+@group cluster
+
+Configuring Phabricator to use multiple SSH servers.
+
+Overview
+========
+
+You can run Phabricator on multiple SSH servers. The advantages of doing this
+are:
+
+  - you can completely survive the loss of multiple SSH hosts.
+
+This configuration is simple, but you must configure repositories first. For
+details, see @{article:Cluster: Repositories}.
+
+SSH servers accept SSH requests from commands like `git clone` and relay them
+to hosts that can serve the requests.
+
+
+Adding SSH Hosts
+================
+
+After configuring repositories in cluster mode, you can add more web hosts
+at any time.
+
+First, deploy the Phabricator software and configuration to a host, then
+register the host as a cluster device if it is not already registered (for
+help, see @{article:Cluster: Devices}.
+
+Once the host is registered, start the SSH server, and then add the host to the
+SSH load balancer pool.
+
+Phabricator SSH servers are stateless, so you can pull them in and out of
+production freely.
+
+You may also want to run web services on these hosts, since the service is very
+similar to SSH, also stateless, and it may be simpler to load balance the
+services together. For details, see @{cluster: Web Servers}.
+
+
+Next Steps
+==========
+
+Continue by:
+
+  - returning to @{article:Clustering Introduction}.

src/docs/user/cluster/cluster_webservers.diviner

@@ -6,9 +6,6 @@ Configuring Phabricator to use multiple web servers.
 Overview
 ========
 
-WARNING: This feature is a very early prototype; the features this document
-describes are mostly speculative fantasy.
-
 You can run Phabricator on multiple web servers. The advantages of doing this
 are:
 
@@ -23,15 +20,21 @@ Adding Web Hosts
 ================
 
 After configuring repositories in cluster mode, you can add more web hosts
-at any time: simply deploy the Phabricator software and configuration to a
-host, start the web server, and then add the host to the load balancer pool.
+at any time.
+
+First, deploy the Phabricator software and configuration to a host, then
+register the host as a cluster device if it is not already registered (for
+help, see @{article:Cluster: Devices}.
+
+Once the host is registered, start the web server, and then add the host to the
+load balancer pool.
 
 Phabricator web servers are stateless, so you can pull them in and out of
 production freely.
 
 You may also want to run SSH services on these hosts, since the service is very
 similar to HTTP, also stateless, and it may be simpler to load balance the
-services together.
+services together. For details, see @{cluster:SSH Servers}.
 
 
 Next Steps