Allow persistent connections to be configured per database host

Summary: Ref T11044. Fixes T11672. In T11672, persistent connections seem to work fine, but they can require `max_connections` and other settings to be raised. Since most users don't need them, make them an advanced option.

Test Plan: Configured persistent connections, loaded some pages, observed persistent connections get used.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044, T11672

Differential Revision: https://secure.phabricator.com/D16913

src/docs/user/cluster/cluster_partitioning.diviner

@@ -31,6 +31,13 @@ with a single master first. Even if you choose not to deploy replication, you
 should review and understand how replication works before you partition. For
 details, see @{Cluster:Databases}.
 
+Databases also support some advanced configuration options. Briefly:
+
+  - `persistent`: Allows use of persistent connections, reducing pressure on
+    outbound ports.
+
+See "Advanced Configuration", below, for additional discussion.
+
 
 What Partitioning Does
 ======================
@@ -145,7 +152,7 @@ Launching a new Partition
 To add a new partition, follow these steps:
 
   - Set up the new database host or hosts.
-  - Add the new database to `cluster.database`, but keep its "partition"
+  - Add the new database to `cluster.databases`, but keep its "partition"
     configuration empty (just an empty list). If this is the first time you
     are partitioning, you will need to configure your existing master as the
     new "default". This will let Phabricator interact with it, but won't send
@@ -196,6 +203,37 @@ meaningful differences in runtime behavior if two applications are on the same
 physical host or different physical hosts.
 
 
+Advanced Configuration
+======================
+
+Separate from partitioning, some advanced configuration is supported. These
+options must be set on database specifications in `cluster.databases`. You can
+configure them without actually building a cluster by defining a cluster with
+only one master.
+
+`persistent` //(bool)// Enables persistent connections. Defaults to off.
+
+With persitent connections enabled, Phabricator will keep a pool of database
+connections open between web requests and reuse them when serving subsequent
+requests.
+
+The primary benefit of using persistent connections is that it will greatly
+reduce pressure on how quickly outbound TCP ports are opened and closed. After
+a TCP port closes, it normally can't be used again for about 60 seconds, so
+rapidly cycling ports can cause resource exuastion. If you're seeing failures
+because requests are unable to bind to an outbound port, enabling this option
+is likely to fix the issue. This option may also slightly increase performance.
+
+The cost of using persistent connections is that you may need to raise the
+MySQL `max_connections` setting: although Phabricator will make far fewer
+connections, the connections it does make will be longer-lived. Raising this
+setting will increase MySQL memory requirements and may run into other limits,
+like `open_files_limit`, which may also need to be raised.
+
+Persistent connections are enabled per-database. If you always want to use
+them, set the flag on each configured database in `cluster.databases`.
+
+
 Next Steps
 ==========
 

src/infrastructure/cluster/PhabricatorClusterDatabasesConfigOptionType.php

@@ -37,6 +37,7 @@ final class PhabricatorClusterDatabasesConfigOptionType
             'disabled' => 'optional bool',
             'master' => 'optional string',
             'partition' => 'optional list<string>',
+            'persistent' => 'optional bool',
           ));
       } catch (Exception $ex) {
         throw new Exception(

src/infrastructure/cluster/PhabricatorDatabaseRef.php

@@ -40,6 +40,7 @@ final class PhabricatorDatabaseRef
   private $applicationMap = array();
   private $masterRef;
   private $replicaRefs = array();
+  private $usePersistentConnections;
 
   public function setHost($host) {
     $this->host = $host;
@@ -171,6 +172,15 @@ final class PhabricatorDatabaseRef
     return $this->isDefaultPartition;
   }
 
+  public function setUsePersistentConnections($use_persistent_connections) {
+    $this->usePersistentConnections = $use_persistent_connections;
+    return $this;
+  }
+
+  public function getUsePersistentConnections() {
+    return $this->usePersistentConnections;
+  }
+
   public function setApplicationMap(array $application_map) {
     $this->applicationMap = $application_map;
     return $this;
@@ -582,7 +592,8 @@ final class PhabricatorDatabaseRef
       ->setPort($default_port)
       ->setIsIndividual(true)
       ->setIsMaster(true)
-      ->setIsDefaultPartition(true);
+      ->setIsDefaultPartition(true)
+      ->setUsePersistentConnections(false);
   }
 
   public static function getAllReplicaDatabaseRefs() {
@@ -672,9 +683,31 @@ final class PhabricatorDatabaseRef
       'database' => null,
       'retries' => $default_retries,
       'timeout' => $default_timeout,
+      'persistent' => $this->getUsePersistentConnections(),
     );
 
-    return self::newRawConnection($spec);
+    $is_cli = (php_sapi_name() == 'cli');
+
+    $use_persistent = false;
+    if (!empty($spec['persistent']) && !$is_cli) {
+      $use_persistent = true;
+    }
+    unset($spec['persistent']);
+
+    $connection = self::newRawConnection($spec);
+
+    // If configured, use persistent connections. See T11672 for details.
+    if ($use_persistent) {
+      $connection->setPersistent($use_persistent);
+    }
+
+    // Unless this is a script running from the CLI, prevent any query from
+    // running for more than 30 seconds. See T10849 for details.
+    if (!$is_cli) {
+      $connection->setQueryTimeout(30);
+    }
+
+    return $connection;
   }
 
   public static function newRawConnection(array $options) {

src/infrastructure/cluster/PhabricatorDatabaseRefParser.php

@@ -58,13 +58,16 @@ final class PhabricatorDatabaseRefParser
       $role = $server['role'];
       $is_master = ($role == 'master');
 
+      $use_persistent = (bool)idx($server, 'persistent', false);
+
       $ref = id(new PhabricatorDatabaseRef())
         ->setHost($host)
         ->setPort($port)
         ->setUser($user)
         ->setPass($pass)
         ->setDisabled($disabled)
-        ->setIsMaster($is_master);
+        ->setIsMaster($is_master)
+        ->setUsePersistentConnections($use_persistent);
 
       if ($is_master) {
         $master_count++;

src/infrastructure/storage/lisk/PhabricatorLiskDAO.php

@@ -73,21 +73,6 @@ abstract class PhabricatorLiskDAO extends LiskDAO {
       $connection->setReadOnly(true);
     }
 
-    // Unless this is a script running from the CLI:
-    //   - (T10849) Prevent any query from running for more than 30 seconds.
-    //   - (T11672) Use persistent connections.
-    if (php_sapi_name() != 'cli') {
-
-      // TODO: For now, disable this until after T11044: it's better at high
-      // load, but causes us to use slightly more connections at low load and
-      // is pushing users over limits like MySQL "max_connections".
-      $use_persistent = false;
-
-      $connection
-        ->setQueryTimeout(30)
-        ->setPersistent($use_persistent);
-    }
-
     return $connection;
   }