Improve setup process / documentation for outbound email configuration

Summary:
ccheever did an install and gave me some feedback about issues he hit. This
tries to:

  - properly document how to configure outbound email;
  - test outbound email configuration in the setup mode;
  - provide basic daemon documentation;
  - document that phabricator.base-uri is required for all installs.

Test Plan:
read documentation, jumped through all the setup branches to test configuration
error detection

Reviewed By: aran
Reviewers: tuomaspelkonen, jungejason, aran, rm
CC: ccheever, aran
Differential Revision: 276

.divinerconfig

@@ -3,7 +3,6 @@
   "src_base" : "https://github.com/facebook/phabricator/blob/master",
   "groups" : {
     "intro" : "Introduction",
-    "install" : "Installing",
     "config" : "Configuration",
     "contrib" : "Contributing",
     "userguide" : "Application User Guides",

conf/default.conf.php

@@ -152,8 +152,8 @@ return array(
   'metamta.can-send-as-user'    => false,
 
   // Adapter class to use to transmit mail to the MTA. The default uses
-  // PHPMailerLite, which will invoke PHP's mail() function. This is appropriate
-  // if mail() actually works on your host, but if you haven't configured mail
+  // PHPMailerLite, which will invoke "sendmail". This is appropriate
+  // if sendmail actually works on your host, but if you haven't configured mail
   // it may not be so great. You can also use Amazon SES, by changing this to
   // 'PhabricatorMailImplementationAmazonSESAdapter', signing up for SES, and
   // filling in your 'amazon-ses.access-key' and 'amazon-ses.secret-key' below.

src/docs/configuration_guide.diviner

@@ -36,6 +36,10 @@ in the file:
     // configure things.
     'phabricator.setup' => true,
 
+    // This will be the base domain for your install, and must be configured.
+    // Use "https://" if you have SSL. See below for some notes.
+    'phabricator.base-uri' => 'http://phabricator.example.com/',
+
   ) + phabricator_read_config_file('production');
 
 For the last line, you can also use ##'development'## instead of
@@ -60,7 +64,8 @@ or edit the Directory entry for the DocumentRoot. It should look something like
 this:
 
   <VirtualHost *>
-    # Change this to the domain which points to your host.
+    # Change this to the domain which points to your host, i.e. the domain
+    # you set as "phabricator.base-uri".
     ServerName phabricator.example.com
 
     # Change this to the path where you put 'phabricator' when you checked it
@@ -125,3 +130,14 @@ that will make upgrading Phabricator more difficult in the future.
 After you have configured Phabricator, you need to upgrade the database
 schema -- see @{article:Upgrading Schema}. You'll also need to do this after you
 update the code in the future.
+
+= Next Steps =
+
+Continue by:
+
+  - upgrading the database schema with @{article:Upgrading Schema}; or
+  - configuring Phabricator so it can send mail with
+    @{article:Configuring Outbound Email}; or
+  - configuring inbound mail with @{article:Configuring Inbound Email}; or
+  - learning about daemons with @{article:Managing Daemons with phd}; or
+  - contributing to Phabricator with @{article:Contributor Introduction}.

src/docs/configuring_outbound_email.diviner

@@ -0,0 +1,145 @@
+@title Configuring Outbound Email
+@group config
+
+Instructions for configuring Phabricator to send mail.
+
+= Overview =
+
+Phabricator can send outbound email via four different adapters:
+
+  - by running ##sendmail## on the local host; or
+  - by using Amazon SES (Simple Email Service); or
+  - via a custom adapter you write; or
+  - by dropping email into a hole and not delivering it.
+
+Of these, ##sendmail## is the default but requires some configuration. SES is
+the easiest but costs money and has some limitations. Writing a custom solution
+requires digging into the code. See below for details on how to set up each
+method.
+
+Phabricator can also send outbound email in two ways:
+
+  - immediately, when messages are generated (default); or
+  - in the background, via a daemon.
+
+Sending mail in the background requires more configuration, but will greatly
+improve the performance of the application if your mail handler is slow. Note
+that Amazon SES commonly takes 1-2 seconds per email. If you use SES,
+**strongly consider** configuring the daemon. You should also configure the
+daemon if commenting on Revisions or Tasks feels slow, as it may significantly
+improve performance.
+
+= Basics =
+
+Regardless of how outbound email is delivered, you should configure these keys
+in your configuration file:
+
+  - **metamta.default-address** determines where mail is sent "From" by
+    default. If your domain is ##example.org##, set this to something like
+    "##noreply@example.org##".
+  - **metamta.domain** should be set to your domain, e.g. "##example.org##".
+  - **metamta.can-send-as-user** should be left as ##false## in most cases,
+    but see the documentation in ##default.conf.php## for details.
+
+= Configuring Mail Adapters =
+
+To choose how mail will be sent, change the **metamta.mail-adapter** key in
+your configuration. Possible values are:
+
+  - ##PhabricatorMailImplementationPHPMailerLiteAdapter##: default, uses
+    "sendmail", see "Adapter: Sendmail".
+  - ##PhabricatorMailImplementationAmazonSESAdapter##: use Amazon SES, see
+    "Adapter: Amazon SES".
+  - ##Some Custom Class You Write##: use a custom adapter you write, see
+    "Adapter: Custom".
+  - ##PhabricatorMailImplementationTestAdapter##: this will
+    **completely disable** outbound mail. You can use this if you don't want to
+    send outbound mail, or want to skip this step for now and configure it
+    later.
+
+= Adapter: Sendmail =
+
+This is the default, and selected by choosing
+##PhabricatorMailImplementationPHPMailerLiteAdapter## as the value for
+**metamta.mail-adapter**. This requires a 'sendmail' binary to be installed on
+the system. Most MTAs (e.g., sendmail, qmail, postfix) should do this, but your
+machine may not have one installed by default. For install instructions, consult
+the documentation for your favorite MTA.
+
+Since you'll be sending the mail yourself, you are subject to things like SPF
+rules, blackholes, and MTA configuration which are beyond the scope of this
+document. If you can already send outbound email from the command line or know
+how to configure it, this option is straightforward. If you have no idea how to
+do any of this, consider using Amazon SES.
+
+= Adapter: Amazon SES =
+
+Amazon SES is Amazon's cloud email service. It is not free, but is easier to
+configure than sendmail and can simplify outbound email configuration. To use
+Amazon SES, you need to sign up for an account with Amazon at
+<http://aws.amazon.com/ses/>.
+
+To configure Phabricator to use Amazon SES, set these configuration keys:
+
+  - **metamta.mail-adapter**: set to
+    "PhabricatorMailImplementationAmazonSESAdapter".
+  - **amazon-ses.access-key**: set to your Amazon SES access key.
+  - **amazon-ses.secret-key**: set to your Amazon SES secret key.
+
+NOTE: Amazon SES is slow to accept mail (often 1-2 seconds) and application
+performance will improve greatly if you configure outbound email to send in
+the background.
+
+= Adapter: Custom =
+
+You can provide a custom adapter by writing a concrete subclass of
+@{class:PhabricatorMailImplementationAdapter} and setting it as the
+**metamta.mail-adapter**.
+
+TODO: This needs to be better documented once extending Phabricator is better
+documented.
+
+= Adapter: Disable Outbound Mail =
+
+You can use the @{class:PhabricatorMailImplementationTestAdapter} to completely
+disable outbound mail, if you don't want to send mail or don't want to configure
+it yet. Just set **metamta.mail-adapter** to
+"PhabricatorMailImplementationTestAdapter".
+
+= Configuring the MetaMTA Daemon =
+
+Regardless of how you are sending outbound email, you can move the handoff to
+the MTA out of the main process and into a daemon. This will greatly improve
+application performance if your mailer is slow, like Amazon SES. In particular,
+commenting on Differential Revisions and Maniphest Tasks sends outbound email.
+
+To use the MetaMTA daemon:
+
+  - set **metamta.send-immediately** to ##false## in your configuration; and
+  - launch a ##metamta## daemon with ##phabricator/bin/phd launch metamta##.
+
+For more information on using daemons, see @{article:Managing Daemons with phd}.
+
+= Testing and Debugging Outbound Email =
+
+Phabricator has a mail log and test console at ##/mail/##, or click the
+**MetaMTA** link from the homepage. This console shows all the mail Phabricator
+has attempted to deliver, plus debugging and error information.
+
+You can use the "Send New Message" button to send mail using the current
+configuration. This can help test that your setup is correct.
+
+NOTE: when you send mail, "to" and "cc" must be valid users of the system, not
+arbitrary email addresses.
+
+You can monitor daemons using the Daemon Console (##/daemon/##, or click
+**Daemon Console** from the homepage).
+
+= Next Steps =
+
+Continue by:
+
+  - @{article:Configuring Inbound Email} so users can reply to email they
+  receive about revisions and tasks to interact with them; or
+  - learning about daemons with @{article:Managing Daemons with phd}; or
+  - returning to the @{article:Configuration Guide}.

src/docs/installation_guide.diviner

@@ -1,5 +1,5 @@
 @title Installation Guide
-@group install
+@group intro
 
 This document contains basic install instructions to get Phabricator up and
 running.
@@ -44,6 +44,11 @@ Now that you have git installed, grab Phabricator and its dependencies:
 
 = Installing Optional Components =
 
+These components are optional and Phabricator will work without them, but works
+better if they are installed. You should read each section and decide if you
+want to install the component. At a minimum, **APC** will greatly improve
+performance.
+
 == APC ==
 
 Like everything else written in PHP, Phabricator will run much faster with APC
@@ -89,4 +94,6 @@ Then install bison normally:
 
 = Next Steps =
 
-If everything is installed, move on to @{article:Configuration Guide}.
+Continue by:
+
+  - configuring Phabricator with the @{article:Configuration Guide}.

src/docs/introduction.diviner

@@ -13,7 +13,7 @@ The major components of Phabricator are:
 
   - **Differential**, a code review tool; and
   - **Diffusion**, a repository browser.
-  
+
 Phabricator also includes a number of smaller tools.
 
 = Why use Phabricator? =
@@ -37,5 +37,8 @@ However, Phabricator may also not be a good solution for you:
     you can get anywhere.
   - If you loathe PHP, well, it's written in PHP.
 
+= Next Steps =
 
-  
+Continue by:
+
+  - installing Phabricator with the @{article:Installation Guide}.

src/docs/managing_daemons.diviner

@@ -0,0 +1,70 @@
+@title Managing Daemons with phd
+@group config
+
+Explains Phabricator daemons and the daemon control program ##phd##.
+
+= Overview =
+
+Phabricator uses daemons (background processing scripts) to handle a number of
+tasks, like:
+
+  - tracking repositories and discovering new commits;
+  - sending mail;
+  - updating objects in the search index; and
+  - custom tasks you define.
+
+Daemons are started and stopped with **phd** (the **Ph**abricator **D**aemon
+launcher). Daemons can be monitored via a web console.
+
+You do not need to run daemons for most parts of Phabricator to work, but a few
+features (principally, repository tracking with Diffusion) require them and
+several features will benefit in performance or stability if you configure
+daemons.
+
+= phd =
+
+**phd** is a command-line script (located at ##phabricator/bin/phd##). To get
+a list of commands, run ##phd help##:
+
+  phabricator/ $ ./bin/phd help
+  NAME
+          phd - phabricator daemon launcher
+  ...
+
+Generally, you will use:
+
+  - **phd launch** to launch daemons;
+  - **phd status** to get a list of running daemons; and
+  - **phd stop** to stop all daemons.
+
+NOTE: When you upgrade Phabricator or change configuration, you should restart
+the daemons by stopping and relaunching them.
+
+NOTE: When you **launch** a daemon, you can type any unique substring of its
+name, so **phd launch metamta** will work correctly.
+
+= Daemon Console =
+
+You can view status and debugging information for daemons in the Daemon Console
+via the web interface. Go to ##/daemon/## in your install or click
+**Daemon Console** from the homepage.
+
+The Daemon Console shows a list of all the daemons that have ever launched, and
+allows you to view log information for them. If you have issues with daemons,
+you may be able to find error information that will help you resolve the problem
+in the console.
+
+= Available Daemons =
+
+You can get a list of launchable daemons with **phd list**:
+
+  - **libphutil test daemons** are not generally useful unless you are
+    developing daemon infrastructure or debugging a daemon problem;
+  - **PhabricatorMetaMTADaemon** sends mail in the background, see
+    @{article:Configuring Outbound Email} for details;
+  - **PhabricatorTaskmasterDaemon** runs a generic task queue; and
+  - **PhabricatorRepository** daemons track repositories.
+
+TODO: Documentation on repository daemons is coming soon. The quick version is:
+run **phd repository-launch-master** after configuring them in the
+**Repositories** tool, then launch a few **taskmaster** daemons.

src/infrastructure/setup/PhabricatorSetup.php

@@ -240,6 +240,114 @@ class PhabricatorSetup {
     self::write("[OKAY] Database configuration OKAY\n");
 
 
+    self::writeHeader("OUTBOUND EMAIL CONFIGURATION");
+
+    $have_adapter = false;
+    $is_ses = false;
+
+    $adapter = PhabricatorEnv::getEnvConfig('metamta.mail-adapter');
+    switch ($adapter) {
+      case 'PhabricatorMailImplementationPHPMailerLiteAdapter':
+
+        $have_adapter = true;
+
+        list($err) = exec_manual('which sendmail');
+        if ($err) {
+          self::writeFailure();
+          self::write(
+            "Setup failure! You don't have a 'sendmail' binary on this system ".
+            "but outbound email is configured to use sendmail. Install an MTA ".
+            "(like sendmail, qmail or postfix) or use a different outbound ".
+            "mail configuration. See this guide for configuring outbound ".
+            "email:\n");
+          self::writeDoc('article/Configuring_Outbound_Email.html');
+          return;
+        } else {
+          self::write(" okay  Sendmail is configured.\n");
+        }
+
+        break;
+      case 'PhabricatorMailImplementationAmazonSESAdapter':
+
+        $is_ses = true;
+        $have_adapter = true;
+
+        if (PhabricatorEnv::getEnvConfig('metamta.can-send-as-user')) {
+          self::writeFailure();
+          self::write(
+            "Setup failure! 'metamta.can-send-as-user' must be false when ".
+            "configured with Amazon SES.");
+          return;
+        } else {
+          self::write(" okay  Sender config looks okay.\n");
+        }
+
+        if (!PhabricatorEnv::getEnvConfig('amazon-ses.access-key')) {
+          self::writeFailure();
+          self::write(
+            "Setup failure! 'amazon-ses.access-key' is not set, but ".
+            "outbound mail is configured to deliver via Amazon SES.");
+          return;
+        } else {
+          self::write(" okay Amazon SES access key is set.\n");
+        }
+
+        if (!PhabricatorEnv::getEnvConfig('amazon-ses.secret-key')) {
+          self::writeFailure();
+          self::write(
+            "Setup failure! 'amazon-ses.secret-key' is not set, but ".
+            "outbound mail is configured to deliver via Amazon SES.");
+          return;
+        } else {
+          self::write(" okay Amazon SES secret key is set.\n");
+        }
+
+        if (PhabricatorEnv::getEnvConfig('metamta.send-immediately')) {
+          self::writeNote(
+            "Your configuration uses Amazon SES to deliver email but tries ".
+            "to send it immediately. This will work, but it's slow. ".
+            "Consider configuring the MetaMTA daemon.");
+        }
+        break;
+      case 'PhabricatorMailImplementationTestAdapter':
+        self::write(" skip  You have disabled outbound email.\n");
+        break;
+      default:
+        self::write(" skip  Configured with a custom adapter.\n");
+        break;
+    }
+
+    if ($have_adapter) {
+      $default = PhabricatorEnv::getEnvConfig('metamta.default-address');
+      if (!$default || $default == 'noreply@example.com') {
+        self::writeFailure();
+        self::write(
+          "Setup failure! You have not set 'metamta.default-address'.");
+        return;
+      } else {
+        self::write(" okay  metamta.default-address is set.\n");
+      }
+
+      if ($is_ses) {
+        self::writeNote(
+          "Make sure you've verified your 'from' address ('{$default}') with ".
+          "Amazon SES. Until you verify it, you will be unable to send mail ".
+          "using Amazon SES.");
+      }
+
+      $domain = PhabricatorEnv::getEnvConfig('metamta.domain');
+      if (!$domain || $domain == 'example.com') {
+        self::writeFailure();
+        self::write(
+          "Setup failure! You have not set 'metamta.domain'.");
+        return;
+      } else {
+        self::write(" okay  metamta.domain is set.\n");
+      }
+
+      self::write("[OKAY] Mail configuration OKAY\n");
+    }
+
     self::writeHeader('SUCCESS!');
     self::write(
       "Congratulations! Your setup seems mostly correct, or at least fairly ".