Support email replies in Phabricator

Summary:
Provides support for per-user x per-object unique reply-to email addresses, plus
SMTP integration.

This does not actually make Phabricator use these in outbound email.

Test Plan:
Used test console to validate in-Phabricator routing and handling.

Piped emails into the "mail_handler.php" script to validate mail parsing.

Configured sendmail and sent mail to Phabricator.

Technically I haven't conducted all parts of this test on the same machine since
I lost the will to configure more SMTP servers after configuring phabricator.com

Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, epriestley, jungejason
Differential Revision: 226

conf/default.conf.php

@@ -253,6 +253,13 @@ return array(
   // you might as well.
   'phabricator.csrf-key'        => '0b7ec0592e0a2829d8b71df2fa269b2c6172eca3',
 
+  // This is hashed with other inputs to generate mail tokens. If you want, you
+  // can change it to some other string which is unique to your install. In
+  // particular, you will want to do this if you accidentally send a bunch of
+  // mail somewhere you shouldn't have, to invalidate all old reply-to
+  // addresses.
+  'phabricator.mail-key'        => '5ce3e7e8787f6e40dfae861da315a5cdf1018f12',
+
   // Version string displayed in the footer. You probably should leave this
   // alone.
   'phabricator.version'         => 'UNSTABLE',

externals/mimemailparser/LICENSE

@@ -0,0 +1,125 @@
+			 The "Artistic License"
+
+				Preamble
+
+The intent of this document is to state the conditions under which a
+Package may be copied, such that the Copyright Holder maintains some
+semblance of artistic control over the development of the package,
+while giving the users of the package the right to use and distribute
+the Package in a more-or-less customary fashion, plus the right to make
+reasonable modifications.
+
+Definitions:
+
+	"Package" refers to the collection of files distributed by the
+	Copyright Holder, and derivatives of that collection of files
+	created through textual modification.
+
+	"Standard Version" refers to such a Package if it has not been
+	modified, or has been modified in accordance with the wishes
+	of the Copyright Holder as specified below.
+
+	"Copyright Holder" is whoever is named in the copyright or
+	copyrights for the package.
+
+	"You" is you, if you're thinking about copying or distributing
+	this Package.
+
+	"Reasonable copying fee" is whatever you can justify on the
+	basis of media cost, duplication charges, time of people involved,
+	and so on.  (You will not be required to justify it to the
+	Copyright Holder, but only to the computing community at large
+	as a market that must bear the fee.)
+
+	"Freely Available" means that no fee is charged for the item
+	itself, though there may be fees involved in handling the item.
+	It also means that recipients of the item may redistribute it
+	under the same conditions they received it.
+
+1. You may make and give away verbatim copies of the source form of the
+Standard Version of this Package without restriction, provided that you
+duplicate all of the original copyright notices and associated disclaimers.
+
+2. You may apply bug fixes, portability fixes and other modifications
+derived from the Public Domain or from the Copyright Holder.  A Package
+modified in such a way shall still be considered the Standard Version.
+
+3. You may otherwise modify your copy of this Package in any way, provided
+that you insert a prominent notice in each changed file stating how and
+when you changed that file, and provided that you do at least ONE of the
+following:
+
+    a) place your modifications in the Public Domain or otherwise make them
+    Freely Available, such as by posting said modifications to Usenet or
+    an equivalent medium, or placing the modifications on a major archive
+    site such as uunet.uu.net, or by allowing the Copyright Holder to include
+    your modifications in the Standard Version of the Package.
+
+    b) use the modified Package only within your corporation or organization.
+
+    c) rename any non-standard executables so the names do not conflict
+    with standard executables, which must also be provided, and provide
+    a separate manual page for each non-standard executable that clearly
+    documents how it differs from the Standard Version.
+
+    d) make other distribution arrangements with the Copyright Holder.
+
+4. You may distribute the programs of this Package in object code or
+executable form, provided that you do at least ONE of the following:
+
+    a) distribute a Standard Version of the executables and library files,
+    together with instructions (in the manual page or equivalent) on where
+    to get the Standard Version.
+
+    b) accompany the distribution with the machine-readable source of
+    the Package with your modifications.
+
+    c) give non-standard executables non-standard names, and clearly
+    document the differences in manual pages (or equivalent), together
+    with instructions on where to get the Standard Version.
+
+    d) make other distribution arrangements with the Copyright Holder.
+
+5. You may charge a reasonable copying fee for any distribution of this
+Package.  You may charge any fee you choose for support of this
+Package.  You may not charge a fee for this Package itself.  However,
+you may distribute this Package in aggregate with other (possibly
+commercial) programs as part of a larger (possibly commercial) software
+distribution provided that you do not advertise this Package as a
+product of your own.  You may embed this Package's interpreter within
+an executable of yours (by linking); this shall be construed as a mere
+form of aggregation, provided that the complete Standard Version of the
+interpreter is so embedded.
+
+6. The scripts and library files supplied as input to or produced as
+output from the programs of this Package do not automatically fall
+under the copyright of this Package, but belong to whoever generated
+them, and may be sold commercially, and may be aggregated with this
+Package.  If such scripts or library files are aggregated with this
+Package via the so-called "undump" or "unexec" methods of producing a
+binary executable image, then distribution of such an image shall
+neither be construed as a distribution of this Package nor shall it
+fall under the restrictions of Paragraphs 3 and 4, provided that you do
+not represent such an executable image as a Standard Version of this
+Package.
+
+7. C subroutines (or comparably compiled subroutines in other
+languages) supplied by you and linked into this Package in order to
+emulate subroutines and variables of the language defined by this
+Package shall not be considered part of this Package, but are the
+equivalent of input as in Paragraph 6, provided these subroutines do
+not change the language in any way that would cause it to fail the
+regression tests for the language.
+
+8. Aggregation of this Package with a commercial distribution is always
+permitted provided that the use of this Package is embedded; that is,
+when no overt attempt is made to make this Package's interfaces visible
+to the end user of the commercial distribution.  Such use shall not be
+construed as a distribution of this Package.
+
+9. The name of the Copyright Holder may not be used to endorse or promote
+products derived from this software without specific prior written permission.
+
+10. THIS PACKAGE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE.

externals/mimemailparser/MimeMailParser.class.php

@@ -0,0 +1,439 @@
+<?php
+
+require_once('attachment.class.php');
+
+/**
+ * Fast Mime Mail parser Class using PHP's MailParse Extension
+ * @author gabe@fijiwebdesign.com
+ * @url http://www.fijiwebdesign.com/
+ * @license http://creativecommons.org/licenses/by-sa/3.0/us/
+ * @version $Id$
+ */
+class MimeMailParser {
+	
+	/**
+	 * PHP MimeParser Resource ID
+	 */
+	public $resource;
+	
+	/**
+	 * A file pointer to email
+	 */
+	public $stream;
+	
+	/**
+	 * A text of an email
+	 */
+	public $data;
+	
+	/**
+	 * Stream Resources for Attachments
+	 */
+	public $attachment_streams;
+	
+	/**
+	 * Inialize some stuff
+	 * @return 
+	 */
+	public function __construct() {
+		$this->attachment_streams = array();
+	}
+	
+	/**
+	 * Free the held resouces
+	 * @return void
+	 */
+	public function __destruct() {
+		// clear the email file resource
+		if (is_resource($this->stream)) {
+			fclose($this->stream);
+		}
+		// clear the MailParse resource
+		if (is_resource($this->resource)) {
+			mailparse_msg_free($this->resource);
+		}
+		// remove attachment resources
+		foreach($this->attachment_streams as $stream) {
+			fclose($stream);
+		}
+	}
+	
+	/**
+	 * Set the file path we use to get the email text
+	 * @return Object MimeMailParser Instance
+	 * @param $mail_path Object
+	 */
+	public function setPath($path) {
+		// should parse message incrementally from file
+		$this->resource = mailparse_msg_parse_file($path);
+		$this->stream = fopen($path, 'r');
+		$this->parse();
+		return $this;
+	}
+	
+	/**
+	 * Set the Stream resource we use to get the email text
+	 * @return Object MimeMailParser Instance
+	 * @param $stream Resource
+	 */
+	public function setStream($stream) {
+
+		// streams have to be cached to file first
+		if (get_resource_type($stream) == 'stream') {
+			$tmp_fp = tmpfile();
+			if ($tmp_fp) {
+				while(!feof($stream)) {
+					fwrite($tmp_fp, fread($stream, 2028));
+				}
+				fseek($tmp_fp, 0);
+				$this->stream =& $tmp_fp;
+			} else {
+				throw new Exception('Could not create temporary files for attachments. Your tmp directory may be unwritable by PHP.');
+				return false;
+			}
+			fclose($stream);
+		} else {
+			$this->stream = $stream;
+		}
+		
+		$this->resource = mailparse_msg_create();
+		// parses the message incrementally low memory usage but slower
+		while(!feof($this->stream)) {
+			mailparse_msg_parse($this->resource, fread($this->stream, 2082));
+		}
+		$this->parse();
+		return $this;
+	}
+	
+	/**
+	 * Set the email text
+	 * @return Object MimeMailParser Instance 
+	 * @param $data String
+	 */
+	public function setText($data) {
+		$this->resource = mailparse_msg_create();
+		// does not parse incrementally, fast memory hog might explode
+		mailparse_msg_parse($this->resource, $data);
+		$this->data = $data;
+		$this->parse();
+		return $this;
+	}
+	
+	/**
+	 * Parse the Message into parts
+	 * @return void
+	 * @private
+	 */
+	private function parse() {
+		$structure = mailparse_msg_get_structure($this->resource);
+		$this->parts = array();
+		foreach($structure as $part_id) {
+			$part = mailparse_msg_get_part($this->resource, $part_id);
+			$this->parts[$part_id] = mailparse_msg_get_part_data($part);
+		}
+	}
+	
+	/**
+	 * Retrieve the Email Headers
+	 * @return Array
+	 */
+	public function getHeaders() {
+		if (isset($this->parts[1])) {
+			return $this->getPartHeaders($this->parts[1]);
+		} else {
+			throw new Exception('MimeMailParser::setPath() or MimeMailParser::setText() must be called before retrieving email headers.');
+		}
+		return false;
+	}
+	/**
+	 * Retrieve the raw Email Headers
+	 * @return string
+	 */
+	public function getHeadersRaw() {
+		if (isset($this->parts[1])) {
+			return $this->getPartHeaderRaw($this->parts[1]);
+		} else {
+			throw new Exception('MimeMailParser::setPath() or MimeMailParser::setText() must be called before retrieving email headers.');
+		}
+		return false;
+	}
+	
+	/**
+	 * Retrieve a specific Email Header
+	 * @return String
+	 * @param $name String Header name
+	 */
+	public function getHeader($name) {
+		if (isset($this->parts[1])) {
+			$headers = $this->getPartHeaders($this->parts[1]);
+			if (isset($headers[$name])) {
+				return $headers[$name];
+			}
+		} else {
+			throw new Exception('MimeMailParser::setPath() or MimeMailParser::setText() must be called before retrieving email headers.');
+		}
+		return false;
+	}
+	
+	/**
+	 * Returns the email message body in the specified format
+	 * @return Mixed String Body or False if not found
+	 * @param $type Object[optional]
+	 */
+	public function getMessageBody($type = 'text') {
+		$body = false;
+		$mime_types = array(
+			'text'=> 'text/plain',
+			'html'=> 'text/html'
+		);
+		if (in_array($type, array_keys($mime_types))) {
+			foreach($this->parts as $part) {
+				if ($this->getPartContentType($part) == $mime_types[$type]) {
+                    $headers = $this->getPartHeaders($part);
+					$body = $this->decode($this->getPartBody($part), array_key_exists('content-transfer-encoding', $headers) ? 
+$headers['content-transfer-encoding'] : '');
+				}
+			}
+		} else {
+			throw new Exception('Invalid type specified for MimeMailParser::getMessageBody. "type" can either be text or html.');
+		}
+		return $body;
+	}
+
+	/**
+	 * get the headers for the message body part.
+	 * @return Array
+	 * @param $type Object[optional]
+	 */
+	public function getMessageBodyHeaders($type = 'text') {
+		$headers = false;
+		$mime_types = array(
+			'text'=> 'text/plain',
+			'html'=> 'text/html'
+		);
+		if (in_array($type, array_keys($mime_types))) {
+			foreach($this->parts as $part) {
+				if ($this->getPartContentType($part) == $mime_types[$type]) {
+					$headers = $this->getPartHeaders($part);
+				}
+			}
+		} else {
+			throw new Exception('Invalid type specified for MimeMailParser::getMessageBody. "type" can either be text or html.');
+		}
+		return $headers;
+	}
+
+	
+	/**
+	 * Returns the attachments contents in order of appearance
+	 * @return Array
+	 * @param $type Object[optional]
+	 */
+	public function getAttachments() {
+		$attachments = array();
+		$dispositions = array("attachment","inline");
+		foreach($this->parts as $part) {
+			$disposition = $this->getPartContentDisposition($part);
+			if (in_array($disposition, $dispositions)) {
+				$attachments[] = new MimeMailParser_attachment(
+					$part['disposition-filename'], 
+					$this->getPartContentType($part), 
+					$this->getAttachmentStream($part),
+					$disposition,
+					$this->getPartHeaders($part)
+				);
+			}
+		}
+		return $attachments;
+	}
+	
+	/**
+	 * Return the Headers for a MIME part
+	 * @return Array
+	 * @param $part Array
+	 */
+	private function getPartHeaders($part) {
+		if (isset($part['headers'])) {
+			return $part['headers'];
+		}
+		return false;
+	}
+	
+	/**
+	 * Return a Specific Header for a MIME part
+	 * @return Array
+	 * @param $part Array
+	 * @param $header String Header Name
+	 */
+	private function getPartHeader($part, $header) {
+		if (isset($part['headers'][$header])) {
+			return $part['headers'][$header];
+		}
+		return false;
+	}
+	
+	/**
+	 * Return the ContentType of the MIME part
+	 * @return String
+	 * @param $part Array
+	 */
+	private function getPartContentType($part) {
+		if (isset($part['content-type'])) {
+			return $part['content-type'];
+		}
+		return false;
+	}
+	
+	/**
+	 * Return the Content Disposition
+	 * @return String
+	 * @param $part Array
+	 */
+	private function getPartContentDisposition($part) {
+		if (isset($part['content-disposition'])) {
+			return $part['content-disposition'];
+		}
+		return false;
+	}
+	
+	/**
+	 * Retrieve the raw Header of a MIME part
+	 * @return String
+	 * @param $part Object
+	 */
+	private function getPartHeaderRaw(&$part) {
+		$header = '';
+		if ($this->stream) {
+			$header = $this->getPartHeaderFromFile($part);
+		} else if ($this->data) {
+			$header = $this->getPartHeaderFromText($part);
+		} else {
+			throw new Exception('MimeMailParser::setPath() or MimeMailParser::setText() must be called before retrieving email parts.');
+		}
+		return $header;
+	}
+	/**
+	 * Retrieve the Body of a MIME part
+	 * @return String
+	 * @param $part Object
+	 */
+	private function getPartBody(&$part) {
+		$body = '';
+		if ($this->stream) {
+			$body = $this->getPartBodyFromFile($part);
+		} else if ($this->data) {
+			$body = $this->getPartBodyFromText($part);
+		} else {
+			throw new Exception('MimeMailParser::setPath() or MimeMailParser::setText() must be called before retrieving email parts.');
+		}
+		return $body;
+	}
+	
+	/**
+	 * Retrieve the Header from a MIME part from file
+	 * @return String Mime Header Part
+	 * @param $part Array
+	 */
+	private function getPartHeaderFromFile(&$part) {
+		$start = $part['starting-pos'];
+		$end = $part['starting-pos-body'];
+		fseek($this->stream, $start, SEEK_SET);
+		$header = fread($this->stream, $end-$start);
+		return $header;
+	}
+	/**
+	 * Retrieve the Body from a MIME part from file
+	 * @return String Mime Body Part
+	 * @param $part Array
+	 */
+	private function getPartBodyFromFile(&$part) {
+		$start = $part['starting-pos-body'];
+		$end = $part['ending-pos-body'];
+		fseek($this->stream, $start, SEEK_SET);
+		$body = fread($this->stream, $end-$start);
+		return $body;
+	}
+	
+	/**
+	 * Retrieve the Header from a MIME part from text
+	 * @return String Mime Header Part
+	 * @param $part Array
+	 */
+	private function getPartHeaderFromText(&$part) {
+		$start = $part['starting-pos'];
+		$end = $part['starting-pos-body'];
+		$header = substr($this->data, $start, $end-$start);
+		return $header;
+	}
+	/**
+	 * Retrieve the Body from a MIME part from text
+	 * @return String Mime Body Part
+	 * @param $part Array
+	 */
+	private function getPartBodyFromText(&$part) {
+		$start = $part['starting-pos-body'];
+		$end = $part['ending-pos-body'];
+		$body = substr($this->data, $start, $end-$start);
+		return $body;
+	}
+	
+	/**
+	 * Read the attachment Body and save temporary file resource
+	 * @return String Mime Body Part
+	 * @param $part Array
+	 */
+	private function getAttachmentStream(&$part) {
+		$temp_fp = tmpfile();   
+
+        array_key_exists('content-transfer-encoding', $part['headers']) ? $encoding = $part['headers']['content-transfer-encoding'] : $encoding = '';
+
+		if ($temp_fp) {
+			if ($this->stream) {
+				$start = $part['starting-pos-body'];
+				$end = $part['ending-pos-body'];
+				fseek($this->stream, $start, SEEK_SET);
+				$len = $end-$start;
+				$written = 0;
+				$write = 2028;
+				$body = '';
+				while($written < $len) {
+					if (($written+$write < $len )) {
+						$write = $len - $written;
+					}
+					$part = fread($this->stream, $write);
+					fwrite($temp_fp, $this->decode($part, $encoding));
+					$written += $write;
+				}
+			} else if ($this->data) {
+				$attachment = $this->decode($this->getPartBodyFromText($part), $encoding);
+				fwrite($temp_fp, $attachment, strlen($attachment));
+			}
+			fseek($temp_fp, 0, SEEK_SET);
+		} else {
+			throw new Exception('Could not create temporary files for attachments. Your tmp directory may be unwritable by PHP.');
+			return false;
+		}
+		return $temp_fp;
+	}
+
+    
+    /**
+     * Decode the string depending on encoding type.
+     * @return String the decoded string.
+     * @param $encodedString    The string in its original encoded state.
+     * @param $encodingType     The encoding type from the Content-Transfer-Encoding header of the part.
+     */
+    private function decode($encodedString, $encodingType) {
+        if (strtolower($encodingType) == 'base64') {
+        	return base64_decode($encodedString);
+        } else if (strtolower($encodingType) == 'quoted-printable') {
+        	 return quoted_printable_decode($encodedString);
+        } else {
+        	return $encodedString;
+        }
+    }
+
+}
+
+
+?>

externals/mimemailparser/README

@@ -0,0 +1,3 @@
+From:
+
+  http://code.google.com/p/php-mime-mail-parser/

externals/mimemailparser/attachment.class.php

@@ -0,0 +1,136 @@
+<?php
+
+/**
+ * Model of an Attachment
+ */
+class MimeMailParser_attachment {
+
+	/**
+	 * @var $filename Filename
+	 */
+	public  $filename;
+	/**
+	 * @var $content_type Mime Type
+	 */
+	public  $content_type;
+	/**
+	 * @var $content File Content
+	 */
+	private  $content;
+	/**
+	 * @var $extension Filename extension
+	 */
+	private $extension;
+	/**
+	 * @var $content_disposition Content-Disposition (attachment or inline)
+	 */
+	public $content_disposition;
+	/**
+	 * @var $headers An Array of the attachment headers
+	 */
+	public $headers;
+	
+	private  $stream;
+
+	public function __construct($filename, $content_type, $stream, $content_disposition = 'attachment', $headers = array()) {
+		$this->filename = $filename;
+		$this->content_type = $content_type;
+		$this->stream = $stream;
+		$this->content = null;
+		$this->content_disposition = $content_disposition;
+		$this->headers = $headers;
+	}
+	
+	/**
+	 * retrieve the attachment filename
+	 * @return String
+	 */
+	public function getFilename() {
+		return $this->filename;
+	}
+	
+	/**
+	 * Retrieve the Attachment Content-Type
+	 * @return String
+	 */
+	public function getContentType() {
+		return $this->content_type;
+	}
+	
+	/**
+	 * Retrieve the Attachment Content-Disposition
+	 * @return String
+	 */
+	public function getContentDisposition() {
+		return $this->content_disposition;
+	}
+	
+	/**
+	 * Retrieve the Attachment Headers
+	 * @return String
+	 */
+	public function getHeaders() {
+		return $this->headers;
+	}
+	
+	/**
+	 * Retrieve the file extension
+	 * @return String
+	 */
+	public function getFileExtension() {
+		if (!$this->extension) {
+			$ext = substr(strrchr($this->filename, '.'), 1);
+			if ($ext == 'gz') {
+				// special case, tar.gz
+				// todo: other special cases?
+				$ext = preg_match("/\.tar\.gz$/i", $ext) ? 'tar.gz' : 'gz';
+			}
+			$this->extension = $ext;
+		}
+		return $this->extension;
+	}
+	
+	/**
+	 * Read the contents a few bytes at a time until completed
+	 * Once read to completion, it always returns false
+	 * @return String
+	 * @param $bytes Int[optional]
+	 */
+	public function read($bytes = 2082) {
+		return feof($this->stream) ? false : fread($this->stream, $bytes);
+	}
+	
+	/**
+	 * Retrieve the file content in one go
+	 * Once you retreive the content you cannot use MimeMailParser_attachment::read()
+	 * @return String
+	 */
+	public function getContent() {
+		if ($this->content === null) {
+			fseek($this->stream, 0);
+			while(($buf = $this->read()) !== false) { 
+				$this->content .= $buf; 
+			}
+		}
+		return $this->content;
+	}
+	
+	/**
+	 * Allow the properties 
+	 * 	MimeMailParser_attachment::$name,
+	 * 	MimeMailParser_attachment::$extension 
+	 * to be retrieved as public properties
+	 * @param $name Object
+	 */
+	public function __get($name) {
+		if ($name == 'content') {
+			return $this->getContent();
+		} else if ($name == 'extension') {
+			return $this->getFileExtension();
+		}
+		return null;
+	}
+	
+}
+
+?>

resources/sql/patches/036.mailkey.sql

@@ -0,0 +1,19 @@
+ALTER TABLE phabricator_differential.differential_revision
+  ADD mailKey VARCHAR(40) binary NOT NULL;
+
+ALTER TABLE phabricator_maniphest.maniphest_task
+  ADD mailKey VARCHAR(40) binary NOT NULL;
+
+CREATE TABLE phabricator_metamta.metamta_receivedmail (
+  id int unsigned not null primary key auto_increment,
+  headers longblob not null,
+  bodies longblob not null,
+  attachments longblob not null,
+  relatedPHID varchar(64) binary,
+  key(relatedPHID),
+  authorPHID varchar(64) binary,
+  key(authorPHID),
+  message longblob,
+  dateCreated int unsigned not null,
+  dateModified int unsigned not null
+) engine=innodb;

scripts/mail/mail_handler.php

@@ -0,0 +1,54 @@
+#!/usr/bin/php
+<?php
+
+/*
+ * Copyright 2011 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$root = dirname(dirname(dirname(__FILE__)));
+require_once $root.'/scripts/__init_script__.php';
+require_once $root.'/scripts/__init_env__.php';
+require_once $root.'/externals/mimemailparser/MimeMailParser.class.php';
+
+phutil_require_module(
+  'phabricator',
+  'applications/metamta/storage/receivedmail');
+phutil_require_module(
+  'phabricator',
+  'applications/files/storage/file');
+
+$parser = new MimeMailParser();
+$parser->setText(file_get_contents('php://stdin'));
+
+$received = new PhabricatorMetaMTAReceivedMail();
+$received->setHeaders($parser->getHeaders());
+$received->setBodies(array(
+  'text' => $parser->getMessageBody('text'),
+  'html' => $parser->getMessageBody('html'),
+));
+
+$attachments = array();
+foreach ($received->getAttachments() as $attachment) {
+  $file = PhabricatorFile::newFromFileData(
+    $attachment->getContent(),
+    array(
+      'name' => $attachment->getFilename(),
+    ));
+  $attachments[] = $file->getPHID();
+}
+$received->setAttachments($attachments);
+$received->save();
+
+$received->processReceivedMail();

src/__phutil_library_map__.php

@@ -334,6 +334,9 @@ phutil_register_library_map(array(
     'PhabricatorMetaMTAMailingList' => 'applications/metamta/storage/mailinglist',
     'PhabricatorMetaMTAMailingListEditController' => 'applications/metamta/controller/mailinglistedit',
     'PhabricatorMetaMTAMailingListsController' => 'applications/metamta/controller/mailinglists',
+    'PhabricatorMetaMTAReceiveController' => 'applications/metamta/controller/receive',
+    'PhabricatorMetaMTAReceivedListController' => 'applications/metamta/controller/receivedlist',
+    'PhabricatorMetaMTAReceivedMail' => 'applications/metamta/storage/receivedmail',
     'PhabricatorMetaMTASendController' => 'applications/metamta/controller/send',
     'PhabricatorMetaMTAViewController' => 'applications/metamta/controller/view',
     'PhabricatorOAuthDefaultRegistrationController' => 'applications/auth/controller/oauthregistration/default',
@@ -743,6 +746,9 @@ phutil_register_library_map(array(
     'PhabricatorMetaMTAMailingList' => 'PhabricatorMetaMTADAO',
     'PhabricatorMetaMTAMailingListEditController' => 'PhabricatorMetaMTAController',
     'PhabricatorMetaMTAMailingListsController' => 'PhabricatorMetaMTAController',
+    'PhabricatorMetaMTAReceiveController' => 'PhabricatorMetaMTAController',
+    'PhabricatorMetaMTAReceivedListController' => 'PhabricatorMetaMTAController',
+    'PhabricatorMetaMTAReceivedMail' => 'PhabricatorMetaMTADAO',
     'PhabricatorMetaMTASendController' => 'PhabricatorMetaMTAController',
     'PhabricatorMetaMTAViewController' => 'PhabricatorMetaMTAController',
     'PhabricatorOAuthDefaultRegistrationController' => 'PhabricatorOAuthRegistrationController',

src/aphront/default/configuration/AphrontDefaultApplicationConfiguration.php

@@ -120,6 +120,8 @@ class AphrontDefaultApplicationConfiguration
         'lists/$' => 'PhabricatorMetaMTAMailingListsController',
         'lists/edit/(?:(?P<id>\d+)/)?$'
           => 'PhabricatorMetaMTAMailingListEditController',
+        'receive/$' => 'PhabricatorMetaMTAReceiveController',
+        'received/$' => 'PhabricatorMetaMTAReceivedListController',
       ),
 
       '/login/' => array(

src/applications/differential/storage/revision/DifferentialRevision.php

@@ -35,6 +35,8 @@ class DifferentialRevision extends DifferentialDAO {
   protected $attached = array();
   protected $unsubscribed = array();
 
+  protected $mailKey;
+
   private $relationships;
   private $commits;
 
@@ -115,6 +117,13 @@ class DifferentialRevision extends DifferentialDAO {
       $this->getID());
   }
 
+  public function save() {
+    if (!$this->getMailKey()) {
+      $this->mailKey = sha1(Filesystem::readRandomBytes(20));
+    }
+    return parent::save();
+  }
+
   public function loadRelationships() {
     if (!$this->getID()) {
       $this->relationships = array();

src/applications/differential/storage/revision/__init__.php

@@ -13,6 +13,7 @@ phutil_require_module('phabricator', 'applications/phid/constants');
 phutil_require_module('phabricator', 'applications/phid/storage/phid');
 phutil_require_module('phabricator', 'storage/queryfx');
 
+phutil_require_module('phutil', 'filesystem');
 phutil_require_module('phutil', 'utils');
 
 

src/applications/maniphest/storage/task/ManiphestTask.php

@@ -29,6 +29,8 @@ class ManiphestTask extends ManiphestDAO {
   protected $title;
   protected $description;
 
+  protected $mailKey;
+
   protected $attached = array();
   protected $projectPHIDs = array();
 
@@ -56,4 +58,11 @@ class ManiphestTask extends ManiphestDAO {
     return nonempty($this->ccPHIDs, array());
   }
 
+  public function save() {
+    if (!$this->mailKey) {
+      $this->mailKey = sha1(Filesystem::readRandomBytes(20));
+    }
+    return parent::save();
+  }
+
 }

src/applications/maniphest/storage/task/__init__.php

@@ -10,6 +10,7 @@ phutil_require_module('phabricator', 'applications/maniphest/storage/base');
 phutil_require_module('phabricator', 'applications/phid/constants');
 phutil_require_module('phabricator', 'applications/phid/storage/phid');
 
+phutil_require_module('phutil', 'filesystem');
 phutil_require_module('phutil', 'utils');
 
 

src/applications/metamta/controller/base/PhabricatorMetaMTAController.php

@@ -34,6 +34,10 @@ abstract class PhabricatorMetaMTAController extends PhabricatorController {
           'name' => 'Mailing Lists',
           'href' => '/mail/lists/',
         ),
+        'received' => array(
+          'name' => 'Received',
+          'href' => '/mail/received/',
+        ),
       ),
       idx($data, 'tab'));
     $page->setGlyph("@");

src/applications/metamta/controller/receive/PhabricatorMetaMTAReceiveController.php

@@ -0,0 +1,91 @@
+<?php
+
+/*
+ * Copyright 2011 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+class PhabricatorMetaMTAReceiveController
+  extends PhabricatorMetaMTAController {
+
+  public function processRequest() {
+
+    $request = $this->getRequest();
+    $user = $request->getUser();
+
+    if ($request->isFormPost()) {
+      $receiver = PhabricatorMetaMTAReceivedMail::loadReceiverObject(
+        $request->getStr('obj'));
+      if (!$receiver) {
+        throw new Exception("No such task or revision!");
+      }
+
+      $hash = PhabricatorMetaMTAReceivedMail::computeMailHash(
+        $receiver,
+        $user);
+
+      $received = new PhabricatorMetaMTAReceivedMail();
+      $received->setHeaders(
+        array(
+          'to' => $request->getStr('obj').'+'.$user->getID().'+'.$hash.'@',
+        ));
+      $received->setBodies(
+        array(
+          'text' => $request->getStr('body'),
+        ));
+      $received->save();
+
+      $received->processReceivedMail();
+
+      $phid = $receiver->getPHID();
+      $handles = id(new PhabricatorObjectHandleData(array($phid)))
+        ->loadHandles();
+      $uri = $handles[$phid]->getURI();
+
+      return id(new AphrontRedirectResponse())->setURI($uri);
+    }
+
+    $form = new AphrontFormView();
+    $form->setUser($request->getUser());
+    $form->setAction('/mail/receive/');
+    $form
+      ->appendChild(
+        '<p class="aphront-form-instructions">This form will simulate '.
+        'sending mail to an object.</p>')
+      ->appendChild(
+        id(new AphrontFormTextControl())
+          ->setLabel('To')
+          ->setName('obj')
+          ->setCaption('e.g. <tt>D1234</tt> or <tt>T1234</tt>'))
+      ->appendChild(
+        id(new AphrontFormTextAreaControl())
+          ->setLabel('Body')
+          ->setName('body'))
+      ->appendChild(
+        id(new AphrontFormSubmitControl())
+          ->setValue('Receive Mail'));
+
+    $panel = new AphrontPanelView();
+    $panel->setHeader('Receive Email');
+    $panel->appendChild($form);
+    $panel->setWidth(AphrontPanelView::WIDTH_WIDE);
+
+    return $this->buildStandardPageResponse(
+      $panel,
+      array(
+        'title' => 'Receive Mail',
+      ));
+  }
+
+}

src/applications/metamta/controller/receive/__init__.php

@@ -0,0 +1,22 @@
+<?php
+/**
+ * This file is automatically generated. Lint this module to rebuild it.
+ * @generated
+ */
+
+
+
+phutil_require_module('phabricator', 'aphront/response/redirect');
+phutil_require_module('phabricator', 'applications/metamta/controller/base');
+phutil_require_module('phabricator', 'applications/metamta/storage/receivedmail');
+phutil_require_module('phabricator', 'applications/phid/handle/data');
+phutil_require_module('phabricator', 'view/form/base');
+phutil_require_module('phabricator', 'view/form/control/submit');
+phutil_require_module('phabricator', 'view/form/control/text');
+phutil_require_module('phabricator', 'view/form/control/textarea');
+phutil_require_module('phabricator', 'view/layout/panel');
+
+phutil_require_module('phutil', 'utils');
+
+
+phutil_require_source('PhabricatorMetaMTAReceiveController.php');

src/applications/metamta/controller/receivedlist/PhabricatorMetaMTAReceivedListController.php

@@ -0,0 +1,93 @@
+<?php
+
+/*
+ * Copyright 2011 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+class PhabricatorMetaMTAReceivedListController
+  extends PhabricatorMetaMTAController {
+
+  public function processRequest() {
+
+    $request = $this->getRequest();
+
+    $pager = new AphrontPagerView();
+    $pager->setOffset($request->getInt('page'));
+    $pager->setURI($request->getRequestURI(), 'page');
+
+    $mails = id(new PhabricatorMetaMTAReceivedMail())->loadAllWhere(
+      '1 = 1 ORDER BY id DESC LIMIT %d, %d',
+      $pager->getOffset(),
+      $pager->getPageSize() + 1);
+    $mails = $pager->sliceResults($mails);
+
+    $phids = array_merge(
+      mpull($mails, 'getAuthorPHID'),
+      mpull($mails, 'getRelatedPHID')
+    );
+    $phids = array_unique(array_filter($phids));
+
+    $handles = id(new PhabricatorObjectHandleData($phids))->loadHandles();
+
+    $rows = array();
+    foreach ($mails as $mail) {
+      $rows[] = array(
+        $mail->getID(),
+        date('M jS Y', $mail->getDateCreated()),
+        date('g:i:s A', $mail->getDateCreated()),
+        $mail->getAuthorPHID()
+          ? $handles[$mail->getAuthorPHID()]->renderLink()
+          : '-',
+        $mail->getRelatedPHID()
+          ? $handles[$mail->getRelatedPHID()]->renderLink()
+          : '-',
+        phutil_escape_html($mail->getMessage()),
+      );
+    }
+
+    $table = new AphrontTableView($rows);
+    $table->setHeaders(
+      array(
+        'ID',
+        'Date',
+        'Time',
+        'Author',
+        'Object',
+        'Message',
+      ));
+    $table->setColumnClasses(
+      array(
+        null,
+        null,
+        'right',
+        null,
+        null,
+        'wide',
+      ));
+
+    $panel = new AphrontPanelView();
+    $panel->setHeader('Received Mail');
+    $panel->setCreateButton('Test Receiver', '/mail/receive/');
+    $panel->appendChild($table);
+    $panel->appendChild($pager);
+
+    return $this->buildStandardPageResponse(
+      $panel,
+      array(
+        'title' => 'Received Mail',
+        'tab'   => 'received',
+      ));
+  }
+}

src/applications/metamta/controller/receivedlist/__init__.php

@@ -0,0 +1,20 @@
+<?php
+/**
+ * This file is automatically generated. Lint this module to rebuild it.
+ * @generated
+ */
+
+
+
+phutil_require_module('phabricator', 'applications/metamta/controller/base');
+phutil_require_module('phabricator', 'applications/metamta/storage/receivedmail');
+phutil_require_module('phabricator', 'applications/phid/handle/data');
+phutil_require_module('phabricator', 'view/control/pager');
+phutil_require_module('phabricator', 'view/control/table');
+phutil_require_module('phabricator', 'view/layout/panel');
+
+phutil_require_module('phutil', 'markup');
+phutil_require_module('phutil', 'utils');
+
+
+phutil_require_source('PhabricatorMetaMTAReceivedListController.php');

src/applications/metamta/storage/receivedmail/PhabricatorMetaMTAReceivedMail.php

@@ -0,0 +1,152 @@
+<?php
+
+/*
+ * Copyright 2011 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+class PhabricatorMetaMTAReceivedMail extends PhabricatorMetaMTADAO {
+
+  protected $headers = array();
+  protected $bodies = array();
+  protected $attachments = array();
+
+  protected $relatedPHID;
+  protected $authorPHID;
+  protected $message;
+
+  public function getConfiguration() {
+    return array(
+      self::CONFIG_SERIALIZATION => array(
+        'headers'     => self::SERIALIZATION_JSON,
+        'bodies'      => self::SERIALIZATION_JSON,
+        'attachments' => self::SERIALIZATION_JSON,
+      ),
+    ) + parent::getConfiguration();
+  }
+
+  public function processReceivedMail() {
+    $to = idx($this->headers, 'to');
+
+    $matches = null;
+    $ok = preg_match(
+      '/^((?:D|T)\d+)\+(\d+)\+([a-f0-9]{16})@/',
+      $to,
+      $matches);
+
+    if (!$ok) {
+      return $this->setMessage("Unrecognized 'to' format: {$to}")->save();
+    }
+
+    $receiver_name = $matches[1];
+    $user_id = $matches[2];
+    $hash = $matches[3];
+
+    $user = id(new PhabricatorUser())->load($user_id);
+    if (!$user) {
+      return $this->setMessage("Invalid user '{$user_id}'")->save();
+    }
+
+    $this->setAuthorPHID($user->getPHID());
+
+    $receiver = self::loadReceiverObject($receiver_name);
+    if (!$receiver) {
+      return $this->setMessage("Invalid object '{$receiver_name}'")->save();
+    }
+
+    $this->setRelatedPHID($receiver->getPHID());
+
+    $expect_hash = self::computeMailHash($receiver, $user);
+    if ($expect_hash != $hash) {
+      return $this->setMessage("Invalid mail hash!")->save();
+    }
+
+    // TODO: Move this into the application logic instead.
+    if ($receiver instanceof ManiphestTask) {
+      $this->processManiphestMail($receiver, $user);
+    } else if ($receiver instanceof DifferentialRevision) {
+      $this->processDifferentialMail($receiver, $user);
+    }
+
+    $this->setMessage('OK');
+
+    return $this->save();
+  }
+
+  private function processManiphestMail(
+    ManiphestTask $task,
+    PhabricatorUser $user) {
+
+    // TODO: implement this
+
+  }
+
+  private function processDifferentialMail(
+    DifferentialRevision $revision,
+    PhabricatorUser $user) {
+
+    // TODO: Support actions
+
+    $editor = new DifferentialCommentEditor(
+      $revision,
+      $user->getPHID(),
+      DifferentialAction::ACTION_COMMENT);
+    $editor->setMessage($this->getCleanTextBody());
+    $editor->save();
+
+  }
+
+  private function getCleanTextBody() {
+    $body = idx($this->bodies, 'text');
+
+    // TODO: Detect quoted content and exclude it.
+
+    return $body;
+  }
+
+  public static function loadReceiverObject($receiver_name) {
+    if (!$receiver_name) {
+      return null;
+    }
+
+    $receiver_type = $receiver_name[0];
+    $receiver_id   = substr($receiver_name, 1);
+
+    $class_obj = null;
+    switch ($receiver_type) {
+      case 'T':
+        $class_obj = newv('ManiphestTask', array());
+        break;
+      case 'D':
+        $class_obj = newv('DifferentialRevision', array());
+        break;
+      default:
+        return null;
+    }
+
+    return $class_obj->load($receiver_id);
+  }
+
+  public static function computeMailHash(
+    $mail_receiver,
+    PhabricatorUser $user) {
+    $global_mail_key = PhabricatorEnv::getEnvConfig('phabricator.mail-key');
+    $local_mail_key = $mail_receiver->getMailKey();
+
+    $hash = sha1($local_mail_key.$global_mail_key.$user->getPHID());
+    return substr($hash, 0, 16);
+  }
+
+
+}

src/applications/metamta/storage/receivedmail/__init__.php

@@ -0,0 +1,18 @@
+<?php
+/**
+ * This file is automatically generated. Lint this module to rebuild it.
+ * @generated
+ */
+
+
+
+phutil_require_module('phabricator', 'applications/differential/constants/action');
+phutil_require_module('phabricator', 'applications/differential/editor/comment');
+phutil_require_module('phabricator', 'applications/metamta/storage/base');
+phutil_require_module('phabricator', 'applications/people/storage/user');
+phutil_require_module('phabricator', 'infrastructure/env');
+
+phutil_require_module('phutil', 'utils');
+
+
+phutil_require_source('PhabricatorMetaMTAReceivedMail.php');

src/docs/configuring_inbound_email.diviner

@@ -0,0 +1,65 @@
+@title Configuring Inbound Email
+@group config
+
+This document contains instructions for configuring inbound email, so users
+may update Differential and Maniphest by replying to messages.
+
+= Preamble =
+
+This is extremely difficult to configure correctly. This is doubly true if
+you use sendmail.
+
+= Installing Mailparse =
+
+You need to install the PECL mailparse extension. In theory, you can do that
+with:
+
+  $ sudo pecl install mailparse
+
+You may run into an error like "needs mbstring". If so, try:
+
+  $ sudo yum install php-mbstring # or equivalent
+  $ sudo pecl install -n mailparse
+
+If you get a linker error like this:
+
+  COUNTEREXAMPLE
+  PHP Warning:  PHP Startup: Unable to load dynamic library
+  '/usr/lib64/php/modules/mailparse.so' - /usr/lib64/php/modules/mailparse.so:
+  undefined symbol: mbfl_name2no_encoding in Unknown on line 0
+
+...you need to edit your php.ini file so that mbstring.so is loaded **before**
+mailparse.so. This is not the default if you have individual files in
+##php.d/##.
+
+= Configuring Sendmail =
+
+Sendmail is very difficult to configure. First, you need to configure it for
+your domain so that mail can be delievered correctly. In broad strokes, this
+probably means something like this:
+
+  - add an MX record;
+  - make sendmail listen on external interfaces;
+  - open up port 25 if necessary (e.g., in your EC2 security policy);
+  - add your host to /etc/mail/local-host-names; and
+  - restart sendmail.
+
+Now, you can actually configure sendmail to deliver to Phabricator. In
+##/etc/aliases##, add an entry like this:
+
+  phabricator: "| PHABRICATOR_ENV=<ENV> /path/to/phabricator/scripts/mail/mail_handler.php"
+
+...where <ENV> is the PHABRICATOR_ENV the script should run under. Run
+##sudo newaliases##. Now you likely need to symlink this script into
+##/etc/smrsh/##:
+
+  sudo ln -s /path/to/phabricator/scripts/mail/mail_handler.php /etc/smrsh/
+
+Finally, edit ##/etc/mail/virtusertable## and add an entry like this:
+
+  @yourdomain.com	phabricator@localhost
+
+That will forward all mail to @yourdomain.com to the Phabricator processing
+script. Run ##sudo /etc/mail/make## or similar and then restart sendmail with
+##sudo /etc/init.d/sendmail restart##
+