Summary:
Uploaded an example email with a lot of accents called 'test_accents.mbox' and
expected headers in the file 'test_accents.headers.txt'.
Better than nothing.
This change also includes a minor refactor in the library loading.
Ref T15960
Test Plan:
Manually run the new unit test and see green lights:
arc unit src/applications/metamta/externals/__tests__/PhabricatorExternalMimeMailParserTestCase.php
Double-check that the new class is already recorded:
arc liberate
Just as extra care, re-apply the same test plan of:
D25839
So, for example, run this, and see no exceptions:
./scripts/mail/mail_handler.php < src/applications/metamta/externals/__tests__/data/test_accents.mbox
Reviewers: aklapper, taavi, O1 Blessed Committers
Reviewed By: aklapper, O1 Blessed Committers
Subscribers: tobiaswiese, Matthew, Cigaryno
Maniphest Tasks: T15960
Differential Revision: https://we.phorge.it/D25844
Summary:
rPa76444a8e238f647dc96f756e6c88aa2fafcdbfe updated our 13 year old copy of the mimemailparser library.
That included a behaviour change in the library not covered by Phorge code: The library now decodes MIME encoded UTF8 data in headers. Phorge passes that header to the `iconv_mime_decode()` PHP function which does not accept already encoded content.
```
EXCEPTION: (RuntimeException) iconv_mime_decode(): Detected an illegal character in input string at [<arcanist>/src/error/PhutilErrorHandler.php:273]
arcanist(head=master, ref.master=29ca3df1122b), phorge(head=master, ref.master=6ec5c88bee24)
#0 PhutilErrorHandler::handleError(integer, string, string, integer)
#1 iconv_mime_decode(string, integer, string) called at [<arcanist>/src/utils/utils.php:1759]
#2 phutil_decode_mime_header(string) called at [<phorge>/scripts/mail/mail_handler.php:64]
```
Closes T15960
Test Plan: * Have an email file called `tmp.mbox` with a UTF-8 encoded `From:` header. In `scripts/mail/mail_handler.php`, replace `file_get_contents('php://stdin')` with `file_get_contents('./tmp.mbox')`. Insert `echo $headers['subject']; echo "\n"; echo $headers['from'];` statements for debugging. Run `php ./mail_handler.php`.
Reviewers: O1 Blessed Committers, taavi, valerio.bozzolan
Reviewed By: O1 Blessed Committers, taavi, valerio.bozzolan
Subscribers: taavi, tobiaswiese, valerio.bozzolan, Matthew, Cigaryno
Maniphest Tasks: T15960
Differential Revision: https://we.phorge.it/D25839
Summary:
Bump to version 8.0.4 from 2024-09-11 per https://github.com/php-mime-mail-parser/php-mime-mail-parser/releases before this ancient code copy falls apart.
`scripts/mail/mail_handler.php` (used for incoming (!) mail) is the only consumer.
Closes T15940
Test Plan: Feed `mail_handler.php` with various test emails (formats: plain text, HTML, multipart; encodings: UTF-8, ASCII, ISO-8859-something) by manually replacing `php://stdin` with corresponding text files and adding some `phlog`s for output checking as I don't have mail server glue handy. Get only expected errors for broken emails.
Reviewers: O1 Blessed Committers, 20after4
Reviewed By: O1 Blessed Committers, 20after4
Subscribers: 20after4, tobiaswiese, valerio.bozzolan, Matthew, Cigaryno
Maniphest Tasks: T15940
Differential Revision: https://we.phorge.it/D25829
Summary:
* Properly handle when no mail headers at all can be parsed
* Properly handle when mail headers can be parsed but no subject line can be found
```
EXCEPTION: (RuntimeException) Undefined index: subject
```
Closes T15769
Test Plan: See T15769
Reviewers: O1 Blessed Committers, valerio.bozzolan
Reviewed By: O1 Blessed Committers, valerio.bozzolan
Subscribers: 20after4, tobiaswiese, valerio.bozzolan, Matthew, Cigaryno
Maniphest Tasks: T15769
Differential Revision: https://we.phorge.it/D25565
Summary: Depends on D20069. Ref T13232. This is a very, very weak dependency and we can reasonably polyfill it.
Test Plan: Grepped for `iconv` in libphutil, arcanist, and Phabricator.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13232
Differential Revision: https://secure.phabricator.com/D20070
Summary:
See D18776. See <https://discourse.phabricator-community.org/t/cant-create-maniphest-tasks-by-email/754/2>.
The change in D18776 to improve handling of non-utf8 HTML parts broke handling of mail with //no// HTML parts. Partly, this is because MimeMailParser has a "traditional" PHP-style API where the return type is an exciting surprise.
Test Plan:
- Sent a text-only message in `Mail.app`.
- Used "Show Raw" to copy it to `mail.txt`, verifying that the raw message contains ONLY a text body.
- Ran `cat mail.txt | ./scripts/mail/mail_handler.php --trace --process-duplicates`.
- Before patch: error about bad `idx()` on a non-array.
- After patch: clean mail processing.
- Did the same with a message with both HTML and text bodies to make sure I didn't break anything.
Ideally we'd probably get test coverage on this, but it's been touched roughly once a year since 2013 so it'll probably hold.
Reviewers: amckinley, alexmv
Reviewed By: amckinley, alexmv
Differential Revision: https://secure.phabricator.com/D18778
Summary:
D1093 did this for just the text/plain part of incoming
email. Most text/html parts choose to either use entity encoding
//or// are already UTF-8, thus obviating the need to transcode the
HTML part. However, this is not always the case, and leads to dropped
messages, by way of:
```
EXCEPTION: (Exception) Failed to JSON encode value (#5: Malformed UTF-8 characters, possibly incorrectly encoded): Dictionary value at key "html" is not valid UTF8, and cannot be JSON encoded: [snip HTML part of message content]```
Generalize the charset transcoding to not apply to just the text/plain part, but
both text/plain and text/html parts.
Test Plan:
Fed in a Windows-1252-encoded text/html part with 0x92
bytes in it; verified that $content only contained valid UTF-8 after
this change.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D18776
Summary:
Fixes T7712. Currently, files sent via email get default policies, like they were dragged and dropped onto the home page.
User expectation is better aligned with giving files more restrictive policies, like they were draggged and dropped directly onto an object.
Make files sent via email have restricted default visibility. Once we identify the sender, set them as the file author. Later, the file will become visible to other users via attachment to a task, revision, etc.
Test Plan: Sent some files via email; verified they got restrictive policies, correct authorship, and appropriate object attachment.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7712
Differential Revision: https://secure.phabricator.com/D12255
Summary: Ran `arc lint --apply-patches --everything` over rP, mainly to change double quotes to single quotes where appropriate. These changes also validate that the `ArcanistXHPASTLinter::LINT_DOUBLE_QUOTE` rule is working as expected.
Test Plan: Eyeballed it.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin, hach-que
Differential Revision: https://secure.phabricator.com/D9431
Summary:
Ref T4107. Two issues:
- With strict MySQL settings, we try to insert `null` into the non-nullable `messageCount` field. Add an `initializeNew...` method.
- If we don't create a new conpherence (for example, because the message body is empty), we fatal on `getPHID()` right now.
Also, make this stuff a little easier to test.
Test Plan: Used `mail_handler.php` to receive empty conpherence mail, and new-thread conpherence mail.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4107
Differential Revision: https://secure.phabricator.com/D7760
Summary:
We/I broke a couple of things here recently (see D5911) and are doing some work here in general (see D5912, etc.).
Generally, this code is pretty oldschool and not especially well architected for modern application-oriented Phabricator. It hardcodes a lot of stuff which should be applications' responsibilites.
Take the first steps toward making it more solid to reduce the risk here. In particular:
- Factor out the "self mail" and "duplicate mail" checks and add unit tests.
- Make Message-ID hash handling automatic.
Test Plan: Ran unit tests.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D5915
Summary: this can happen if you have Phabricator and email lists co-mingling such that Phabricator receives an email multiple times. we can prevent this from then spamming everyone or otherwise taking the action multiple times by storing a message id hash and dropping the message if we have more than one message that matches.
Test Plan: simulated sending the same email multiple times on the command line. noted only the first one made it through.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Maniphest Tasks: T1726
Differential Revision: https://secure.phabricator.com/D4328
Summary:
This commit doesn't change license of any file. It just makes the license implicit (inherited from LICENSE file in the root directory).
We are removing the headers for these reasons:
- It wastes space in editors, less code is visible in editor upon opening a file.
- It brings noise to diff of the first change of any file every year.
- It confuses Git file copy detection when creating small files.
- We don't have an explicit license header in other files (JS, CSS, images, documentation).
- Using license header in every file is not obligatory: http://www.apache.org/dev/apply-license.html#new.
This change is approved by Alma Chao (Lead Open Source and IP Counsel at Facebook).
Test Plan: Verified that the license survived only in LICENSE file and that it didn't modify externals.
Reviewers: epriestley, davidrecordon
Reviewed By: epriestley
CC: aran, Korvin
Maniphest Tasks: T2035
Differential Revision: https://secure.phabricator.com/D3886
Summary: See D3252.
Test Plan: This one is nasty to test, I'm going to make some coffee first.
Reviewers: davidreuss, vrana, btrahan
Reviewed By: davidreuss
CC: aran
Maniphest Tasks: T452
Differential Revision: https://secure.phabricator.com/D3254
Summary:
If mails are not sent in UTF-8 we cannot just it verbatim, so we have to
encode it into UTF-8 if it is not the case. Mime headers use different
encodings like "quoted-printable", which we have to handle.
It looks like "Subject: =?iso-8859-1?Q?opr=E6t_s=E5_den_task?=", and can
be decoded by ##iconv_mime_decode##.
Furthermore the body of the email might be in various encodings as well,
which we attempt to pull from the content-type header of the plain text
part of the mail.
Test Plan:
Attempted receiving mails in a variety of flavors. These could be
converted to test-cases once i know if this is a sane solution. Got
expected results from mails sent with Windows-1252 and ISO-8859-1.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, epriestley, davidreuss
Differential Revision: 1093
Summary: by default it was looking for php to be in /usr/bin/php on some systems
this is incorrect. It should be using /usr/bin/env php for calling php with the
correct path.
Test Plan: patch and try to run.
Reviewers: epriestley
Reviewed By: epriestley
CC: svemir, aran, epriestley
Differential Revision: 1136
Summary:
This makes debugging issues a lot easier, since any exceptions thrown
gets logged and can be seen at /mail/received/, for inspection.
Test Plan:
Created a task via the public-author functionality, and saw no errors
without this patch applied. Even worse, the exception was never shown at
command line either. Not sure what's up with that. Output buffering, or
whatever?
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, epriestley, davidreuss
Differential Revision: 1042
Summary: There are currently two files, but all scripts require both of them,
which is clearly silly. In the longer term I want to rewrite all of this init
stuff to be more structured (e.g., merge webroot/index.php and __init_script__
better) but this reduces the surface area of the ad-hoc "include files" API we
have now, at least.
Test Plan:
- Grepped for __init_env__.php (no hits)
- Ran a unit test (to test unit changes)
- Ran a daemon (to test daemon changes)
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: jungejason
CC: aran, jungejason
Differential Revision: 976
Summary:
Mail clients can send messages where the body is represented as 'inline'
attachments. Don't treat any such text attachments as actual attachments.
Test Plan:
toulouse, can you verify this fixes the issue?
Reviewed By: toulouse
Reviewers: toulouse
CC: aran, toulouse, epriestley
Differential Revision: 441
Summary:
Allow files to be attached to a task by attaching them to an email reply to the
task.
Test Plan:
Applied this patch live since I haven't managed to get inbound email configured
locally, then attached files to a task via email.
Reviewed By: jungejason
Reviewers: tuomaspelkonen, jungejason, aran
CC: anjali, aran, jungejason
Differential Revision: 369
Summary:
Sendmail isn't actually OK with passing ENV stuff via 'aliases', accept it as an
argument instead.
Test Plan:
Sent real email to a real server, got differential updates!
Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, jungejason
Differential Revision: 233
Summary:
Provides support for per-user x per-object unique reply-to email addresses, plus
SMTP integration.
This does not actually make Phabricator use these in outbound email.
Test Plan:
Used test console to validate in-Phabricator routing and handling.
Piped emails into the "mail_handler.php" script to validate mail parsing.
Configured sendmail and sent mail to Phabricator.
Technically I haven't conducted all parts of this test on the same machine since
I lost the will to configure more SMTP servers after configuring phabricator.com
Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, epriestley, jungejason
Differential Revision: 226
