Summary:
Uploaded an example email with a lot of accents called 'test_accents.mbox' and
expected headers in the file 'test_accents.headers.txt'.
Better than nothing.
This change also includes a minor refactor in the library loading.
Ref T15960
Test Plan:
Manually run the new unit test and see green lights:
arc unit src/applications/metamta/externals/__tests__/PhabricatorExternalMimeMailParserTestCase.php
Double-check that the new class is already recorded:
arc liberate
Just as extra care, re-apply the same test plan of:
D25839
So, for example, run this, and see no exceptions:
./scripts/mail/mail_handler.php < src/applications/metamta/externals/__tests__/data/test_accents.mbox
Reviewers: aklapper, taavi, O1 Blessed Committers
Reviewed By: aklapper, O1 Blessed Committers
Subscribers: tobiaswiese, Matthew, Cigaryno
Maniphest Tasks: T15960
Differential Revision: https://we.phorge.it/D25844
Summary:
rPa76444a8e238f647dc96f756e6c88aa2fafcdbfe updated our 13 year old copy of the mimemailparser library.
That included a behaviour change in the library not covered by Phorge code: The library now decodes MIME encoded UTF8 data in headers. Phorge passes that header to the `iconv_mime_decode()` PHP function which does not accept already encoded content.
```
EXCEPTION: (RuntimeException) iconv_mime_decode(): Detected an illegal character in input string at [<arcanist>/src/error/PhutilErrorHandler.php:273]
arcanist(head=master, ref.master=29ca3df1122b), phorge(head=master, ref.master=6ec5c88bee24)
#0 PhutilErrorHandler::handleError(integer, string, string, integer)
#1 iconv_mime_decode(string, integer, string) called at [<arcanist>/src/utils/utils.php:1759]
#2 phutil_decode_mime_header(string) called at [<phorge>/scripts/mail/mail_handler.php:64]
```
Closes T15960
Test Plan: * Have an email file called `tmp.mbox` with a UTF-8 encoded `From:` header. In `scripts/mail/mail_handler.php`, replace `file_get_contents('php://stdin')` with `file_get_contents('./tmp.mbox')`. Insert `echo $headers['subject']; echo "\n"; echo $headers['from'];` statements for debugging. Run `php ./mail_handler.php`.
Reviewers: O1 Blessed Committers, taavi, valerio.bozzolan
Reviewed By: O1 Blessed Committers, taavi, valerio.bozzolan
Subscribers: taavi, tobiaswiese, valerio.bozzolan, Matthew, Cigaryno
Maniphest Tasks: T15960
Differential Revision: https://we.phorge.it/D25839
Summary:
Bump to version 8.0.4 from 2024-09-11 per https://github.com/php-mime-mail-parser/php-mime-mail-parser/releases before this ancient code copy falls apart.
`scripts/mail/mail_handler.php` (used for incoming (!) mail) is the only consumer.
Closes T15940
Test Plan: Feed `mail_handler.php` with various test emails (formats: plain text, HTML, multipart; encodings: UTF-8, ASCII, ISO-8859-something) by manually replacing `php://stdin` with corresponding text files and adding some `phlog`s for output checking as I don't have mail server glue handy. Get only expected errors for broken emails.
Reviewers: O1 Blessed Committers, 20after4
Reviewed By: O1 Blessed Committers, 20after4
Subscribers: 20after4, tobiaswiese, valerio.bozzolan, Matthew, Cigaryno
Maniphest Tasks: T15940
Differential Revision: https://we.phorge.it/D25829
Summary:
* Properly handle when no mail headers at all can be parsed
* Properly handle when mail headers can be parsed but no subject line can be found
```
EXCEPTION: (RuntimeException) Undefined index: subject
```
Closes T15769
Test Plan: See T15769
Reviewers: O1 Blessed Committers, valerio.bozzolan
Reviewed By: O1 Blessed Committers, valerio.bozzolan
Subscribers: 20after4, tobiaswiese, valerio.bozzolan, Matthew, Cigaryno
Maniphest Tasks: T15769
Differential Revision: https://we.phorge.it/D25565
Summary: Depends on D20069. Ref T13232. This is a very, very weak dependency and we can reasonably polyfill it.
Test Plan: Grepped for `iconv` in libphutil, arcanist, and Phabricator.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13232
Differential Revision: https://secure.phabricator.com/D20070
Summary:
See D18776. See <https://discourse.phabricator-community.org/t/cant-create-maniphest-tasks-by-email/754/2>.
The change in D18776 to improve handling of non-utf8 HTML parts broke handling of mail with //no// HTML parts. Partly, this is because MimeMailParser has a "traditional" PHP-style API where the return type is an exciting surprise.
Test Plan:
- Sent a text-only message in `Mail.app`.
- Used "Show Raw" to copy it to `mail.txt`, verifying that the raw message contains ONLY a text body.
- Ran `cat mail.txt | ./scripts/mail/mail_handler.php --trace --process-duplicates`.
- Before patch: error about bad `idx()` on a non-array.
- After patch: clean mail processing.
- Did the same with a message with both HTML and text bodies to make sure I didn't break anything.
Ideally we'd probably get test coverage on this, but it's been touched roughly once a year since 2013 so it'll probably hold.
Reviewers: amckinley, alexmv
Reviewed By: amckinley, alexmv
Differential Revision: https://secure.phabricator.com/D18778
Summary:
D1093 did this for just the text/plain part of incoming
email. Most text/html parts choose to either use entity encoding
//or// are already UTF-8, thus obviating the need to transcode the
HTML part. However, this is not always the case, and leads to dropped
messages, by way of:
```
EXCEPTION: (Exception) Failed to JSON encode value (#5: Malformed UTF-8 characters, possibly incorrectly encoded): Dictionary value at key "html" is not valid UTF8, and cannot be JSON encoded: [snip HTML part of message content]```
Generalize the charset transcoding to not apply to just the text/plain part, but
both text/plain and text/html parts.
Test Plan:
Fed in a Windows-1252-encoded text/html part with 0x92
bytes in it; verified that $content only contained valid UTF-8 after
this change.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D18776
Summary: Use `PhutilClassMaQuery` instead of `PhutilSymbolLoader`, mostly for consistency. Depends on D13588.
Test Plan: Poked around a bunch of pages.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D13589
Summary:
Fixes T7712. Currently, files sent via email get default policies, like they were dragged and dropped onto the home page.
User expectation is better aligned with giving files more restrictive policies, like they were draggged and dropped directly onto an object.
Make files sent via email have restricted default visibility. Once we identify the sender, set them as the file author. Later, the file will become visible to other users via attachment to a task, revision, etc.
Test Plan: Sent some files via email; verified they got restrictive policies, correct authorship, and appropriate object attachment.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7712
Differential Revision: https://secure.phabricator.com/D12255
Summary: Ran `arc lint --apply-patches --everything` over rP, mainly to change double quotes to single quotes where appropriate. These changes also validate that the `ArcanistXHPASTLinter::LINT_DOUBLE_QUOTE` rule is working as expected.
Test Plan: Eyeballed it.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin, hach-que
Differential Revision: https://secure.phabricator.com/D9431
Summary:
Ref T2015. Not directly related to Drydock, but I bumped into this. All these scripts currently enumerate their workflows explicitly.
Instead, use `PhutilSymbolLoader` to automatically discover workflows. This reduces code duplication and errors (see all the bad `extends` this diff fixes) and lets third parties add new workflows (not clearly valuable?).
Test Plan: Ran `bin/x help` for each modified script.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2015
Differential Revision: https://secure.phabricator.com/D7840
Summary:
Ref T4107. Two issues:
- With strict MySQL settings, we try to insert `null` into the non-nullable `messageCount` field. Add an `initializeNew...` method.
- If we don't create a new conpherence (for example, because the message body is empty), we fatal on `getPHID()` right now.
Also, make this stuff a little easier to test.
Test Plan: Used `mail_handler.php` to receive empty conpherence mail, and new-thread conpherence mail.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4107
Differential Revision: https://secure.phabricator.com/D7760
Summary:
Ref T3306. Moves this from the web to the CLI, which is a tiny bit clunkier but way better as far as policies go and more repeatable for development.
See discussion in D6413.
Test Plan: Ran `bin/mail receive-test`, verified mail was received. Used and abused various options.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3306
Differential Revision: https://secure.phabricator.com/D6417
Summary: We can't show this stuff on the web UI because it has password reset links and private reply-to addresses, but we can provide easier CLI tools than "root around in the database". Land a rough version of `bin/mail show-inbound` and `bin/mail show-outbound`.
Test Plan: Used both commands to examine mail from the CLI.
Reviewers: btrahan
Reviewed By: btrahan
CC: tido, euresti, aran
Differential Revision: https://secure.phabricator.com/D5963
Summary:
We/I broke a couple of things here recently (see D5911) and are doing some work here in general (see D5912, etc.).
Generally, this code is pretty oldschool and not especially well architected for modern application-oriented Phabricator. It hardcodes a lot of stuff which should be applications' responsibilites.
Take the first steps toward making it more solid to reduce the risk here. In particular:
- Factor out the "self mail" and "duplicate mail" checks and add unit tests.
- Make Message-ID hash handling automatic.
Test Plan: Ran unit tests.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D5915
Summary:
Fixes T2458. Ref T2843. @tido's email from T2843 has exhausted its retries and failed, but we want to try it again with the patch from D5464 to capture the actual error. This sort of thing has come up a few times in debugging, too.
Also fixed some stuff that came up while debugging this.
Test Plan:
- Ran command with no args.
- Ran resend with no args.
- Ran resend with bad IDs.
- Ran resend with already-queued messages, got "already queued" error.
- Ran resend with already-sent message, got requeue.
Reviewers: btrahan, tido
Reviewed By: tido
CC: aran
Maniphest Tasks: T2458, T2843
Differential Revision: https://secure.phabricator.com/D5493
Summary: this can happen if you have Phabricator and email lists co-mingling such that Phabricator receives an email multiple times. we can prevent this from then spamming everyone or otherwise taking the action multiple times by storing a message id hash and dropping the message if we have more than one message that matches.
Test Plan: simulated sending the same email multiple times on the command line. noted only the first one made it through.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Maniphest Tasks: T1726
Differential Revision: https://secure.phabricator.com/D4328
Summary:
This commit doesn't change license of any file. It just makes the license implicit (inherited from LICENSE file in the root directory).
We are removing the headers for these reasons:
- It wastes space in editors, less code is visible in editor upon opening a file.
- It brings noise to diff of the first change of any file every year.
- It confuses Git file copy detection when creating small files.
- We don't have an explicit license header in other files (JS, CSS, images, documentation).
- Using license header in every file is not obligatory: http://www.apache.org/dev/apply-license.html#new.
This change is approved by Alma Chao (Lead Open Source and IP Counsel at Facebook).
Test Plan: Verified that the license survived only in LICENSE file and that it didn't modify externals.
Reviewers: epriestley, davidrecordon
Reviewed By: epriestley
CC: aran, Korvin
Maniphest Tasks: T2035
Differential Revision: https://secure.phabricator.com/D3886
Summary:
Currently, when taskmasters complete a task it is immediately deleted. This prevents us from doing some general things, like:
- Supporting the idea of permanent failure (e.g., after N failures just stop trying).
- Showing the user how fast taskmasters are completing tasks.
- Showing the user how long tasks took to complete.
Having better visibility into this is important to Drydock, which builds on the task system. Also, generally buff debug output for task execution.
Test Plan: Ran `bin/phd debug taskmaster`. Ran `bin/phd debug garbage`. Queued some tasks via various systems.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2015
Differential Revision: https://secure.phabricator.com/D3852
Summary: See D3252.
Test Plan: This one is nasty to test, I'm going to make some coffee first.
Reviewers: davidreuss, vrana, btrahan
Reviewed By: davidreuss
CC: aran
Maniphest Tasks: T452
Differential Revision: https://secure.phabricator.com/D3254
Summary:
When switching from using the MetaMTADaemon to a Taskmaster for sending mail,
if there are messages queued for delivery, they need to be re-queued into the
task system. This patch does that.
Task ID: #
Blame Rev:
Test Plan:
Ran it.
Revert Plan:
Tags:
Reviewers: epriestley, btrahan, jungejason
Reviewed By: epriestley
CC: aran, epriestley
Differential Revision: https://secure.phabricator.com/D1780
Summary:
If mails are not sent in UTF-8 we cannot just it verbatim, so we have to
encode it into UTF-8 if it is not the case. Mime headers use different
encodings like "quoted-printable", which we have to handle.
It looks like "Subject: =?iso-8859-1?Q?opr=E6t_s=E5_den_task?=", and can
be decoded by ##iconv_mime_decode##.
Furthermore the body of the email might be in various encodings as well,
which we attempt to pull from the content-type header of the plain text
part of the mail.
Test Plan:
Attempted receiving mails in a variety of flavors. These could be
converted to test-cases once i know if this is a sane solution. Got
expected results from mails sent with Windows-1252 and ISO-8859-1.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, epriestley, davidreuss
Differential Revision: 1093
Summary: by default it was looking for php to be in /usr/bin/php on some systems
this is incorrect. It should be using /usr/bin/env php for calling php with the
correct path.
Test Plan: patch and try to run.
Reviewers: epriestley
Reviewed By: epriestley
CC: svemir, aran, epriestley
Differential Revision: 1136
Summary:
This makes debugging issues a lot easier, since any exceptions thrown
gets logged and can be seen at /mail/received/, for inspection.
Test Plan:
Created a task via the public-author functionality, and saw no errors
without this patch applied. Even worse, the exception was never shown at
command line either. Not sure what's up with that. Output buffering, or
whatever?
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, epriestley, davidreuss
Differential Revision: 1042
Summary: There are currently two files, but all scripts require both of them,
which is clearly silly. In the longer term I want to rewrite all of this init
stuff to be more structured (e.g., merge webroot/index.php and __init_script__
better) but this reduces the surface area of the ad-hoc "include files" API we
have now, at least.
Test Plan:
- Grepped for __init_env__.php (no hits)
- Ran a unit test (to test unit changes)
- Ran a daemon (to test daemon changes)
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: jungejason
CC: aran, jungejason
Differential Revision: 976
Summary:
Mail clients can send messages where the body is represented as 'inline'
attachments. Don't treat any such text attachments as actual attachments.
Test Plan:
toulouse, can you verify this fixes the issue?
Reviewed By: toulouse
Reviewers: toulouse
CC: aran, toulouse, epriestley
Differential Revision: 441
Summary:
Allow files to be attached to a task by attaching them to an email reply to the
task.
Test Plan:
Applied this patch live since I haven't managed to get inbound email configured
locally, then attached files to a task via email.
Reviewed By: jungejason
Reviewers: tuomaspelkonen, jungejason, aran
CC: anjali, aran, jungejason
Differential Revision: 369
Summary:
Sendmail isn't actually OK with passing ENV stuff via 'aliases', accept it as an
argument instead.
Test Plan:
Sent real email to a real server, got differential updates!
Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, jungejason
Differential Revision: 233
Summary:
Provides support for per-user x per-object unique reply-to email addresses, plus
SMTP integration.
This does not actually make Phabricator use these in outbound email.
Test Plan:
Used test console to validate in-Phabricator routing and handling.
Piped emails into the "mail_handler.php" script to validate mail parsing.
Configured sendmail and sent mail to Phabricator.
Technically I haven't conducted all parts of this test on the same machine since
I lost the will to configure more SMTP servers after configuring phabricator.com
Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, epriestley, jungejason
Differential Revision: 226
