Summary:
Fixes T12844. This code is misleading: the daemon insert is happening on a different connection, and is not inside the transaction on the Mail connection.
What actually happens is this:
- (Connection A) `BEGIN`
- (Connection A) `INSERT INTO mail ...`
- (Connection B) `INSERT INTO worker ...` <-- This is a different connection, and it is NOT in a transaction!
- There's a race window here: the worker row is globally visible but the mail row is still isolated inside the transaction.
- (Connection A) `COMMIT`
- Now we're clear: the mail row is globally visible.
Change this code to reflect what's actually happening.
This means that if the worker row insert fails for some reason, we'll now throw with a mail row written to the database. But this is fine: it doesn't send on its own (so it can't cause mail loops or anything) and it can be re-queued with `bin/mail resend` if necessary without too much trouble.
Test Plan: See T12844 for particulars. Made some comments on tasks, saw the daemons send mail.
Reviewers: chad, amckinley, jmeador
Reviewed By: jmeador
Maniphest Tasks: T12844
Differential Revision: https://secure.phabricator.com/D18124
Summary:
Fixes T12803. An install is having difficulty diagnosing mail failures, and one component is that permanent task failures aren't reaching the log.
It's reasonable to send these to the log even when "phd.verbose" is off. See T12803 for a rough review of when we generate these failrues today.
Test Plan:
- Faked some exceptions.
- Got a result in the log (P2058) with `phd.verbose` turned off.
Reviewers: chad, amckinley
Reviewed By: chad
Maniphest Tasks: T12803
Differential Revision: https://secure.phabricator.com/D18106
Summary: Try to dis-ambiguate various button types and colors. Moves `simple` to `phui-button-simple` and moves colors to `button-color`.
Test Plan: Grep for buttons still inline, UIExamples, PHUIX, Herald, and Email Preferences.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D18077
Summary:
We are submitting `epriestley (Evan Priestley) <noreply@meta.phacility.com>`, but should be submitting `"epriestley (Evan Priestley)" <noreply@meta.phacility.com>`.
Add the missing quotes.
Test Plan: Locally, this makes the API calls work against the Mailgun sandbox domain.
Reviewers: chad, amckinley
Reviewed By: chad, amckinley
Differential Revision: https://secure.phabricator.com/D17831
Summary: Fixes T12579. Unclear why the user ran this command.
Test Plan: Ran with `--id cat`. Ran with `--id 123`.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12579
Differential Revision: https://secure.phabricator.com/D17719
Summary: Ref T12509. This encourages code to move away from HMAC+SHA1 by making the method name more obviously undesirable.
Test Plan: `grep`, browsed around.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12509
Differential Revision: https://secure.phabricator.com/D17632
Summary: Ref T10390. Simplifies dropdown by rolling out canUseInPanel in useless panels
Test Plan: Add a query panel, see less options.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T10390
Differential Revision: https://secure.phabricator.com/D17341
Summary:
Ref T12237. This adds a UI cue for users who have unverified primary addresses, since we no longer send them mail.
Also adds a new `bin/mail unverify` to unverify an address (for example, because mail is bouncing).
Test Plan:
- Unverified my address, saw setup issue.
- Verified my address, no more setup issue.
{F2861820}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12237
Differential Revision: https://secure.phabricator.com/D17344
Summary:
Ref T12237. This tightens our delivery rules, which previously sent normal mail to unverified addresses:
- We sent general mail to unverified addresses so that you wouldn't miss anything between the time you sign up (or have an account created) and the time you verify your address. This was imagined as a slight convenience for users.
- We sent automatic reply mail to unverified addresses if they sent mail to us first, saying "we don't recognize that address". This was imagined as a convenience for users who accidentally send mail "From" the wrong address (personal vs work, for example).
I think both behaviors are probably a little better for users on the balance, but not having mail providers randomly shut us off without warning is better for me, personally -- so stop doing this stuff.
This creates a problem which we likely need to solve before the release is cut:
- On installs which do not require mail verification, mail to you will now mostly-silently be dropped if you never bothered to verify your address.
I'd like to solve this by adding some kind of per-user alert that says "We recently tried to send you some mail but you haven't verified your address.", and giving them links to verify the address and review the mail. I'll pursue this after restoring mail service to `secure.phabricator.com`.
Test Plan:
- Added a unit test.
- Unverified my address, sent mail, saw it get dropped.
- Reverified my address, sent mail, saw it go through.
- Verified that important mail (password reset, invite, confirm-this-address) either uses "Force Delivery" (skips this check) or "Raw To Addresses" (also skips this check).
- Verified that Phacility instance stuff is also covered: it uses the same invite flow.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12237
Differential Revision: https://secure.phabricator.com/D17329
Summary:
Fixes T9719. Currently, the Herald "Test Console" has a big `instanceof` thing, so new adapters (like a Calendar adapter, or third-party adapters) aren't available automatically. Instead, do a standard modular thing: load the available adapters, ask which ones can test the object the user selected, then let the user pick which one they want to move forward with.
Additionally, it isn't very clear that you can't test "commit hook" rules because they rely on push state which we don't really have a good way to simulate. When the user picks a commit, we now show them the "Hook" events, but the options are disabled and explain why they can not be selected.
Test Plan:
- Ran test rules for revisions, commits, mocks, tasks, wiki documents, questions, and outbound mail.
- Plugged in a commit, got a more-helpful choice screen explaining why you do a test run of hook rules.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9719
Differential Revision: https://secure.phabricator.com/D16360
Summary: Ref T10628. This moves everything else over. I'll clean up the cruft in the next diff.
Test Plan:
- Viewed Conduit API page, toggled tabs.
- Viewed Harbormaster build, toggled tabs.
- Viewed a Drydock lease, swapped tabs.
- Viewed a Drydock resource, swapped tabs.
- Viewed mail, swapped tabs.
- Grepped for `addPropertyList(...)`, looked for any remaining calls with a second argument.
- Also checked rSAAS for any calls, but we don't have anything there that uses tabs.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10628
Differential Revision: https://secure.phabricator.com/D16207
Summary:
Ref T11098. Mixture of issues here:
- Similar problem to D16112, where users with no settings at all could fail to fall back to the global defaults.
- I made `UserPreferencesQuery` responsible for building defaults instead to simplify this, since we have 4 or 5 callsites which need to do it and they aren't easily reducible.
- Handle cases where `metamta.one-mail-per-recipient` is off (and thus users can not have any custom settings) more explicitly.
- When `metamta.one-mail-per-recipient` is off, remove the "Email Format" panel for users only -- administrators can still access it in global preferences.
Test Plan:
- Deleted a user's preferences, changed globals, purged cache, made sure defaults reflected global defaults.
- Changed global mail tags, sent mail to the user, verified it was dropped in accordinace with global settings.
- Changed user's settings to get the mail instead, verified mail was sent.
- Toggled user's Re / Vary settings, verified mail subject lines reflected user settings.
- Disabled `metamta.one-mail-per-recipient`, verified user "Email Format" panel vanished.
- Edited "Email Format" in single-mail-mode in global prefs as an administrator.
- Sent more mail, verified mail respected new global settings.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T11098
Differential Revision: https://secure.phabricator.com/D16118
Summary: Ref T11098. There is no reason to maintain these as separate values now that they can be configured in global settings.
Test Plan:
- Hit and read setup issue.
- Fiddled with settings.
- I'll vet this more throughly in the next diff since I need to fix an issue with global defaults in mail and can explicitly test this at the same time.
Reviewers: chad
Reviewed By: chad
Subscribers: eadler
Maniphest Tasks: T11098
Differential Revision: https://secure.phabricator.com/D16117
Summary:
Ref T4103. Ref T10078. Currently, when a user misses a cache we just build it for them.
This is the behavior we want for the the viewer (so we don't have to build every cache up front if we don't actually need them), but not the right behavior for other users (since it allows performance problems to go undetected).
Make inline cache generation strict by default, then make sure all the things that rely on cache data request the correct data (well, all of the things identified by unit tests, at least: there might be some more stuff I haven't hit yet).
This fixes test failures in D16040, and backports a piece of that change.
Test Plan: Identified and then fixed failures with `arc unit --everything`.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4103, T10078
Differential Revision: https://secure.phabricator.com/D16042
Summary:
Ref T4103. This isn't completely perfect but should let us move forward without also expanding scope into "too much mail".
I split the existing "Mail Preferences" into two panels: a "Mail Delivery" panel for the EditEngine settings, and a "2000000 dropdowns" panel for the two million dropdowns. This one retains the old code more or less unmodified.
Test Plan:
- Ran unit tests, which cover most of this stuff.
- Grepped for all removed constants.
- Ran migrations, inspected database results.
- Changed settings in both modified panels.
- This covers a lot of ground, but anything I missed will hopefully be fairly obvious.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4103
Differential Revision: https://secure.phabricator.com/D16038
Summary:
Ref T4103. These are currently stored on the user, for historic/performance reasons.
Since I want administrators to be able to set defaults for translations and timezones at a minimum and there's no longer a meaningful performance penalty for moving them off the user record, turn them into real preferences and then nuke the columns.
Test Plan:
- Set settings to unusual values.
- Ran migrations.
- Verified my unusual settings survived.
- Created a new user.
- Edited all settings with old and new UIs.
- Reconciled client/server timezone disagreement.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4103
Differential Revision: https://secure.phabricator.com/D16005
Summary:
Ref T4103. This give preferences a PHID, policy/transaction interfaces, a transaction table, and a Query class.
This doesn't actually change how they're edited, yet.
Test Plan:
- Ran migrations.
- Inspected database for date created, date modified, PHIDs.
- Changed some of my preferences.
- Deleted a user's preferences, verified they reset properly.
- Set some preferences as a new user, got a new row.
- Destroyed a user, verified their preferences were destroyed.
- Sent Conpherence messages.
- Send mail.
- Tried to edit another user's settings.
- Tried to edit a bot's settings as a non-admin.
- Edited a bot's settings as an admin (technically, none of the editable settings are actually stored in the settings table, currently).
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4103
Differential Revision: https://secure.phabricator.com/D15991
Summary: Ref T10694. Switch default mode to HTML since it has a number of significant advantages and we haven't seen reports of significant problems.
Test Plan:
- Switched preference to default (saw "HTML" in UI).
- Sent myself some mail.
- Got HTML mail.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10694
Differential Revision: https://secure.phabricator.com/D15885
Summary:
Ref T10939. Fixes T7834.
- Make packages into mailable objects, like projects and users.
- Packages resolve recipients by resolving project and user owners into recipients.
Test Plan:
- Added a comment to a revision with a package subscriber.
- Used `bin/mail show-outbound` to see that owners got mail.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T7834, T10939
Differential Revision: https://secure.phabricator.com/D15912
Summary:
Ref T10694. Ref T9790. When generating inline diff context, highlight it and then mangle the highlighted output into `style="..."` so it works in HTML.
Also try to tighten up some spacing/formatting stuff.
Test Plan:
Got some output in this vein:
{F1259937}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9790, T10694
Differential Revision: https://secure.phabricator.com/D15852
Summary: Swaps over to new hotness
Test Plan: Pull up mail view, see new UI
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15616
Summary:
Ref T10709. Two issues:
- If a user sends an invalid `!command`, we can throw, which means we don't return HTTP 200. This makes Mailgun re-send the mail later.
- We don't parse headers of the modern API correctly, so the "Message-ID" failsafe doesn't work. Parse them correctly. I //believe// Mailgun's API changed at some point.
Test Plan:
This is difficult to test exhaustively in isolation. I used Mailgun's web tools to verify the format of the hook request, and faked some requests locally.
I'll keep an eye on this as it goes to production and make sure the fix is correct there.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10709
Differential Revision: https://secure.phabricator.com/D15575
Summary: Runs through Meta, cleaned up policies and editing email addresses to new UI
Test Plan: Set a new Email address for Maniphest, edit policies.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15569
Summary:
Ref T10537. For Nuance, I want to introduce new sources (like "GitHub" or "GitHub via Nuance" or something) but this needs to modularize eventually.
Split ContentSource apart so applications can add new content sources.
Test Plan:
This change has huge surface area, so I'll hold it until post-release. I think it's fairly safe (and if it does break anything, the breaks should be fatals, not anything subtle or difficult to fix), there's just no reason not to hold it for a few hours.
- Viewed new module page.
- Grepped for all removed functions/constants.
- Viewed some transactions.
- Hovered over timestamps to get content source details.
- Added a comment via Conduit.
- Added a comment via web.
- Ran `bin/storage upgrade --namespace XXXXX --no-quickstart -f` to re-run all historic migrations.
- Generated some objects with `bin/lipsum`.
- Ran a bulk job on some tasks.
- Ran unit tests.
{F1190182}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10537
Differential Revision: https://secure.phabricator.com/D15521
Summary: Fixes T10539. When building mail targets, we fail to preserve whether a recipient was originally "To" or "Cc", and just move everyone to "To".
Test Plan:
Added a comment to a task with a "To" user and a "Cc" user, with `metamta.placeholder-to-recipient` set and `metamta.one-mail-per-recipient` set.
Got mail with me Cc'd as the Cc'd user:
{F1172020}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10539
Differential Revision: https://secure.phabricator.com/D15465
Summary: This inverts colors and icons a bit, so they're not as harsh. So instead of a dark green item with white icon, its now light green with a dark green icon. I've also changed all text and comment boxes to be "grey" visually to separate out the UI from converation/actions. Give it a spin and let me know how this feels. I still need to update the comment UI.
Test Plan:
UIExamples, lots of various tasks and diffs.
{F1163837}
{F1163839}
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15442
Summary: Mostly for consistency, we're not using other forms of icons and this makes all classes that use an icon call it in the same way.
Test Plan: tested uiexamples, lots of other random pages.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15125
Summary:
Ref T10054. Ref T6113. Users can currently subscribe to projects, which causes them to receive:
# mail about project membership changes, description changes, etc; and
# mail to the project, e.g. when the project is added as a subscriber on a task, or a reviewer on a revision.
Almost no one cares about (1), and after D15061 you can use Herald to get this stuff if you really want it. (It will get progressively more annoying in the future with external membership sources causing automated project membership updates.)
A lot of users are confused about (2) and how it relates to membership, watching, etc, and most users who want (2) don't want (1).
Instead, add an explicit option for this and explain what it does.
This is fairly verbose but I've hidden it on the member/watch screen, which is now the "explain how projects work" screen, I guess.
Test Plan:
{F1064929}
{F1064930}
{F1064931}
- Disabled/enabled mail for a project.
- Sent mail to a project with mail disabled, verified I didn't get a copy.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T6113, T10054
Differential Revision: https://secure.phabricator.com/D15065
Summary:
Ref T10121. If a user sends mail with duplicate headers, like:
```
X-Duplicate: A
X-Duplicate: B
```
...and we process it with `mail_handler.php`, we may end up with `array('A', 'B')` as the header value. When we try to write this back into an error response mail, it fails.
Test Plan:
- Generated a message with duplicate headers.
- Piped it into `mail_handler.php` with `--process-duplicates` and `--trace` to get a look at it.
- Faked an exception.
- Before patch: bad error email.
- After patch: clean error email showing multiple header values.
```
$ ./bin/mail show-outbound --id 12386
PROPERTIES
ID: 12386
Status: queued
Related PHID:
Message:
PARAMETERS
sensitive: 1
is-error: 1
force: 1
subject: Error Processing Mail (Exception)
raw-to: ["epriestley@phacility.com"]
HEADERS
X-Phabricator-Sent-This-Message: Yes
X-Mail-Transport-Agent: MetaMTA
X-Auto-Response-Suppress: All
TEXT BODY
Your email to Phabricator was not processed, because an error occurred while
trying to handle it:
Exception: TEST
-- Original Message Body -----------------------------------------------------
testy testy
-- Original Message Headers --------------------------------------------------
from: Evan Priestley <epriestley@phacility.com>
content-type: text/plain; charset=us-ascii
content-transfer-encoding: 7bit
x-smtp-server: smtp.gmail.com:epriestley@phacility.com
subject: test outbound mail
message-id: 7isvptmllqvdvtdxthvdwzg3woj5au7csyuh3hopypjv6y6hqb32qm4bcrd4jtid
x-universally-unique-identifier: 4E489E20-F674-49B2-94BA-0DE44F504EAA
date: Mon, 11 Jan 2016 09:50:12 -0800
date: Mon, 11 Jan 2016 09:50:13 -0800
date: Mon, 11 Jan 2016 09:50:14 -0800
date: Mon, 11 Jan 2016 09:50:15 -0800
to: epriestley@yghe.net
mime-version: 1.0 (Mac OS X Mail 8.2 \(2104\))
HTML BODY
(This message has no HTML body.)
```
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10121
Differential Revision: https://secure.phabricator.com/D14996
Summary:
Fixes T9156.
- Fix hashtag generation.
- Fix various badnesses.
- Improve project name generator.
Test Plan:
```
$ ./bin/lipsum generate projects
GENERATORS Selected generators: Projects.
WARNING This command generates synthetic test data, including user accounts. It is intended for use in development environments so you can test features more easily. There is no easy way to delete this data or undo the effects of this command. If you run it in a production environment, it will pollute your data with large amounts of meaningless garbage that you can not get rid of.
Are you sure you want to generate piles of garbage? [y/N] y
LIPSUM Generating synthetic test objects forever. Use ^C to stop when satisfied.
Generated "Project": Self-Flying Data Center Swag Performance
Generated "Project": Optimize Cars
Generated "Project": Triaging Culture Optimization
Generated "Project": Automating Experience
Generated "Project": Accelerating NUX Performance
Generated "Project": Optimizing Culture Optimization
Generated "Project": Optimize Hardware
```
{F1042949}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9156
Differential Revision: https://secure.phabricator.com/D14874
Summary: Via HackerOne. See D14025. I missed this comparison when making the original change.
Test Plan:
- Used `cat mail.txt | scripts/mail/mail_handler.php --process-duplicates` to pipe mail in a whole lot of times.
- Tried bad hashes, saw rejections.
- Tried good hash, saw mail accepted.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14455
Summary:
We don't use these for anything, we're inconsistent about recording them, and there's some mild interaction with privacy concerns and data retention. Every other log we store any kind of information in can be given a custom retention policy after recent GC changes.
If we did put this back eventually it would probably be better to store a session identifier anyway, since that's more granular and more detailed.
You can fetch this info out of access logs anyway, too.
Test Plan: Left a couple of comments.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14315
Summary: Updates metamta for handleRequest
Test Plan: Unable to test this, but looks safe?
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14256
Summary:
Fixes T9494. This:
- Removes all the random GC.x.y.z config.
- Puts it all in one place that's locked and which you use `bin/garbage set-policy ...` to adjust.
- Makes every TTL-based GC configurable.
- Simplifies the code in the actual GCs.
Test Plan:
- Ran `bin/garbage collect` to collect some garbage, until it stopped collecting.
- Ran `bin/garbage set-policy ...` to shorten policy. Saw change in web UI. Ran `bin/garbage collect` again and saw it collect more garbage.
- Set policy to indefinite and saw it not collect garabge.
- Set policy to default and saw it reflected in web UI / `collect`.
- Ran `bin/phd debug trigger` and saw all GCs fire with reasonable looking queries.
- Read new docs.
{F857928}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9494
Differential Revision: https://secure.phabricator.com/D14219
Summary: You can already pass other icons, but this makes it a bit simpler.
Test Plan: Test Maniphest, Badges
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14131
Summary: Fixes T9369.
Test Plan:
- Sent a mail with Mail.app to `bugs@local.phacility.com`.
- Used "View Raw Mail", copy-pasted it into `mail.txt` on disk.
- Ran `cat mail.txt | ./scripts/mail/manage_mail.php --process-duplicates`.
- Saw task get created and me get added as CC.
- Changed "To" to include another user, ran command again, saw task get created and other user get added as CC.
Reviewers: chad
Reviewed By: chad
Subscribers: Korvin
Maniphest Tasks: T9369
Differential Revision: https://secure.phabricator.com/D14086
Summary: See D14025. In all cases where we compare hashes, use strict, constant-time comparisons.
Test Plan: Logged in, logged out, added TOTP, ran Conduit, terminated sessions, submitted forms, changed password. Tweaked CSRF token, got rejected.
Reviewers: chad
Reviewed By: chad
Subscribers: chenxiruanhai
Differential Revision: https://secure.phabricator.com/D14026
Summary:
Email is so exciting I can't wait 30 days for initial results.
ref T9161
Test Plan:
* `./bin/mail volume --days 60` took longer and gave plausibly larger
results.
* `./bin/mail volume --days 0` quickly told me no mail had been sent.
* `./bin/mail volume` Said it was still looking 30 days back.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley
Maniphest Tasks: T9161
Differential Revision: https://secure.phabricator.com/D13901
Summary:
Fixes T9251. Old mail could get saved with bad parameters for two reasons that I can come up with:
- Nothing ever set a parameter on it -- not sure this could ever actually happen; or
- some field contained non-UTF8 data prior to D13939 and we silently failed to encode it.
My guess is that the second case is probably the culprit here.
In any case, recover from this so `20150622.metamta.5.actor-phid-mig.php` can proceed.
Test Plan: Same effective patch as user patch in T9251; looked at some mail to make sure it was still pulling parameters properly.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9251
Differential Revision: https://secure.phabricator.com/D13990
Summary: Ref T5791. This is still very basic (no global actions, no support for matching headers/bodies/recipients/etc) but gets the core in.
Test Plan:
{F715209}
{F715211}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T5791
Differential Revision: https://secure.phabricator.com/D13897
Summary: Ref T5791. This collects outbound mail status in one place and makes the list view a little spiffier.
Test Plan: Looked at list and detail views. Grepped for changed classes/constants.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T5791
Differential Revision: https://secure.phabricator.com/D13884
