Summary:
Ref T6755. This is a partial fix, but:
- Allow netblocks to be blacklisted instead of making the feature all-or-nothing.
- Default to disallow requests to all reserved private/local/special IP blocks. This should generally be a "safe" setting.
- Explain the risks better.
- Improve the errors rasied by Macro when failing.
- Removed `security.allow-outbound-http`, as it is superseded by this setting and is somewhat misleading.
- We still make outbound HTTP requests to OAuth.
- We still make outbound HTTP requests for repositories.
From a technical perspective:
- Separate URIs that are safe to link to or redirect to (basically, not "javascript://") from URIs that are safe to fetch (nothing in a private block).
- Add the default blacklist.
- Be more careful with response data in Macro fetching, and don't let the user see it if it isn't ultimately valid.
Additionally:
- I want to do this check before pulling repositories, but that's enough of a mess that it should go in a separate diff.
- The future implementation of T4190 needs to perform the fetch check.
Test Plan:
- Fetched a valid macro.
- Fetched a non-image, verified it didn't result in a viewable file.
- Fetched a private-ip-space image, got an error.
- Fetched a 404, got a useful-enough error without additional revealing response content (which is usually HTML anyway and not useful).
- Fetched a bad protocol, got an error.
- Linked to a local resource, a phriction page, a valid remote site, all worked.
- Linked to private IP space, which worked fine (we want to let you link and redierect to other private services, just not fetch them).
- Added and executed unit tests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6755
Differential Revision: https://secure.phabricator.com/D12136
Summary: Fixes T7609. When we moved the sidenav background to z-index -1, the footer then reappeared. Make the hiding more specific in CSS.
Test Plan: test mobile layout of homepage, check footer still exists on all other pages.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7609
Differential Revision: https://secure.phabricator.com/D12138
Summary: Ref T7627. This centralizes this transaction construction code so the unit tests and Instances can both use it.
Test Plan: See D12116.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7627
Differential Revision: https://secure.phabricator.com/D12118
Summary: Fixes T7625. The way the regexp worked, "unusual" terminal characters required at least one character as a prefix in order to match. Allow 0 instead, so `#a1` matches.
Test Plan: Added and executed unit test.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7625
Differential Revision: https://secure.phabricator.com/D12123
Summary: Ref T7611. This should let us figure out the root cause, hopefully.
Test Plan: iiam
Reviewers: joshuaspence, btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7611
Differential Revision: https://secure.phabricator.com/D12124
Summary:
Fixes T7485. Before applications had proper policies, we gated access by requiring the viewer be an administrator.
This is now redundant (CAN_EDIT on applications has the same effect, and performs the same check), and may some day be wrong (we might let administrators configure a different policy to control who can configure applications). Today, it gets the policy dialog wrong.
Test Plan:
Clicked "Edit Policies" as a non-administrator, was unable to, got nice error:
{F346598}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7485
Differential Revision: https://secure.phabricator.com/D12125
Summary: Fixes T7620. I don't fully understand exactly what's going on here, but we don't actually need to call `flush()`.
Test Plan:
- Put timing code around the `echo`.
- Made a fake page that emitted a lot of data.
- Saw the `echo` block proportionate to data size under `curl --limit-rate ...`.
- See T7620.
- Downloaded a large file, got a reasonable progress bar and no obvious memory use issues.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: jlarouche, rbalik, epriestley
Maniphest Tasks: T7620
Differential Revision: https://secure.phabricator.com/D12127
Summary: Fixes T7647. We float both the time and description here and want the fluidity.
Test Plan: Review a new Conpherence in FF, Safari, Chrome.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7647
Differential Revision: https://secure.phabricator.com/D12137
Summary:
Fixes T7563. Fixes T5201. Reframe this as two separate operations:
- Move before or after a task.
- Move to the beginning or end of a priority.
Then:
- Make all the order queries unambiguous and properly reversible, with an explicit `id` order.
- Just reuse `ManiphestTask` to get results in the correct order.
- Simplify the actual transaction apply logic.
- Detect and recover from cases where tasks have identical or similar subpriorities.
Test Plan:
- Wrote and executed unit tests.
- Dragged and dropped tasks within priorities and between priorities in the main Maniphest view.
- Dragged and dropped tasks within priorities in the workboard view, when ordered by priority.
- Also poked at the "natural" order, but that shouldn't be affected.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: chad, epriestley
Maniphest Tasks: T5201, T7563
Differential Revision: https://secure.phabricator.com/D12121
Summary:
Via HackerOne. We aren't correctly escaping the date, so a user can XSS themselves by setting their date format creatively.
This construction is very unusual and I don't think we do anything similar elsewhere, so I can't come up with a systematic change which would prevent this in the general case.
Test Plan: Set date format to tag junk, got self-XSS before patch and proper escaping after the patch.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D12117
Summary:
Fixes T7621. The engine selection code started out making sense, but didn't make as much sense by the time I was done with it.
Specifically, from the vanilla file upload, we may incorrectly try to write directly to the chunk storage engine. This is incorrect, and produces a confusing/bad error.
Make chunk storage engines explicit and don't try to do single-file one-shot writes to them.
Test Plan:
- Tried to upload a large file with vanilla uploader, got better error message.
- Uploaded small and large files with drag and drop.
- Viewed {nav Files > Help/Options}.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7621
Differential Revision: https://secure.phabricator.com/D12110
Summary:
Ref T7522. This is mostly useful in the cluster, but could be useful for external installs too.
If you want to import an instance into a test/dry-run state (in the cluster, to test an import; in the general case, to do something like test new hardware or configuration), you currently risk spamming users with a lot of duplicate notifications. In particular, if Phabricator tracks remotes, both instances will continue importing commits and sending email about them. Both instances will try to publish to mirrors, too, which could be bad news, and both instances will try to update linked services.
Instead, provide a flag to let an instance run in "silent mode", which disables all outbound messaging and data.
We need to remember to support this flag on any new outbound channels, but we add about one of those per year so I think that's reasonable.
Test Plan:
- Flipped config.
- Saw it void email, feed and mirroring.
- Didn't test SMS since it's not really in use yet and not convenient to test.
- (Can you think of any publishing I missed?)
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7522
Differential Revision: https://secure.phabricator.com/D12109
Summary: Fixes T7618. The "button" needs to be a PHUIButtonView later on.
Test Plan: Forced condition, loaded page, saw button instead of fatal.
Reviewers: chad, btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7618
Differential Revision: https://secure.phabricator.com/D12108
Summary:
Indigo apparently used to be pink and was fixed in D10707.
This looks like it never got changed
Test Plan: Flag was purple, then it was pink
Reviewers: chad, #blessed_reviewers
Reviewed By: chad, #blessed_reviewers
Subscribers: epriestley, #flags
Differential Revision: https://secure.phabricator.com/D12101
Summary:
Ref T7149. When users give us dumpfiles for import, they will almost inevitably use the `phabricator` namespace. They need to be renamed to use an instance namespace.
We can do this either by:
- importing the data first, then renaming; or
- renaming first, then importing.
This implements the second one, basically `storage renamespace --in dump.sql --from phabricator --to instancename > instance.sql`.
Renaming first is a little hackier since we have to `preg_match()` a SQL dump file, but I think it's better overall:
- With only one database, it lets you dump/import without downtime.
- If you have development stuff in a development environment in the `phabricator` namespace, you don't have to move it aside to do an import.
- No possibility that two people doing an import at the same time on the same box will collide with each other.
- You can do the rename once and then repeat the import process with the renamed dump more easily.
- No tricky stuff with modern Phabricator running against an old dump and the database names not matching up.
None of this is super important, but it just makes large dumps a bit easier to work with, and the dumpfile format is regular enough that this seems unlikely to ever really not work.
Test Plan: Renamespaced a dump, did a `diff -u`, saw all the relevant parts changed (and only those parts changed).
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12105
Summary: Ref T7582. Also adds the basic logic for "rooms" implementation. Also makes sure we use the initializeNewThread method as appropriate.
Test Plan: made a new conpherence and it worked!
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7582
Differential Revision: https://secure.phabricator.com/D12103
Summary: this typo broke (at least) renaming the thread from the durable column.
Test Plan: renamed a thread from durable column and it worked
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12104
Summary: Fixes T7583. We also add `key_room`, which uses isRoom and dateModified since a very common view of rooms is going to be ordered by last updated.
Test Plan: made the conpherence view controller query specify `withIsRoom(true)` and `withIsRoom(false)`. The former made the controller correctly 404 while the latter had no change in functionality.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7583
Differential Revision: https://secure.phabricator.com/D12102
Summary:
Ref T7607. Ref T7522.
- For the import tools, I want to send from "Phacility Support <support@phacility.com>".
- In the general case, I want to send billing mail from merchants (T7607) later on.
Test Plan: Sent an email and saw the desired "From" address.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7607, T7522
Differential Revision: https://secure.phabricator.com/D12100
Summary:
Ref T7522. This seems like the least-bad approach to a messy issue:
- When backfilling accounts from an imported instance, I need to write ExternalAccount rows to the instance to link instance accounts with upstream accounts.
- We do this in the daemons in some other cases, which lets us run all the code in the context of the instance. However, I really want to do this in-process here because it's way way simpler and we need to do writes to //both// the instance and the upstream, and they're interleaved, and they depend on one another.
- I can hard-code the query with `qsprintf()` but that feels like 100x worse than this.
This allows me to do this:
```
id(new PhabricatorExternalAccount())
->setForcedConnnection($instance_conn)
->...
->save();
```
...and get a write to the instance database, which is at least not completely a minefield.
Test Plan: Backfilled instance accounts and got interleaved instance and upstream writes as expected.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7522
Differential Revision: https://secure.phabricator.com/D12098
Summary: Conpherence Full modernize pass, setting standard space and colors on all widget panels. Moved menu back to 240px as the narrow column wasn't really usefull. Removed 'subtitle' on menu, seems simpler but almost under-designed. Subtitle isn't particularly useful and I plan on adding audience icons next (single, group, project, public) so I think this is the right direction.
Test Plan:
Tested with and without number columns on the menu, test with files, calendar dates, removing participants.
{F337941}
{F337942}
{F337943}
Reviewers: epriestley, btrahan
Reviewed By: btrahan
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12078
Summary:
Fixes T7561. Basically everytime we load some transactions in the thread manager, kick off an async thread to update the notification panel.
Should I consolidate this little bit of code into something like this._handleTransactionResponse(r)... ? I just want to keep the JS clear for other engineers and I wasn't sure if that was hiding a bit too much detail.
Test Plan: user a opened durable column. user b sent user a a few messages. reloaded user a page and noted the "N" count became N-1 as the message loaded. Switched messages and saw N-2, N-3, etc as I loaded up the messages.
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7561
Differential Revision: https://secure.phabricator.com/D12099
Summary:
Ref T7149. Currently, global drag and drop always uses the most open visibility policy on the install. This was appropriate before the application preference was introduced, but default to the application preference now.
In particular, this supports a default value of "Administrators" in the Phacility cluster.
Also simplify/clean up some code.
Test Plan:
- Set application default policy to "Adminstrators".
- Uploaded file via drag-and-drop, saw "administrators" policy.
- Uploaded file via `arc upload`, saw "administrators" policy.
- Saw better URI for a text file upload after patch.
- Uploaded file via drag-and-drop-to-textarea, saw "only viewer" policy.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12093
Summary: Rewrite this expression so that the second parameter to `qsprintf` is a scalar.
Test Plan: `arc lint`
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11913
Summary:
Fixes T6713. The idea is to keep checking what's going on in the update paths that touch the DOM. If we're doing an update or should be doing a different update, then we bail early.
This is the type of code + testing that makes me dizzy after awhile, but I think it works...
Test Plan:
added a "forceStall" parameter to the column view controller, which when specified sleeps for seconds before returning. I then augmented the JS such that the "send message" code for the durable column would specifiy this parameter.
For actual testing, I then spammed the heck out of the durable column channel and saw each message only once. I also spammed the column, switched browsers to a user on the same thread in the normal "speedy" view, sent messages there, and also only received one copy
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T6713
Differential Revision: https://secure.phabricator.com/D12092
Summary:
Fixes T7545. Turns out we had the right logic to handle this basically, and just needed to variablize the CSS class that gets added / removed as appropos.
Note the new behavior is to keep the icon highlighted just with no number. This emulates how it would work if e.g. there was no unread message in the first place and you just clicked the message icon to invoke the message menu.
Test Plan: had a durable conpherence open for user A with user B. used a separate browser to send message as user B. reloaded as user A - saw new message in conpherence durable column and the "1" unread icon. I then clicked the "1" and saw it disappear as expected
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7545
Differential Revision: https://secure.phabricator.com/D12091
Summary: Ref T7538. I got this half correct but not fully correct: when you press enter in an empty text box, do nothing (instead of: sending an empty message, or writing a literal newline).
Test Plan: Hit enter in empty chat column box.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Maniphest Tasks: T7538
Differential Revision: https://secure.phabricator.com/D12089
These parameters were set inconsistently. Use the value that the storage task uses.
Also, allow `bin/phortune invoice` to invoice in the past and future to aid testing.
Summary:
Ref T7149. We can simplify configuration somewhat by removing the upload limit setting, now that we support arbitrarily large files.
- Merge configuration documentation.
- Tell users to set things to at least 32MB. This is 8MB maximum one-shot file + 4x headroom. Chunk sizes are 4MB.
Test Plan:
- Faked all the setup warnings.
- Read documentation.
- Uploaded some files.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12083
Summary: Ref T7149. This works now, so enable it.
Test Plan:
- Uploaded large and small files in Firefox, Safari and Chrome.
- Uploaded large files with `arc upload`.
- Stopped/resumed large files with all clients.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12079
Summary: Ref T7149. This was just to make testing easier, but chunking substantially works now.
Test Plan: `grep`
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12076
Summary:
Right now, if a daemon dies it can leave the setup warning around for like 10 minutes or something until we reap it.
Tighten the warning so we only care about actively running daemons.
Test Plan: Checked setup issues.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D12088
Summary: Normalizes size and colors. Default to non-bold names, grey text, and hide the time in column. Also re-evaluated header spacing in Photoshop.
Test Plan: Lots of photoshop, tested full in desktop, mobile and tablet and normal durable column. This was sadly 2 hours of work.
Reviewers: epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12087
Summary:
- Don't show a loading state on the whole column while sending chat. We could show some kind of minor loading state, but standard JX.Busy stuff will kick in after a couple seconds anyway.
- Blank the textarea immediately on submit so you can start typing more text.
- Don't disable the form while submiting; disabling it prevents you from typing more text.
- Hide the placeholder while the textarea is focused. If we don't do this, the placeholder reappearing after submitting text feels weird to me.
Test Plan:
- Sent a lot of text.
- Real fast.
- Focused and unfocused the area.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D12086
Summary: Ref T7538. We can figure out whether to backport this to main Conpherence later and/or remove buttons, etc., but this behavior seems pretty clearly good.
Test Plan:
- Pressed enter (sent message).
- Pressed shift+enter (newline).
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Maniphest Tasks: T7538
Differential Revision: https://secure.phabricator.com/D12085
Summary: Fixes T7529. I think stylesheet order got juggled at some point and made this more specific, but we don't actually need it.
Test Plan:
- Column now looks reasonable.
- Everything else does too.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Maniphest Tasks: T7529
Differential Revision: https://secure.phabricator.com/D12084
Summary: Currently we punch down Dashboard columns on smaller displays. This adds another set of rules for if durable-column is present.
Test Plan: Test breakpoints at 1300 and 1000 pixel wide.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7531
Differential Revision: https://secure.phabricator.com/D12056
Summary: Ref T7149. Return a real iterator from the Chunk engine, which processes chunks sequentially.
Test Plan:
This is a bit hard to read, but shows the underlying chunks being accessed one at a time and only some being accessed when requesting a range of a file:
```
$ ./bin/files cat F878 --trace --begin 100 --end 256
...
>>> [10] <query> SELECT * FROM `file_storageblob` WHERE `id` = 85
<<< [10] <query> 240 us
better software.
Phabricat>>> [11] <query> SELECT * FROM `file_storageblob` WHERE `id` = 84
<<< [11] <query> 205 us
or includes applications for:
>>> [12] <query> SELECT * FROM `file_storageblob` WHERE `id` = 83
<<< [12] <query> 226 us
- reviewing and auditing source>>> [13] <query> SELECT * FROM `file_storageblob` WHERE `id` = 82
<<< [13] <query> 203 us
code;
- hosting and browsing >>> [14] <query> SELECT * FROM `file_storageblob` WHERE `id` = 81
<<< [14] <query> 231 us
repositories;
- tracking bugs;
```
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12073
Summary:
Ref T7149. This still buffers the whole file, but is reaaaaal close to not doing that.
Allow Responses to be streamed, and rewrite the range stuff in the FileResponse so it does not rely on having the entire content available.
Test Plan:
- Artificially slowed down downloads, suspended/resumed them (works in chrome, not so much in Safari/Firefox?)
- Played sounds in Safari/Chrome.
- Viewed a bunch of pages and files in every browser.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12072
