Summary:
Fixes T7621. The engine selection code started out making sense, but didn't make as much sense by the time I was done with it.
Specifically, from the vanilla file upload, we may incorrectly try to write directly to the chunk storage engine. This is incorrect, and produces a confusing/bad error.
Make chunk storage engines explicit and don't try to do single-file one-shot writes to them.
Test Plan:
- Tried to upload a large file with vanilla uploader, got better error message.
- Uploaded small and large files with drag and drop.
- Viewed {nav Files > Help/Options}.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7621
Differential Revision: https://secure.phabricator.com/D12110
Summary:
Ref T7522. This is mostly useful in the cluster, but could be useful for external installs too.
If you want to import an instance into a test/dry-run state (in the cluster, to test an import; in the general case, to do something like test new hardware or configuration), you currently risk spamming users with a lot of duplicate notifications. In particular, if Phabricator tracks remotes, both instances will continue importing commits and sending email about them. Both instances will try to publish to mirrors, too, which could be bad news, and both instances will try to update linked services.
Instead, provide a flag to let an instance run in "silent mode", which disables all outbound messaging and data.
We need to remember to support this flag on any new outbound channels, but we add about one of those per year so I think that's reasonable.
Test Plan:
- Flipped config.
- Saw it void email, feed and mirroring.
- Didn't test SMS since it's not really in use yet and not convenient to test.
- (Can you think of any publishing I missed?)
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7522
Differential Revision: https://secure.phabricator.com/D12109
Summary: Fixes T7618. The "button" needs to be a PHUIButtonView later on.
Test Plan: Forced condition, loaded page, saw button instead of fatal.
Reviewers: chad, btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7618
Differential Revision: https://secure.phabricator.com/D12108
Summary:
Indigo apparently used to be pink and was fixed in D10707.
This looks like it never got changed
Test Plan: Flag was purple, then it was pink
Reviewers: chad, #blessed_reviewers
Reviewed By: chad, #blessed_reviewers
Subscribers: epriestley, #flags
Differential Revision: https://secure.phabricator.com/D12101
Summary:
Ref T7149. When users give us dumpfiles for import, they will almost inevitably use the `phabricator` namespace. They need to be renamed to use an instance namespace.
We can do this either by:
- importing the data first, then renaming; or
- renaming first, then importing.
This implements the second one, basically `storage renamespace --in dump.sql --from phabricator --to instancename > instance.sql`.
Renaming first is a little hackier since we have to `preg_match()` a SQL dump file, but I think it's better overall:
- With only one database, it lets you dump/import without downtime.
- If you have development stuff in a development environment in the `phabricator` namespace, you don't have to move it aside to do an import.
- No possibility that two people doing an import at the same time on the same box will collide with each other.
- You can do the rename once and then repeat the import process with the renamed dump more easily.
- No tricky stuff with modern Phabricator running against an old dump and the database names not matching up.
None of this is super important, but it just makes large dumps a bit easier to work with, and the dumpfile format is regular enough that this seems unlikely to ever really not work.
Test Plan: Renamespaced a dump, did a `diff -u`, saw all the relevant parts changed (and only those parts changed).
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12105
Summary: Ref T7582. Also adds the basic logic for "rooms" implementation. Also makes sure we use the initializeNewThread method as appropriate.
Test Plan: made a new conpherence and it worked!
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7582
Differential Revision: https://secure.phabricator.com/D12103
Summary: this typo broke (at least) renaming the thread from the durable column.
Test Plan: renamed a thread from durable column and it worked
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12104
Summary: Fixes T7583. We also add `key_room`, which uses isRoom and dateModified since a very common view of rooms is going to be ordered by last updated.
Test Plan: made the conpherence view controller query specify `withIsRoom(true)` and `withIsRoom(false)`. The former made the controller correctly 404 while the latter had no change in functionality.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7583
Differential Revision: https://secure.phabricator.com/D12102
Summary:
Ref T7607. Ref T7522.
- For the import tools, I want to send from "Phacility Support <support@phacility.com>".
- In the general case, I want to send billing mail from merchants (T7607) later on.
Test Plan: Sent an email and saw the desired "From" address.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7607, T7522
Differential Revision: https://secure.phabricator.com/D12100
Summary:
Ref T7522. This seems like the least-bad approach to a messy issue:
- When backfilling accounts from an imported instance, I need to write ExternalAccount rows to the instance to link instance accounts with upstream accounts.
- We do this in the daemons in some other cases, which lets us run all the code in the context of the instance. However, I really want to do this in-process here because it's way way simpler and we need to do writes to //both// the instance and the upstream, and they're interleaved, and they depend on one another.
- I can hard-code the query with `qsprintf()` but that feels like 100x worse than this.
This allows me to do this:
```
id(new PhabricatorExternalAccount())
->setForcedConnnection($instance_conn)
->...
->save();
```
...and get a write to the instance database, which is at least not completely a minefield.
Test Plan: Backfilled instance accounts and got interleaved instance and upstream writes as expected.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7522
Differential Revision: https://secure.phabricator.com/D12098
Summary: Conpherence Full modernize pass, setting standard space and colors on all widget panels. Moved menu back to 240px as the narrow column wasn't really usefull. Removed 'subtitle' on menu, seems simpler but almost under-designed. Subtitle isn't particularly useful and I plan on adding audience icons next (single, group, project, public) so I think this is the right direction.
Test Plan:
Tested with and without number columns on the menu, test with files, calendar dates, removing participants.
{F337941}
{F337942}
{F337943}
Reviewers: epriestley, btrahan
Reviewed By: btrahan
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12078
Summary:
Fixes T7561. Basically everytime we load some transactions in the thread manager, kick off an async thread to update the notification panel.
Should I consolidate this little bit of code into something like this._handleTransactionResponse(r)... ? I just want to keep the JS clear for other engineers and I wasn't sure if that was hiding a bit too much detail.
Test Plan: user a opened durable column. user b sent user a a few messages. reloaded user a page and noted the "N" count became N-1 as the message loaded. Switched messages and saw N-2, N-3, etc as I loaded up the messages.
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7561
Differential Revision: https://secure.phabricator.com/D12099
Summary:
Ref T7149. Currently, global drag and drop always uses the most open visibility policy on the install. This was appropriate before the application preference was introduced, but default to the application preference now.
In particular, this supports a default value of "Administrators" in the Phacility cluster.
Also simplify/clean up some code.
Test Plan:
- Set application default policy to "Adminstrators".
- Uploaded file via drag-and-drop, saw "administrators" policy.
- Uploaded file via `arc upload`, saw "administrators" policy.
- Saw better URI for a text file upload after patch.
- Uploaded file via drag-and-drop-to-textarea, saw "only viewer" policy.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12093
Summary: Rewrite this expression so that the second parameter to `qsprintf` is a scalar.
Test Plan: `arc lint`
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11913
Summary:
Fixes T6713. The idea is to keep checking what's going on in the update paths that touch the DOM. If we're doing an update or should be doing a different update, then we bail early.
This is the type of code + testing that makes me dizzy after awhile, but I think it works...
Test Plan:
added a "forceStall" parameter to the column view controller, which when specified sleeps for seconds before returning. I then augmented the JS such that the "send message" code for the durable column would specifiy this parameter.
For actual testing, I then spammed the heck out of the durable column channel and saw each message only once. I also spammed the column, switched browsers to a user on the same thread in the normal "speedy" view, sent messages there, and also only received one copy
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T6713
Differential Revision: https://secure.phabricator.com/D12092
Summary:
Fixes T7545. Turns out we had the right logic to handle this basically, and just needed to variablize the CSS class that gets added / removed as appropos.
Note the new behavior is to keep the icon highlighted just with no number. This emulates how it would work if e.g. there was no unread message in the first place and you just clicked the message icon to invoke the message menu.
Test Plan: had a durable conpherence open for user A with user B. used a separate browser to send message as user B. reloaded as user A - saw new message in conpherence durable column and the "1" unread icon. I then clicked the "1" and saw it disappear as expected
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7545
Differential Revision: https://secure.phabricator.com/D12091
Summary: Ref T7538. I got this half correct but not fully correct: when you press enter in an empty text box, do nothing (instead of: sending an empty message, or writing a literal newline).
Test Plan: Hit enter in empty chat column box.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Maniphest Tasks: T7538
Differential Revision: https://secure.phabricator.com/D12089
These parameters were set inconsistently. Use the value that the storage task uses.
Also, allow `bin/phortune invoice` to invoice in the past and future to aid testing.
Summary:
Ref T7149. We can simplify configuration somewhat by removing the upload limit setting, now that we support arbitrarily large files.
- Merge configuration documentation.
- Tell users to set things to at least 32MB. This is 8MB maximum one-shot file + 4x headroom. Chunk sizes are 4MB.
Test Plan:
- Faked all the setup warnings.
- Read documentation.
- Uploaded some files.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12083
Summary: Ref T7149. This works now, so enable it.
Test Plan:
- Uploaded large and small files in Firefox, Safari and Chrome.
- Uploaded large files with `arc upload`.
- Stopped/resumed large files with all clients.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12079
Summary: Ref T7149. This was just to make testing easier, but chunking substantially works now.
Test Plan: `grep`
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12076
Summary:
Right now, if a daemon dies it can leave the setup warning around for like 10 minutes or something until we reap it.
Tighten the warning so we only care about actively running daemons.
Test Plan: Checked setup issues.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D12088
Summary: Normalizes size and colors. Default to non-bold names, grey text, and hide the time in column. Also re-evaluated header spacing in Photoshop.
Test Plan: Lots of photoshop, tested full in desktop, mobile and tablet and normal durable column. This was sadly 2 hours of work.
Reviewers: epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12087
Summary:
- Don't show a loading state on the whole column while sending chat. We could show some kind of minor loading state, but standard JX.Busy stuff will kick in after a couple seconds anyway.
- Blank the textarea immediately on submit so you can start typing more text.
- Don't disable the form while submiting; disabling it prevents you from typing more text.
- Hide the placeholder while the textarea is focused. If we don't do this, the placeholder reappearing after submitting text feels weird to me.
Test Plan:
- Sent a lot of text.
- Real fast.
- Focused and unfocused the area.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D12086
Summary: Ref T7538. We can figure out whether to backport this to main Conpherence later and/or remove buttons, etc., but this behavior seems pretty clearly good.
Test Plan:
- Pressed enter (sent message).
- Pressed shift+enter (newline).
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Maniphest Tasks: T7538
Differential Revision: https://secure.phabricator.com/D12085
Summary: Fixes T7529. I think stylesheet order got juggled at some point and made this more specific, but we don't actually need it.
Test Plan:
- Column now looks reasonable.
- Everything else does too.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Maniphest Tasks: T7529
Differential Revision: https://secure.phabricator.com/D12084
Summary: Currently we punch down Dashboard columns on smaller displays. This adds another set of rules for if durable-column is present.
Test Plan: Test breakpoints at 1300 and 1000 pixel wide.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7531
Differential Revision: https://secure.phabricator.com/D12056
Summary: Ref T7149. Return a real iterator from the Chunk engine, which processes chunks sequentially.
Test Plan:
This is a bit hard to read, but shows the underlying chunks being accessed one at a time and only some being accessed when requesting a range of a file:
```
$ ./bin/files cat F878 --trace --begin 100 --end 256
...
>>> [10] <query> SELECT * FROM `file_storageblob` WHERE `id` = 85
<<< [10] <query> 240 us
better software.
Phabricat>>> [11] <query> SELECT * FROM `file_storageblob` WHERE `id` = 84
<<< [11] <query> 205 us
or includes applications for:
>>> [12] <query> SELECT * FROM `file_storageblob` WHERE `id` = 83
<<< [12] <query> 226 us
- reviewing and auditing source>>> [13] <query> SELECT * FROM `file_storageblob` WHERE `id` = 82
<<< [13] <query> 203 us
code;
- hosting and browsing >>> [14] <query> SELECT * FROM `file_storageblob` WHERE `id` = 81
<<< [14] <query> 231 us
repositories;
- tracking bugs;
```
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12073
Summary:
Ref T7149. This still buffers the whole file, but is reaaaaal close to not doing that.
Allow Responses to be streamed, and rewrite the range stuff in the FileResponse so it does not rely on having the entire content available.
Test Plan:
- Artificially slowed down downloads, suspended/resumed them (works in chrome, not so much in Safari/Firefox?)
- Played sounds in Safari/Chrome.
- Viewed a bunch of pages and files in every browser.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12072
Summary: Ref T7149. A couple diffs down the line, this will let us emit chunked files without doing all the work up front or holding the entire file in RAM.
Test Plan:
(Some newlines added for clarity.)
```
$ ./bin/files cat F942
ABCDEFGHIJKLMNOPQRSTUVWXYZ
$ ./bin/files cat F942 --begin 1
BCDEFGHIJKLMNOPQRSTUVWXYZ
$ ./bin/files cat F942 --end 10
ABCDEFGHIJ
$ ./bin/files cat F942 --begin 3 --end 5
DE
$
```
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12071
Summary: Ref T7149. We can't compute hashes of large files efficiently, but we can resume uploads by the same author, with the same name and file size, which are only partially completed. This seems like a reasonable heuristic that is unlikely to ever misfire, even if it's a little magical.
Test Plan:
- Forced chunking on.
- Started uploading a chunked file.
- Closed the browser window.
- Dropped it into a new window.
- Upload resumed //(!!!)//
- Did this again.
- Downloaded the final file, which successfully reconstructed the original file.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, chad, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12070
Summary: Changes the text to just "Stay", which is still obvious what it means, with less copy. Fixes T7027
Test Plan: Now works on mobile.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7027
Differential Revision: https://secure.phabricator.com/D12075
Summary: Fixes T7556, adds margin instead of padding to links.
Test Plan: Set my status as away, see proper spacing.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7556
Differential Revision: https://secure.phabricator.com/D12069
Summary:
Ref T7149. This adds chunking support to drag-and-drop uploads. It never activates right now unless you hack things up, since the chunk engine is still hard-coded as disabled.
The overall approach is the same as `arc upload` in D12061, with some slight changes to the API return values to avoid a few extra HTTP calls.
Test Plan:
- Enabled chunk engine.
- Uploaded some READMEs in a bunch of tiny 32 byte chunks.
- Worked out of the box in Safari, Chrome, Firefox.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12066
Summary:
Ref T7149. This flags allocated but incomplete files and doesn't explode when trying to download them.
Files are marked complete when the last chunk is uploaded.
I added a key on `<authorPHID, isPartial>` so we can show you a list of partially uploaded files and prompt you to resume them at some point down the road.
Test Plan: Massaged debugging settings and uploaded README.md very slowly in 32b chunks. Saw the file lose its "Partial" flag when the last chunk finished.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12063
Summary:
Ref T7149. This makes debugging some of this stuff a bit easier by removing the HTTP part in the middle.
Particularly, I anticipate having this stream data chunk-by-chunk in the near future.
Test Plan: Ran `files cat F23`, got output.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12062
Summary:
Ref T7149. This isn't complete and isn't active yet, but does basically work. I'll shore it up in the next few diffs.
The new workflow goes like this:
> Client, file.allocate(): I'd like to upload a file with length L, metadata M, and hash H.
Then the server returns `upload` (a boolean) and `filePHID` (a PHID). These mean:
| upload | filePHID | means |
|---|---|---|
| false | false | Server can't accept file.
| false | true | File data already known, file created from hash.
| true | false | Just upload normally.
| true | true | Query chunks to start or resume a chunked upload.
All but the last case are uninteresting and work like exising uploads with `file.uploadhash` (which we can eventually deprecate).
In the last case:
> Client, file.querychunks(): Give me a list of chunks that I should upload.
This returns all the chunks for the file. Chunks have a start byte, an end byte, and a "complete" flag to indicate that the server already has the data.
Then, the client fills in chunks by sending them:
> Client, file.uploadchunk(): Here is the data for one chunk.
This stuff doesn't work yet or has some caveats:
- I haven't tested resume much.
- Files need an "isPartial()" flag for partial uploads, and the UI needs to respect it.
- The JS client needs to become chunk-aware.
- Chunk size is set crazy low to make testing easier.
- Some debugging flags that I'll remove soon-ish.
- Downloading works, but still streams the whole file into memory.
- This storage engine is disabled by default (hardcoded as a unit test engine) because it's still sketchy.
- Need some code to remove the "isParital" flag when the last chunk is uploaded.
- Maybe do checksumming on chunks.
Test Plan:
- Hacked up `arc upload` (see next diff) to be chunk-aware and uploaded a readme in 18 32-byte chunks. Then downloaded it. Got the same file back that I uploaded.
- File UI now shows some basic chunk info for chunked files:
{F336434}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12060
Summary: D12058 for all its brilliance should have applied this css rule to all instances of this, not just once the column is toggled
Test Plan: tried to make the flickering happen WITHOUT the column toggled and failed; uploaded successfully
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D12059
Summary:
Ref T7539. This stuff is pretty whack - the "dragEnter" and "dragLeave" fire over and over and over despite not moving the mouse sometimes from JX.Mask interaction, causing that neat flickering effect. Fix this mess by disabling pointer events for the mask.
Test Plan: dragged a file to the durable column textarea. it uploaded and inlined the Fxxx into the message. tried uploading a few times and it worked repeatedly.
Reviewers: epriestley, chad
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7539
Differential Revision: https://secure.phabricator.com/D12058
Summary: Fixes T7539. We need to set the "with-column" css class on the document body to make things like the jx-mask style-able. Also, make the global upload control only do it for the standard phabrcator page and not the document body.
Test Plan: dragged a file to conpherence column and it worked! uploaded a file to homepage with column open and it worked! uploaded a file to /file/ with column open and it worked!
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7539
Differential Revision: https://secure.phabricator.com/D12055
Summary:
Fixes T5843. File storage engines use a very old "selector" mechanism which makes them difficult to extend.
This mechanism predates widespread use of `PhutilSymbolLoader` to discover available implementations at runtime. Runtime discovery has generally proven more flexible and easier to use than explicit selection (although it sometimes needs more UI to support it in cases where order or enabled/disabled flags can not be directly determined).
Use a modern runtime discovery mechanism instead of an explicit selector. This might break any installs which subclassed the `Selector`, but I believe almost no such installs exist, and they'll receive a meaningful exception upon upgrading (any custom engines will no longer implement all of the required methods).
Looking forward, this modernizes infrastructure to prepare for new "virtual" chunked-storage engines, with the eventual goal of supporting very large file uploads and data import into the Phacility cluster.
This uses D12051 to add UI to make it easier to understand the state of storage engines.
Test Plan:
Used new UI panel to assess storage engines:
{F336270}
- Uploaded a small file, saw it go to MySQL engine.
- Uploaded a larger file, saw it go to S3 engine.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5843
Differential Revision: https://secure.phabricator.com/D12053
