1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-18 11:30:55 +01:00
phorge-phorge/src/applications/files
epriestley 8b8a3142b3 Support export of data in files larger than 8MB
Summary:
Depends on D18952. Ref T13049. For files larger than 8MB, we need to engage the chunk storage engine. `PhabricatorFile::newFromFileData()` always writes a single chunk, and can't handle files larger than the mandatory chunk threshold (8MB).

Use `IteratorUploadSource`, which can, and "stream" the data into it. This should raise the limit from 8MB to 2GB (maximum size of a string in PHP).

If we need to go above 2GB we could stream CSV and text pretty easily, and JSON without too much trouble, but Excel might be trickier. Hopefully no one is trying to export 2GB+ datafiles, though.

Test Plan:
  - Changed the JSON exporter to just export 8MB of the letter "q": `return str_repeat('q', 1024 * 1024 * 9);`.
  - Before change: fatal, "no storage engine can store this file".
  - After change: export works cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18953
2018-01-29 15:58:34 -08:00
..
action Rate limit outbound requests in Macros 2015-03-26 11:11:52 -07:00
application Move Files editing and commenting to EditEngine 2017-04-04 16:15:11 -07:00
applicationpanel Allow diffusion.filecontentquery to load data for arbitrarily large files 2016-01-21 09:52:43 -08:00
builtin Don't require the "gd" extension be installed in order to run unit tests 2017-11-30 13:51:31 -08:00
capability Add default policy to Files application 2014-11-21 11:17:20 -08:00
conduit Stop trying to assess the image dimensions of large files and file chunks 2017-12-18 09:17:32 -08:00
config Prevent enormous changes from being pushed to repositoires by default 2018-01-04 10:02:29 -08:00
constants Prettier file embeds 2016-11-27 14:57:06 -08:00
controller Don't show personalized menu items until users establish a full session 2017-11-28 10:01:58 -08:00
edge Modernize remaining edge types 2015-01-03 10:58:20 +11:00
editor Make daemons perform file deletion 2017-04-18 11:09:41 -07:00
engine Kick off indexing for File objects on creation 2017-04-18 08:38:34 -07:00
exception Store and verify content integrity checksums for files 2017-04-05 11:12:31 -07:00
format Support HMAC+SHA256 with automatic key generation and management 2017-04-06 15:42:59 -07:00
garbagecollector Implement ngram search for File objects 2017-04-17 17:37:20 -07:00
iconset Convert all calls to 'IconFont' to just 'Icon' 2016-01-27 20:59:27 -08:00
keyring Support AES256 at-rest encryption in Files 2016-06-16 08:08:56 -07:00
lipsum Make bin/lipsum generate hanldle generator keys and arguments more clearly 2017-02-27 09:09:28 -08:00
mail phtize all the things 2015-05-22 21:16:39 +10:00
management Fix spelling 2017-10-09 10:48:04 -07:00
markup Add download link to embedded files 2017-01-03 10:50:26 -08:00
phid Mark PhabricatorPHIDType::getPHIDTypeApplicationClass() as abstract 2015-11-03 06:47:12 +11:00
query Add a cache purger for builtin files 2017-06-22 11:13:23 -07:00
storage Support export of data in files larger than 8MB 2018-01-29 15:58:34 -08:00
transform Add unit tests for file thumbnail generation 2017-04-23 11:02:21 -07:00
typeahead Add a basic icon typeahead 2017-02-03 09:19:29 -08:00
uploadsource Support export of data in files larger than 8MB 2018-01-29 15:58:34 -08:00
view Allow users to drop .ics files on calendar views to import them 2016-10-18 15:26:44 -07:00
worker Make daemons perform file deletion 2017-04-18 11:09:41 -07:00
xaction Switch File deletion to use ModularTransactions 2017-04-18 13:01:51 -07:00
PhabricatorImageTransformer.php Make the Files "TTL" API more structured 2017-04-04 16:16:28 -07:00