1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-28 01:32:42 +01:00
phorge-phorge/src/applications/files/uploadsource
epriestley 8b8a3142b3 Support export of data in files larger than 8MB
Summary:
Depends on D18952. Ref T13049. For files larger than 8MB, we need to engage the chunk storage engine. `PhabricatorFile::newFromFileData()` always writes a single chunk, and can't handle files larger than the mandatory chunk threshold (8MB).

Use `IteratorUploadSource`, which can, and "stream" the data into it. This should raise the limit from 8MB to 2GB (maximum size of a string in PHP).

If we need to go above 2GB we could stream CSV and text pretty easily, and JSON without too much trouble, but Excel might be trickier. Hopefully no one is trying to export 2GB+ datafiles, though.

Test Plan:
  - Changed the JSON exporter to just export 8MB of the letter "q": `return str_repeat('q', 1024 * 1024 * 9);`.
  - Before change: fatal, "no storage engine can store this file".
  - After change: export works cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18953
2018-01-29 15:58:34 -08:00
..
PhabricatorExecFutureFileUploadSource.php Allow diffusion.filecontentquery to load data for arbitrarily large files 2016-01-21 09:52:43 -08:00
PhabricatorFileUploadSource.php Support export of data in files larger than 8MB 2018-01-29 15:58:34 -08:00
PhabricatorFileUploadSourceByteLimitException.php Fix some issues with Diffusion file data limits 2018-01-22 11:52:37 -08:00
PhabricatorIteratorFileUploadSource.php Support pushing data into Git LFS 2016-03-18 09:36:34 -07:00