1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-20 20:40:56 +01:00
phorge-phorge/src/applications/files/conduit/FileConduitAPIMethod.php

121 lines
2.7 KiB
PHP
Raw Normal View History

<?php
abstract class FileConduitAPIMethod extends ConduitAPIMethod {
final public function getApplication() {
return PhabricatorApplication::getByClass('PhabricatorFilesApplication');
}
Add a chunking storage engine for files Summary: Ref T7149. This isn't complete and isn't active yet, but does basically work. I'll shore it up in the next few diffs. The new workflow goes like this: > Client, file.allocate(): I'd like to upload a file with length L, metadata M, and hash H. Then the server returns `upload` (a boolean) and `filePHID` (a PHID). These mean: | upload | filePHID | means | |---|---|---| | false | false | Server can't accept file. | false | true | File data already known, file created from hash. | true | false | Just upload normally. | true | true | Query chunks to start or resume a chunked upload. All but the last case are uninteresting and work like exising uploads with `file.uploadhash` (which we can eventually deprecate). In the last case: > Client, file.querychunks(): Give me a list of chunks that I should upload. This returns all the chunks for the file. Chunks have a start byte, an end byte, and a "complete" flag to indicate that the server already has the data. Then, the client fills in chunks by sending them: > Client, file.uploadchunk(): Here is the data for one chunk. This stuff doesn't work yet or has some caveats: - I haven't tested resume much. - Files need an "isPartial()" flag for partial uploads, and the UI needs to respect it. - The JS client needs to become chunk-aware. - Chunk size is set crazy low to make testing easier. - Some debugging flags that I'll remove soon-ish. - Downloading works, but still streams the whole file into memory. - This storage engine is disabled by default (hardcoded as a unit test engine) because it's still sketchy. - Need some code to remove the "isParital" flag when the last chunk is uploaded. - Maybe do checksumming on chunks. Test Plan: - Hacked up `arc upload` (see next diff) to be chunk-aware and uploaded a readme in 18 32-byte chunks. Then downloaded it. Got the same file back that I uploaded. - File UI now shows some basic chunk info for chunked files: {F336434} Reviewers: btrahan Reviewed By: btrahan Subscribers: joshuaspence, epriestley Maniphest Tasks: T7149 Differential Revision: https://secure.phabricator.com/D12060
2015-03-13 19:30:02 +01:00
protected function loadFileByPHID(PhabricatorUser $viewer, $file_phid) {
$file = id(new PhabricatorFileQuery())
->setViewer($viewer)
->withPHIDs(array($file_phid))
->executeOne();
if (!$file) {
throw new Exception(pht('No such file "%s"!', $file_phid));
}
return $file;
}
protected function loadFileChunks(
PhabricatorUser $viewer,
PhabricatorFile $file) {
return $this->newChunkQuery($viewer, $file)
->execute();
}
protected function loadFileChunkForUpload(
PhabricatorUser $viewer,
PhabricatorFile $file,
$start,
$end) {
$start = (int)$start;
$end = (int)$end;
$chunks = $this->newChunkQuery($viewer, $file)
->withByteRange($start, $end)
->execute();
if (!$chunks) {
throw new Exception(
pht(
'There are no file data chunks in byte range %d - %d.',
$start,
$end));
}
if (count($chunks) !== 1) {
phlog($chunks);
throw new Exception(
pht(
'There are multiple chunks in byte range %d - %d.',
$start,
$end));
}
$chunk = head($chunks);
if ($chunk->getByteStart() != $start) {
throw new Exception(
pht(
'Chunk start byte is %d, not %d.',
$chunk->getByteStart(),
$start));
}
if ($chunk->getByteEnd() != $end) {
throw new Exception(
pht(
'Chunk end byte is %d, not %d.',
$chunk->getByteEnd(),
$end));
}
if ($chunk->getDataFilePHID()) {
throw new Exception(
pht(
'Chunk has already been uploaded.'));
}
return $chunk;
}
protected function decodeBase64($data) {
$data = base64_decode($data, $strict = true);
if ($data === false) {
throw new Exception(pht('Unable to decode base64 data!'));
}
return $data;
}
protected function loadAnyMissingChunk(
PhabricatorUser $viewer,
PhabricatorFile $file) {
return $this->newChunkQuery($viewer, $file)
->withIsComplete(false)
->setLimit(1)
->execute();
}
Add a chunking storage engine for files Summary: Ref T7149. This isn't complete and isn't active yet, but does basically work. I'll shore it up in the next few diffs. The new workflow goes like this: > Client, file.allocate(): I'd like to upload a file with length L, metadata M, and hash H. Then the server returns `upload` (a boolean) and `filePHID` (a PHID). These mean: | upload | filePHID | means | |---|---|---| | false | false | Server can't accept file. | false | true | File data already known, file created from hash. | true | false | Just upload normally. | true | true | Query chunks to start or resume a chunked upload. All but the last case are uninteresting and work like exising uploads with `file.uploadhash` (which we can eventually deprecate). In the last case: > Client, file.querychunks(): Give me a list of chunks that I should upload. This returns all the chunks for the file. Chunks have a start byte, an end byte, and a "complete" flag to indicate that the server already has the data. Then, the client fills in chunks by sending them: > Client, file.uploadchunk(): Here is the data for one chunk. This stuff doesn't work yet or has some caveats: - I haven't tested resume much. - Files need an "isPartial()" flag for partial uploads, and the UI needs to respect it. - The JS client needs to become chunk-aware. - Chunk size is set crazy low to make testing easier. - Some debugging flags that I'll remove soon-ish. - Downloading works, but still streams the whole file into memory. - This storage engine is disabled by default (hardcoded as a unit test engine) because it's still sketchy. - Need some code to remove the "isParital" flag when the last chunk is uploaded. - Maybe do checksumming on chunks. Test Plan: - Hacked up `arc upload` (see next diff) to be chunk-aware and uploaded a readme in 18 32-byte chunks. Then downloaded it. Got the same file back that I uploaded. - File UI now shows some basic chunk info for chunked files: {F336434} Reviewers: btrahan Reviewed By: btrahan Subscribers: joshuaspence, epriestley Maniphest Tasks: T7149 Differential Revision: https://secure.phabricator.com/D12060
2015-03-13 19:30:02 +01:00
private function newChunkQuery(
PhabricatorUser $viewer,
PhabricatorFile $file) {
$engine = $file->instantiateStorageEngine();
if (!$engine->isChunkEngine()) {
throw new Exception(
pht(
'File "%s" does not have chunks!',
$file->getPHID()));
}
return id(new PhabricatorFileChunkQuery())
->setViewer($viewer)
->withChunkHandles(array($file->getStorageHandle()));
}
}