Rename Conduit classes
Summary: Ref T5655. Rename Conduit classes and provide a `getAPIMethodName` method to declare the API method.
Test Plan:
```
> echo '{}' | arc --conduit-uri='http://phabricator.joshuaspence.com' call-conduit user.whoami
Waiting for JSON parameters on stdin...
{"error":null,"errorMessage":null,"response":{"phid":"PHID-USER-lioqffnwn6y475mu5ndb","userName":"josh","realName":"Joshua Spence","image":"http:\/\/phabricator.joshuaspence.com\/res\/1404425321T\/phabricator\/3eb28cd9\/rsrc\/image\/avatar.png","uri":"http:\/\/phabricator.joshuaspence.com\/p\/josh\/","roles":["admin","verified","approved","activated"]}}
```
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin, hach-que
Maniphest Tasks: T5655
Differential Revision: https://secure.phabricator.com/D9991
2014-07-25 02:54:15 +02:00
|
|
|
<?php
|
|
|
|
|
|
|
|
abstract class FileConduitAPIMethod extends ConduitAPIMethod {
|
|
|
|
|
|
|
|
final public function getApplication() {
|
2014-07-25 03:03:59 +02:00
|
|
|
return PhabricatorApplication::getByClass('PhabricatorFilesApplication');
|
Rename Conduit classes
Summary: Ref T5655. Rename Conduit classes and provide a `getAPIMethodName` method to declare the API method.
Test Plan:
```
> echo '{}' | arc --conduit-uri='http://phabricator.joshuaspence.com' call-conduit user.whoami
Waiting for JSON parameters on stdin...
{"error":null,"errorMessage":null,"response":{"phid":"PHID-USER-lioqffnwn6y475mu5ndb","userName":"josh","realName":"Joshua Spence","image":"http:\/\/phabricator.joshuaspence.com\/res\/1404425321T\/phabricator\/3eb28cd9\/rsrc\/image\/avatar.png","uri":"http:\/\/phabricator.joshuaspence.com\/p\/josh\/","roles":["admin","verified","approved","activated"]}}
```
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin, hach-que
Maniphest Tasks: T5655
Differential Revision: https://secure.phabricator.com/D9991
2014-07-25 02:54:15 +02:00
|
|
|
}
|
|
|
|
|
Add a chunking storage engine for files
Summary:
Ref T7149. This isn't complete and isn't active yet, but does basically work. I'll shore it up in the next few diffs.
The new workflow goes like this:
> Client, file.allocate(): I'd like to upload a file with length L, metadata M, and hash H.
Then the server returns `upload` (a boolean) and `filePHID` (a PHID). These mean:
| upload | filePHID | means |
|---|---|---|
| false | false | Server can't accept file.
| false | true | File data already known, file created from hash.
| true | false | Just upload normally.
| true | true | Query chunks to start or resume a chunked upload.
All but the last case are uninteresting and work like exising uploads with `file.uploadhash` (which we can eventually deprecate).
In the last case:
> Client, file.querychunks(): Give me a list of chunks that I should upload.
This returns all the chunks for the file. Chunks have a start byte, an end byte, and a "complete" flag to indicate that the server already has the data.
Then, the client fills in chunks by sending them:
> Client, file.uploadchunk(): Here is the data for one chunk.
This stuff doesn't work yet or has some caveats:
- I haven't tested resume much.
- Files need an "isPartial()" flag for partial uploads, and the UI needs to respect it.
- The JS client needs to become chunk-aware.
- Chunk size is set crazy low to make testing easier.
- Some debugging flags that I'll remove soon-ish.
- Downloading works, but still streams the whole file into memory.
- This storage engine is disabled by default (hardcoded as a unit test engine) because it's still sketchy.
- Need some code to remove the "isParital" flag when the last chunk is uploaded.
- Maybe do checksumming on chunks.
Test Plan:
- Hacked up `arc upload` (see next diff) to be chunk-aware and uploaded a readme in 18 32-byte chunks. Then downloaded it. Got the same file back that I uploaded.
- File UI now shows some basic chunk info for chunked files:
{F336434}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12060
2015-03-13 19:30:02 +01:00
|
|
|
protected function loadFileByPHID(PhabricatorUser $viewer, $file_phid) {
|
|
|
|
$file = id(new PhabricatorFileQuery())
|
|
|
|
->setViewer($viewer)
|
|
|
|
->withPHIDs(array($file_phid))
|
|
|
|
->executeOne();
|
|
|
|
if (!$file) {
|
|
|
|
throw new Exception(pht('No such file "%s"!', $file_phid));
|
|
|
|
}
|
|
|
|
|
|
|
|
return $file;
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function loadFileChunks(
|
|
|
|
PhabricatorUser $viewer,
|
|
|
|
PhabricatorFile $file) {
|
|
|
|
return $this->newChunkQuery($viewer, $file)
|
|
|
|
->execute();
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function loadFileChunkForUpload(
|
|
|
|
PhabricatorUser $viewer,
|
|
|
|
PhabricatorFile $file,
|
|
|
|
$start,
|
|
|
|
$end) {
|
|
|
|
|
|
|
|
$start = (int)$start;
|
|
|
|
$end = (int)$end;
|
|
|
|
|
|
|
|
$chunks = $this->newChunkQuery($viewer, $file)
|
|
|
|
->withByteRange($start, $end)
|
|
|
|
->execute();
|
|
|
|
|
|
|
|
if (!$chunks) {
|
|
|
|
throw new Exception(
|
|
|
|
pht(
|
|
|
|
'There are no file data chunks in byte range %d - %d.',
|
|
|
|
$start,
|
|
|
|
$end));
|
|
|
|
}
|
|
|
|
|
|
|
|
if (count($chunks) !== 1) {
|
|
|
|
phlog($chunks);
|
|
|
|
throw new Exception(
|
|
|
|
pht(
|
|
|
|
'There are multiple chunks in byte range %d - %d.',
|
|
|
|
$start,
|
|
|
|
$end));
|
|
|
|
}
|
|
|
|
|
|
|
|
$chunk = head($chunks);
|
|
|
|
if ($chunk->getByteStart() != $start) {
|
|
|
|
throw new Exception(
|
|
|
|
pht(
|
|
|
|
'Chunk start byte is %d, not %d.',
|
|
|
|
$chunk->getByteStart(),
|
|
|
|
$start));
|
|
|
|
}
|
|
|
|
|
|
|
|
if ($chunk->getByteEnd() != $end) {
|
|
|
|
throw new Exception(
|
|
|
|
pht(
|
|
|
|
'Chunk end byte is %d, not %d.',
|
|
|
|
$chunk->getByteEnd(),
|
|
|
|
$end));
|
|
|
|
}
|
|
|
|
|
|
|
|
if ($chunk->getDataFilePHID()) {
|
|
|
|
throw new Exception(
|
|
|
|
pht(
|
|
|
|
'Chunk has already been uploaded.'));
|
|
|
|
}
|
|
|
|
|
|
|
|
return $chunk;
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function decodeBase64($data) {
|
|
|
|
$data = base64_decode($data, $strict = true);
|
|
|
|
if ($data === false) {
|
|
|
|
throw new Exception(pht('Unable to decode base64 data!'));
|
|
|
|
}
|
|
|
|
return $data;
|
|
|
|
}
|
|
|
|
|
2015-03-13 19:30:24 +01:00
|
|
|
protected function loadAnyMissingChunk(
|
|
|
|
PhabricatorUser $viewer,
|
|
|
|
PhabricatorFile $file) {
|
|
|
|
|
|
|
|
return $this->newChunkQuery($viewer, $file)
|
|
|
|
->withIsComplete(false)
|
|
|
|
->setLimit(1)
|
|
|
|
->execute();
|
|
|
|
}
|
|
|
|
|
Add a chunking storage engine for files
Summary:
Ref T7149. This isn't complete and isn't active yet, but does basically work. I'll shore it up in the next few diffs.
The new workflow goes like this:
> Client, file.allocate(): I'd like to upload a file with length L, metadata M, and hash H.
Then the server returns `upload` (a boolean) and `filePHID` (a PHID). These mean:
| upload | filePHID | means |
|---|---|---|
| false | false | Server can't accept file.
| false | true | File data already known, file created from hash.
| true | false | Just upload normally.
| true | true | Query chunks to start or resume a chunked upload.
All but the last case are uninteresting and work like exising uploads with `file.uploadhash` (which we can eventually deprecate).
In the last case:
> Client, file.querychunks(): Give me a list of chunks that I should upload.
This returns all the chunks for the file. Chunks have a start byte, an end byte, and a "complete" flag to indicate that the server already has the data.
Then, the client fills in chunks by sending them:
> Client, file.uploadchunk(): Here is the data for one chunk.
This stuff doesn't work yet or has some caveats:
- I haven't tested resume much.
- Files need an "isPartial()" flag for partial uploads, and the UI needs to respect it.
- The JS client needs to become chunk-aware.
- Chunk size is set crazy low to make testing easier.
- Some debugging flags that I'll remove soon-ish.
- Downloading works, but still streams the whole file into memory.
- This storage engine is disabled by default (hardcoded as a unit test engine) because it's still sketchy.
- Need some code to remove the "isParital" flag when the last chunk is uploaded.
- Maybe do checksumming on chunks.
Test Plan:
- Hacked up `arc upload` (see next diff) to be chunk-aware and uploaded a readme in 18 32-byte chunks. Then downloaded it. Got the same file back that I uploaded.
- File UI now shows some basic chunk info for chunked files:
{F336434}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T7149
Differential Revision: https://secure.phabricator.com/D12060
2015-03-13 19:30:02 +01:00
|
|
|
private function newChunkQuery(
|
|
|
|
PhabricatorUser $viewer,
|
|
|
|
PhabricatorFile $file) {
|
|
|
|
|
|
|
|
$engine = $file->instantiateStorageEngine();
|
|
|
|
if (!$engine->isChunkEngine()) {
|
|
|
|
throw new Exception(
|
|
|
|
pht(
|
|
|
|
'File "%s" does not have chunks!',
|
|
|
|
$file->getPHID()));
|
|
|
|
}
|
|
|
|
|
|
|
|
return id(new PhabricatorFileChunkQuery())
|
|
|
|
->setViewer($viewer)
|
|
|
|
->withChunkHandles(array($file->getStorageHandle()));
|
|
|
|
}
|
|
|
|
|
2015-03-13 19:30:24 +01:00
|
|
|
|
Rename Conduit classes
Summary: Ref T5655. Rename Conduit classes and provide a `getAPIMethodName` method to declare the API method.
Test Plan:
```
> echo '{}' | arc --conduit-uri='http://phabricator.joshuaspence.com' call-conduit user.whoami
Waiting for JSON parameters on stdin...
{"error":null,"errorMessage":null,"response":{"phid":"PHID-USER-lioqffnwn6y475mu5ndb","userName":"josh","realName":"Joshua Spence","image":"http:\/\/phabricator.joshuaspence.com\/res\/1404425321T\/phabricator\/3eb28cd9\/rsrc\/image\/avatar.png","uri":"http:\/\/phabricator.joshuaspence.com\/p\/josh\/","roles":["admin","verified","approved","activated"]}}
```
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin, hach-que
Maniphest Tasks: T5655
Differential Revision: https://secure.phabricator.com/D9991
2014-07-25 02:54:15 +02:00
|
|
|
}
|