1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-15 19:32:40 +01:00
Commit graph

129 commits

Author SHA1 Message Date
epriestley
47b14c9bde Convert inline profile image transforms to new transformations
Summary:
Ref T7707. Fixes T7879. Fixes T4406. When creating profile images:

  - Use the new transforms;
  - mark them as "profile" images so they're forced to the most-open policies.

Test Plan:
  - Set restrictive default file policies.
  - Changed profile picture, project pictures, etc. Verified they were visible to logged-out users.
  - Registered via OAuth.
  - Updated a Conpherence thread image.
  - Browsed around looking for profile images, fixed sizing on everything I could find.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7879, T7707, T4406

Differential Revision: https://secure.phabricator.com/D12821
2015-05-13 11:38:46 -07:00
epriestley
7e365eb8ae Convert "profile" image transforms to the new pathway
Summary:
Ref T7707. This ends up being sort of complicated: to support 100x100 images in T4406, we need to scale small images //up// so they look OK when we scale them back down with `background-size` in CSS.

The rest of it is mostly straightforward.

Test Plan:
  - Did an OAuth handshake and saw a scaled-up, scaled-down profile picture that looked correct.
  - Used Pholio, edited pholio, embedded pholio.
  - Uploaded a bunch of small/weird/big images and regenerated all their transforms.
  - Uploaded some text files into Pholio.
  - Grepped for removed methods, etc.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7707

Differential Revision: https://secure.phabricator.com/D12818
2015-05-13 11:38:46 -07:00
epriestley
75f6211233 Convert "preview" image transforms to new pathway
Summary: Ref T7707. Move the 220px (file uploads) and 100px (Pholio thumbgrid) previews over to the new stuff.

Test Plan: Uploaded a bunch of images to remarkup and Pholio; they generated reasonable results in the web UI.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7707

Differential Revision: https://secure.phabricator.com/D12814
2015-05-13 11:38:46 -07:00
epriestley
c998e44b5a Modularize file transforms and provide a "transforms" UI
Summary:
Ref T7707. Available transforms are currently relatively hard-coded and don't really have any support UI.

Modularize them so we can build some support UI.

This doesn't actually //use// any of the new stuff yet: I want to make a clean cutover once I fix the aspect ratio stuff so I can pick up a cachekey/URI change as a side effect.

Test Plan: {F400524}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: chad, epriestley

Maniphest Tasks: T7707

Differential Revision: https://secure.phabricator.com/D12808
2015-05-13 11:38:45 -07:00
epriestley
ae32d1afb8 Remove unused file transforms
Summary:
Ref T7707.

  - Modernize the file transform endpoint a bit.
  - Delete transforms which are no longer used in the product.

Test Plan:
  - Used Pholio (navigation, inline thumbs).
  - Uploaded images (embed thumb).
  - Changed profile picture (profile thumb).

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7707

Differential Revision: https://secure.phabricator.com/D12807
2015-05-13 11:38:45 -07:00
lkassianik
55ff197f2a File names should be editable.
Summary: Fixes T7480, File names should be editable and the event should show up in feed.

Test Plan: Upload a file, view file details, edit file, change file name by adding a space and a word to the name, save changes, file name should retain space and not normalize the name, file details should show the edit event, install feed should correctly show an event for the action.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Maniphest Tasks: T7480

Differential Revision: https://secure.phabricator.com/D12561
2015-04-26 15:24:29 -07:00
epriestley
40fb0f98df Mostly defuse DNS rebinding attack for outbound requests
Summary: Ref T6755. I'll add some notes there about specifics.

Test Plan:
  - Made connections to HTTP and HTTPS URIs.
  - Added some debugging code to verify that HTTP URIs were pre-resolved.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6755

Differential Revision: https://secure.phabricator.com/D12169
2015-03-26 11:12:22 -07:00
epriestley
4f8147dbb8 Improve protection against SSRF attacks
Summary:
Ref T6755. This improves our resistance to SSRF attacks:

  - Follow redirects manually and verify each component of the redirect chain.
  - Handle authentication provider profile picture fetches more strictly.

Test Plan:
  - Tried to download macros from various URIs which issued redirects, etc.
  - Downloaded an actual macro.
  - Went through external account workflow.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6755

Differential Revision: https://secure.phabricator.com/D12151
2015-03-24 18:49:01 -07:00
epriestley
1c32c9b965 Improve granluarity and defaults of security.allow-outbound-http
Summary:
Ref T6755. This is a partial fix, but:

  - Allow netblocks to be blacklisted instead of making the feature all-or-nothing.
  - Default to disallow requests to all reserved private/local/special IP blocks. This should generally be a "safe" setting.
  - Explain the risks better.
  - Improve the errors rasied by Macro when failing.
  - Removed `security.allow-outbound-http`, as it is superseded by this setting and is somewhat misleading.
    - We still make outbound HTTP requests to OAuth.
    - We still make outbound HTTP requests for repositories.

From a technical perspective:

  - Separate URIs that are safe to link to or redirect to (basically, not "javascript://") from URIs that are safe to fetch (nothing in a private block).
  - Add the default blacklist.
  - Be more careful with response data in Macro fetching, and don't let the user see it if it isn't ultimately valid.

Additionally:

  - I want to do this check before pulling repositories, but that's enough of a mess that it should go in a separate diff.
  - The future implementation of T4190 needs to perform the fetch check.

Test Plan:
  - Fetched a valid macro.
  - Fetched a non-image, verified it didn't result in a viewable file.
  - Fetched a private-ip-space image, got an error.
  - Fetched a 404, got a useful-enough error without additional revealing response content (which is usually HTML anyway and not useful).
  - Fetched a bad protocol, got an error.
  - Linked to a local resource, a phriction page, a valid remote site, all worked.
  - Linked to private IP space, which worked fine (we want to let you link and redierect to other private services, just not fetch them).
  - Added and executed unit tests.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6755

Differential Revision: https://secure.phabricator.com/D12136
2015-03-23 10:44:03 -07:00
epriestley
7482d260b0 Rewrite file documentation to be chunk-aware
Summary:
Ref T7149. We can simplify configuration somewhat by removing the upload limit setting, now that we support arbitrarily large files.

  - Merge configuration documentation.
  - Tell users to set things to at least 32MB. This is 8MB maximum one-shot file + 4x headroom. Chunk sizes are 4MB.

Test Plan:
  - Faked all the setup warnings.
  - Read documentation.
  - Uploaded some files.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7149

Differential Revision: https://secure.phabricator.com/D12083
2015-03-15 11:37:47 -07:00
epriestley
c19bb57730 Stream chunks when sending chunked files
Summary: Ref T7149. Return a real iterator from the Chunk engine, which processes chunks sequentially.

Test Plan:
This is a bit hard to read, but shows the underlying chunks being accessed one at a time and only some being accessed when requesting a range of a file:

```
$ ./bin/files cat F878 --trace --begin 100 --end 256
...
>>> [10] <query> SELECT * FROM `file_storageblob` WHERE `id` = 85
<<< [10] <query> 240 us
 better software.

Phabricat>>> [11] <query> SELECT * FROM `file_storageblob` WHERE `id` = 84
<<< [11] <query> 205 us
or includes applications for:

 >>> [12] <query> SELECT * FROM `file_storageblob` WHERE `id` = 83
<<< [12] <query> 226 us
 - reviewing and auditing source>>> [13] <query> SELECT * FROM `file_storageblob` WHERE `id` = 82
<<< [13] <query> 203 us
 code;
  - hosting and browsing >>> [14] <query> SELECT * FROM `file_storageblob` WHERE `id` = 81
<<< [14] <query> 231 us
repositories;
  - tracking bugs;
```

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: joshuaspence, epriestley

Maniphest Tasks: T7149

Differential Revision: https://secure.phabricator.com/D12073
2015-03-14 08:29:30 -07:00
epriestley
2aefb43843 Support a file data iteration interface for large files
Summary: Ref T7149. A couple diffs down the line, this will let us emit chunked files without doing all the work up front or holding the entire file in RAM.

Test Plan:
(Some newlines added for clarity.)

```
$ ./bin/files cat F942
ABCDEFGHIJKLMNOPQRSTUVWXYZ
$ ./bin/files cat F942 --begin 1
BCDEFGHIJKLMNOPQRSTUVWXYZ
$ ./bin/files cat F942 --end 10
ABCDEFGHIJ
$ ./bin/files cat F942 --begin 3 --end 5
DE
$
```

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: joshuaspence, epriestley

Maniphest Tasks: T7149

Differential Revision: https://secure.phabricator.com/D12071
2015-03-14 08:28:59 -07:00
epriestley
32d8d67535 Support resuming JS uploads of chunked files
Summary: Ref T7149. We can't compute hashes of large files efficiently, but we can resume uploads by the same author, with the same name and file size, which are only partially completed. This seems like a reasonable heuristic that is unlikely to ever misfire, even if it's a little magical.

Test Plan:
  - Forced chunking on.
  - Started uploading a chunked file.
  - Closed the browser window.
  - Dropped it into a new window.
  - Upload resumed //(!!!)//
  - Did this again.
  - Downloaded the final file, which successfully reconstructed the original file.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: joshuaspence, chad, epriestley

Maniphest Tasks: T7149

Differential Revision: https://secure.phabricator.com/D12070
2015-03-14 08:28:46 -07:00
epriestley
aa4adf3ab8 Add support for partially uploaded files
Summary:
Ref T7149. This flags allocated but incomplete files and doesn't explode when trying to download them.

Files are marked complete when the last chunk is uploaded.

I added a key on `<authorPHID, isPartial>` so we can show you a list of partially uploaded files and prompt you to resume them at some point down the road.

Test Plan: Massaged debugging settings and uploaded README.md very slowly in 32b chunks. Saw the file lose its "Partial" flag when the last chunk finished.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: joshuaspence, epriestley

Maniphest Tasks: T7149

Differential Revision: https://secure.phabricator.com/D12063
2015-03-13 11:30:24 -07:00
epriestley
4aed453b06 Add a chunking storage engine for files
Summary:
Ref T7149. This isn't complete and isn't active yet, but does basically work. I'll shore it up in the next few diffs.

The new workflow goes like this:

> Client, file.allocate(): I'd like to upload a file with length L, metadata M, and hash H.

Then the server returns `upload` (a boolean) and `filePHID` (a PHID). These mean:

| upload | filePHID | means |
|---|---|---|
| false | false | Server can't accept file.
| false | true | File data already known, file created from hash.
| true | false | Just upload normally.
| true | true | Query chunks to start or resume a chunked upload.

All but the last case are uninteresting and work like exising uploads with `file.uploadhash` (which we can eventually deprecate).

In the last case:

> Client, file.querychunks(): Give me a list of chunks that I should upload.

This returns all the chunks for the file. Chunks have a start byte, an end byte, and a "complete" flag to indicate that the server already has the data.

Then, the client fills in chunks by sending them:

> Client, file.uploadchunk(): Here is the data for one chunk.

This stuff doesn't work yet or has some caveats:

  - I haven't tested resume much.
  - Files need an "isPartial()" flag for partial uploads, and the UI needs to respect it.
  - The JS client needs to become chunk-aware.
  - Chunk size is set crazy low to make testing easier.
  - Some debugging flags that I'll remove soon-ish.
  - Downloading works, but still streams the whole file into memory.
  - This storage engine is disabled by default (hardcoded as a unit test engine) because it's still sketchy.
  - Need some code to remove the "isParital" flag when the last chunk is uploaded.
  - Maybe do checksumming on chunks.

Test Plan:
  - Hacked up `arc upload` (see next diff) to be chunk-aware and uploaded a readme in 18 32-byte chunks. Then downloaded it. Got the same file back that I uploaded.
  - File UI now shows some basic chunk info for chunked files:

{F336434}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: joshuaspence, epriestley

Maniphest Tasks: T7149

Differential Revision: https://secure.phabricator.com/D12060
2015-03-13 11:30:02 -07:00
epriestley
e2296a0ff7 Modernize file storage engine selection
Summary:
Fixes T5843. File storage engines use a very old "selector" mechanism which makes them difficult to extend.

This mechanism predates widespread use of `PhutilSymbolLoader` to discover available implementations at runtime. Runtime discovery has generally proven more flexible and easier to use than explicit selection (although it sometimes needs more UI to support it in cases where order or enabled/disabled flags can not be directly determined).

Use a modern runtime discovery mechanism instead of an explicit selector. This might break any installs which subclassed the `Selector`, but I believe almost no such installs exist, and they'll receive a meaningful exception upon upgrading (any custom engines will no longer implement all of the required methods).

Looking forward, this modernizes infrastructure to prepare for new "virtual" chunked-storage engines, with the eventual goal of supporting very large file uploads and data import into the Phacility cluster.

This uses D12051 to add UI to make it easier to understand the state of storage engines.

Test Plan:
Used new UI panel to assess storage engines:

{F336270}

  - Uploaded a small file, saw it go to MySQL engine.
  - Uploaded a larger file, saw it go to S3 engine.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5843

Differential Revision: https://secure.phabricator.com/D12053
2015-03-12 13:28:53 -07:00
epriestley
d1eda610fa Identify builtin files and give them open policies
Summary:
Fixes T7379. Currently, builtin files generate with a "users" view policy even if an install is public.

Because these files TTL after 7 days, there's no migration here. Installs won't see the fix actually happen for up to 7 days after updating, though.

Test Plan:
  - Deleted a builtin.
  - Loaded projects page to regenerate it.
  - Saw new builtin had most open policy and was marked as a builtin.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7379

Differential Revision: https://secure.phabricator.com/D11917
2015-03-01 12:12:38 -08:00
epriestley
5b1ea8c8d5 Pass instance through file transform URIs
Summary:
This makes thumbnail URIs work on instanced, CDN'd installs like Phacility cluster instances.

Some of these transforms can proabably be removed, but the underlying code to generate the transform should be cleaned up too and we have some other tasks filed elsewhere about this anyway.

Test Plan: CDN'd local install now loads thumbnails properly.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D11719
2015-02-09 15:31:47 -08:00
epriestley
e6fb1dc1e9 When an install is instanced, include the instance identifier in the URI for file data
Summary:
This allows us to CDN the cluster.

General problem is that we can't easily give each instance its own CDN URI (`giraffe.phcdn.net`) in Cloudfront, because it requires that you enumerate all aliases (and there's a limit of 100) and depends on SNI (a newish feature of SSL which allows one server to serve multiple certificates, but which doesn't have full support everywhere yet).

It's //possible// that we could eventually work around this, or use Cloudflare instead (which has a different model that seems like a slightly easier fit for CDN-domain-per-instance), but I don't want to sink a ton of work into this and want to keep things on AWS insofar as we reasonably can.

The easiest way to fix this is just to put the instance identity into URIs, then read it out when handling CDN requests. This has no effect on installs without cluster instance configuration, which is all of them except ours.

It's also slightly desirable to share this stuff, since we get to share the cache for static resources, which are always identical across instances.

So requests go from the Cloudfront gateway ("xyz.cloudfront.com") to the LB with a hard-coded instance name ("cdn.phacility.com"), which gets them routed to a balanced web machine. The web machine picks the correct instance name out of the URI, acts as that instance, and does the correct thing.

The messiest part of this is that we need "cdn.phacility.com" to be a real instance so it can serve static resources, but that's not a big deal. We have a few other hard-codes which have to be real resources for now, like we must have a merchant named "Phacility".

Test Plan:
  - Viewed files with `security.alternate-file-domain` off (i.e., no file tokens).
  - Viewed pages and files with `security.alternate-file-domain` on. Saw correct resource behavior, @isntance generation of URIs, and correct token redirect behavior for files.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D11668
2015-02-03 14:55:46 -08:00
Joshua Spence
36b760cd8a Fix method visiblity for PhabricatorFileTestCase::getPhabricatorTestCaseConfiguration
Summary: Ref T6822.

Test Plan: `grep` for `->getPhabricatorTestCaseConfiguration(`.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11366
2015-01-14 07:04:36 +11:00
Joshua Spence
d6b882a804 Fix visiblity of LiskDAO::getConfiguration()
Summary: Ref T6822.

Test Plan: `grep`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: hach-que, Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11370
2015-01-14 06:54:13 +11:00
Joshua Spence
7c2a7d0365 Modernize remaining edge types
Summary: Modernize remaining edges to subclass `PhabricatorEdgeType`. Largely based on D11045.

Test Plan: Browsed around and performed various actions include subscribing, unsubscribing and watching.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D11116
2015-01-03 10:58:20 +11:00
Joshua Spence
8dee37a132 Fix some linter violations
Summary: Self-explanatory.

Test Plan: `arc lint`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11142
2015-01-03 09:11:41 +11:00
Chad Little
b925f42c55 Remove docs sprite, replace with FontAwesome
Summary: Removes the docs sprite in Conpherence with FontAwesome, adds additional icons. Unsure what happens if someone customized this config option.

Test Plan: Added images and files to a Conpherence, saw new icons.

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11028
2014-12-20 20:59:24 -08:00
Bob Trahan
6ab3f06b6e Transactions - adding willRenderTimeline to handle tricky cases
Summary: Fixes T6693.

Test Plan:
Made a bunch of comments on a diff with differential, being sure to leave inlines here and there. This reproduced the issue in T6693. With this patch this issue no longer reproduces!

Successfully "showed older changes" in Maniphest too.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6693

Differential Revision: https://secure.phabricator.com/D10931
2014-12-04 13:58:52 -08:00
Bob Trahan
5e8600442d Transactions - land buildTransactionTimeline in a bunch more apps
Summary:
Ref T4712. Specifically...

 - Dashboards
  - two objects needed PhabricatorApplicationTransactionInterface
 - Macros
 - Paste
 - Phlux
  - one object needed PhabricatorApplicationTransactionInterface
  - added setShouldTerminate(true)
 - Files
  - one object needed PhabricatorApplicationTransactionInterface
 - Passphrase
  - one object needed PhabricatorApplicationTransactionInterface
  - added setShouldTerminate(true)
 - Drydock
  - one object needed PhabricatorApplicationTransactionInterface
  - added setShouldTerminate(true)

Test Plan: foreach application, verify that the timeline(s) showed up correctly, including with appropriate setShouldTerminate-ness

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T4712

Differential Revision: https://secure.phabricator.com/D10924
2014-12-03 13:16:15 -08:00
Chad Little
1a8d87699c Add default policy to Files application
Summary: **WIP** This adds default capability to the Files application

Test Plan: Set default to public, go to Files page, see public preset. Upload File. Doesn't work.

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6564

Differential Revision: https://secure.phabricator.com/D10888
2014-11-21 11:17:20 -08:00
epriestley
8fa8415c07 Automatically build all Lisk schemata
Summary:
Ref T1191. Now that the whole database is covered, we don't need to do as much work to build expected schemata. Doing them database-by-database was helpful in converting, but is just reudndant work now.

Instead of requiring every application to build its Lisk objects, just build all Lisk objects.

I removed `harbormaster.lisk_counter` because it is unused.

It would be nice to autogenerate edge schemata, too, but that's a little trickier.

Test Plan: Database setup issues are all green.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10620
2014-10-02 09:51:20 -07:00
epriestley
4fcc634a99 Fix almost all remaining schemata issues
Summary:
Ref T1191. This fixes nearly every remaining blocker for utf8mb4 -- primarily, overlong keys.

Remaining issue is https://secure.phabricator.com/T1191#77467

Test Plan: I'll annotate inline.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T6099, T6129, T6133, T6134, T6150, T6148, T6147, T6146, T6105, T1191

Differential Revision: https://secure.phabricator.com/D10601
2014-10-01 08:18:36 -07:00
epriestley
943c62d1e9 Add missing expected keys and uniqueness
Summary:
Ref T1191.

  - Adds definitions for missing keys and keys with wrong uniqueness. Generally, I defined these before fixing the key query to actually pull all keys and support uniqueness.
  - Moves "key uniqueness" to note severity; this is fixable (probably?) and there are no remaining issues.
  - Moves "Missing Key" to note severity; missing keys are fixable and all remaining missing keys are really missing (either missing edge keys, or missing PHID keys):

{F210089}

  - Moves "Surplus Key" to note seveirty; surplus keys are fixable all remaining surplus keys are really surplus (duplicate key in Harbormaster, key on unused column in Worker):

{F210090}

Test Plan:
  - Vetted missing/surplus/unique messages.
  - 146 issues remaining.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10590
2014-10-01 07:53:50 -07:00
epriestley
e9ac3f436a Add expected schemata for Fund, Files, Flags and Legalpad
Summary: Ref T1191. Nothing too exciting in these.

Test Plan: Saw more blue in UI.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10521
2014-09-19 05:44:40 -07:00
epriestley
3af442e4ac Don't require an actor in PhabricatorFile::attachToObject()
Summary:
Ref T6013. A very long time ago, edges were less clearly low-level infrastructure, and some user-aware stuff got built around edge edits.

This was kind of a mess and I eventually removed it, during or prior to T5245. The big issue was that control flow was really hard to figure out as things went all the way down to the deepest level of infrastructure and then came back up the stack to events and transactions. The new stuff is more top-down and generally seems a lot easier and cleaner.

Consequently, actors are no longer required for edge edits. Remove the parameter.

Test Plan: Poked around; ran unit tests.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T6013

Differential Revision: https://secure.phabricator.com/D10412
2014-09-04 12:51:33 -07:00
epriestley
2e0361bd98 Bind file thumbnail visibility to the visibility of the original files
Summary:
Ref T6013. Currently, when we create a thumbnail, it gets its own (default) file visibility policy.

In particular, this causes the issue in T6013: thumbnails get "all users" visibility, which does not include logged-out users.

Instead, a thumbnail should just have the same visibility as the original file does. Enforce this:

  - When loading thumbnails, reject thumbnails with invisible originals.
  - When filtering thumbnails, permit thumbnails with visible originals.

Test Plan: As a logged-out user, thumbnails are now visible when the original files are attached to visible objects.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6013

Differential Revision: https://secure.phabricator.com/D10410
2014-09-04 12:49:31 -07:00
epriestley
f43355855c Add bin/files compact for sharing file data storage
Summary:
Fixes T5912. When we write files, we attempt to share storage if two files have the same content.

In some cases, we may not share storage. Examples include:

  - Files migrated with `bin/files migrate` (it's simpler not to try to dedupe them).
  - Old files, from before storage was sharable (the mechanism did not exist).
  - Files broken by the bug fixed in T5912.

Add a script to compact files by pointing files with the same content hash at the same file contnet.

In the particular case of files broken by the bug in T5912, we know the hash of the file's content and will only point them at a file that we can load the data for, so this fixes them.

Compaction is not hugely useful in general, but this script isn't too complex and the ability to fix damage from the bug in T5912 is desirable. We could remove this capability eventually.

Test Plan:
  - Ran `files compact --all --dry-run` and sanity checked a bunch of the duplicates for actually being duplicates.
  - Migrated individual files with `files compact Fnnn --trace` and verified the storage compacted and all files survived the process.
  - Verified unused storage was correctly destroyed after removing the last reference to it.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5912

Differential Revision: https://secure.phabricator.com/D10327
2014-08-21 11:47:59 -07:00
epriestley
66fa59d04d Fix an issue where migrating files could prematurely destroy duplicates
Summary:
Fixes T5912. When migrating files, we try to clean up the old data. However, this code isn't aware of reference counting, and unconditionally destroys the old data.

For example, if you migrate files `F1` and `F2` and they have the same data, we'll delete the shared data when we migrate `F1`. Then you'll get an error when you migrate `F2`.

Since this only affects duplicate files, it primarily hits default profile pictures, which are the most numerous duplicate files on most installs.

Test Plan:
  - Verified that the theory was correct by uploading two copies of a file and migrating the first one, before applying the patch. The second one's data was nuked and it couldn't be migrated.
  - Applied patch.
  - Uploaded two copies of a new file, migrated the first one (no data deletion), migrated the second one (data correctly deleted).
  - Uploaded two copies of another new file, `bin/remove destory'd` the first one (no data deletion), then did it to the second one (data correctly deleted).

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5912

Differential Revision: https://secure.phabricator.com/D10312
2014-08-20 15:32:32 -07:00
Bob Trahan
8dd4d5cfe5 Files - make file info page public
Summary: and for bonus, finesse some URIs a tad. Fixes T5922.

Test Plan: viewed F1 logged out and it worked! viewed the ugly URI for F1 and got redirected to the pretty URI.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T5922

Differential Revision: https://secure.phabricator.com/D10309
2014-08-20 13:18:21 -07:00
epriestley
94cdddc211 Cover redirects to files in more cases
Summary: Ref T5894. We have a couple more similar cases. Make them all do a decision-based redirect for now.

Test Plan: Did "View Raw File" and such, and also made sure thumbnails still work.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T5894

Differential Revision: https://secure.phabricator.com/D10301
2014-08-19 15:53:15 -07:00
epriestley
e8ece70ee0 Support bin/remove destroy Fnnn for files
Summary: Straightforward (this is the one object type we do let you delete from the web UI) implemetation of `PhabricatorDestructibleInterface`.

Test Plan: Used `bin/remove destroy` to destory several files. Used `--trace` to verify they wiped file data.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D10300
2014-08-19 15:44:27 -07:00
epriestley
607e99490b Migrate "cancdn" to "canCDN" in the database
Summary: Ref T5884. We migrated with "canCDN" and then had live writes with "cancdn". Move everything to "canCDN" for consistency.

Test Plan: Ran migration, verified DB only has "canCDN" afterward.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5884

Differential Revision: https://secure.phabricator.com/D10273
2014-08-15 11:07:40 -07:00
epriestley
ae1a821b65 Fix cancdn vs canCDN flag
Summary:
Ref T5884. We migrated to add a `canCDN` flag, but the code looks for a `cancdn` flag.

If this fixes the issue, I'll migrate `cancdn` to `canCDN` in the next diff.

Test Plan: Viewed some files, including old files, and saw the cacheability I expected.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5884

Differential Revision: https://secure.phabricator.com/D10264
2014-08-14 12:13:26 -07:00
epriestley
eb9dcd6fba Consolidate handling of special properties for newly uploaded files
Summary:
Fixes T5849. When a new file is created, we might have to actually write the data to a storage engine, or we might be able to just point at data which is already there.

Currently, these two paths handle `$params` with different code and mild behavioral differences. Instead, have them call the same code so they get the same behavior.

Test Plan:
  - Uploaded the same file multiple times to home page.
  - Uploaded the same file multiple times as profile picture.
  - Generated files via Diffusion.
  - All the files got the expected properties, whether they were reusing data or not.

Reviewers: btrahan, 20after4

Reviewed By: 20after4

Subscribers: epriestley

Maniphest Tasks: T5849

Differential Revision: https://secure.phabricator.com/D10216
2014-08-11 09:39:40 -07:00
Mukunda Modell
25ae4c458d Protect file data with a one-time-token
Test Plan: currently untested work in progress

Reviewers: #blessed_reviewers, epriestley

Subscribers: rush898, aklapper, Korvin, epriestley

Projects: #wikimedia

Maniphest Tasks: T5685

Differential Revision: https://secure.phabricator.com/D10054
2014-08-11 07:32:17 -07:00
Mukunda Modell
12aaa942ac Add a CanCDN flag to uploaded files
Summary:
CanCDN flag indicates that a file can be served + cached
via anonymous content distribution networks.

Once D10054 lands, any files that lack the CanCDN flag
will require a one-time-use token and headers will
prohibit cache to protect sensitive files from
unauthorized access.

This diff separates the CanCDN changes from the code that
enforces these restrictions in D10054 so that the changes
can be tested and refined independently.

Test Plan: Work in progress

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: rush898, qgil, epriestley, aklapper, Korvin

Maniphest Tasks: T5685

Differential Revision: https://secure.phabricator.com/D10166
2014-08-07 18:56:20 -07:00
epriestley
4c04d4d019 Allow users to set view policies on files explicitly
Summary: Ref T4589. Allow users to adjust visibility settings on files explicitly. This makes it easier to understand and manage upcoming changes in T4589.

Test Plan: Changed the view policy for a file several times.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4589

Differential Revision: https://secure.phabricator.com/D10129
2014-08-02 14:45:50 -07:00
Joshua Spence
97a8700e45 Rename PHIDType classes
Summary: Ref T5655. Rename `PhabricatorPHIDType` subclasses for clarity (see discussion in D9839). I'm not too keen on some of the resulting class names, so feel free to suggest alternatives.

Test Plan: Ran unit tests.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin, hach-que

Maniphest Tasks: T5655

Differential Revision: https://secure.phabricator.com/D9986
2014-07-24 08:05:46 +10:00
epriestley
8cbfb49b4e Remove all edge events
Summary:
Ref T5245. These were a bad idea.

We no longer need actors for edge edits either, so remove those. Generally, edges have fit into the policy model as pure/low-level infrastructure, and they do not have any policy or capability information in and of themselves.

Test Plan: `grep`

Reviewers: chad, btrahan, joshuaspence

Reviewed By: joshuaspence

Subscribers: epriestley

Maniphest Tasks: T5245

Differential Revision: https://secure.phabricator.com/D9840
2014-07-17 15:41:42 -07:00
James Rhodes
9cb6b2cfcc Remove user-independent date and time functions from Phabricator
Summary: These have been moved into libphutil.

Test Plan: Browsed Phabricator, didn't see a crash.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9907
2014-07-13 12:03:17 +10:00
Joshua Spence
8756d82cf6 Remove @group annotations
Summary: I'm pretty sure that `@group` annotations are useless now... see D9855. Also fixed various other minor issues.

Test Plan: Eye-ball it.

Reviewers: #blessed_reviewers, epriestley, chad

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D9859
2014-07-10 08:12:48 +10:00
epriestley
dcf3b4d322 Reduce Pholio brokenness for non-image files
Summary:
Ref T5359. When users upload non-image file types (PDFs, text files, whatever), Pholio currently chokes in a few places. Make most of these behaviors more reasonable:

  - Provide thumbs in the required sizes.
  - Predict the thumb size of these files correctly.
  - Disable inline comments.
  - Make "View Fullsize" and "Download" into buttons. These mostly-work. Download should probaly really download, but CSRF on forms is a bit of a pain right now.

Test Plan: See screenshots.

Reviewers: chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T5359

Differential Revision: https://secure.phabricator.com/D9548
2014-06-15 08:03:04 -07:00
epriestley
7ce8a1f437 Turn thumbs into a history grid thing
Summary:
This could probably use some refinement (and, like, explanatory text, and stronger cues about what rows and columns mean) but feels fairly good to me, at least on test data.

I didn't do any scrolling for now since we have to do full height on mobile anyway I think. I did swap it so the newer ones are on top.

Left/right navigate you among current images only, but you can click any thumb to review history.

Removed history view since it's no longer useful.

Some things that would probably help:

  - Some kind of header explaining what this is ("Mock History" or something).
  - Stronger visual cue that columns are related by being the same image.
  - Clearer cues about obsolete/deleted images (e.g., on the stage itself?)
  - Maybe general tweaks.
  - Maybe a placeholder (like a grey "X") for images which have been deleted.

(I'm planning to add comment counts too, which I think will be pretty useful, but that felt good to put in another diff.)

Test Plan: See screenshots.

Reviewers: chad

Reviewed By: chad

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D9543
2014-06-14 21:12:19 -07:00