1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-02 11:42:42 +01:00
Commit graph

631 commits

Author SHA1 Message Date
epriestley
5784e3d3c0 Omit "type" attribute from "<source />" tags in "<video>" to trick Chrome into playing them
Summary:
Fixes T13135. See PHI633. For at least some video files with legitimate MIME type "video/quicktime", Chrome can play them but refuses to if the `<source />` tag has a `type="video/quicktime"` attribute.

To trick Chrome into giving these videos the old college try, omit the "type" attribute. Chrome then tries to play the video, seems to realize it can, and we're back on track.

Since the "type" attribute is theoretically only useful to help browsers select among multiple different alternatives and we're only presenting one alternative, this seems likely safe and reasonable. Omitting "type" also validates. It's hard to be certain that this won't cause any collateral damage, but intuitively it seems like it should be safe and I wasn't able to identify any problems.

Test Plan:
  - Watched a "video/quicktime" MP4 cat video in Chrome/Safari/Firefox.
  - See T13135 for discussion, context, and discussion of the behavior of some smaller reproduction cases.

Reviewers: amckinley, asherkin

Reviewed By: amckinley

Maniphest Tasks: T13135

Differential Revision: https://secure.phabricator.com/D19424
2018-05-04 09:28:47 -07:00
epriestley
332f4ab66d Restore support for using "arc download" to fetch files with no "security.alternate-file-domain"
Summary:
Fixes T13132. I removed this branch in D19156 when tightening the logic for the new CSP header, but there's a legitimate need for it: downloading files via `arc download`, or more generally being an API consumer of files.

This is not completely safe, but attacks I'm aware of (particularly, cookie fixation, where an attacker could potentially force a victim to become logged in to an account they control) are difficult and not very powerful. We already issue clear setup advice about the importance of configuring this option ("Phabricator is currently configured to serve user uploads directly from the same domain as other content. This is a security risk.") and I think there's significant value in letting API clients just GET file data without having to jump through a lot of weird hoops.

Test Plan:
  - With `security.alternate-file-domain` off, tried to `arc download` a file.
  - Before: downloaded an HTML dialog page.
  - After: downloaded the file.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13132

Differential Revision: https://secure.phabricator.com/D19421
2018-05-01 10:08:05 -07:00
epriestley
afc3099ee7 Add a view option to disable blame in Diffusion and fix some view transition bugs
Summary:
See PHI604. Ref T13130. Ref T13105. There's currently no way to turn blame off in Diffusion. Add a "Hide Blame" option to the "View Options" dropdown so it can be toggled off.

Also fix a couple of bugs around this: for example, if you loaded a Jupyter notebook and then switched to "Source" view, blame would incorrectly fail to activate because the original rendering of the "stage" used an asynchronous engine so `willRenderRef()` wasn't called to populate blame.

Test Plan:
  - Viewed a source file, toggled blame off/on, reloaded page to see state stick in URL.
  - Viewed a Jupyter notebook, toggled to "Source" view, saw blame.
  - Viewed stuff in Files (no blame UI options).
  - Tried to do some invalid stuff like toggle blame on a non-blame engine (options disable properly).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130, T13105

Differential Revision: https://secure.phabricator.com/D19414
2018-04-30 15:32:23 -07:00
epriestley
223d7b84dd Recover more gracefully when favicon configuration points at a corrupt/damaged file
Summary:
Ref T13103. Locally, I managed to break the data for a bunch of files by doing `git clean -df` in a working copy that I'd updated to a commit from many many years ago. Since `conf/local.json` wasn't on the gitignore list many years ago, this removed it, and I lost my encryption keyring.

I've symlinked my local config to a version-controlled file now to avoid this specific type of creative self-sabotage in the future, but this has exposed a few cases where we could handle things more gracefully.

One issue is that if your favicon is customized but the file it points at can't actually be loaded, we fail explosively and you really can't do anything to move forward except somehow guess that you need to fix your favicon. Instead, recover more gracefully.

Test Plan:
  - Configure file encryption.
  - Configure a favicon.
  - Remove the encryption key from your keyring.
  - Purge Phabricator's caches.
  - Before: you pretty much dead-end on a fatal that's hard to understand/fix.
  - After: everything works except your favicon.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13103

Differential Revision: https://secure.phabricator.com/D19406
2018-04-27 12:02:32 -07:00
epriestley
665529ab60 Restore coverage reporting to Diffusion browse UI
Summary:
Depends on D19377. Ref T13125. Ref T13124. Ref T13105. Coverage reporting in Diffusion didn't initially survive the transition to Document Engine; restore it.

This adds some tentative/theoretical support for multiple columns of coverage, but no way to actually produce them in the UI. For now, the labels, codes, and colors are hard coded.

Test Plan:
Added coverage with `diffusion.updatecoverage`, saw coverage in the UI:

{F5525542}

Hovered over coverage, got labels and highlighting.

Double-checked labels for "N" (Not Executable) and "U" (Uncovered). See PHI577.

Faked some multi-column coverage, but you can't currently get this yourself today:

{F5525544}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13125, T13124, T13105

Differential Revision: https://secure.phabricator.com/D19378
2018-04-17 14:51:47 -07:00
epriestley
70056a9072 When creating a file by downloading a URI, truncate the length of the default name
Summary:
See <https://discourse.phabricator-community.org/t/embedding-external-images-url-show-error-for-long-urls/1339>.

When we download a file from a URI, we provide a default name based on the URI. However, if the URI is something like `http://example.com/very-very-very-....-long.jpg` with more than 255 characters, we may suggest a name which won't fit into the `name` column of `PhabricatorFile`.

Instead, suggest a default name no longer than 64 bytes.

Test Plan:
  - Used the `{image ...}` example from the Discourse report locally; got an image with a truncated name.
  - Used a normal `{image ...}`, got an image file with a normal name.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19353
2018-04-12 13:29:53 -07:00
epriestley
09c6d42b95 Mostly make blame work with DocumentEngine
Summary: Ref T13105. This needs refinement but blame sort of works again, now.

Test Plan: Viewed files in Diffusion and Files; saw blame in Diffusion when viewing in source mode.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19309
2018-04-09 04:48:21 -07:00
epriestley
90a614778c Make repository symbol references work with DocumentEngine
Summary: Ref T13105. Ref T13047. This makes symbol indexes work with DocumentEngine in Files, and restores support in Diffusion.

Test Plan: Command-clicked stuff, got taken to the symbol index with reasonable metadata in Diffusion, Differential and Files.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105, T13047

Differential Revision: https://secure.phabricator.com/D19307
2018-04-09 04:47:28 -07:00
epriestley
0363febeb2 Disable default syntax highlighting for large files in DocumentEngine
Summary: Ref T13105. See also T7895. When users render very large files as source via DocumentEngine, skip highlighting.

Test Plan: Fiddled with the limit, viewed files, saw highlighting degrade.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19306
2018-04-09 04:47:08 -07:00
epriestley
6dea2ba3b3 Fix DocumentEngine line behaviors in Diffusion
Summary:
Ref T13105. Fixes some issues with line linking and highlighting under DocumentEngine:

  - Adding `$1-3` to the URI didn't work correctly with query parameters.
  - Reading `$1-3` from the URI didn't work correctly because Diffusion parses them slightly abnormally.

Test Plan: Clicked/dragged lines to select them. Observed URI. Reloaded page, got the right selection.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19305
2018-04-09 04:46:47 -07:00
epriestley
1fde4a9450 Move Diffusion browse rendering to DocumentEngine, breaking almost all features
Summary:
Ref T13105. This breaks about 9,000 features but moves Diffusion to DocumentEngine for rendering. See T13105 for a more complete list of all the broken stuff.

But you can't bake a software without breaking all the features every time you make a change, right?

Test Plan: Viewed various files in Diffusion, used DocumentEngine features like highlighting and rendering engine selection.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Subscribers: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19302
2018-04-09 04:46:26 -07:00
epriestley
245132a0b2 Pull file Document Engine rendering out of "Files" application controllers
Summary:
Ref T13105. This separates document rendering from the Controllers which trigger it so it can be reused elsewhere (notably, in Diffusion).

This shouldn't cause any application behavior to change, it just pulls the rendering logic out so it can be reused elsewhere.

Test Plan: Viewed various types of files in Files; toggled rendering, highlighting, and encoding.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19301
2018-04-09 04:45:58 -07:00
epriestley
7189cb7ba8 Support text encoding and syntax highlighting options in document rendering
Summary: Depends on D19273. Ref T13105. Adds "Change Text Encoding..." and "Highlight As..." options when rendering documents, and makes an effort to automatically detect and handle text encoding.

Test Plan:
  - Uploaded a Shift-JIS file, saw it auto-detect as Shift-JIS.
  - Converted files between encodings.
  - Highlighted various things as "Rainbow", etc.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19274
2018-03-30 11:28:52 -07:00
epriestley
ccbc8a430f Make Jupyter notebooks use the fast builtin Python highlighter
Summary:
Ref T13105. This is silly, but "py" and "python" end up in different places today, and "py" is ~100x faster than "python".

See also T3626 for longer-term plans on this.

Test Plan: Reloaded a Jupyter notebook, saw it render almost instantly instead of taking a few seconds.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19273
2018-03-30 11:26:48 -07:00
epriestley
c5b244bfd0 Render directly embedded image data represented as a string in Jupyter notebooks
Summary: Depends on D19259. Ref T13105. Some examples represent image data as `["da", "ta"]` while others represent it as `"data"`. Accept either.

Test Plan: Rendered example notebooks with both kinds of images.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19260
2018-03-28 15:08:44 -07:00
epriestley
38999e25ac Support logged-out access to the document rendering endpoint
Summary: Ref T13105. Currently, logged-out users can't render documents via the endpoint even if they otherwise have access to the file.

Test Plan: Viewed a file as a logged-out user and re-rendered it via Ajax.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19258
2018-03-28 15:01:36 -07:00
epriestley
bba1b185f8 Improve minor client behaviors for document rendering
Summary:
Ref T13105. This adds various small client-side improvements to document rendering.

  - In the menu, show which renderer is in use.
  - Make linking to lines work.
  - Make URIs persist information about which rendering engine is in use.
  - Improve the UI feedback for transitions between document types.
  - Load slower documents asynchronously by default.
  - Discard irrelevant requests if you spam the view menu.

Test Plan: Loaded files, linked to lines, swapped between modes, copy/pasted URLs.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19256
2018-03-23 14:09:31 -07:00
epriestley
4906364751 Add a JSON document rendering engine
Summary: Depends on D19254. This engine just formats JSON files in a nicer, more readable way.

Test Plan: Looked at some JSON files, saw them become formatted nicely.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Differential Revision: https://secure.phabricator.com/D19255
2018-03-23 12:29:05 -07:00
epriestley
d2727d24da Add an abstract "Text" document engine and a "Source" document engine
Summary: Ref T13105. Allow normal text files to be rendered as documents, and add a "source code" rendering engine.

Test Plan: Viewed some source code.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19254
2018-03-23 12:28:43 -07:00
epriestley
cbf3d3c371 Add a very rough, proof-of-concept Jupyter notebook document engine
Summary:
Depends on D19252. Ref T13105. This very roughly renders Jupyter notebooks.

It's probably better than showing the raw JSON, but not by much.

Test Plan:
  - Viewed various notebooks with various cell types, including markdown, code, stdout, stderr, images, HTML, and Javascript.
  - HTML and Javascript are not live-fired since they're wildly dangerous.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19253
2018-03-23 07:14:45 -07:00
epriestley
fb4ce851c4 Add a PDF document "rendering" engine
Summary:
Depends on D19251. Ref T13105. This adds rendering engine support for PDFs.

It doesn't actually render them, it just renders a link which you can click to view them in a new window. This is much easier than actually rendering them inline and at least 95% as good most of the time (and probably more-than-100%-as-good some of the time).

This makes PDF a viewable MIME type by default and adds a narrow CSP exception for it. See also T13112.

Test Plan:
  - Viewed PDFs in Files, got a link to view them in a new tab.
  - Clicked the link in Safari, Chrome, and Firefox; got inline PDFs.
  - Verified primary CSP is still `object-src 'none'` with `curl ...`.
  - Interacted with the vanilla lightbox element to check that it still works.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19252
2018-03-23 07:14:17 -07:00
epriestley
8b658706a8 Add a basic Remarkup document rendering engine
Summary:
Ref T13105. Although Markdown is trickier to deal with, we can handle Remarkup easily.

This may need some support for encoding options.

Test Plan: Viewed `.remarkup` files, got remarkup document presentation by default. Viewed other text files, got an option to render as remarkup.

Reviewers: avivey

Reviewed By: avivey

Subscribers: mydeveloperday, avivey

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19251
2018-03-23 07:07:50 -07:00
epriestley
4aafce6862 Add filesize limits for document rendering engines and support partial/complete rendering
Summary:
Depends on D19238. Ref T13105. Give document engines some reasonable automatic support for degrading gracefully when someone tries to hexdump a 100MB file or similar.

Also, make "Video" sort above "Audio" for files which could be rendered either way.

Test Plan: Viewed audio, video, image, and other files. Adjusted limits and saw full, partial, and fallback/error rendering.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19239
2018-03-19 15:18:34 -07:00
epriestley
f646153f4d Add an async driver for document rendering and a crude "Hexdump" document engine
Summary: Depends on D19237. Ref T13105. This adds a (very basic) "Hexdump" engine (mostly just to have a second option to switch to) and a selector for choosing view modes.

Test Plan: Viewed some files, switched between audio/video/image/hexdump.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19238
2018-03-19 15:18:05 -07:00
epriestley
01f22a8d06 Roughly modularize document rendering in Files
Summary:
Ref T13105. This change begins modularizing document rendering. I'm starting in Files since it's the use case with the smallest amount of complexity.

Currently, we hard-coding the inline rendering for images, audio, and video. Instead, use the modular engine pattern to make rendering flexible and extensible.

There aren't any options for switching modes yet and none of the renderers do anything fancy. This API is also probably very unstable.

Test Plan: Viewwed images, audio, video, and other files. Saw reasonable renderings, with "nothing can render this" for any other file type.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19237
2018-03-19 15:17:04 -07:00
epriestley
d80a53abcc Skip loading file transform sources when we know a file is not transformed
Summary:
Depends on D19223. Ref T13106. When we're loading a file, we currently test if it's a transformed version of another file (usually, a thumbnail) and apply policy behavior if it is.

We know that builtins and profile images are never transforms and that the policy behavior for these files doesn't matter anyway. Skip loading transforms for these files.

Test Plan: Saw local queries drop from 146 to 139 with no change in behavior.

Maniphest Tasks: T13106

Differential Revision: https://secure.phabricator.com/D19224
2018-03-14 12:45:06 -07:00
epriestley
31bd3679f0 Skip loading attached objects for files when we know the file is visible
Summary:
Depends on D19222. Ref T13106. We currently execute an edge query (and possibly an object query) when loading builtin files, but this is never necessary because we know these files are always visible.

Instead, skip this logic for builtin files and profile image files; these files have global visibility and will never get a different policy result because of file attachment information.

(In theory, we could additionally skip this for files with the most open visibility policy or some other trivially visible policy like the user's PHID, but we do actually care about the attachment data some of the time.)

Test Plan: Saw queries drop from 151 to 145 on local test page. Checked file attachment data in Files, saw it still working correctly.

Maniphest Tasks: T13106

Differential Revision: https://secure.phabricator.com/D19223
2018-03-14 12:36:46 -07:00
epriestley
3c4f31e4b9 Dynamically composite favicons from customizable sources
Summary: Ref T13103. Make favicons customizable, and perform dynamic compositing to add marker to indicate things like "unread messages".

Test Plan: Viewed favicons in Safari, Firefox and Chrome. With unread messages, saw pink dot composited into icon.

Maniphest Tasks: T13103

Differential Revision: https://secure.phabricator.com/D19209
2018-03-12 15:28:41 -07:00
epriestley
a3d282d33e Somewhat improve meme transform code so it is merely very bad
Summary: Depends on D19200. Fixes T5258. Ref T13101. Attempt to simplify and modernize this code and improve error handling.

Test Plan: did real hard dank memes

Maniphest Tasks: T13101, T5258

Differential Revision: https://secure.phabricator.com/D19201
2018-03-08 11:08:55 -08:00
epriestley
a099a06265 Remove some old image transform code with no callsites
Summary: Ref T13101. Ref T5258. This old image transform code no longer has callsites.

Test Plan: Grepped for removed methods, no hits.

Maniphest Tasks: T13101, T5258

Differential Revision: https://secure.phabricator.com/D19198
2018-03-08 11:04:53 -08:00
epriestley
6095d88998 Don't require prototypes for "{image ...}"
Summary: Depends on D19194. Fixes T4190. This should be in good-enough shape now to release and support more generally.

Test Plan: Used `{image ...}` in remarkup.

Maniphest Tasks: T4190

Differential Revision: https://secure.phabricator.com/D19195
2018-03-08 07:04:23 -08:00
epriestley
b30535a36f When rendering "{image ...}" images, check the cache and just render a direct "<img />" tag if possible
Summary: Depends on D19193. Ref T13101. Fixes T4190. Before we render a fancy AJAX placeholder, check if we already have a valid cache for the image. If we do, render a direct `<img />` tag. This is a little cleaner and, e.g., avoids flicker in Safari, at least.

Test Plan: Rendered `{image ...}` rules in remarkup with new and existing URIs.

Maniphest Tasks: T13101, T4190

Differential Revision: https://secure.phabricator.com/D19194
2018-03-08 07:03:55 -08:00
epriestley
9d3a722eb1 When proxying an "{image ...}" image fails, show the user an error message
Summary:
Depends on D19192. Ref T4190. Ref T13101. Instead of directly including the proxy endpoint with `<img src="..." />`, emit a placeholder and use AJAX to make the request. If the proxy fetch fails, replace the placeholder with an error message.

This isn't the most polished implementation imaginable, but it's much less mysterious about errors.

Test Plan: Used `{image ...}` for valid and invalid images, got images and useful error messages respectively.

Maniphest Tasks: T13101, T4190

Differential Revision: https://secure.phabricator.com/D19193
2018-03-08 07:03:26 -08:00
epriestley
01bbd71b96 Separate the "{img ...}" remarkup rule into separate parse and markup phases
Summary:
Ref T13101. Ref T4190. This rule is currently single-phase but I'd like to check for a valid proxied image in cache already and just emit an `<img ... />` tag pointing at it if we have one.

To support batching these lookups, split the rule into a parse phase (where we extract URIs) and a markup phase (where we build tags).

Test Plan: Used `{img ...}` in Remarkup with no apparent behavioral changes. (This change should do nothing on its own.)

Maniphest Tasks: T13101, T4190

Differential Revision: https://secure.phabricator.com/D19192
2018-03-08 07:02:59 -08:00
epriestley
14fe941c34 Reduce the cost of generating default user profile images
Summary:
See PHI413. You can pre-generate these with `bin/people profileimage --all`, but they're needlessly expensive to generate.

Streamline the workflow and cache some of the cacheable parts to reduce the generation cost.

Test Plan:
  - Ran `bin/people profileimage --all` and saw cost drop from {nav 15.801s > 4.839s}.
  - Set `defaultProfileImagePHID` to `NULL` in `phabricator_user.user` and purged caches with `bin/cache purge --all`.
  - Loaded user directory.
  - Saw default images regenerate relatively quickly.

Differential Revision: https://secure.phabricator.com/D19168
2018-03-01 16:53:17 -08:00
epriestley
ab579f2511 Never generate file download forms which point to the CDN domain, tighten "form-action" CSP
Summary:
Depends on D19155. Ref T13094. Ref T4340.

We can't currently implement a strict `form-action 'self'` content security policy because some file downloads rely on a `<form />` which sometimes POSTs to the CDN domain.

Broadly, stop generating these forms. We just redirect instead, and show an interstitial confirm dialog if no CDN domain is configured. This makes the UX for installs with no CDN domain a little worse and the UX for everyone else better.

Then, implement the stricter Content-Security-Policy.

This also removes extra confirm dialogs for downloading Harbormaster build logs and data exports.

Test Plan:
  - Went through the plain data export, data export with bulk jobs, ssh key generation, calendar ICS download, Diffusion data, Paste data, Harbormaster log data, and normal file data download workflows with a CDN domain.
  - Went through all those workflows again without a CDN domain.
  - Grepped for affected symbols (`getCDNURI()`, `getDownloadURI()`).
  - Added an evil form to a page, tried to submit it, was rejected.
  - Went through the ReCaptcha and Stripe flows again to see if they're submitting any forms.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13094, T4340

Differential Revision: https://secure.phabricator.com/D19156
2018-02-28 17:20:12 -08:00
epriestley
afc98f5d5d Remove defunct "download" route in Files pointing to nonexistent controller
Summary:
Depends on D19154. Ref T13094. This controller was removed at some point and this route no longer works.

I plan to add a new `download/` route to let us tighten the `form-action` Content Security Policy.

Test Plan: Grepped for the route and controller, no hits.

Maniphest Tasks: T13094

Differential Revision: https://secure.phabricator.com/D19155
2018-02-28 17:19:52 -08:00
epriestley
05a4c55c52 Explicitly add rel="noreferrer" to all external links
Summary: See D19117. Instead of automatically figuring this out inside `phutil_tag()`, explicitly add rel="noreferrer" at the application level to all external links.

Test Plan:
  - Grepped for `_blank`, `isValidRemoteURIForLink`, checked all callsites for user-controlled data.
  - Created a link menu item, verified noreferrer in markup.
  - Created a link custom field, verified no referrer in markup.
  - Verified noreferrer for `{nav href=...}`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19118
2018-02-17 17:46:11 -08:00
epriestley
f090fa7426 Use object PHIDs for "Thread-Topic" headers in mail
Summary:
Depends on D19009. Ref T13053. For "Must Encrypt" mail, we must currently strip the "Thread-Topic" header because it sometimes contains sensitive information about the object.

I don't actually know if this header is useful or anyting uses it. My understanding is that it's an Outlook/Exchange thing, but we also implement "Thread-Index" which I think is what Outlook/Exchange actually look at. This header may have done something before we implemented "Thread-Index", or maybe never done anything. Or maybe older versions of Excel/Outlook did something with it and newer versions don't, or do less. So it's possible that an even better fix here would be to simply remove this, but I wasn't able to convince myself of that after Googling for 10 minutes and I don't think it's worth hours of installing Exchange/Outlook to figure out. Instead, I'm just trying to simplify our handling of this header for now, and maybe some day we'll learn more about Exchange/Outlook and can remove it.

In a number of cases we already use the object monogram or PHID as a "Thread-Topic" without users ever complaining, so I think that if this header is useful it probably isn't shown to users, or isn't shown very often (e.g., only in a specific "conversation" sub-view?). Just use the object PHID (which should be unique and stable) as a thread-topic, everywhere, automatically.

Then allow this header through for "Must Encrypt" mail.

Test Plan: Processed some local mail, saw object PHIDs for "Thread-Topic" headers.

Reviewers: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19012
2018-02-08 06:21:00 -08:00
epriestley
8b8a3142b3 Support export of data in files larger than 8MB
Summary:
Depends on D18952. Ref T13049. For files larger than 8MB, we need to engage the chunk storage engine. `PhabricatorFile::newFromFileData()` always writes a single chunk, and can't handle files larger than the mandatory chunk threshold (8MB).

Use `IteratorUploadSource`, which can, and "stream" the data into it. This should raise the limit from 8MB to 2GB (maximum size of a string in PHP).

If we need to go above 2GB we could stream CSV and text pretty easily, and JSON without too much trouble, but Excel might be trickier. Hopefully no one is trying to export 2GB+ datafiles, though.

Test Plan:
  - Changed the JSON exporter to just export 8MB of the letter "q": `return str_repeat('q', 1024 * 1024 * 9);`.
  - Before change: fatal, "no storage engine can store this file".
  - After change: export works cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18953
2018-01-29 15:58:34 -08:00
epriestley
6a62797056 Fix some issues with Diffusion file data limits
Summary:
See <https://discourse.phabricator-community.org/t/files-created-from-repository-contents-slightly-over-one-chunk-in-size-are-truncated-to-exactly-one-chunk-in-size/988/1>. Three issues here:

  - When we finish reading `git cat-file ...` or whatever, we can end up with more than one chunk worth of bytes left in the internal buffer if the read is fast. Use `while` instead of `if` to make sure we write the whole buffer.
  - Limiting output with `setStdoutSizeLimit()` isn't really a reliable way to limit the size if we're also reading from the buffer. It's also pretty indirect and confusing. Instead, just let the `FileUploadSource` explicitly implement a byte limit in a straightforward way.
  - We weren't setting the time limit correctly on the main path.

Overall, this could cause >4MB files to "write" as 4MB files, with the rest of the file left in the UploadSource buffer. Since these files were technically under the limit, they could return as valid. This was intermittent.

Test Plan:
  - Pushed a ~4.2MB file.
  - Reloaded Diffusion a bunch, sometimes saw the `while/if` buffer race and produce a 4MB file with a prompt to download it. (Other times, the buffer worked right and the page just says "this file is too big, sorry").
  - Applied patches.
  - Reloaded Diffusion a bunch, no longer saw bad behavior or truncated files.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18885
2018-01-22 11:52:37 -08:00
epriestley
53b25db918 Prevent enormous changes from being pushed to repositoires by default
Summary:
Fixes T13031. "Enormous" changes are basically changes which are too large to hold in memory, although the actual definition we use today is "more than 1GB of change text or `git diff` runs for more than 15 minutes".

If an install configures a Herald content rule like "when content matches /XYZ/, do something" and then a user pushes a 30 GB source file, we can't put it into memory to `preg_match()` it. Currently, the way to handle this case is to write a separate Herald rule that rejects enormous changes. However, this isn't obvious and means the default behavior is unsafe.

Make the default behavior safe by rejecting these changes with a message, similar to how we reject "dangerous" changes (which permanently delete or overwrite history) by default.

Also, change a couple of UI strings from "Enormous" to "Very Large" to reduce ambiguity. See <https://discourse.phabricator-community.org/t/herald-enormous-check/822>.

Test Plan: Changed the definition of "enormous" from 1GB to 1 byte. Pushed a change; got rejected. Allowed enormous changes, pushed, got rejected by a Herald rule. Disabled the Herald rule, pushed, got a clean push. Prevented enormous changes again. Grepped for "enormous" elsewhere in the UI.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: joshuaspence

Maniphest Tasks: T13031

Differential Revision: https://secure.phabricator.com/D18850
2018-01-04 10:02:29 -08:00
epriestley
e1d6bad864 Stop trying to assess the image dimensions of large files and file chunks
Summary:
Depends on D18828. Ref T7789. See <https://discourse.phabricator-community.org/t/git-lfs-fails-with-large-images/584>.

Currently, when you upload a large (>4MB) image, we may try to assess the dimensions for the image and for each individual chunk.

At best, this is slow and not useful. At worst, it fatals or consumes a ton of memory and I/O we don't need to be using.

Instead:

  - Don't try to assess dimensions for chunked files.
  - Don't try to assess dimensions for the chunks themselves.
  - Squelch errors for bad data, etc., that `gd` can't actually read, since we recover sensibly.

Test Plan:
  - Created a 2048x2048 PNG in Photoshop using the "Random Noise" filter which weighs 8.5MB.
  - Uploaded it.
  - Before patch: got complaints in log about `imagecreatefromstring()` failing, although the actual upload went OK in my environment.
  - After patch: clean log, no attempt to detect the size of a big image.
  - Also uploaded a small image, got dimensions detected properly still.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7789

Differential Revision: https://secure.phabricator.com/D18830
2017-12-18 09:17:32 -08:00
epriestley
f786c86a6a Don't require the "gd" extension be installed in order to run unit tests
Summary:
Ref T10405. If `gd` is not installed, a number of unit tests currently fail because they generate synthetic users and try to composite profile pictures for them.

Instead, just fall back to a static default if `gd` is not available.

Test Plan: Faked this pathway, created a new user, saw a default profile picture.

Reviewers: amckinley, lpriestley, avivey

Reviewed By: avivey

Maniphest Tasks: T10405

Differential Revision: https://secure.phabricator.com/D18812
2017-11-30 13:51:31 -08:00
epriestley
e919233b31 Don't show personalized menu items until users establish a full session
Summary:
Depends on D18792. Fixes T13024. Fixes T89198. Currently, when users are logging in initially (for example, need to enter MFA) we show more menu items than we should.

Notably, we may show some personalized/private account details, like the number of unread notifications (probably not relevant) or a user's saved queries (possibly sensitive). At best these are misleading (they won't work yet) and there's an outside possibility they leak a little bit of private data.

Instead, nuke everything except "Log Out" when users have partial sessions.

Test Plan:
Hit a partial session (MFA required, email verification required) and looked at the menu. Only saw "Log Out".

{F5297713}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024

Differential Revision: https://secure.phabricator.com/D18793
2017-11-28 10:01:58 -08:00
epriestley
b04d9fdc0b Add a missing key to PhabricatorFile for destroying Files
Summary: See PHI176. Depends on D18733. We issue a query when deleting files that currently doesn't hit any keys.

Test Plan:
Ran `./bin/remove destroy --force --trace F56376` to get the query.

Ran ##SELECT * FROM `file` WHERE storageEngine = 'blob' AND storageHandle = '23366' LIMIT 1## before and after the change.

Before:

```
mysql> explain SELECT * FROM `file` WHERE storageEngine = 'blob' AND storageHandle = '23366' LIMIT 1;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | file  | ALL  | NULL          | NULL | NULL    | NULL | 33866 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.01 sec)
```

After:

```
mysql> explain SELECT * FROM `file` WHERE storageEngine = 'blob' AND storageHandle = '23366' LIMIT 1;
+----+-------------+-------+------+---------------+------------+---------+-------------+------+------------------------------------+
| id | select_type | table | type | possible_keys | key        | key_len | ref         | rows | Extra                              |
+----+-------------+-------+------+---------------+------------+---------+-------------+------+------------------------------------+
|  1 | SIMPLE      | file  | ref  | key_engine    | key_engine | 388     | const,const |  190 | Using index condition; Using where |
+----+-------------+-------+------+---------------+------------+---------+-------------+------+------------------------------------+
1 row in set (0.00 sec)
```

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18734
2017-10-26 18:20:12 -07:00
epriestley
9777c66576 Allow Phabricator to run with "enable_post_data_reading" disabled
Summary:
Ref T13008. Depends on D18701. The overall goal here is to make turning `enable_post_data_reading` off not break things, so we can run rate limiting checks before we read file uploads.

The biggest blocker for this is that turning it off stops `$_FILES` from coming into existence.

This //appears// to mostly work. Specifically:

  - Skip the `max_post_size` check when POST is off, since it's meaningless.
  - Don't read or scrub $_POST at startup when POST is off.
  - When we rebuild REQUEST and POST before processing requests, do multipart parsing if we need to and rebuild FILES.
  - Skip the `is_uploaded_file()` check if we built FILES ourselves.

This probably breaks a couple of small things, like maybe `__profile__` and other DarkConsole triggers over POST, and probably some other weird stuff. The parsers may also need more work than they've received so far.

I also need to verify that this actually works (i.e., lets us run code without reading the request body) but I'll include that in the change where I update the actual rate limiting.

Test Plan:
  - Disabled `enable_post_data_reading`.
  - Uploaded a file with a vanilla upload form (project profile image).
  - Uploaded a file with drag and drop.
  - Used DarkConsole.
  - Submitted comments.
  - Created a task.
  - Browsed around.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13008

Differential Revision: https://secure.phabricator.com/D18702
2017-10-13 13:11:11 -07:00
Dmitri Iouchtchenko
9bd6a37055 Fix spelling
Summary: Noticed a couple of typos in the docs, and then things got out of hand.

Test Plan:
  - Stared at the words until my eyes watered and the letters began to swim on the screen.
  - Consulted a dictionary.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18693
2017-10-09 10:48:04 -07:00
Chad Little
d3db3fff59 Update file edit UI
Summary: New white box

Test Plan: /file/upload/

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18557
2017-09-07 11:35:40 -07:00
epriestley
8443366f32 Remove bin/files purge workflow
Summary:
Fixes T12948. See that task for substantial discussion and context. Briefly:

  - This workflow is very old, and won't work for large (>2GB) files.
  - This workflow has become more dangerous than it once was, and can fail in several ways that delete data and/or make recovery much more difficult (see T12948 for more discussion).
  - This was originally added in D6068, which is a bit muddled, but looks like "one install ran into a weird issue so I wrote a script for them"; this would be a Consulting/Support issue and not come upstream today. I can't identify any arguments for retaining this workflow there, at least.

Test Plan:
  - Grepped for `files purge`, got nothing.
  - Grepped for `purge`, looked for anything that looked like instructions or documentation, got nothing.

I don't recall recommending anyone run this script in many years, and didn't even remember that it existed or what it did when T12948 was reported, so I believe it is not in widespread use.

Reviewers: joshuaspence, chad

Reviewed By: joshuaspence

Maniphest Tasks: T12948

Differential Revision: https://secure.phabricator.com/D18384
2017-08-10 08:49:06 -07:00
Chad Little
ba4b936dff Use Log In vs. Login when it's a verb
Summary: Cursory research indicates that "login" is a noun, referring to a form, and "log in" is a verb, referring to the action of logging in. I went though every instances of 'login' I could find and tried to clarify all this language. Also, we have "Phabricator" on the registration for like 4-5 times, which is a bit verbose, so I tried to simplify that language as well.

Test Plan: Tested logging in and logging out. Pages feel simpler.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18322
2017-08-02 12:26:47 -07:00
Chad Little
250aaabd64 Choose default project image by icon
Summary: Builds out a map for icon->image in Projects, selects the icon's image as the default project image if there is no custom image chosen by the user.

Test Plan: Select various icons, see image change. Test choose picture, pick a new image.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18174
2017-07-09 11:41:02 -07:00
epriestley
224c4692ee Add a cache purger for builtin files
Summary: Fixes T12859.

Test Plan:
  - Loaded Diffusion builtin icons before recent updates, saw cached builtins.
  - Ran `bin/cache purge --caches builtin-file`.
  - Reloaded, saw new files.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12859

Differential Revision: https://secure.phabricator.com/D18147
2017-06-22 11:13:23 -07:00
epriestley
17fc447503 Don't compute MIME type of noninitial chunks from diffusion.filecontentquery
Summary:
Ref T12857. This is generally fairly fuzzy for now, but here's something concrete: when we build a large file with `diffusion.filecontentquery`, we compute the MIME type of all chunks, not just the initial chunk.

Instead, pass a dummy MIME type to non-initial chunks so we don't try to compute them. This mirrors logic elsewhere, in `file.uploadchunk`. This should perhaps be centralized at some point, but it's a bit tricky since the file doesn't know that it's a chunk until later.

Also, clean up the `TempFile` immediately -- this shouldn't actually affect anything, but we don't need it to live any longer than this.

Test Plan:
  - Made `hashFileContent()` return `null` to skip the chunk cache.
  - Added `phlog()` to the MIME type computation.
  - Loaded a 12MB file in Diffusion.
  - Before patch: Saw 3x MIME type computations, one for each 4MB chunk.
  - After patch: Saw 1x MIME type computation, for initial chunk only.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12857

Differential Revision: https://secure.phabricator.com/D18138
2017-06-20 05:45:20 -07:00
epriestley
4b15871ced Fix specification of ttl.relative via LFS
Summary: Fixes T12828. This API was tightened up D17616, but I missed this callsite.

Test Plan: I've just got a good feeling in my bones.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12828

Differential Revision: https://secure.phabricator.com/D18119
2017-06-12 11:24:20 -07:00
Chad Little
38904ea4ad More button grey conversions
Summary: Ran across a few straglers. Convert to the correct color.

Test Plan: grep for profile-image-button, check profile image selection page

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18096
2017-06-07 09:34:50 -07:00
Chad Little
d3c464a610 Separate button CSS classes
Summary: Try to dis-ambiguate various button types and colors. Moves `simple` to `phui-button-simple` and moves colors to `button-color`.

Test Plan: Grep for buttons still inline, UIExamples, PHUIX, Herald, and Email Preferences.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18077
2017-06-05 20:14:34 +00:00
Chad Little
00400ae6f9 Search and Replace calls to setShade
Summary: grep for setShade and update to setColor. Add deprecated warning.

Test Plan: Diffusion, Workboards, Maniphest, Project tags, tokenizer, uiexamples

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, O14 ATC Monitoring

Differential Revision: https://secure.phabricator.com/D17995
2017-05-22 18:59:53 +00:00
Austin McKinley
eb84bf98a4 Migrate Project image to modular transactions
Summary: I'm not sure you can actually remove a project's image (maybe via the API?), but I kept the code for rendering the relevant title/feed anyway.

Test Plan: Unit tests + adding/changing project pictures.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D17954
2017-05-18 16:08:37 -07:00
Chad Little
d6a620be45 Update Maniphest for modular transactions
Summary: Ref T12671. This modernized Maniphest transactions to modular transactions.

Test Plan:
- Create Task
- Edit Task
- Raise Priority
- Change Status
- Merge as a duplicate
- Create Subtask
- Claim Task
- Assign Project
- Move on Workboard
- Set a cover image
- Assign story points
- Change story points
- Generate lots via lipsum
- Bulk edit tasks
- Leave comments
- Award Token

I'm sure I'm missing something.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: hazelyang, Korvin

Maniphest Tasks: T12671

Differential Revision: https://secure.phabricator.com/D17844
2017-05-15 10:29:06 -07:00
epriestley
bb2b91d28e Add unit tests for file thumbnail generation
Summary: Fixes T12614.

Test Plan:
  - Ran tests, saw them pass.
  - Changed a thumbnail to 800x800, saw tests detect that the default image was missing.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12614

Differential Revision: https://secure.phabricator.com/D17773
2017-04-23 11:02:21 -07:00
epriestley
b479941ceb Make "Range" HTTP header work for Celerity static resource requests
Summary:
Ref T7567. In T8266 I fixed a bunch of obscure "Range" issues, but only for file downloads -- not for Celerity.

Extend all that stuff to Celerity, which is fortunately much easier.

I believe this will fix Conpherence sounds in Safari.

Test Plan:
  - Wrote out an HTTP request in a text file with `Range: bytes=0-1` and similar, piped it to localhost with `cat request.txt | nc localhost 80`, saw server return appropriate range responses consistent with file behavior after T8266, which all seems to work.
  - Also did that for files to try to make sure I wasn't breaking anything.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T7567

Differential Revision: https://secure.phabricator.com/D17724
2017-04-18 13:46:27 -07:00
Austin McKinley
ece9579d25 Switch File deletion to use ModularTransactions
Summary: Fixes T12587. Adds a new `PhabricatorFileDeleteTransaction` that enqueues `File` delete tasks.

Test Plan:
  - hack `PhabricatorFileQuery` to ignore isDeleted state
  - stop daemons
  - upload a file, delete it from the UI
  - check that the DB has updated isDeleted = 1
  - check timeline rendering in `File` detail view
  - start daemons
  - confirm rows are deleted from DB

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, thoughtpolice

Maniphest Tasks: T12587

Differential Revision: https://secure.phabricator.com/D17723
2017-04-18 13:01:51 -07:00
Austin McKinley
be00264ae7 Make daemons perform file deletion
Summary:
Deletion is a possibly time-intensive process, especially with large
files that are backed by high-latency, chunked storage (such as
S3). Even ~200mb objects take minutes to delete, which makes for an
unhappy experience. Fixes T10828.

Test Plan:
Delete a large file, and stare in awe of the swiftness with
which I am redirected to the main file application.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: thoughtpolice, Korvin

Maniphest Tasks: T10828

Differential Revision: https://secure.phabricator.com/D15743
2017-04-18 11:09:41 -07:00
Austin McKinley
b54adc6161 Kick off indexing for File objects on creation
Summary: Ensures that newly-made `File` objects get indexed into the new ngrams index. Fixes T8788.

Test Plan:
  - uploaded a file with daemons stopped; confirmed no new rows in ngrams table
  - started daemons; confirmed indexing of previously-uploaded files happened
  - uploaded a new file with daemons running; confirmed it got added to the index

Not sure how to test the changes to `PhabricatorFileUploadSource->writeChunkedFile()` and `PhabricatorChunkedFileStorageEngine->allocateChunks()`. I spent a few minutes trying to find their callers, but the first looks like it requires a Diffusion repo and the 2nd is only accessible via Conduit. I can test that stuff if necessary, but it's such a small change that I'm not worried about it.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T8788

Differential Revision: https://secure.phabricator.com/D17718
2017-04-18 08:38:34 -07:00
Austin McKinley
976fbee877 Implement ngram search for File objects
Summary: Follows the outline in D15656 for implementing ngram search for names of File objects. Also created FileFullTextEngine, because without implementing `PhabricatorFulltextInterface`, `./bin/search` complains that `File` is not an indexable type.

Test Plan:
  - ran `./bin/storage upgrade` to apply the schema change
  - confirmed the presence of a new `file_filename_ngrams` table
  - added a couple file objects
  - ran `bin/search index --type file --force`
  - confirmed the presence of rows in `file_filename_ngrams`
  - did a few keyword searches and saw expected results

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T8788

Differential Revision: https://secure.phabricator.com/D17702
2017-04-17 17:37:20 -07:00
epriestley
21709a2bbc Remove 'isPartial' parameter with no effect
Summary: Fixes T12536. Nothing reads this parameter; `PhabricatorFile::newChunkedFile` sets the `isPartial` flag automatically.

Test Plan: Grepped for `isPartial`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12536

Differential Revision: https://secure.phabricator.com/D17654
2017-04-11 11:29:23 -07:00
epriestley
a7a068f84c Correct two parameter strictness issues with file uploads
Summary:
Fixes T12531. Strictness fallout from adding typechecking in D17616.

  - `chunkedHash` is not a real parameter, so the new typechecking was unhappy about it.
  - `mime-type` no longer allows `null`.

Test Plan:
  - Ran `arc upload --conduit-uri ... 12MB.zero` on a 12MB file full of zeroes.
  - Before patch: badness, failure, fallback to one-shot uploads.
  - After patch: success and glory.

Reviewers: chad

Subscribers: joshuaspence

Maniphest Tasks: T12531

Differential Revision: https://secure.phabricator.com/D17651
2017-04-10 16:01:15 -07:00
epriestley
49132b884b Sell Yellow! Buy Indigo!
Summary: Fixes T12504. Replaces all tags with indigo.

Test Plan: {F4849487}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12504

Differential Revision: https://secure.phabricator.com/D17649
2017-04-10 15:01:10 -07:00
epriestley
3d816e94df Rename "PhabricatorHash::digest()" to "weakDigest()"
Summary: Ref T12509. This encourages code to move away from HMAC+SHA1 by making the method name more obviously undesirable.

Test Plan: `grep`, browsed around.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17632
2017-04-06 15:43:33 -07:00
epriestley
d450a08890 Support HMAC+SHA256 with automatic key generation and management
Summary:
Ref T12509. This adds support for HMAC+SHA256 (instead of HMAC+SHA1). Although HMAC+SHA1 is not currently broken in any sense, SHA1 has a well-known collision and it's good to look at moving away from HMAC+SHA1.

The new mechanism also automatically generates and stores HMAC keys.

Currently, HMAC keys largely use a per-install constant defined in `security.hmac-key`. In theory this can be changed, but in practice essentially no install changes it.

We generally (in fact, always, I think?) don't use HMAC digests in a way where it matters that this key is well-known, but it's slightly better if this key is unique per class of use cases. Principally, if use cases have unique HMAC keys they are generally less vulnerable to precomputation attacks where an attacker might generate a large number of HMAC hashes of well-known values and use them in a nefarious way. The actual threat here is probably close to nonexistent, but we can harden against it without much extra effort.

Beyond that, this isn't something users should really have to think about or bother configuring.

Test Plan:
  - Added unit tests.
  - Used `bin/files integrity` to verify, strip, and recompute hashes.
  - Tampered with a generated HMAC key, verified it invalidated hashes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17630
2017-04-06 15:42:59 -07:00
epriestley
08a4225437 Provide "bin/files integrity" for debugging, maintaining and backfilling integrity hashes
Summary:
Ref T12470. Provides an "integrity" utility which runs in these modes:

  - Verify: check that hashes match.
  - Compute: backfill missing hashes.
  - Strip: remove hashes. Useful for upgrading across a hash change.
  - Corrupt: intentionally corrupt hashes. Useful for debugging.
  - Overwrite: force hash recomputation.

Users normally shouldn't need to run any of this stuff, but this provides a reasonable toolkit for managing integrity hashes.

I'll recommend existing installs use `bin/files integrity --compute all` in the upgrade guidance to backfill hashes for existing files.

Test Plan:
  - Ran the script in many modes against various files, saw expected operation, including:
  - Verified a file, corrupted it, saw it fail.
  - Verified a file, stripped it, saw it have no hash.
  - Stripped a file, computed it, got a clean verify.
  - Stripped a file, overwrote it, got a clean verify.
  - Corrupted a file, overwrote it, got a clean verify.
  - Overwrote a file, overwrote again, got a no-op.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12470

Differential Revision: https://secure.phabricator.com/D17629
2017-04-06 15:42:43 -07:00
epriestley
d1a971e221 Support "Range: bytes=123-" requests
Summary:
Ref T12219. We currently only support Range requests like "bytes=123-456", but "bytes=123-", meaning "until end of file", is valid, and Chrome can send these requests.

I suspect this is the issue with T12219.

Test Plan: Used `nc local.phacility.com 80` to pipe raw requests, saw both "bytes=123-456" and "bytes=123-" requests satisfied correctly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12219

Differential Revision: https://secure.phabricator.com/D17626
2017-04-05 11:25:44 -07:00
epriestley
63828f5806 Store and verify content integrity checksums for files
Summary:
Ref T12470. This helps defuse attacks where an adversary can directly take control of whatever storage engine files are being stored in and change data there. These attacks would require a significant level of access.

Such attackers could potentially attack ranges of AES-256-CBC encrypted files by using Phabricator as a decryption oracle if they were also able to compromise a Phabricator account with read access to the files.

By storing a hash of the data (and, in the case of AES-256-CBC files, the IV) when we write files, and verifying it before we decrypt or read them, we can detect and prevent this kind of tampering.

This also helps detect mundane corruption and integrity issues.

Test Plan:
  - Added unit tests.
  - Uploaded new files, saw them get integrity hashes.
  - Manually corrupted file data, saw it fail. Used `bin/files cat --salvage` to read it anyway.
  - Tampered with IVs, saw integrity failures.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12470

Differential Revision: https://secure.phabricator.com/D17625
2017-04-05 11:12:31 -07:00
epriestley
45fc4f6f64 Iterate over ranges correctly for encryped files
Summary:
Fixes T12079. Currently, when a file is encrypted and a request has "Content-Range", we apply the range first, //then// decrypt the result. This doesn't work since you can't start decrypting something from somewhere in the middle (at least, not with our cipher selection).

Instead: decrypt the result, //then// apply the range.

Test Plan: Added failing unit tests, made them pass

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12079

Differential Revision: https://secure.phabricator.com/D17623
2017-04-05 09:56:30 -07:00
epriestley
58011a4e8e Upgrade File content hashing to SHA256
Summary:
Ref T12464. This defuses any possible SHA1-collision attacks by using SHA256, for which there is no known collision.

(SHA256 hashes are larger -- 256 bits -- so expand the storage column to 64 bytes to hold them.)

Test Plan:
  - Uploaded the same file twice, saw the two files generate the same SHA256 content hash and use the same underlying data.
  - Tried with a fake hash algorihtm ("quackxyz") to make sure the failure mode worked/degraded correctly if we don't have SHA256 for some reason. Got two valid files with two copies of the same data, as expected.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12464

Differential Revision: https://secure.phabricator.com/D17620
2017-04-04 16:23:08 -07:00
epriestley
440ef5b7a7 Remove SHA1 file content hashing and make Files work without any hashing
Summary:
Ref T12464. We currently use SHA1 to detect when two files have the same content so we don't have to store two copies of the data.

Now that a SHA1 collision is known, this is theoretically dangerous. T12464 describes the shape of a possible attack.

Before replacing this with something more robust, shore things up so things work correctly if we don't hash at all. This mechanism is entirely optional; it only helps us store less data if some files are duplicates.

(This mechanism is also less important now than it once was, before we added temporary files.)

Test Plan: Uploaded multiple identical files, saw the uploads work and the files store separate copies of the same data.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12464

Differential Revision: https://secure.phabricator.com/D17619
2017-04-04 16:22:10 -07:00
epriestley
1e181f0781 Deprecate "file.uploadhash"
Summary:
Ref T12464. This is a very old method which let you create a file on the server by referring to data which already existed in another file.

Basically, long ago, `arc` could say "Do you already have a file with hash X?" and just skip some work if the server did.

`arc` has not called this method since D13017, in May 2015.

Since it's easy to do so, just make this method pretend that it never has the file. Very old clients will continue to work, since they would expect this response in the common case and continue by uploading data.

Test Plan:
  - Grepped for `uploadhash` in Phabricator and Arcanist.
  - Called the method with the console, verified it returned `null`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12464

Differential Revision: https://secure.phabricator.com/D17618
2017-04-04 16:18:26 -07:00
epriestley
873b39be82 Remove PhabricatorFile::buildFromFileDataOrHash()
Summary:
Ref T12464. This is a very old method which can return an existing file instead of creating a new one, if there's some existing file with the same content.

In the best case this is a bad idea. This being somewhat reasonable predates policies, temporary files, etc. Modern methods like `newFromFileData()` do this right: they share underlying data in storage, but not the actual `File` records.

Specifically, this is the case where we get into trouble:

  - I upload a private file with content "X".
  - You somehow generate a file with the same content by, say, viewing a raw diff in Differential.
  - If the diff had the same content, you get my file, but you don't have permission to see it or whatever so everything breaks and is terrible.

Just get rid of this.

Test Plan:
  - Generated an SSH key.
  - Viewed a raw diff in Differential.
  - (Did not test Phragment.)

Reviewers: chad

Reviewed By: chad

Subscribers: hach-que

Maniphest Tasks: T12464

Differential Revision: https://secure.phabricator.com/D17617
2017-04-04 16:18:00 -07:00
epriestley
45b386596e Make the Files "TTL" API more structured
Summary:
Ref T11357. When creating a file, callers can currently specify a `ttl`. However, it isn't unambiguous what you're supposed to pass, and some callers get it wrong.

For example, to mean "this file expires in 60 minutes", you might pass either of these:

  - `time() + phutil_units('60 minutes in seconds')`
  - `phutil_units('60 minutes in seconds')`

The former means "60 minutes from now". The latter means "1 AM, January 1, 1970". In practice, because the GC normally runs only once every four hours (at least, until recently), and all the bad TTLs are cases where files are normally accessed immediately, these 1970 TTLs didn't cause any real problems.

Split `ttl` into `ttl.relative` and `ttl.absolute`, and make sure the values are sane. Then correct all callers, and simplify out the `time()` calls where possible to make switching to `PhabricatorTime` easier.

Test Plan:
- Generated an SSH keypair.
- Viewed a changeset.
- Viewed a raw diff.
- Viewed a commit's file data.
- Viewed a temporary file's details, saw expiration date and relative time.
- Ran unit tests.
- (Didn't really test Phragment.)

Reviewers: chad

Reviewed By: chad

Subscribers: hach-que

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17616
2017-04-04 16:16:28 -07:00
epriestley
2896da384c Only require POST to fetch file data if the viewer is logged in
Summary:
Ref T11357. In D17611, I added `file.search`, which includes a `"dataURI"`. Partly, this is building toward resolving T8348.

However, in some cases you can't GET this URI because of a security measure:

  - You have not configured `security.alternate-file-domain`.
  - The file isn't web-viewable.
  - (The request isn't an LFS request.)

The goal of this security mechanism is just to protect against session hijacking, so it's also safe to disable it if the viewer didn't present any credentials (since that means there's nothing to hijack). Add that exception, and reorganize the code a little bit.

Test Plan:
  - From the browser (with a session), tried to GET a binary data file. Got redirected.
  - Got a download with POST.
  - From the CLI (without a session), tried to GET a binary data file. Go a download.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17613
2017-04-04 16:16:01 -07:00
epriestley
2369fa38e1 Provide a modern ("v3") API for querying files ("file.search")
Summary: Ref T11357. Implements a modern `file.search` for files, and freezes `file.info`.

Test Plan: Ran `file.search` from the Conduit console.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17612
2017-04-04 16:15:36 -07:00
epriestley
260a08a128 Move Files editing and commenting to EditEngine
Summary:
Ref T11357. This moves editing and commenting (but not creation) to EditEngine.

Since only the name is really editable, this is pretty straightforward.

Test Plan: Renamed files; commented on files.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17611
2017-04-04 16:15:11 -07:00
epriestley
8500f78e45 Move Files to ModularTransactions
Summary: Ref T11357. A lot of file creation doesn't go through transactions, so we only actually have one real transaction type: editing a file name.

Test Plan:
Created and edited files.

{F4559287}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11357

Differential Revision: https://secure.phabricator.com/D17610
2017-04-04 10:25:05 -07:00
epriestley
9326b4d131 Fix some range issues and 32-bit issues with avatar generation
Summary:
Ref T12444. A few issues:

   - `x % (y - z)` doesn't generate values in the full range: the largest value is never generated. Instead, use `x % (1 + y - z)`.
   - `digestToRange(1, count)` never generates 0. After fixing the first bug, it could generate `count`. The range of the arrays is `0..(count-1)`, inclusive. Generate the correct range instead.
   - `unpack('L', ...)` can unpack a negative number on a 32-bit system. Use `& 0x7FFFFFFF` to mask off the sign bit so the result is always a positive integer.
   - FileFinder might return arbitrary keys, but we rely on sequential keys (0, 1, 2, ...)

Test Plan:
  - Used `bin/people profileimage ... --force` to regenerate images.
  - Added some debugging to verify that the math seemed to be working.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12444

Differential Revision: https://secure.phabricator.com/D17543
2017-03-23 10:51:33 -07:00
Chad Little
3a868940c7 Add a profileimage generation workflow for the cli
Summary: Ref T10319. This adds a basic means of generating default profile images for users. You can generate them for everyone, a group of users, or force updates. This only generated images and stores them in files. It does not assign them to users.

Test Plan:
`bin/people profileimage --all` to generate all images.
`bin/people profileimage --users chad` to generate a user.
`bin/people profileimage --all --force` to force rebuilding all images.

{F3662810}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T10319

Differential Revision: https://secure.phabricator.com/D17464
2017-03-04 15:43:13 -08:00
Chad Little
f095a81b00 Allow custom image generation when choosing a profile image
Summary: Ref T10319. This swaps the default in the Picture Chooser to allow picking of the custom unique avatar. We're currently going with 100k unique possibilities. The logic roughly hashes a user name and picks an image pack, color, and border. Based on that, we select the first character of their username, or fall back to Psyduck if not [a-z][0-9].

Test Plan:
Set the following usernames from ProfilePicture as a test: chad, epriestley, sally, 007, _cat_, -doggie-.

{F3453979}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T10319

Differential Revision: https://secure.phabricator.com/D17430
2017-03-03 20:21:31 -08:00
Chad Little
1460f2b85c Add more icon choices to Badges
Summary: Ref T9010. This adds more icons and lets the IconChooser handle more icons more easier.

Test Plan: Test Project Icons, Badges Icons

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T9006, T9010

Differential Revision: https://secure.phabricator.com/D17456
2017-03-03 13:45:53 -08:00
epriestley
c102620a29 Lock files.video-mime-types config option for consistency
Summary:
This is a consistency change to make this option consistent with `audio-mime-types`, `image-mime-types` and `icon-mime-types`, all of which are locked.

(They're locked because SVG is definitely dangerous, and other types might be dangerous or might become dangerous in the future, although I'm not aware of any actual dangers from video types today.)

Test Plan: Viewed `files.video-mime-types` in Config, saw it was locked.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17454
2017-03-03 08:38:02 -08:00
Chad Little
05377bea19 Add an avatar builtin file generator
Summary: Ref T10319. This builds out a reasonably decent avatar generator. 256 colors x 74 images x 2 borders, 38k options. Not completely sure though how names disburse though, so likely half that number. I can add lowercase lettering to double the footprint if needed though.

Test Plan:
UIExamples. Color generator here: http://tools.medialab.sciences-po.fr/iwanthue/

{F3416622}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T10319

Differential Revision: https://secure.phabricator.com/D17418
2017-02-27 11:09:30 -08:00
epriestley
99bcf5f112 Make bin/lipsum generate hanldle generator keys and arguments more clearly
Summary:
Ref T12319. Currently, `bin/lipsum` uses substring matches against human-readable text to chose which objects to generate.

Instead:

  - Use separate selector keys which are guaranteed to be unique.
  - When a match is exact, select only that generator.
  - When a match is ambiguous, fail and warn the user.

Test Plan: Generated several types of objects, tried to generate ambiguous objects like "e".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12319

Differential Revision: https://secure.phabricator.com/D17420
2017-02-27 09:09:28 -08:00
epriestley
1b2c047ce0 Correct spelling of "phabrictor" in Lipsum and elsewhere
Summary: Ref T12319. The product name is misspelled in some methods, and a few places in the documentation.

Test Plan: `grep`

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12319

Differential Revision: https://secure.phabricator.com/D17419
2017-02-27 09:09:13 -08:00
Chad Little
bf44210dc8 Reduce application search engine results list for Dashboards
Summary: Ref T10390. Simplifies dropdown by rolling out canUseInPanel in useless panels

Test Plan: Add a query panel, see less options.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T10390

Differential Revision: https://secure.phabricator.com/D17341
2017-02-22 12:42:43 -08:00
Chad Little
f9163bf065 Allow lightbox comments to be viewed logged out
Summary: Fixes T12160. Lightbox thread view should be visible if file is public.

Test Plan:
Add a file to a task, log out, click on file in task, get lightbox and no error. Expand comments, see login box.

{F2867067}

{F2867088}

{F2867098}

{F2867114}

{F2867124}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12160

Differential Revision: https://secure.phabricator.com/D17347
2017-02-13 13:54:13 -08:00
Chad Little
f54dfe7567 Add a basic icon typeahead
Summary: Fixes T11971, adds a basic typeahead for selecting an icon.

Test Plan:
http://local.phacility.com/typeahead/browse/PhabricatorIconDatasource/

{F2561013}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11971

Differential Revision: https://secure.phabricator.com/D17301
2017-02-03 09:19:29 -08:00
Chad Little
e980b94a2f Add a Picture Menu Item for Profiles
Summary: Just making profiles a little nicer, adds a big picture, easier mechanism for updating photos. Also larger profile pictures... need to re-thumb?

Test Plan:
View my profile, edit my picture, view a stranger, see profile. Check mobile, tablet, desktop. Check action menu on mobile.

{F2559394}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17295
2017-02-02 08:52:38 -08:00
Chad Little
489587d607 Add download link to embedded files
Summary: Ref T3612. Doesn't render correctly, need help please. Adds a download icon into the renderfilelinkview to allow easier downloads.

Test Plan: Click on link, get download, click on file, get lightbox.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T3612

Differential Revision: https://secure.phabricator.com/D16980
2017-01-03 10:50:26 -08:00
Chad Little
dece7af50b Prettier file embeds
Summary: Spruce up the file embeds a little more, hover state, icons, file size.

Test Plan:
Add a psd and pdf, see new icons. Check differential, still see icons there too. Test mobile, desktop.

{F2042539}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16950
2016-11-27 14:57:06 -08:00
epriestley
4dadad53ae Prevent media from autoplaying when rendered as a feed story
Summary: Fixes T11845. Users can still embed a text panel on the home page to give it some ambiance.

Test Plan: Wrote an autoplay video as a comment, saw it in feed. Before change: autoplay. After change: no auto play. On task: still autoplay.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11845

Differential Revision: https://secure.phabricator.com/D16920
2016-11-22 14:19:56 -08:00
Chad Little
0aa3f8b86b Allow commenting on non-viewable lightbox files
Summary: Removes the viewable restriction on embedded files. Builds a basic lightbox UI for commenting.

Test Plan:
Add psd, pdf to Maniphest task, clicked on download, comment, left comment. Closed box.

{F1943726}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T3612

Differential Revision: https://secure.phabricator.com/D16917
2016-11-22 11:29:36 -08:00