1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-11 08:06:13 +01:00
Commit graph

16 commits

Author SHA1 Message Date
epriestley
917fedafe6 Support exporting custom "Options/Select" fields to Excel/JSON/CSV/etc
Summary:
See <https://discourse.phabricator-community.org/t/exporting-custom-select-fields-as-part-of-query-exports/2466/>.

In JSON, export both the internal key and the visible value. For other formats, export the visible label.

Test Plan:
  - Added a custom options/select field.
  - Exported CSV, JSON, Text, got sensible output.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20316
2019-03-25 12:58:14 -07:00
epriestley
4928c34d00 Allow "bin/bulk export" to merge multiple queries and accept more flexible flags
Summary:
Ref T13210. Minor usability improvements to "bin/bulk export":

  - Allow `--class task` to work (previously, only `--class ManiphestTaskSearchEngine` worked).
  - If you run `--query jXIlzQyOYHPU`, don't require `--class`, since the query identifies the class on its own.
  - Allow users to call `--query A --query B --query C` and get a union of all results.

Test Plan:
  - Ran `--class task`, `--query A --query B`, `--query X` (with no `--class`), got good results.
  - Ran various flavors of bad combinations (queries from different engines, invalid engines, query and class differing, ambiguous/invalid `--class` name) and got sensible errors.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19738
2018-10-10 09:14:14 -07:00
epriestley
397645b273 Export task point values as double, not int
Summary:
See <https://discourse.phabricator-community.org/t/maniphest-non-integer-point-values-in-csv-export/1443>.

We currently export the Maniphest "points" field as an integer, but allow it to accept decimal values (e.g. "6.25").

Also fix a bug where we wouldn't roll over from "..., X, Y, Z, AA, AB, ..." correctly for Excel column names if sheet had more than 26 columns.

Test Plan:
  - Set a task point value to 6.25.
  - Exported to text, JSON, XLS.
  - Saw 6.25 represented accurately in exports.
  - Exported an excel sheet with 27+ columns.
  - Manually printed the first 200 column names to check that the algorithm looks correct.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19434
2018-05-08 15:49:40 -07:00
epriestley
ab579f2511 Never generate file download forms which point to the CDN domain, tighten "form-action" CSP
Summary:
Depends on D19155. Ref T13094. Ref T4340.

We can't currently implement a strict `form-action 'self'` content security policy because some file downloads rely on a `<form />` which sometimes POSTs to the CDN domain.

Broadly, stop generating these forms. We just redirect instead, and show an interstitial confirm dialog if no CDN domain is configured. This makes the UX for installs with no CDN domain a little worse and the UX for everyone else better.

Then, implement the stricter Content-Security-Policy.

This also removes extra confirm dialogs for downloading Harbormaster build logs and data exports.

Test Plan:
  - Went through the plain data export, data export with bulk jobs, ssh key generation, calendar ICS download, Diffusion data, Paste data, Harbormaster log data, and normal file data download workflows with a CDN domain.
  - Went through all those workflows again without a CDN domain.
  - Grepped for affected symbols (`getCDNURI()`, `getDownloadURI()`).
  - Added an evil form to a page, tried to submit it, was rejected.
  - Went through the ReCaptcha and Stripe flows again to see if they're submitting any forms.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13094, T4340

Differential Revision: https://secure.phabricator.com/D19156
2018-02-28 17:20:12 -08:00
epriestley
f9336e5694 Mangle cells that look a little bit like formulas in CSV files
Summary:
Fixes T12800. See that task for discussion. When a cell in a CSV begins with "=", "+", "-", or "@", mangle the content to discourage Excel from executing it.

This is clumsy, but we support other formats (e.g., JSON) which preserve the data faithfully and you should probably be using JSON if you're going to do anything programmatic with it.

We could add two formats or a checkbox or a warning or something but cells with these symbols are fairly rare anyway.

Some possible exceptions I can think of are "user monograms" (but we don't export those right now) and "negative numbers" (but also no direct export today). We can add exceptions for those as they arise.

Test Plan: Exported a task named `=cmd|'/C evil.exe'!A0`, saw the title get mangled with "(!)" in front.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12800

Differential Revision: https://secure.phabricator.com/D18974
2018-01-31 15:33:11 -08:00
epriestley
84df122085 When exporting more than 1,000 records, export in the background
Summary:
Depends on D18961. Ref T13049. Currently, longer exports don't give the user any feedback, and exports that take longer than 30 seconds are likely to timeout.

For small exports (up to 1,000 rows) continue doing the export in the web process.

For large exports, queue a bulk job and do them in the workers instead. This sends the user through the bulk operation UI and is similar to bulk edits. It's a little clunky for now, but you get your data at the end, which is far better than hanging for 30 seconds and then fataling.

Test Plan: Exported small result sets, got the same workflow as before. Exported very large result sets, went through the bulk flow, got reasonable results out.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18962
2018-01-29 16:08:02 -08:00
epriestley
c00838878a Implement common infrastructure fields as export extensions
Summary:
Depends on D18959. Ref T13049. Provide tags, subscribers, spaces, and created/modified as automatic extensions for all objects which support them.

(Also, for JSON export, be a little more consistent about exporting `null` instead of empty string when there's no value in a text field.)

Test Plan: Exported users and tasks, saw relevant fields in the export.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18960
2018-01-29 16:05:32 -08:00
epriestley
2ac4e1991b Support new data export infrastructure in Maniphest
Summary: Depends on D18958. Ref T13049. Support the new stuff. There are a couple more fields this needs to strictly improve on the old export, but I'll add them as extensions shortly.

Test Plan: Exported tasks to Excel, saw reasonble-looking data in the export.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18959
2018-01-29 16:04:39 -08:00
epriestley
00b4eae1f4 When PHPExcel is not installed, detect it and provide install instructions
Summary:
Depends on D18957. Ref T13049. To do Excel exports, PHPExcel needs to be installed on the system somewhere.

This library is enormous (1K files, ~100K SLOC), which is why we don't just include it in `externals/`. This install process is a little weird and we could improve it, but users don't seem to have too much difficulty with it. This shouldn't be worse than the existing workflow in Maniphest, and I tried to make it at least slightly more clear.

Test Plan: Uninstalled PHPExcel, got it marked "Unavailable" and got reasonably-helpful-ish guidance on how to get it to work. Reinstalled, exported, got a sheet.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18958
2018-01-29 16:03:34 -08:00
epriestley
61b8c12970 Make the data export format selector remember your last setting
Summary:
Depends on D18956. Ref T13049. Make the "Export Format" selector sticky.

This is partly selfish, since it makes testing format changes a bit easier.

It also seems like it's probably a good behavior in general: if you export to Excel once, that's probably what you're going to pick next time.

Test Plan: Exported to excel. Exported again, got excel as the default option.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18957
2018-01-29 16:01:54 -08:00
epriestley
5b61f863fd Organize the export code into subdirectories
Summary:
Depends on D18955. Ref T13049. This directory was getting a little cluttered with different kinds of code.

Put the formats (csv, json, ...), the field types (int, string, epoch, ...) and the engine-related stuff in subdirectories.

Test Plan: wow so aesthetic

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18956
2018-01-29 16:01:02 -08:00
epriestley
0409279595 Support Excel as a data export format
Summary:
Depends on D18954. Ref T13049. This brings over the existing Maniphest Excel export pipeline in a generic way.

The `<Type>ExportField` classes know directly that `PHPExcel` exists, which is a little sketchy, but writing an Excel indirection layer sounds like a lot of work and I don't anticipate us changing Excel backends anytime soon, so trying to abstract this feels YAGNI.

This doesn't bring over the install instructions for PHPExcel or the detection of whether or not it exists. I'll bring that over in a future change.

Test Plan: Exported users as Excel, opened them up, got a sensible-looking Excel sheet.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18955
2018-01-29 16:00:41 -08:00
epriestley
a067f64ebb Support export engine extensions and implement an extension for custom fields
Summary:
Depends on D18953. Ref T13049. Allow applications and infrastructure to supplement exportable fields for objects.

Then, implement an extension for custom fields. Only a couple field types (int, string) are supported for now.

Test Plan: Added some custom fields to Users, populated them, exported users. Saw custom fields in the export.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18954
2018-01-29 15:59:58 -08:00
epriestley
0de6210808 Give data exporters a header row
Summary:
Depends on D18951. Ref T13049. When we export to CSV or plain text, add a header row in the first line of the file to explain what each column means. This often isn't obvious with PHIDs, etc.

JSON has keys and is essentially self-labeling, so don't do anything special.

Test Plan: Exported CSV and text, saw new headers. Exported JSON, no changes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18952
2018-01-29 15:17:30 -08:00
epriestley
a79bb55f3f Support CSV, JSON, and tab-separated text as export formats
Summary: Depends on D18919. Ref T13046. Adds some simple modular exporters.

Test Plan: Exported pull logs in each format.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046

Differential Revision: https://secure.phabricator.com/D18934
2018-01-26 11:16:52 -08:00
epriestley
c0b8e4784b Add a basic, general-purpose export workflow for all objects with SearchEngine support
Summary:
Depends on D18918. Ref T13046. Ref T5954. Pull logs can currently be browsed in the web UI, but this isn't very powerful, especially if you have thousands of them.

Allow SearchEngine implementations to define exportable fields so that users can "Use Results > Export Data" on any query. In particular, they can use this workflow to download a file with pull logs.

In the future, this can replace the existing "Export to Excel" feature in Maniphest.

For now, we hard-code JSON as the only supported datatype and don't actually make any effort to format the data properly, but this leaves room to add more exporters (CSV, Excel) and data type awareness (integer casting, date formatting, etc) in the future.

For sufficiently large result sets, this will probably time out. At some point, I'll make this use the job queue (like bulk editing) when the export is "large" (affects more than 1K rows?).

Test Plan: Downloaded pull logs in JSON format.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046, T5954

Differential Revision: https://secure.phabricator.com/D18919
2018-01-26 11:15:59 -08:00