3ef270b292
Summary: Ref T8672. Ref T9187. Root issue in at least one case is: - User makes a commit including a file with some non-UTF8 text (say, a Japanese file full of Shift-JIS). - We pass the file to the TransactionEditor so it can inline or attach the patch if the server is configured for these things. - When inlining patches, we convert them to UTF8 before inlining. We must do this since the rest of the mail is UTF8. - When attaching patches, we send them in the original encoding (as file attachments). This is correct, and means we need to give the worker the raw patch in whatever encoding it was originally in: we can't just convert it to utf8 earlier, or we'd attach the wrong patch in some cases. - TransactionEditor does its thing (e.g., creates the commit), then gets ready to send mail about whatever it did. - The publishing work now happens in the daemon queue, so we prepare to queue a PublishWorker and pass it the patch (with some other data). - When we queue workers, we serialize the state data with JSON. So far, so good. But this is where things go wrong: - JSON can't encode binary data, and can't encode Shift-JIS. The encoding silently fails and we ignore it. Then we get to the worker, and things go wrong-er: - Since the data is bad, we fatal. This isn't a permanent failure, so we continue retrying the task indefinitely. This applies several fixes: # When queueing tasks, fail loudly when JSON encoding fails. # In the worker, fail permanently when data can't be decoded. # Allow Editors to specify that some of their data is binary and needs special handling. This is fairly messy, but some simpler alternatives don't seem like good ways forward: - We can't convert to UTF8 earlier, because we need the original raw patch when adding it as an attachment. - We could encode //only// this field, but I suspect some other fields will also need attention, so that adding a mechanism will be worthwhile. In particular, I suspect filenames //may// be causing a similar problem in some cases. - We could convert task data to always use a serialize()-based binary safe encoding, but this is a larger change and I think it's correct that things are UTF8 by default, even if it makes a bit of a mess. I'd rather have an explicit mess like this than a lot of binary data floating around. The change to make `LiskDAO` will almost certainly catch some other problems too, so I'm going to hold this until after `stable` is cut. These problems were existing problems (i.e., the code was previously breaking or destroying data) so it's definitely correct to catch them, but this will make the problems much more obvious/urgent than they previously were. Test Plan: - Created a commit with a bunch of Shift-JIS stuff in a file. - Tried to import it. Prior to patch: - Broken PublishWorker with distant, irrelevant error message. With patch partially applied (only new error checking): - Explicit, local error message about bad key in serialized data. With patch fully applied: - Import went fine and mail generated. Reviewers: chad Reviewed By: chad Subscribers: devurandom, nevogd Maniphest Tasks: T8672, T9187 Differential Revision: https://secure.phabricator.com/D13939 |
||
|---|---|---|
| bin | ||
| conf | ||
| externals | ||
| resources | ||
| scripts | ||
| src | ||
| support | ||
| webroot | ||
| .arcconfig | ||
| .arclint | ||
| .arcunit | ||
| .editorconfig | ||
| .gitignore | ||
| LICENSE | ||
| NOTICE | ||
| README.md | ||
Phabricator is an open source collection of web applications which help software companies build better software.
Phabricator includes applications for:
- reviewing and auditing source code;
- hosting and browsing repositories;
- tracking bugs;
- managing projects;
- conversing with team members;
- assembling a party to venture forth;
- writing stuff down and reading it later;
- hiding stuff from coworkers; and
- also some other things.
You can learn more about the project (and find links to documentation and resources) at Phabricator.org
Phabricator is developed and maintained by Phacility.
BUG REPORTS
Please update your install to HEAD before filing bug reports. Follow our bug reporting guide for complete instructions.
FEATURE REQUESTS
We're big fans of feature requests that state core problems, not just 'add this'. We've compiled a short guide to effective upstream requests here.
COMMUNITY CHAT
Please visit our IRC Channel (#phabricator on FreeNode) to talk with other members of the Phabricator community. There might be someone there who can help you with setup issues or what image to choose for a macro.
SECURITY ISSUES
Phabricator participates in HackerOne and may pay out for various issues reported there. You can find out more information on our HackerOne page.
PULL REQUESTS
We do not accept pull requests through GitHub. If you would like to contribute code, please read our Contributor's Guide for more information.
LICENSE
Phabricator is released under the Apache 2.0 license except as otherwise noted.