1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-14 19:02:41 +01:00
Commit graph

1356 commits

Author SHA1 Message Date
epriestley
1c32c9b965 Improve granluarity and defaults of security.allow-outbound-http
Summary:
Ref T6755. This is a partial fix, but:

  - Allow netblocks to be blacklisted instead of making the feature all-or-nothing.
  - Default to disallow requests to all reserved private/local/special IP blocks. This should generally be a "safe" setting.
  - Explain the risks better.
  - Improve the errors rasied by Macro when failing.
  - Removed `security.allow-outbound-http`, as it is superseded by this setting and is somewhat misleading.
    - We still make outbound HTTP requests to OAuth.
    - We still make outbound HTTP requests for repositories.

From a technical perspective:

  - Separate URIs that are safe to link to or redirect to (basically, not "javascript://") from URIs that are safe to fetch (nothing in a private block).
  - Add the default blacklist.
  - Be more careful with response data in Macro fetching, and don't let the user see it if it isn't ultimately valid.

Additionally:

  - I want to do this check before pulling repositories, but that's enough of a mess that it should go in a separate diff.
  - The future implementation of T4190 needs to perform the fetch check.

Test Plan:
  - Fetched a valid macro.
  - Fetched a non-image, verified it didn't result in a viewable file.
  - Fetched a private-ip-space image, got an error.
  - Fetched a 404, got a useful-enough error without additional revealing response content (which is usually HTML anyway and not useful).
  - Fetched a bad protocol, got an error.
  - Linked to a local resource, a phriction page, a valid remote site, all worked.
  - Linked to private IP space, which worked fine (we want to let you link and redierect to other private services, just not fetch them).
  - Added and executed unit tests.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6755

Differential Revision: https://secure.phabricator.com/D12136
2015-03-23 10:44:03 -07:00
epriestley
6eadfe6a6f Allow repositories to be ordered by commit count
Summary: Fixes T7640.

Test Plan: {F346553}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7640

Differential Revision: https://secure.phabricator.com/D12122
2015-03-23 09:10:34 -07:00
epriestley
bd2eaad04f Add "phabricator.silent" for stopping all outbound events from an install
Summary:
Ref T7522. This is mostly useful in the cluster, but could be useful for external installs too.

If you want to import an instance into a test/dry-run state (in the cluster, to test an import; in the general case, to do something like test new hardware or configuration), you currently risk spamming users with a lot of duplicate notifications. In particular, if Phabricator tracks remotes, both instances will continue importing commits and sending email about them. Both instances will try to publish to mirrors, too, which could be bad news, and both instances will try to update linked services.

Instead, provide a flag to let an instance run in "silent mode", which disables all outbound messaging and data.

We need to remember to support this flag on any new outbound channels, but we add about one of those per year so I think that's reasonable.

Test Plan:
  - Flipped config.
  - Saw it void email, feed and mirroring.
  - Didn't test SMS since it's not really in use yet and not convenient to test.
  - (Can you think of any publishing I missed?)

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7522

Differential Revision: https://secure.phabricator.com/D12109
2015-03-18 07:09:43 -07:00
epriestley
924b135d31 Add a storage renamespace for mangling SQL dumpfiles into a new namespace
Summary:
Ref T7149. When users give us dumpfiles for import, they will almost inevitably use the `phabricator` namespace. They need to be renamed to use an instance namespace.

We can do this either by:

  - importing the data first, then renaming; or
  - renaming first, then importing.

This implements the second one, basically `storage renamespace --in dump.sql --from phabricator --to instancename > instance.sql`.

Renaming first is a little hackier since we have to `preg_match()` a SQL dump file, but I think it's better overall:

  - With only one database, it lets you dump/import without downtime.
  - If you have development stuff in a development environment in the `phabricator` namespace, you don't have to move it aside to do an import.
  - No possibility that two people doing an import at the same time on the same box will collide with each other.
  - You can do the rename once and then repeat the import process with the renamed dump more easily.
  - No tricky stuff with modern Phabricator running against an old dump and the database names not matching up.

None of this is super important, but it just makes large dumps a bit easier to work with, and the dumpfile format is regular enough that this seems unlikely to ever really not work.

Test Plan: Renamespaced a dump, did a `diff -u`, saw all the relevant parts changed (and only those parts changed).

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7149

Differential Revision: https://secure.phabricator.com/D12105
2015-03-17 18:29:01 -07:00
epriestley
827c0ce081 Allow LiskDAO to be forced to use a specific connection
Summary:
Ref T7522. This seems like the least-bad approach to a messy issue:

  - When backfilling accounts from an imported instance, I need to write ExternalAccount rows to the instance to link instance accounts with upstream accounts.
  - We do this in the daemons in some other cases, which lets us run all the code in the context of the instance. However, I really want to do this in-process here because it's way way simpler and we need to do writes to //both// the instance and the upstream, and they're interleaved, and they depend on one another.
  - I can hard-code the query with `qsprintf()` but that feels like 100x worse than this.

This allows me to do this:

```
id(new PhabricatorExternalAccount())
  ->setForcedConnnection($instance_conn)
  ->...
  ->save();
```

...and get a write to the instance database, which is at least not completely a minefield.

Test Plan: Backfilled instance accounts and got interleaved instance and upstream writes as expected.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7522

Differential Revision: https://secure.phabricator.com/D12098
2015-03-17 14:43:08 -07:00
epriestley
6b86f81fe4 Increase the visibility of permanent task failures in task queue
Make permanent failures always reach the log.
Make `bin/worker execute` report exceptions properly.
2015-03-15 13:27:05 -07:00
epriestley
21aa086b69 Improve translation of some file strings
Summary: Ref T7149. hue hue hue hue

Test Plan: hue hue

Reviewers: chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T7149

Differential Revision: https://secure.phabricator.com/D12080
2015-03-15 11:37:30 -07:00
epriestley
106ca70acb Fix an issue where subpriority paging could be truncated
Ref T7548. Subpriority is a float, but we're truncating it to an int, which can cause reselection of the same row while paging.
2015-03-14 13:42:06 -07:00
epriestley
dd501117e8 When deleting inline comments, offer "undo" instead of prompting
Summary:
Ref T2009. Ref T1460.

Fixes T2618. When users hit "Delete" on inline comments, delete immediately and offer them "Undo". If they delete indirectly (e.g., by clicking "Delete" from the preview at the bottom of the page), we still prompt them, because the "Undo" action either won't be available or may not be easy to find. This is a "refdelete".

Fixes T6464. This was just a mess. Make it not as much of a mess. It should work now. Pretty sure.

Fixes T4999. We did not refresh these links often enough to find targets for them, so they could race with content. Reevaluate them after loading new changes.

Test Plan:
  - Deleted and undid deletion of inlines from main view and preview.
  - Clicked "View" on inlines.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6464, T4999, T2618, T1460, T2009

Differential Revision: https://secure.phabricator.com/D12032
2015-03-09 17:27:51 -07:00
epriestley
daa893e508 Extend TransactionCommentQuery for Diffusion
Summary: Ref T2009. Ref T1460. Reduces the amount of garbage involved in loading inline comments and routes more pathways through the proper Query layer.

Test Plan: Viewed, edited, previewed, submitted inline comments in Diffusion.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2009, T1460

Differential Revision: https://secure.phabricator.com/D12028
2015-03-09 14:11:22 -07:00
epriestley
7427a6e648 Extend TransactionCommentQuery for Differential
Summary: Ref T2009. Ref T1460. Replace hard-coded garbage with a real Query-layer query.

Test Plan: Submitted inline comments in Differential.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2009, T1460

Differential Revision: https://secure.phabricator.com/D12027
2015-03-09 14:11:20 -07:00
epriestley
4d86d51125 Prepare TransactionCommentQuery for extension
Summary:
Ref T2009. Ref T1460. The way Diffusion and Differential load inlines is horrible garbage right now:

  - Differential does an ad-hoc query to get the PHIDs, then does a real load to policy check.
  - Diffusion completely fakes things. In practice this is not a policy violation, but it's dangerous.

Make TransactionCommentQuery extensible so we can subclass it and get the query building correctly in the right Query layer.

Specifically, the Diffusion and Differential subclasses of this Query will add appropriate `withX()` methods to let us express the query in SQL.

Test Plan: Loaded, previewed, edited, and submitted inlines in Differential and Diffusion

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2009, T1460

Differential Revision: https://secure.phabricator.com/D12026
2015-03-09 14:11:18 -07:00
epriestley
2972894a4d Write "hasReplies" to database for inline comments
Summary:
Ref T1460. Ref T2618.

When publishing a draft inline, mark the inline it replies to (if any) as replied to.

Also, don't load deleted comments as drafts (sets the stage for T2618).

I'll make an effort to clean up the loading mess here in the next revision, and find some more appropriate home for the shared code.

Test Plan: Made and replied to comments in Differential and Diffusion. Saw comments get marked as "Has Replies" and "Is Reply".

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2618, T1460

Differential Revision: https://secure.phabricator.com/D12025
2015-03-09 14:11:16 -07:00
epriestley
f1f2c5d01d Reduce code duplication in inline right/left side tracking
Summary: Ref T2009. These subclasses have a mixture of similar methods, move them all to the base class.

Test Plan: Created/edited/undo/submitted comments on the left and right sides of a diff.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D12024
2015-03-09 12:53:40 -07:00
epriestley
56a9709008 Reduce code duplication for inline "Undo"
Summary:
Ref T2009. This is another almost-identical copy of the row scaffolding, which has the same 1up/2up bugs as the 8 other copies of this code.

Turn the "undo" element into an InlineCommentView so we can scaffold it.

Then, scaffold it with the same code as everything else.

Test Plan: Hit "Undo", swapped from 1up to 2up, hit "undo" again, swapped back, tried left/right, everything rendered with proper scaffolding.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D12019
2015-03-09 10:26:53 -07:00
epriestley
355142fcbf Reduce code duplication on comment editing UI
Summary: Ref T2009. This has two more copies of the scaffolding.

Test Plan: Created, edited, deleted, replied to inline comments.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D12018
2015-03-09 10:26:51 -07:00
epriestley
082b7f95e6 Explicitly track inline comment reply threading
Summary:
Ref T1460. Track and store which comments are threaded replies to other comments, vs merely appearing on the same lines.

This doesn't actually write `hasReplies` yet, since that needs to happen when we un-draft comments on submission.

Test Plan: Made inline comments in Differential and Diffusion, including replies. Replies were marked as "Is Reply".

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1460

Differential Revision: https://secure.phabricator.com/D12017
2015-03-09 10:26:50 -07:00
epriestley
1df321bf00 Fix left/right detection of inline comments in unified view
Summary:
Ref T2009. Currently, the code figures out if a comment is on the left or right by looking at the `<th />` preceeding the enclosing `<td />`.

This gets the right result in 2-up, but in 1-up rows are always `<th />`, `<th />`, `<td />`, so it always detects every inline as being in the new file.

Because "old" and "new" cells aren't inherently distingushable in the 1up view, we can't use a DOM test for this at all. Instead, just track this state explicitly.

Test Plan:
  - Made left/right comments in 1up view and 2up view.
  - Viewed them in 1up and 2up views.
  - Hovered in 1up and 2up views.
  - Diff-of-diff'd and reviewed old/new comments, then made some more.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D12011
2015-03-07 14:37:57 -08:00
epriestley
f9cb366f00 Remove duplicate inline scaffold in 2up renderer
Summary: Ref T2009. Remove the 4 (!!) copies of this code.

Test Plan:
  - Added, edited, and removed inline comments in 2up view.
  - Stacked a bunch of comments on the same line and saw the JS place them correctly.
  - Created an image diff and added, edited and removed inlines on it.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D12000
2015-03-06 15:00:43 -08:00
epriestley
1352be827e Begin separating inline comment scaffolding from other renderers
Summary:
Ref T2009. Inline comments have "scaffolding", which is basically some empty table cells/rows around them to get the layout correct.

The scaffolding depends on the renderer, since the cells are different for side-by-side vs unified diffs.

This is currently duplicated all over the place:

  - Edit view has 1up/2up.
  - Detail view has 1up/2up.
  - 1up renderer has 1up.
  - 2up renderer has four separate copies of the 2up logic.

These all have subtle differences, which are mostly bugs. Start making the scaffolding more composable so we can get rid of that mess.

Test Plan: Added, edited, and removed inline comments on unified and side-by-side diffs.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D11997
2015-03-06 15:00:33 -08:00
epriestley
1088d34e58 Rename inline comment views to "PHUIDiff" and give them a base class
Summary:
Ref T2009. These classes are "Differential" now, but are used elsewhere in diff infrastructure (e.g., Diffusion).

  - Rename them to "PHUIDiff".
  - Move them to "src/infrastructure/".
  - Give them a base class.

Test Plan: Interacted with inlines in unified and side-by-side views.

Reviewers: btrahan

Subscribers: epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D11996
2015-03-06 15:00:14 -08:00
Chad Little
85f4bdc2ac Update Conpherence Main UI
Summary: Uses standard sidenav width, more spacing in labels, added background around textarea, make background work in Firefox.

Test Plan:
Test Desktop, Mobile, and Tablet break points. Test Firefox and Chrome.

{F331201}

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11993
2015-03-05 17:09:07 -08:00
epriestley
9564b0a40e Improve behavior of inline rendering with unified views
Summary:
Ref T2009. This reduces how buggy inlines are. They're still buggy.

Specifically, the inline endpoint didn't know how to scaffold inlines before, so some of them ended up rendering in the wrong rows or breaking layouts.

This passes the current renderer through to the inline editor endpoint, so it can at least get the layout correct.

Test Plan: Interacted with inlines in unified and side-by-side views.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D11988
2015-03-05 14:11:51 -08:00
epriestley
ad3c94dd45 Make "Show Context" persist rendering, whitespace, encoding, etc
Summary:
Ref T2009. Currently, we do not persist view parameters when making context rendering requests.

The big one is the renderer (1up vs 2up). This makes context on unified diffs come in with too many columns.

However, it impacts other parameters too. For example, at HEAD, if you change highlighting to "rainbow" and then load more context, the context uses the original highlighter instead of the rainbow highlighter.

This moves context loads into ChangesetViewManager, which maintains view parameters and can provide them correctly.

  - This removes "ref"; it is no longer required, as the ChangesetViewManager tracks it.
  - This removes URI management from `behavior-show-more`; it is no longer required, since the ChangesetViewManager knows how to render.
  - This removes "whitespace" since this is handled properly by the view manager.

Test Plan:
  - Used "Show Top" / "Show All" / "Show Bottom" in 1-up and 2-up views.
  - Changed file highlighting to rainbow, loaded stuff, saw rainbow stick.
  - Used "Show Entire File" in 1-up and 2-up views.
    - Saw loading chrome.
    - No loading chrome normally.
  - Made inlines, verified `copyRows()` code runs.
  - Poked around Diffusion -- it is missing some parameter handling, but works OK.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D11977
2015-03-05 14:03:00 -08:00
epriestley
06df75ebbd Render "Show Context" blocks in unified view
Summary:
Ref T2009. This basically copy/pastes them for now. Plans is:

  - Make this actually work all the way.
  - Add test coverage after D11970.
  - Move 2-up here after test coverage.

Clicking the links does not work yet, because they use the 2-up renderer. I'll fix this in the next diff.

Test Plan: Viewed diffs in unified, saw links to show more.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: chad, epriestley

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D11976
2015-03-05 14:02:29 -08:00
epriestley
e1d09fd035 Show change details for "Remarkup" standard custom field edits
Summary: Fixes T7436.

Test Plan: {F328222}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7436

Differential Revision: https://secure.phabricator.com/D11952
2015-03-03 10:39:32 -08:00
epriestley
a65667443b Fix quickstart.sql for old MySQL
Summary:
Fixes T7422. After the recent fix for "sort" columns, we can end up with invalid SQL in some cases when running quickstart.

In particular, we do "COLLATE binary CHARACTER SET utf8_general_ci" (which is invalid).

Preprocess these so we get "COLLATE utf8 CHARACTER SET utf8_general_ci" (which is valid and correct).

Test Plan: Ran `bin/storage upgrade -f --namespace blahblhbaba` with and without `--disable-utf8mb4`.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7422

Differential Revision: https://secure.phabricator.com/D11929
2015-03-02 09:57:38 -08:00
epriestley
9e82cfcc21 Use utf8_general_ci for "sort" columns in old MySQL
Summary:
Fixes T7287. This trades off 4-byte character support for case insensitivity in these columns, which is a much better trade on the balance.

Also adds more warnings about old MySQL. Note that we already issue a warning when you run "storage adjust" (which I've made stronger) and already "strongly recommend" MySQL 5.5 or newer in the install documentation.

Test Plan:
  - Ran `storage adjust --disable-utf8mb4` to go to old definitions, then ran `storage adjust` to get back to the new ones. Everything seemed OK in both cases.
  - Verified that utf8mb4 data can be migrated out of these colums with `--unsafe` (which will truncate).
  - Verified that manual explains this.
  - Faked my way into the setup warning.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7287

Differential Revision: https://secure.phabricator.com/D11893
2015-02-26 10:18:54 -08:00
epriestley
38636a39cf Allow modern phd stop to stop old daemons cleanly
Summary:
Ref T7352. Make sure modern `phd stop` can still read the old PID file format and stop the daemons, at least for now.

Without this, `stop` still detects them and tells you to `stop --force`, which works, but this makes things a good deal cleaner.

Test Plan: Ran `phd stop` from master, then `phd stop` from this revision. Saw old daemons stop cleanly.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7352

Differential Revision: https://secure.phabricator.com/D11873
2015-02-24 14:50:40 -08:00
epriestley
a3518e19a5 Merge GC daemon into Trigger daemon
Summary:
Fixes T7352. This reduces the memory footprint for instances by combining these two similar daemons into one daemon which handles the responsibilities of both.

The fit isn't 100% perfect here but it's pretty close, and the GC daemon is fairly trivial.

Test Plan:
  - Adjusted all the numbers to small numbers (5 second sleep, 120 second GC length).
  - Added a ton of logging.
  - Started trigger daemon.
    - Saw it run a GC cycle.
    - Saw it reschedule another cycle after 120 seconds (adjusted down from 4 hours).
  - Reverted all the logging/small numbers.
  - Ran `bin/phd start`, saw stable trigger daemon running.
  - Grepped for removed daemon class name.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7352

Differential Revision: https://secure.phabricator.com/D11872
2015-02-24 14:50:39 -08:00
epriestley
af303f458b Convert taskmasters to use an autoscale pool
Summary: Ref T7352. This is pretty straightforward. I renamed `phd.start-taskmasters` to `phd.taskmasters` for clarity.

Test Plan:
  - Ran `phd start`, `phd start --autoscale-reserve 0.25`, `phd restart --autoscale-reserve 0.25`, etc.
  - Examined PID file to see options were passed.
  - I'm defaulting this off (0 reserve) and making it a flag rather than an option because it's a very advanced feature which is probably not useful outside of instancing.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7352

Differential Revision: https://secure.phabricator.com/D11871
2015-02-24 14:50:38 -08:00
epriestley
a354e5fa6b Track daemon unique IDs in Phabricator daemon logs
Summary:
Ref T7352. We were previously identifying things by `<daemonClass, overseerPID, startTime>` but that's not unique in a world where one overseer can run multiple daemons.

We already have an internal "daemonID", it just doesn't get written into the DB right now.

Start writing it, then use it to clean up `phd status`.

Test Plan: Ran `phd status`, got more accurate/useful output than previously.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7352

Differential Revision: https://secure.phabricator.com/D11865
2015-02-24 14:50:37 -08:00
epriestley
48fc3126a1 Support autoscaling daemons in phd
Summary: Ref T7352. This supports passing autoscaling configuration to daemons, and adds `debug --autoscale`.

Test Plan: See D11711.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7352

Differential Revision: https://secure.phabricator.com/D11860
2015-02-24 14:50:34 -08:00
epriestley
c2d66f29cd Make phd more aware of multiple daemons under a single overseer
Summary: Ref T7352. This makes `phd stop` and `phd status` produce more reasonable output with the new PID file format.

Test Plan: Ran `phd stop`, `phd status`, etc.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7352

Differential Revision: https://secure.phabricator.com/D11856
2015-02-24 14:50:32 -08:00
epriestley
09f3d0bb7e Pass overseer configuration over stdin
Summary:
Ref T7352. This changes `phd` to pass configuration to overseers over stdin. We still run one overseer per daemon.

The "status" stuff needs some cleanup, but it's mostly just UI/cosmetic.

Test Plan:
  - Ran `phd debug`, `phd launch`, `phd start`, `phd status`, `phd stop`, etc.
  - Verified PID files write in a reasonable format.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7352

Differential Revision: https://secure.phabricator.com/D11855
2015-02-24 14:50:30 -08:00
epriestley
59ae91a5ce Disable caching of remarkup previews
Summary:
We currently cache previews, but the vast majority of previews are never rendered again (e.g., they're a preview of someone partway through typing a comment).

Especially when editing large documents (Legalpad, Phriction), this can bloat the markup cache with data that will never be read and won't get purged for 30 days.

In particular, most of the data on `admin.phacility.com` is currently 1,000 previews of legalpad documents as I made minor edits to them over the course of several hours.

This isn't a big concern, but it's a very easy fix.

Test Plan:
  - Previewed a legalpad document, verified that cache rows were not written as I mashed the keyboard.
  - Saved the document, verified a new cache row was written.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D11832
2015-02-22 05:39:25 -08:00
epriestley
29fd3f136b Allow columns to be marked as nonmutable (so save() will not change them)
Summary:
Ref T6840. This feels a little dirty; open to alternate suggestions.

We currently have a race condition where multiple daemons may load a commit and then save it at the same time, when processing "reverts X" text. Prior to this feature, two daemons would never load a commit at the same time.

The "reverts X" load/save has no effect (doesn't change any object properties), but it will set the state back to the loaded state on save(). This overwrites any flag updates made to the commit in the meantime, and can produce the race in T6840.

In other cases (triggers, harbormaster, repositories) we deal with this kind of problem with "append-only-updates + single-consumer", or a bunch of locking. There isn't really a good place to add a single consumer for commits, since a lot of daemons need to access them. We could move the flags column to a separate table, but this feels pretty complicated. And locking is messy, also mostly because we have so many consumers.

Just exempting this column (which has unusual behavior) from `save()` feels OK-ish? I don't know if we'll have other use cases for this, and I like it even less if we never do, but this patch is pretty small and feels fairly understandable (that said, I also don't like that it can make some properties just silently not update if you aren't on the lookout).

So, this is //a// fix, and feels simplest/least-bad for the moment to me, I thiiink.

Test Plan: Added and executed unit tests.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6840

Differential Revision: https://secure.phabricator.com/D11822
2015-02-19 10:37:17 -08:00
Bob Trahan
a77127ab63 Projects - fix translation strings in watcher edge class
Summary: Fixes T7319. These need a "%s" for the count where they had a "%d"

Test Plan: plan in D11812 is no longer a lie! (watcher added / removed strings render correctly)

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T7319

Differential Revision: https://secure.phabricator.com/D11813
2015-02-18 15:44:54 -08:00
Bob Trahan
7ef5c52934 Projects - add translation for watcher strings
Summary: Fixes T7319. ...except I can't get this working in my sandbox? Changes to the translation file don't seem to show up. TEST PLAN IS A LIE

Test Plan: became a watcher, un became a watcher - saw sensical translated strings

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T7319

Differential Revision: https://secure.phabricator.com/D11812
2015-02-18 15:33:52 -08:00
epriestley
2cd77b5b58 Improve taskmaster behavior on empty queues
Summary:
Right now, taskmasters on empty queues sleep for 30 seconds. With a default setup (4 taskmasters), this averages out to 7.5 seconds between the time you do anything that queues something and the time that the taskmasters start work on it.

On instances, which currently launch a smaller number of taskmasters, this wait is even longer.

Instead, sleep for the number of seconds that there are taskmasters, with a random offset. This makes the average wait to start a task from an empty queue 1 second, and the average maximum load of an empty queue also one query per second.

On loaded instances this doesn't matter, but this should dramatically improve behavior for less-loaded instances without any real tradeoffs.

Test Plan: Started several taskmasters, saw them jitter out of sync and then use short sleeps to give an empty queue about a 1s delay.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D11772
2015-02-16 11:30:49 -08:00
epriestley
36494d4e2e Add a "did verify email" event to Phabricator
Summary: Ref T7152. Gives us an event hook so we can go make users a member of any instance they've been invited to as soon as they verify an email address.

Test Plan:
  - Used `bin/auth verify` to trigger the event.
  - Build out the invite flow in rSERVICES.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7152

Differential Revision: https://secure.phabricator.com/D11752
2015-02-11 14:39:06 -08:00
epriestley
d4680a7e4e Update Phabricator to work with more modular translations
Summary:
Ref T7152. Ref T1139. This updates Phabricator so third-party libraries can translate their own stuff. Also:

  - Hide "All Caps" when not in development mode, since some users have found this a little confusing.
  - With other changes, adds a "Raw Strings" mode (development mode only).
  - Add an example silly translation to make sure the serious business flag works.
  - Add a basic British English translation.
  - Simplify handling of translation overrides.

Test Plan:
  - Flipped serious business / development on and off and saw silly/development translations drop off.
  - Switched to "All Caps" and saw all caps.
  - Switched to Very English, Wow!
  - Switched to British english and saw "colour".

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7152, T1139

Differential Revision: https://secure.phabricator.com/D11747
2015-02-11 13:02:35 -08:00
epriestley
6f90fbdef8 Send emails for email invites
Summary:
Ref T7152. Ref T3554.

  - When an administrator clicks "send invites", queue tasks to send the invites.
  - Then, actually send the invites.
  - Make the links in the invites work properly.
  - Also provide `bin/worker execute` to make debugging one-off workers like this easier.
  - Clean up some UI, too.

Test Plan:
We now get as far as the exception which is a placeholder for a registration workflow.

{F291213}

{F291214}

{F291215}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T3554, T7152

Differential Revision: https://secure.phabricator.com/D11736
2015-02-11 06:06:09 -08:00
Joshua Spence
84b0c8e6db Fix a pht method call
Summary: Ref T7046. I missed this in D11680.

Test Plan: `arc lint`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T7046

Differential Revision: https://secure.phabricator.com/D11731
2015-02-11 06:54:10 +11:00
Joshua Spence
aaf8d73ec7 Fix pht method calls
Summary: Ref T7046. This is mainly a proof-of-concept for D11661.

Test Plan: `arc lint`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T7046

Differential Revision: https://secure.phabricator.com/D11680
2015-02-10 18:57:45 +11:00
Chad Little
cdd8dcbf17 Update InlineCommentSummary UI
Summary: Minor spring cleaning, improve the visual feel of the comments table, more consistent structure.

Test Plan:
Test multiple comments, long comments, short comments, and multiple lines.

{F282462}

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: hach-que, Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11666
2015-02-09 08:38:51 -08:00
epriestley
8c568d88d7 Reduce severity of auth provider warning
Summary:
Ref T7208. Now that we have approvals (new installs are safe by default), take those into account when generating this warning.

Try to soften the warning to cover the case discussed in T7208, hopefully without requiring additional measures.

Test Plan:
{F286014}

{F286015}

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T7208

Differential Revision: https://secure.phabricator.com/D11708
2015-02-07 14:45:27 -08:00
Eric
28b23fd789 Use --hex-blob flag in bin/storage dump
Summary: mysqldump output can end up having weird encoding issues when raw BLOBs are in the output, preventing the backup restoration from succeeding. This hex-encodes blobs in the dump from the backup workflow causing the output file to only contain ASCII and ensure imports are successful.

Test Plan: Had issues restoring a backup from the original `mysqldump` command issued by this workflow. Ran the same command with this flag added and I was able to restore the backup.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D11704
2015-02-06 12:56:23 -08:00
epriestley
74ea59235a Make the "daemons and web have different config" warning more specific
Summary:
I'm hitting this in the cluster and couldn't figure it out after staring at it for a couple minutes. Produce a better error.

This dumps a hash of each configuration key value which is set to a non-default value into the daemon log. This is much more compact than the full config, and doesn't spread secrets around, so it seems like a good balance between providing information and going crazy with it.

Test Plan: {F284139}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D11699
2015-02-05 14:07:35 -08:00
Bob Trahan
137b0ebc53 Phabot / Conduit - stop using deprecated conduit api paste.info
Summary: Fixes T7111. Also nails out the TODO to show the username

Test Plan: fired up the ole phabot, chatted "P123" and saw "P123: https://secure.phabricator.com/P123 - Masterwork From Distant Lands by epriestley"...!

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T7111

Differential Revision: https://secure.phabricator.com/D11664
2015-02-03 13:44:20 -08:00