1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-25 08:12:40 +01:00
Commit graph

10287 commits

Author SHA1 Message Date
epriestley
005d8493b0 Pass GIT_ENVIRONMENTAL_MAGIC through to hook subprocesses to support Git 2.11.0
Summary:
Fixes T11940. In 2.11.0, Git has made a change so that newly-pushed changes are held in a temporary area until the hook accepts or rejects them.

This magic temporary area is only readable if the appropriate `GIT_ENVIRONMENTAL_MAGIC` variables are available. When executing `git` commands, pass them through from the calling context.

We're intentionally conservative about which variables we pass, and with good reason (see "httpoxy" in T11359). I think this continues to be the correct default behavior.

Test Plan:
  - Upgraded to Git 2.11.0.
  - Tried to push over SSH, got a hook error.
  - Applied patch.
  - Pulled and pushed over SSH.
  - Pulled and pushed over HTTP.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11940

Differential Revision: https://secure.phabricator.com/D16988
2016-12-05 12:45:30 -08:00
epriestley
6058d3305f Normalize remote IP addresses when writing to logs, etc
Summary:
Ref T11939. IPv4 addresses can normally only be written in one way, but IPv6 addresses have several formats.

For example, the addresses "FFF::", "FfF::", "fff::", "0ffF::", "0fFf:0::", and "0FfF:0:0:0:0:0:0:0" are all the same address.

Normalize all addresses before writing them to logs, etc, so we store the most-preferred form ("fff::", above).

Test Plan:
Ran an SSH clone over IPv6:

```
$ git fetch ssh://local@::1/diffusion/26/locktopia.git
```

It worked; verified that address read out of `SSH_CLIENT` sensibly.

Faked my remote address as a non-preferred-form IPv6 address using `preamble.php`.

Failed to login, verified that the preferred-form version of the address appeared in the user activity log.

Made IPv6 requests over HTTP:

```
$ curl -H "Host: local.phacility.com" "http://[::1]/"
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11939

Differential Revision: https://secure.phabricator.com/D16987
2016-12-05 11:20:29 -08:00
epriestley
5a060b34df Add IPv6 reserved addresses to the default outbound blacklist
Summary:
Ref T11939. Depends on D16984. Now that CIDRLists can contain IPv6 addresses, blacklist all of the reserved IPv6 space.

This reserved blacklist is used to prevent users from accessing internal services via "Import Calendar" or "Add Macro".

They can't actually reach IPv6 addresses via these mechanisms yet because we need to do more work to support outbound IPv6 requests, but make sure reserved IPv6 space is blacklisted already when that support eventaully arrives.

Also, clean up some error messages (e.g., for trying to hit a bad URI in "Add Macro").

Test Plan:
  - Loaded pages with default blacklist.
  - Tried to make requests into IPv6 space.
  - Currently, this is impossible because of `parse_url()` and `gethostynamel()` calls.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11939

Differential Revision: https://secure.phabricator.com/D16986
2016-12-05 11:20:13 -08:00
epriestley
4a6229ee69 Remove some no-op "canUninstall()" Application methods
Summary: The default behavior of these methods is to return `true`, so these overrides have no effect.

Test Plan: `grep`; poked around.

Reviewers: chad

Reviewed By: chad

Subscribers: hach-que

Differential Revision: https://secure.phabricator.com/D16985
2016-12-05 11:02:25 -08:00
epriestley
5f593aafb1 Allow logged-out users to load global preferences on installs without public viewers
Summary:
Fixes T11946. When a logged-out viewer is loading a page on a non-public install, there are two policy issues which prevent them from loading global settings:

  - They can not see the Settings application itself.
  - They can not see the global settings object.

Allow them to see Settings by making mandatory applications always visible. (This doesn't make any application pages public.)

Allow them to see the global settings object explicitly.

Test Plan:
Changed default language, viewed logged-out page:

{F2076924}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11946

Differential Revision: https://secure.phabricator.com/D16983
2016-12-05 11:00:39 -08:00
epriestley
faf983614c Improve error messages for running git clone against a Mercurial repository
Summary:
Fixes T11938.

Note that there's a subcase here: if you `hg clone` or `svn checkout` a short `/source/` URI that ends in `.git`, we miss the lookup and don't get this far, so you still get a generic error message.

Hopefully it is clear enough on its own that `proto://.../blah.git` is, in fact, a Git repository, since it says ".git" at the end.

If that doesn't prove to be true, we can be more surgical about this.

Test Plan:
```
$ git clone ssh://local@localvault.phacility.com/source/quack.notgit/
Cloning into 'quack.notgit'...
phabricator-ssh-exec: This repository ("quack.notgit") is not a Git repository. Use "hg" to interact with this repository.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

```
$ hg clone ssh://local@localvault.phacility.com/source/phabx
remote: phabricator-ssh-exec: This repository ("phabx") is not a Mercurial repository. Use "git" to interact with this repository.
abort: no suitable response from remote hg!
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11938

Differential Revision: https://secure.phabricator.com/D16976
2016-12-02 07:30:03 -08:00
epriestley
7c37377e0d Set the viewer timezone properly on Calendar event RecurrenceSet objects
Summary: Ref T11801. In some cases, this could lead to us failing to generate the first recurrence in a series.

Test Plan: Imported `weekly.ics` (from D16974) and saw an event correctly occur on Aug 18, with my local timezone set to "America/Los_Angeles".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11801

Differential Revision: https://secure.phabricator.com/D16975
2016-12-02 07:29:48 -08:00
epriestley
99c6b53ab2 Explicitly update the repository URI index after making a URI edit
Summary:
Fixes T11936. After editing a repository URI, we were not correctly updating the URI index.

Any other edit to the repository //would// update the index, and this index is only really used by `arc` to figure out which repository a working copy belongs to, so that's how this evaded detection for this long. In particular, creating a repository would usually have an edit after any URI edits, to activate it, which would build the index correctly.

Test Plan:
  - Added a new URI to a repository.
  - Verified it was immediately reflected in the `repository_uriindex` table.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11936

Differential Revision: https://secure.phabricator.com/D16972
2016-12-01 14:29:39 -08:00
epriestley
dc73785c4f Add a "--force" argument to "bin/config done"
Summary:
Ref T11922. When we deploy on Saturday I need to rebuild all the cluster indexes, but some instances won't have anything indexed so they won't actually trigger the activity.

Add a `--force` flag that just clears an activity even if the activity is not required.

Test Plan: Ran `bin/config done reindex --force` several times.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11922

Differential Revision: https://secure.phabricator.com/D16970
2016-12-01 13:53:33 -08:00
epriestley
9730f5a34f Allow custom Sites to have custom 404 controllers
Summary:
Currently, custom Sites must match `.*` or similar to handle 404's, since the fallback is always generic.

This locks them out of the "redirect to canonicalize to `path/` code", so they currently have a choice between a custom 404 page or automatic correction of `/`.

Instead, allow the 404 controller to be constructed explicitly. Sites can now customize 404 by implementing this method and not matching everything.

(Sites can still match everything with a catchall rule if they don't want this behavior for some reason, so this should be strictly more powerful than the old behavior.)

See next diff for CORGI.

Test Plan:
  - Visited real 404 (like "/asdfafewfq"), missing-slash-404 (like "/maniphest") and real page (like "/maniphest/") URIs on blog, main, and CORGI sites.
  - Got 404 behavior, redirects, and real pages, respectively.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16966
2016-11-30 15:25:09 -08:00
epriestley
29a3cd5121 Add "Manual Activities", to tell administrators to rebuild the search index
Summary:
Ref T11922. After updating to HEAD of `master`, you need to manually rebuild the index. We don't do this during `bin/storage upgrade` because it can take a very long time (`secure.phabricator.com` took roughly an hour) and can happen while Phabricator is running.

However, if we don't warn users about this they'll just get a broken index unless they go read the changelog (or file an issue, then we tell them to go read the changelog).

This adds a very simple table for notes to administrators so we can write a "you need to go rebuild the index" note, then adds one.

Administrators clear the note by completing the activity and running `bin/config done reindex`. This isn't automatic because there are various strategies you can use to approach the issue, which I'll discuss in greater detail in the linked documentation.

Also, fix an issue where `bin/storage upgrade --apply <patch>` could try to re-mark an already-applied patch as applied.

Test Plan:
  - Ran storage ugrades.
  - Got instructions to rebuild search index.
  - Cleared instructions with `bin/config done reindex`.

Reviewers: chad

Reviewed By: chad

Subscribers: avivey

Maniphest Tasks: T11922

Differential Revision: https://secure.phabricator.com/D16965
2016-11-30 11:23:54 -08:00
Chad Little
eeb80ba96b Add sidenav back to workboards
Summary: This is still reasonably functional and useful to people, and we don't have better mechanics to offset the change.

Test Plan: New Workboard, set Workboard color, test mobile, desktop.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16964
2016-11-30 09:56:55 -08:00
epriestley
23a202866a When running a fulltext query with no query, enforce order by document creation date
Summary:
Fixes T11929. When running with a query, we no longer enforce an order on the subquery join to produce results more quickly when searching for common strings.

However, this means that empty queries (like those issued by "Close as Duplicate") don't order subquery results.

Restore a `dateCreated` order if there is no query text.

Test Plan: Artificially set limit to 10, still saw 10 most recent tasks.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11929

Differential Revision: https://secure.phabricator.com/D16960
2016-11-29 12:14:57 -08:00
epriestley
5cc4f6407c Make the modal choice between "Edit This Event" and "Edit Future Events" more clear
Summary:
Fixes T11909. Ref T11816. Instead of offering a dropdown with choices between "Edit/Cancel/Reinstate This Event" and "Edit/Cancel/Reinstate Future Events", make the choice more explicit.

This dialog ends up pretty wordy but this edit is rare, so I think that's alright.

Test Plan: {F2046863}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816, T11909

Differential Revision: https://secure.phabricator.com/D16956
2016-11-28 10:47:58 -08:00
epriestley
22a566f732 Ignore Calendar date edits which just change the internal date timezone without rescheduling it
Summary:
Ref T11816. Currently, if someone in California creates an event and then someone in New York edits it, we generate a no-op "<user> changed the start time from 3PM to 3PM." transaction.

This is because the internal timezone of the event is changing, but the actual absolute time is not.

Instead, when an edit wouldn't reschedule an event and would only change the internal timezone, ignore the edit.

Test Plan:
  - Edited non-all-day events in PST / EST with out making changes (ignored).
  - Edited non-all-day events in PST / EST with changes (changes worked).
  - Performed the same edits with all-day events, which also were ignored and worked, respectively.
  - Pulled events in and out of all-day mode in different timezones, behavior seemeed reasonable.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16955
2016-11-28 10:33:59 -08:00
epriestley
c5162074a1 Fix an issue where internal Calendar DateTimes would not be correctly set to all-day
Summary:
Ref T11816. I don't really know what happened here, maybe I rewrote and broke this at the last second?

In most cases, we directly respect the `isAllDay` flag on the event, so the internal date state doesn't matter too much.

However, in the case of mail notifications, the raw internal state is relevant. This should fix mail notifications for all-day events.

(I might still turn them off since I'm not sure they're too useful, but it's good to have them working.)

Test Plan:
  - Created a new all-day event, verified database values wrote correctly.
  - Ran `bin/calendar notify --trace`, verified it picked up an all-day event tomorrow with a large enough `--minutes` value.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16954
2016-11-28 08:54:28 -08:00
Chad Little
dece7af50b Prettier file embeds
Summary: Spruce up the file embeds a little more, hover state, icons, file size.

Test Plan:
Add a psd and pdf, see new icons. Check differential, still see icons there too. Test mobile, desktop.

{F2042539}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16950
2016-11-27 14:57:06 -08:00
epriestley
b2cdebefea Fix two errors from the error logs
Summary: Found these in the `secure` error logs: one bad call, one bad column.

Test Plan: Searched for empty string. Double-checked method name.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16948
2016-11-26 07:50:57 -08:00
epriestley
7c5b5327c8 Use stemming in the MySQL fulltext search engine
Summary:
Ref T6740. When we index a document, also save a copy of the stemmed version.

When querying, search the combined corpus for the terms.

(We may need to tune this a bit later since it's possible for literal, quoted terms to match in the stemmed section, but I think this wil rarely cause issues in practice.)

A downside here is that search sort of breaks if you upgrade into this and don't reindex. I wasn't able to find a way to issue the query that remained compatible with older indexes and didn't have awful performance, so my plan is:

  - Put this on `secure`.
  - Rebuild the index.
  - If things look good after a couple of days, add a way that we can tell people they need to rebuild the search index with a setup warning.

We might get some reports between now and then, but if this is super awful we should know by the end of the weekend.

Test Plan:
WOW AMAZING

{F2021466}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T6740

Differential Revision: https://secure.phabricator.com/D16947
2016-11-25 15:30:50 -08:00
epriestley
d54c14c644 If InnoDB FULLTEXT is available, use it for for fulltext indexes
Summary: Ref T11741. I'll wait until the release cut to land this; it just adds a test for InnoDB FULLTEXT being available instead of always returning `false`.

Test Plan:
  - Ran with InnoDB fulltext locally for a day and a half without issues.
  - Ran `bin/storage upgrade`, saw it detect InnoDB fulltext.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11741

Differential Revision: https://secure.phabricator.com/D16946
2016-11-25 15:29:14 -08:00
epriestley
2b7ec1deea Boost search result title matches
Summary: Ref T6740. When a query matches a document title, boost results.

Test Plan:
  - Searched for `button bar`.
  - Before:

{F2019463}

  - After:

{F2019470}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T6740

Differential Revision: https://secure.phabricator.com/D16945
2016-11-25 15:25:49 -08:00
epriestley
54470a12d4 Execute fulltext queries using a subquery instead of by ordering the entire result set
Summary:
Ref T6740. Currently, we issue fulltext queries with an "ORDER BY <score>" on the entire result set.

For very large result sets, this can require MySQL to do a lot of work. However, this work is generally useless: if you search for some common word like "diff" or "internet" or whatever and match 4,000 documents, the chance that we can score whatever thing you were thinking of at the top of the result set is nearly nothing. It's more useful to return quickly, and let the user see that they need to narrow their query to get useful results.

Instead of doing all that work, let MySQL find up to 1,000 results, then pick the best ones out of those.

This actual change is a little flimsy, since our index isn't really big enough to suffer indexing issues. However, searching for common terms on my local install (where I have some large repositories imported and indexed) drops from ~40ms to ~10ms.

My hope is to improve downstream performance for queries like "translatewiki" here, particularly:

<https://phabricator.wikimedia.org/T143863>

That query matches about 300 trillion documents but there's a ~0% chance that the one the user wants is at the top. It takes a couple of seconds to execute, for me. Better to return quickly and let the user refine their results.

I think this will also make some other changes related to stemming easier.

This also removes the "list users first" ordering on the query, which made performance more complicated and seems irrelevant now that we have the typeahead.

Test Plan:
  - Searched for some common terms like "code" locally, saw similar results with better performance.
  - Searched for useful queries (e.g., small result set), got identical results.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T6740

Differential Revision: https://secure.phabricator.com/D16944
2016-11-25 15:19:54 -08:00
epriestley
48a34eced2 Prepare for InnoDB FULLTEXT support
Summary:
Ref T11741. This makes everything work if we switch to InnoDB, but never actually switches yet.

Since the default minimum word length (3) and stopword list (36 common English words) in InnoDB are generally pretty reasonable, I just didn't add any setup advice for them. I figure we're better off with simpler setup until we identify some real problem that the builtin stopwords create.

Test Plan: Swapped the `false` to `true`, ran `storage adjust`, got InnoDB fulltext indexes, searched for stuff, got default "AND" behavior.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11741

Differential Revision: https://secure.phabricator.com/D16942
2016-11-25 15:18:26 -08:00
epriestley
9d0752063e Allow bin/storage adjust to adjust table engines
Summary:
Ref T11741. On recent-enough versions of MySQL, we would prefer to use InnoDB for fulltext indexes instead of MyISAM.

Allow `bin/storage adjust` to read actual and expected table engines, and apply adjustments as necessary.

We have one existing bad table that uses the wrong engine, `metamta_applicationemail`. This change corrects that table.

Test Plan:
  - Ran `bin/storage upgrade`.
  - Saw the adjustment phase apply this change properly:

```
>>>[463] <query> ALTER TABLE `local_metamta`.`metamta_applicationemail` COLLATE = 'utf8mb4_bin', ENGINE = 'InnoDB'
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11741

Differential Revision: https://secure.phabricator.com/D16941
2016-11-25 15:13:40 -08:00
epriestley
ff3333548f Create and populate a stopwords table for InnoDB fulltext indexes to use in the future
Summary:
Ref T11741. InnoDB uses a stopwords table instead of a stopwords file.

During `storage upgrade`, synchronize the table from the stopwords file on disk.

Test Plan:
  - Ran `storage upgrade`.
  - Ran `select * from stopwords`, saw stopwords.
  - Added some garbage to the table.
  - Ran `storage upgrade`, saw it remove it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11741

Differential Revision: https://secure.phabricator.com/D16940
2016-11-25 15:13:08 -08:00
epriestley
a956047989 Use PhutilQueryCompiler in Phabricator fulltext search
Summary:
Ref T11741. Fixes T10642. Parse and compile user queries with a consistent ruleset, then submit queries to the backend using whatever ruleset MySQL is configured with.

This means that `ft_boolean_syntax` no longer needs to be configured (we'll just do the right thing in all cases).

This should improve behavior with RDS immediately (T10642), and allow us to improve behavior with InnoDB in the future (T11741).

Test Plan:
  - Ran various queries in the UI, saw the expected results.
  - Ran bad queries, got useful errors.
  - Searched threads in Conpherence.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10642, T11741

Differential Revision: https://secure.phabricator.com/D16939
2016-11-25 14:46:10 -08:00
epriestley
eac49e421a Fix an issue where an excessively long TTL was computed for "not attending anything" event caches
Summary:
Fixes T11894. Currently, if you aren't attending any events for a while, we can cache that you are free for the next 72 hours, even if you have an event in a few hours.

Instead, only cache "user is free" until the next event, if one exists.

Test Plan: Dumped cache TTLs, saw 52 minutes instead of ~4300 minutes with a near-upcoming event.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11894

Differential Revision: https://secure.phabricator.com/D16937
2016-11-23 15:22:13 -08:00
epriestley
a1025ca52e Make logged-out users use global settings, not default settings
Summary: Fixes T11917. Give logged-out / omnipotent users the global settings, not the default settings.

Test Plan: Changed applications and language, logged out, saw changes as a public user.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11917

Differential Revision: https://secure.phabricator.com/D16936
2016-11-23 14:26:00 -08:00
epriestley
8958d68ec6 When a Phortune subscription has a removed payment method, be more explicit about it
Summary:
Currently, when a payment method is invalid we still render the full name and let you save the form without making changes. This can be confusing.

Instead:

  - Render "<Deleted Payment Method>", literally.
  - Render an error immediately.
  - Prevent the form from being saved without changing the method.

Test Plan: {F1955487}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16935
2016-11-23 14:07:40 -08:00
epriestley
f199243104 Clean up another insufficiently-general exception
Summary:
Ref T11044. This is still catching the older exceptions, which are now more general.

If you loaded the web UI without MySQL running, this meant you got a less-helpful error.

Test Plan: Stopped MySQL, loaded web UI, got a more-helpful error.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16930
2016-11-23 10:41:19 -08:00
epriestley
4dadad53ae Prevent media from autoplaying when rendered as a feed story
Summary: Fixes T11845. Users can still embed a text panel on the home page to give it some ambiance.

Test Plan: Wrote an autoplay video as a comment, saw it in feed. Before change: autoplay. After change: no auto play. On task: still autoplay.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11845

Differential Revision: https://secure.phabricator.com/D16920
2016-11-22 14:19:56 -08:00
epriestley
faf42cbe8f Allow "Wait for Message" to be unset on build plans
Summary: Fixes T11910. I spent a couple of minutes looking for the root cause without much luck, but this will all be obsoleted by an eventual upgrade to `EditEngine` anyway.

Test Plan: Set and unset "Wait for Message", which now worked.

Reviewers: chad, avivey

Reviewed By: avivey

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11910

Differential Revision: https://secure.phabricator.com/D16919
2016-11-22 14:02:37 -08:00
Chad Little
0aa3f8b86b Allow commenting on non-viewable lightbox files
Summary: Removes the viewable restriction on embedded files. Builds a basic lightbox UI for commenting.

Test Plan:
Add psd, pdf to Maniphest task, clicked on download, comment, left comment. Closed box.

{F1943726}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T3612

Differential Revision: https://secure.phabricator.com/D16917
2016-11-22 11:29:36 -08:00
epriestley
0ed767b967 Fix a couple of partition migration bugs
Summary:
Ref T11044. Few issues here:

  - The `PhutilProxyException` is missing an argument (hit this while in read-only mode).
  - The `$ref_key` is unused.
  - When you add a new master to an existing cluster, we can incorrectly apply `.php` patches which we should not reapply. Instead, mark them as already-applied.

Test Plan:
  - Poked this locally, but will initialize `secure004` as an empty master to be sure.

Reviewers: chad, avivey

Reviewed By: avivey

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16916
2016-11-22 10:57:24 -08:00
Aviv Eyal
256d14c7ea Move testcase file to right place
Test Plan: `arc unit`, see test name in list.

Reviewers: chad, epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16915
2016-11-22 18:54:15 +00:00
epriestley
e6bfa1bd23 Remove "mysql.configuration-provider" configuration option
Summary:
Ref T11044. This was old Facebook cruft for reading configuration from SMC (and maybe doing some other questionable things). See D183.

(See also D175 for discussion of this from 2011.)

In modern Phabricator, you can subclass `SiteConfig` to provide dynamic configuration, and we do so in the Phacility cluster. This lets you change any config, and change in response to requests (e.g., for instancing) and is generally more powerful than this mechanism was.

This configuration provider theoretically let you roll your own replication or partitioning, but in practice I believe no one ever did, and no one ever could have anyway without more support in the upstream (for migrations, read-after-write, etc).

Test Plan:
  - Grepped for removed option.
  - Browsed around with clustering off.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16911
2016-11-22 09:24:46 -08:00
epriestley
4da74166fe When storage is partitioned, refuse to serve requests unless web and databases agree on partitioning
Summary:
Ref T11044. One popular tool in a modern operations environment is Puppet. The primary purpose of this tool is to randomly revert hosts to older or different configurations.

Introducing an element of chaotic unpredictability into operations trains staff to be on high alert at all times, rather than lulled into complacency by predictability or consistency.

When Puppet reverts a Phabricator host's configuration to an older version, we might start writing data to a lot of crazy places where it shouldn't go. This will create a big sticky mess that is virtually impossible to undo, mostly because we'll get two files with ID 123 or two tasks with ID 456 or whatever else and good luck with that.

Instead, after changing the partition layout, require `bin/storage partition` to be run. This writes a copy of the config everywhere.

Then, when we start serving web requests, make sure every database has the exact same config. This will foil Puppet by refusing to run requests on hosts it has reverted.

Test Plan:
  - Changed partition configuration.
  - Ran Phabricator.
  - FOILED!
  - Ran `bin/storage partition` to sync config.
  - Things worked again.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16910
2016-11-22 04:15:46 -08:00
epriestley
bac27fb403 Remove "mysql.implementation" configuration
Summary:
Ref T11044. Fixes T10931. This option has essentially never been useful for anything, and we've picked the best implementation for a long time (MySQLi if available, MySQL if not).

I am not aware of any reason to ever set this manually. If someone comes up with some bizarre but legitimate use case that I haven't thought of, we can modularize it.

Test Plan: Browsed around. Grepped for `mysql.implementation`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10931, T11044

Differential Revision: https://secure.phabricator.com/D16909
2016-11-22 04:15:34 -08:00
Chad Little
88a966993b Ajax commenting on lightbox images
Summary: Adds a comment box, you can put text into it, hit enter, and see it come back.

Test Plan: Put text into box, see it come back.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T3612

Differential Revision: https://secure.phabricator.com/D16907
2016-11-21 20:19:37 -08:00
epriestley
bcfd515b32 Run all minor setup checks on all configured database hosts
Summary:
Fixes T10759. Fixes T11817. This runs all the general sanity/configuration checks on all the active servers.

None of these warnings are very important, and this doesn't change any logical stuff.

Depends on D16904.

Test Plan: Painstakingly triggered each warning, verified that they rendered correctly and that messages told me which host was affected.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10759, T11817

Differential Revision: https://secure.phabricator.com/D16905
2016-11-21 15:55:54 -08:00
epriestley
326d5bf800 Detect replicating masters and fatal (also, warn on nonreplicating replicas)
Summary:
Ref T10759. Check master/replica status during startup.

After D16903, this also means that we check this status after a database comes back online after being unreachable.

If a master is replicating, fatal (since this can do a million kinds of bad things).

If a replica is not replicating, warn (this just means the replica is behind so some data is at risk).

Also: if your masters were actually configured properly (mine weren't until this change detected it), we would throw away patches as we applied them, so they would only apply to the //first// master. Instead, properly apply all migration patches to all masters.

Test Plan:
  - Started Phabricator with a replicating master, got a fatal.
  - Stopped replication on a replica, got a warning.
  - With two non-replicating masters, upgraded storage.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10759

Differential Revision: https://secure.phabricator.com/D16904
2016-11-21 15:55:22 -08:00
epriestley
bc4187d709 When we "discover" new fatal setup issues, stop serving traffic
Summary:
Ref T10759. We may "discover" the presence of a fatal setup error later, after starting Phabricator.

This can happen in a few ways, but most are unlikely. The one I'm immediately concerned about is:

  - Phabricator starts up during a disaster with some databases unreachable.
  - We start with warnings (unreachable databases are generally not fatal, since it's OK for some subset of hosts to be down in replicated/partitioned setups).
  - The unreachable databases later recover and become accessible again.
  - When we run checks against them, we discover that they are misconfigured.

Currently, "fatal" setup issues are not truly fatal if we're "in flight" -- we've survived setup checks at least once in the past. This is bad in the scenario above.

Especially with partitioning, it could lead to mangled data in a disaster scenario where operations staff makes a small configuration mistake while trying to get things running again.

Instead, if we "discover" a fatal error while already "in flight", reset the whole setup process as though the webserver had just restarted. Don't serve requests again until we can make it through setup without hitting fatals.

Test Plan:
  - Started Phabricator with multiple masters, one of which was down and broken.
  - Got a warning about the bad master.
  - Revived the master.
  - Before: Phabricator detects the fatal, but keeps serving requests.
  - After: Phabricator detects the fatal, resets the webserver, and stops serving requests until the fatal is resolved.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10759

Differential Revision: https://secure.phabricator.com/D16903
2016-11-21 15:54:40 -08:00
epriestley
78040e0ff5 Run "DatabaseSetup" checks against all configured hosts
Summary:
Ref T10759. Currently, these checks run only against configured masters. Instead, check every host.

These checks also sort of cheat through restart during a recovery, when some hosts will be unreachable: they test for "disaster" by seeing if no masters are reachable, and just skip all the checks in that case.

This is bad for at least two reasons:

  - After recent changes, it is possible that //some// masters are dead but it's still OK to start. For example, "slowvote" may have no master, but everything else is reachable. We can safely run without slowvote.
  - It's possible to start during a disaster and miss important setup checks completely, since we skip them, get a clean bill of health, and never re-test them.

Instead:

  - Test each host individually.
  - Fundamental problems (lack of InnoDB, bad schema) are fatal on any host.
  - If we can't connect, raise it as a //warning// to make sure we check it later. If you start during a disaster, we still want to make sure that schemata are up to date if you later recover a host.

In particular, I'm going to add these checks soon:

  - Fatal if a "master" is replicating.
  - Fatal if a "replica" is not replicating.
  - Fatal if a database partition config differs from web partition config.
  - When we let a database off with a warning because it's down, and later upgrade it to a fatal because we discover it is broken after it comes up again, fatal everything. Currently, we keep running if we "discover" the presence of new fatals after surviving setup checks for the first time.

Test Plan:
  - Configured with multiple masters, intentionally broke one (simulating a disaster where one master is lost), saw Phabricator still startup.
  - Tested individual setup checks by intentionally breaking them.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10759

Differential Revision: https://secure.phabricator.com/D16902
2016-11-21 15:49:07 -08:00
epriestley
bf1cbc2499 Don't let users pick "whatever.git" as a repository short name, make "." work
Summary:
Fixes T11902.

  - Periods now work in short names.
  - If you try to name something ".git", no dice.

Test Plan:
  - Tried to name something "quack.git", was politely rejected.
  - Named something "quack.notgit", and it worked fine.
  - Cloned Mercurial and Git repositories over SSH with ".git" and non-".git" variants without hitting any issues.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11902

Differential Revision: https://secure.phabricator.com/D16908
2016-11-21 15:47:20 -08:00
epriestley
55e21565b5 Support application partitioning across multiple masters
Summary:
Ref T11044. I'm going to hold this until after the release cut, but I think it's good to go.

This allows installs to configure multiple masters in `cluster.databases` and partition applications across them (for example, put Maniphest on a dedicated database).

When we make a Maniphest connection we go look up which master we should be hitting first, then connect to it.

This has at least approximately been planned for many years, so the actual change is largely just making sure that your config makes sense.

Test Plan:
  - Configured `db001.epriestley.com` and `db002.epriestley.com` as master/master.
  - Partitioned applications between them.
  - Interacted with various applications, saw writes go to the correct host.
  - Viewed "Database Servers" and saw partitioning information.
  - Ran schema upgrades.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16876
2016-11-19 14:14:39 -08:00
epriestley
97cd7a98b1 Strip restricted and incomplete handles from the "Mentions" tab on Maniphest tasks
Summary:
Ref T8345. See T8345#201048 for discussion.

This rule (don't show mentions of or from restricted objects) is more consistent with how we render mentions in the timeline and I think generally a better behavior.

Test Plan:
  - Mentioned a task on a public task and a private task.
  - Privileged user (foreground) sees both.
  - Public user (background) sees only the public mention.

{F1929485}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8345

Differential Revision: https://secure.phabricator.com/D16900
2016-11-18 14:08:20 -08:00
Chad Little
8aeb7aa525 Show file comments on file lightboxes
Summary: Basic work in progress, but should show timeline comments for files when in lightbox mode. Looks reasonable.

Test Plan: click on images, see comments from timeline.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T3612

Differential Revision: https://secure.phabricator.com/D16896
2016-11-18 13:24:03 -08:00
epriestley
39509648f2 In Calendar mobile month view, color circles green for attending events too
Summary: Ref T11816. We currently color circles green if you're invited, but should color them green if you're attending, too.

Test Plan: Viewed calendar mobile month view, saw attending events in green.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16899
2016-11-18 11:56:40 -08:00
epriestley
132b0803cb Fix a couple of calendar export daterange issues
Summary:
Ref T11816. In some cases, Calendar would only export a subset of events because the "export" flag was ignored or the "display" parameter applied an improper date range to the query.

  - Make sure the `export` flag gets processed, even though it isn't a "real" field on the search engine.
  - Clear the "display" parameter to avoid date range windowing coming from the day/month logic.

Test Plan: Exported a "display=month" view, verified future events came with it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16898
2016-11-18 10:07:35 -08:00
epriestley
0033fe6667 When a field isn't lockable, just freeze the lock status instead of removing any lock
Summary:
See downstream issue here: <https://phabricator.wikimedia.org/T150992>

In at least one case (project milestones) we have a locked, non-lockable field. This means "this is locked, and you can't change the fact that it is locked".

At least for now, preserve this behavior.

Test Plan: Created a new milestone of an existing project. This worked correctly with the patch.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16895
2016-11-17 15:04:18 -08:00
epriestley
2befd239a8 Add session and request hooks to PhabricatorAuthSessionEngine
Summary: This supports doing a bunch of sales funnel tracking on Phacility.

Test Plan: See next diff.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16890
2016-11-17 13:09:29 -08:00
epriestley
79132311f4 Generate slightly shorter summaries in the typeahead browse dialog
Summary: Ref T11034. Try to produce a roughly-one-sentence summary instead of a roughly-one-paragraph summary for the browse dialog.

Test Plan:
  - Added unit tests, ran unit tests.
  - Wrote a longer summary for a project, browsed to it, saw a shorter summary.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11034

Differential Revision: https://secure.phabricator.com/D16892
2016-11-17 13:08:08 -08:00
epriestley
d69a1b95e7 Fix an EditEngine issue with unlocking fields which can't be locked
Summary: This code should go inside the field-locking loop. Otherwise, it only applies to the last field, and fatals if there are no fields.

Test Plan: Carefuller inspection.

Reviewers: chad

Reviewed By: chad

Subscribers: 20after4

Differential Revision: https://secure.phabricator.com/D16889
2016-11-17 10:29:52 -08:00
epriestley
625d5235a5 Prevent typeahead sources from querying against empty tokens
Summary:
Certain unusual queries, like `[-]`, could tokenize into a list which included the empty string.

This would then convert into a query for `... LIKE "%"` which just joins the entire table.

Instead: tokenize smarter; never return the empty token; add some test cases.

Test Plan: Ran unit tests. Queried for `[[blah blah]]`, saw a reasonable query come out the other end.

Reviewers: chad

Reviewed By: chad

Subscribers: 20after4

Differential Revision: https://secure.phabricator.com/D16888
2016-11-17 09:45:16 -08:00
epriestley
b4faf2e63e Allow "harbormaster.createartifact" to decode raw HTTP parameter types of artifact properties
Summary:
Ref T11887. This isn't a great fix but makes the method behave properly until I get around to a real fix.

In the longer term, I want to convert all of this pluggable Harbormaster/Drydock stuff (blueprints, artifacts, build plans) to use EditEngine + EditField instead of the weird mishmash of older/custom stuff it currently uses. However, this is a more involved project to execute and I'd like to be in that area of the codebase first so it gets adequate testing.

Until that happens, just put a reasonble-ish mechanism in place to let artifacts correct inbound types. This is the only artifact type and only parameter which needs casting.

Test Plan:
  - Made a `curl` call to `harbormaster.createartifact` to create a URI artifact with `?...&ui.external=1`.
  - Before patch: type error on `ui.external` not being a boolean.
  - After patch: artifact created successfully.

Reviewers: chad

Reviewed By: chad

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11887

Differential Revision: https://secure.phabricator.com/D16887
2016-11-17 08:03:01 -08:00
epriestley
b02f64f6ee Make project token sorting and normalization a little less hacky
Summary:
Ref T8510. Use "\n" as a delimiter between name sections. Specifically, project "AAA" with tag "zzz" should be a better match for query "AAA" than project "AAA BBB" is.

Make use of this delimiter slighlty more obvious in the UI.

Test Plan:
  - Created projects "Phacility" and "Phacility Core Access".
  - Typed "Phacility".
  - Before patch: first hit is "Phacility Core Access".
  - After patch: first hit is "Phacility".
  - Viewed debugging output table, saw visual explanation of behavior.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8510

Differential Revision: https://secure.phabricator.com/D16886
2016-11-17 08:02:23 -08:00
epriestley
7c4c76d32a Fix a Quicksand/Lightbox contention issue
Summary:
Fixes T11785. Lightbox calls `JX.Stratcom.pass()` to let other handlers react, but should not. At least today, we never put, e.g., links inside a lightbox.

This code appears in the original commit so it was probably just copy/pasted from somewhere and I missed it in review.

(Or there's some edge case I'm not thinking of and we'll figure it out soon enough.)

Additionally, blacklist `/file/data/` from Quicksand naviagtion: Quicksand should never fetch these URIs.

Test Plan:
  - Disabled `security.alternate-file-domain`.
  - Enabled Quicksand ("Persistent Chat").
  - Clicked an image thumbnail on a task.
    - Repeated that until things flipped out a bit.
    - After the patch: no issues.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11785

Differential Revision: https://secure.phabricator.com/D16884
2016-11-16 17:10:56 -08:00
Chad Little
d2f3e7f7f3 Remove sidenav from Workboards
Summary: Visually, I think these are much cleaner (with colors), and provide more workspace. I also don't really use the sidenav here and if I did, it would be to go back to the project homepage. I think this is overall better. If navigation page to project home is difficult or hard to find, we can maybe make a better header / crumbs bar to reduce that.

Test Plan: New project -> basic new board. Existing project -> color board. Desktop, Mobile, Fullscreen

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16882
2016-11-16 16:43:46 -08:00
epriestley
f33b5c30eb Use the same date rendering display logic for both tooltips and subheaders
Summary: Ref T11816. This could be a little cleaner, but we currently have two copies of the logic. Get them using the same code. Once that's actually working I can go make the code a little prettier.

Test Plan: Viewed Calendar month view tooltips, saw the same values as subheaders.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16880
2016-11-16 08:37:25 -08:00
epriestley
91ee6b78df Make some confusing/weird Calendar fields not configurable on custom EditEngine forms
Summary:
Ref T11816.

  - Host gets weird behavior around defaulting to the viewer.
  - Invitees get weird behavior around defaulting to the viewer.
  - "All Day" is just sort of weird since start / end date aren't customizable.
  - Recurring/Frequency are weird here and don't make much sense.

I can't immediately come up with reasons that any of these are particularly useful/valuable to default. More of them can be made editable after T10222 gets sorted out.

Test Plan: Edited edit engine custom forms for Calendar events, saw a more sensible list of customizable fields (e.g., policy stuff).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16878
2016-11-16 07:49:00 -08:00
epriestley
6e5565b3ff Clean up a little more Calendar display logic
Summary: Uh, non-all-day-events said 1:30 - 2:29 PM, which is real silly.

Test Plan: Looked at a non-all-day-event.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16874
2016-11-15 14:42:35 -08:00
Chad Little
ce0cb115ca Add Hero Image to Phame Post
Summary: Adds a headerimage and lets you set it on posts for added reverence. Is that a word?

Test Plan:
Add an image, see an image.

{F1923010}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16873
2016-11-15 13:44:47 -08:00
epriestley
c7f2e4a924 Document calendar summary icons
Summary:
Fixes T11809. Ref

  - Explicitly document the summary icon hints -- I don't think these are too hard to figure out (and maybe this stuff should just go in the tooltips) but we can start here.
  - Use color + shape to distinguish between "cancelled" and "declined", not just color (for users with vision accessibility issues).
  - Translate a "minute(s)" string into sensible English.
  - Use RSVP status on the month view green circle thing.

Test Plan:
  - Read docs.
  - Looked at month view.
  - Read reminder mail.
  - Viewed month view mobile view.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11809

Differential Revision: https://secure.phabricator.com/D16872
2016-11-15 13:44:20 -08:00
epriestley
e713cc08e6 Make Calendar query for indirect invites/RSVPs by default, like Differential
Summary:
Ref T11816. Since the dashboard got updated, Differential now interprets "Responsible Users: epriestley" to mean "epriestley, or any project or package epriestley is part of". You can query for just "epriestley" with "exact(epriestley)".

Give Calendar invites the same behavior: "epriestley" means "any event epriestley is invited to, or a project they are a member of is invited to". Individual invites can be queried with "exact(epriestley)".

This is a little bit copy-pastey but I want to wait for a third use case to clean it up since I think I'm going to have to do a bunch of generalization around "how does an individual PHID get turned into a bunch of PHIDs".

Test Plan: Queried for "Invited: dog", "invited: viewer", "invited; exact(dog)", etc.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16870
2016-11-15 12:47:01 -08:00
Chad Little
015ead6e7b Add subtitle to PhamePost
Summary: No view engine yet (adding header image next), but adds subtitle to display like PhameBlog

Test Plan: Add a subtitle, remove a subtitle.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16871
2016-11-15 12:02:03 -08:00
epriestley
edba4bb8d4 Improve Calendar event behavior for group invites
Summary:
Ref T11816. Projects can be invited to an event, but the UI is currently fairly agnostic about them.

Instead, introduce the idea of "RSVPs", which are basically invites for you as an individual or for any group you're a part of. When we go to check if you're invited, we check for you individually first, then check for any groups you belong to if you haven't already accepted/declined.

On the calendar detail page:

  - Show the quick "Join" / "Decline" buttons if any project you're a member of is invited.
  - If you're invited, highlight any projects which you're a member of to make that more clear.

On other calendar views:

  - If you're invited as part of a project, show the "multiple users" icon.
  - If it's just you, continue showing the "add one user" icon.

Test Plan: Viewed month view, day view, detail view. Invited groups and individuals. Invited "Dog Project", accepted invite as user "Dog".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16868
2016-11-15 11:16:55 -08:00
epriestley
6464934cd6 Fix a bug on Calendar event day views for all day events in certain timezones
Summary:
Ref T11816. This logic was correct, we just did all the work and then mostly threw away the results. This worked correctly anyway in some timezones.

Instead, actually use `$min_date` and `$max_date`.

Test Plan: In "America/Toronto" with server in a more-western timezone, viewed a "Nov 11 - Nov 12" all-day event, saw those dates.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16867
2016-11-15 11:10:20 -08:00
epriestley
885805f340 Make Passphrase "token" credentials accessible via the API
Summary: Fixes T11867. This should really be on the `CredentialType` itself, but just punt that for now until the API endpoint gets updated. We'll need the actual code here anyway in some form.

Test Plan: {F1922728}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11867

Differential Revision: https://secure.phabricator.com/D16864
2016-11-15 09:12:35 -08:00
epriestley
508d86aab6 Don't send Phurl mail to the URL object itself
Summary: Fixes T11868. This is silly and does not make sense.

Test Plan: Edited a Phurl URL, verified mail only went to me, not to the object itself.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11868

Differential Revision: https://secure.phabricator.com/D16863
2016-11-15 09:12:15 -08:00
epriestley
7097abbe57 Add a bunch of Phacility-specific code to the upstream, thinly veiled as generic code
Summary:
Ref T9304. This adds a "GuidanceEngine" which can generate "Guidance".

In practice, this lets third-party code (rSERVICES) remove and replace instructions in the UI, which is basically only usefulf or us to tell users to go read the documentation in the Phacility cluster.

The next diff tailors the help on the "Auth Providers" and "Create New User" pages to say "PHACILITY PHACILITY PHACILITY PHACILITY".

Test Plan: Browed to "Auth Providers" and "Create New User" on instanced and non-instanced installs, saw appropriate guidance.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9304

Differential Revision: https://secure.phabricator.com/D16861
2016-11-15 09:11:22 -08:00
epriestley
e6c82c0994 Fix an issue with generating browser URIs in an SVN repository
Summary: Fixes T11866. This got converted wrong when doing the `/source/` stuff.

Test Plan: Browsed the root directory of a Subversion repository in Diffusion.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11866

Differential Revision: https://secure.phabricator.com/D16860
2016-11-15 07:15:20 -08:00
Aviv Eyal
d5a72ca98e Don't show "clone-name" as "Short Name"
Summary: See D16851 - there's now a difference in their meaning, so don't unite them in the UI.

Test Plan: Load manage page of repos

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16858
2016-11-14 22:46:40 +00:00
epriestley
7165e4da90 Discard stdout/stderr from the aphlict subprocess when running in daemon (normal) mode
Summary:
Fixes T11818. We don't discard output, so once we read more than 2GB of output we'll exceed the maximum size of a string in an internal buffer.

Instead, configure the future so output is discarded.

Test Plan: Added logging to `libphutil/`, saw internal buffer grow steadily before this change and stay constant at 0 after this change.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11818

Differential Revision: https://secure.phabricator.com/D16855
2016-11-13 16:43:42 -08:00
epriestley
6a62fca950 Support slightly prettier repository URIs in Diffusion
Summary: Fixes T4245. When a repository has a short name, use `/source/shortname/` as its primary URI.

Test Plan:
  - Cloned Git repositories from shortnames via HTTP and SSH.
  - Cloned Mercurial repositories from shortnames via HTTP and SSH.
  - Cloned Subversion repositories from shortnames via SSH.
  - Browsed Git, Mercurial and Subversion repositories.
  - Added and removed short names to various repositories.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D16851
2016-11-13 12:42:12 -08:00
epriestley
c9e140e283 Restore green color on mobile calendar month view when a day contains events you are invited to
Summary: Ref T11816. This got dropped somewhere along the way, so the mobile month view no longer showed a green-colored hint if a day has events you're invited to.

Test Plan: Viewed Calendar month view on mobile, saw green circles for days with invited events.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16852
2016-11-13 12:41:55 -08:00
epriestley
558d194302 Update bin/storage workflows to accommodate multiple masters
Summary: Depends on D16847. Ref T11044. This updates the remaining storage-related workflows from the CLI to accommodate multiple masters.

Test Plan:
  - Configured multiple masters.
  - Ran all `bin/storage` workflows.
  - Ran `arc unit --everything`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16848
2016-11-12 16:37:47 -08:00
epriestley
bc15eee3f2 Update SchemaQuery and the web UI to accommodate multiple master databases
Summary:
Depends on D16115. Ref T11044. In the brave new world of multiple masters, we need to check the schemata on each master when looking for missing storage patches, keys, schema changes, etc.

This realigns all the "check out what's up with that schema" calls to work for multiple hosts, and updates the web UI to include a "Server" column and allow you to browse per-server.

This doesn't update `bin/storage`, so it breaks things on its own (and unit tests probably won't pass). I'll update that in the next change.

Test Plan: Configured local environment in cluster mode with multiple masters, saw both hosts' status reported in web UI.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16847
2016-11-12 16:36:52 -08:00
epriestley
ecc598f18d Support multiple database masters and convert easy callers
Summary:
Ref T11044. This moves toward partitioned application databases:

  - You can define multiple masters.
  - Convert all the easily-convertible code to become multi-master aware.

This doesn't convert most of `bin/storage` or "Config > Database (Stuff)" yet, as both are quite involved. They still work for now, but only operate on the first master instead of all masters.

Test Plan: Configured multiple masters, browsed around, ran `bin/storage` commands, ran `bin/storage --host ...`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16115
2016-11-12 16:30:20 -08:00
epriestley
745429aac1 Fix some slop with Differential field lists
Summary: I moved and then un-moved this incorrectly in D16846.

Test Plan: Looked at the old code, which worked better.

Reviewers: jacksongabbard, chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16849
2016-11-12 16:27:03 -08:00
epriestley
6dfc7e48ae Don't let users write summaries or test plans which will become ambiguous in commit messages
Summary:
Ref T11085. To recreate the issue:

  - From the web UI, click "Edit Revision".
  - Write something like this as your "Summary" (i.e., put another field marker, like "Test Plan:", into the summary):

> This is a test of the
> Test Plan: field to see
> if it works.

  - Save changes.

Later, when the summary is amended into a commit message, the parser will see two "Test Plan:" fields and fail to parse the message.

Instead, prevent users from making this edit.

Test Plan: {F1917640}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11085

Differential Revision: https://secure.phabricator.com/D16846
2016-11-12 08:03:46 -08:00
epriestley
4c540fb01b Fix some policy CSS
Summary:
Ref T11853. My CSS change for the more enormous policy dialog was a little too broad, and affected the "You shall not pass!" dialog too.

Narrow the scope of the CSS rules.

Also add a missing "." that I caught.

Test Plan:
  - Looked at policy exception dialogs.
  - Looked at policy explanation dialogs.
  - Looked at the end of that sentence.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11853

Differential Revision: https://secure.phabricator.com/D16841
2016-11-11 13:43:13 -08:00
epriestley
40d3bcb891 Fix a complicated object caching issue with the policy filter
Summary:
Fixes T11853. To set this up:

  - Create "Project A".
  - Join "Project A".
  - Create a subproject, "Project A Subproject 1".
    - This causes Project A to become a parent project.
    - This moves you to be a member of "Project A Subproject 1" instead of "Project A" directly.
  - Create another subproject, "Project A Subproject 2".
    - Do not join this subproject.
  - Set the second subproject's policy to "Visible To: Members of Project A".
  - Try to edit the second subproject.

Before this change, this fails:

  - When querying projects, we sometime try to skip loading the viewer's membership in ancestor projects as a small optimization.
  - Via `PhabricatorExtendedPolicyInterface`, we may then return the parent project to the policy filter for extended checks.
  - The PolicyFilter has an optimization: if we're checking an object, and we already have that object, we can just use the object we already have. This is common and useful.
  - However, in this case it causes us to reuse an incomplete object (an object without proper membership information). We fail a policy check which we should pass.

Instead, don't skip loading the viewer's membership in ancestor projects.

Test Plan:
  - Did all that stuff above.
  - Could edit the subproject.
  - Ran `arc unit --everything`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11853

Differential Revision: https://secure.phabricator.com/D16840
2016-11-11 13:42:18 -08:00
epriestley
ff677c1964 Fix two error strings in the diffusion.uri.edit Conduit method
Summary: Fixes T11839. Both are missing a parameter and one is a copy/paste slop.

Test Plan:
{F1913812}

{F1913813}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11839

Differential Revision: https://secure.phabricator.com/D16837
2016-11-10 08:55:12 -08:00
epriestley
9a1d59ad5b Separate sever-side typeahead queries into "prefix" and "content" phases
Summary:
Ref T8510. When users type "platypus" into a typeahead, they want "Platypus Playground" to be a higher-ranked match than "AAA Platypus", even though the latter is alphabetically first.

Specifically, the rule is: results which match the query as a prefix of the result text should rank above results which do not.

I believe we now always get this right on the client side. However, WMF has at least one case (described in T8510) where we do not get it right on the server side, and thus the user sees the wrong result.

The remaining issue is that if "platypus" matches more than 100 results, the result "Platypus Playground" may not appear in the result set at all, beacuse there are 100 copies of "AAA Platypus 1", "AAA Platypus 2", etc., first. So even though the client will apply the correct sort, it doesn't have the result the user wants and can't show it to them.

To fix this, split the server-side query into two phases:

  - In the first phase, the "prefix" phase, we find results that **start with** "platypus".
  - In the second phase, the "content" phase, we find results that contain "platypus" anywhere.

We skip the "prefix" phase if the user has not typed a query (for example, in the browse view).

Test Plan:
This is a lot of stuff, but the new ranking here puts projects which start with "w" at the top of the list. Lower down the list, you can see some projects which contain "w" but do not appear at the top (like "Serious Work").

{F1913931}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8510

Differential Revision: https://secure.phabricator.com/D16838
2016-11-10 08:54:59 -08:00
epriestley
663629e8ad Use Doritos™ Brand® perfect circles to indicate Busy/Away/Disabled
Summary:
Fixes T11829.

  - Currently in use: Doritos™ Brand® "Nacho Cheese"® perfect circles: •
  - Available alternative: Doritos™ Brand® "Cool Ranch"® perfect circles: ●

Test Plan: {F1913116}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11829

Differential Revision: https://secure.phabricator.com/D16834
2016-11-09 17:13:10 -08:00
Aviv Eyal
e634812a6d Remove plain-text file view of Diffusion files.
Summary:
fixes T11792.
There's no good reason any more to have this option, so just drop it.

Test Plan: Load a file, toggle remaining "blame" button. Load search results page and an image too, which are serviced by the same controller.

Reviewers: chad, #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin

Maniphest Tasks: T11792

Differential Revision: https://secure.phabricator.com/D16833
2016-11-10 00:40:09 +00:00
epriestley
706c21375e Remove empty implementations of describeAutomaticCapabilities()
Summary:
This has been replaced by `PolicyCodex` after D16830. Also:

  - Rebuild Celerity map to fix grumpy unit test.
  - Fix one issue on the policy exception workflow to accommodate the new code.

Test Plan:
  - `arc unit --everything`
  - Viewed policy explanations.
  - Viewed policy errors.

Reviewers: chad

Reviewed By: chad

Subscribers: hach-que, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D16831
2016-11-09 15:24:22 -08:00
epriestley
4811e6e7c1 Require several advanced postgraduate degrees to understand object policies
Summary:
Fixes T11836. See some prior discussion in T8376#120613.

The policy hint in headers in the UI is not exhaustive, and can not reasonably be exhaustive. For example, on a revision, it may say "All Users", but really mean "All users who can see the space this object is in and the repository it belongs to, plus the revision author and reviewers".

These rules are explained if you click (and, often, in the documentation), but "All Users" is still at least somewhat misleading.

I don't think there's any perfect solution here that balances the needs of both new and experienced users perfectly, but this change tries to do a bit better about avoiding cases where we say something very open (like "All Users") when the real policy is not very open.

Specifically, I've made these changes to the header:

  - Spaces are now listed in the tag, so it will say `(S3 > All Users)` instead of `(All Users)`. They're already listed in the header, this just makes it more explicit that Spaces are a policy container and part of the view policy.
  - Extended policy objects are now listed in the tag, so it will say `(S3 > rARC > All Users)` for a revision in the Arcanist repository which is also in Space 3.
  - Objects can now provide a "Policy Codex", which is an object that represents a rulebook of more sophisticated policy descriptions. This codex can replace the tag with something else.
    - Imported calendar events now say "Uses Import Policy" instead of, e.g., "All Users".

I've made these changes to the policy dialog:

  - Split it into more visually separate sections.
  - Added an explicit section for extended policies ("You must also have access to these other objects: ...").
  - Broken the object policy rules into a "Special Rules" section (for rules like "you can only see a revision if you can see the repository it is part of") and an "Object Policy" section (for the actual object policy).
  - Tried to make it a little more readable?
  - The new policy dialogs are great to curl up with in front of a fire with a nice cup of cocoa.

I've made these changes to infrastructure:

  - Implementing `PhabricatorPolicyInterface` no longer requires you to implement `describeAutomaticCapability()`.
  - Instead, implement `PhabricatorPolicyCodexInterface` and return a `PhabricatorPolicyCodex` object.
  - This "codex" is a policy rulebook which can set all the policy icons, labels, colors, rules, etc., to properly explain complex policies.
  - Broadly, the old method was usually either not useful (most objects have no special rules) or not powerful enough (objects with special rules often need to do more in order to explain them).

Test Plan:
{F1912860}

{F1912861}

{F1912862}

{F1912863}

Reviewers: chad

Reviewed By: chad

Subscribers: avivey

Maniphest Tasks: T11836

Differential Revision: https://secure.phabricator.com/D16830
2016-11-09 15:05:38 -08:00
Chad Little
d78802f3ab Redesign Comment Box
Summary: Redesign the action comment box for better use in two column, mobile, nuance.

Test Plan: Test in mobile / desktop / tablet, adding and removing actions. Actionless comment boxes, etc.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: cspeckmim, Korvin

Differential Revision: https://secure.phabricator.com/D16811
2016-11-09 10:36:25 -08:00
epriestley
d032eea216 Discourage new users from exploring too much
Summary: Fixes T11834. Actually adding the step wasn't in the `if (...)` block. Also, typo fix.

Test Plan: Saw only one "Explore" on `/guides/`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11834

Differential Revision: https://secure.phabricator.com/D16828
2016-11-08 13:27:15 -08:00
epriestley
afa1bb2860 Fix some grammatical gender constants
Summary: Ref T5267. I missed these in the variable types conversion.

Test Plan: `arc unit --everything`

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T5267

Differential Revision: https://secure.phabricator.com/D16824
2016-11-08 08:45:14 -08:00
epriestley
9803674525 Extract variable type information from pht() calls
Summary:
Ref T5267. When extrating data from `pht()` calls, also extract the argument types and export them into the map so they can be used by consumers.

We recognize plurals (`phutil_count()`, `new PhutilNumber`) and genders (`phutil_person()`). We'll need to annotate the codebase for those, since they're currently runtime-only.

Test Plan:
Rebuilt extraction maps, got data like this (note "number" type annotation).

```
  "Scaling pool \"%s\" up to %s daemon(s).": {
    "uses": [
      {
        "file": "/daemon/PhutilDaemonOverseer.php",
        "line": 378
      }
    ],
    "types": [
      null,
      "number"
    ]
  },
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T5267

Differential Revision: https://secure.phabricator.com/D16823
2016-11-08 08:33:15 -08:00
epriestley
3f5109b668 In prose diff dialogs (like "Show Details" in transactions), show "old", "new" and "diff" tabs
Summary:
Ref T7643. When you do something like this:

  - Edit a task description.
  - Click "Show Details" on the resulting transaction.
  - Get a prose diff dialog showing the change.

...now add some "Old" and "New" tabs. These are useful for:

  - reverting to the old text by copy/pasting;
  - reading just the new/old text if the diff is noisy;
  - sometimes just nice to have?

(This looks a little rough but I didn't want to put a negative margin on tab groups inside dialogs? Not sure what the best fix here is.)

Test Plan: {F1909390}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7643

Differential Revision: https://secure.phabricator.com/D16817
2016-11-07 15:18:19 -08:00
epriestley
6a7dde03cc On @username mentions in remarkup, show the "busy" dot color
Summary:
Ref T11809. I missed this when adding a "Busy" status.

Also the other dot is orange? Just make them all orange for consistency.

Test Plan: Viewed `@username` of busy users (orange), away users (red).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11809

Differential Revision: https://secure.phabricator.com/D16819
2016-11-07 14:57:32 -08:00
epriestley
729492a8ff Allow transactions to specialize their mail headers for diff sections
Summary: Ref T7643. When we send mail about a change to a package description, allow it to say "CHANGES TO PACKAGE DESCRIPTION" instead of "EDIT DETAILS". Smooth!

Test Plan: {F1909417}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7643

Differential Revision: https://secure.phabricator.com/D16818
2016-11-07 12:16:39 -08:00
epriestley
87c4efdb63 Probably fix some display issues with all-day events?
Summary:
Ref T11801. These are pretty fiddly because users expect to see the end time for timed events ("10 AM - 11 AM" is ONE hour long) but not for all-day events ("Nov 2 - Nov 3" is TWO days long!)

We also want to store the thing the user actually entered so we don't lose data if they un-all-day the event later.

This may take a little more fiddling since it feels a little shaky, but I couldn't break this version immediately.

Test Plan: Imported a French holiday, got proper display in the UI.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11801

Differential Revision: https://secure.phabricator.com/D16815
2016-11-07 10:55:30 -08:00
epriestley
7ddd570fa5 Provide a standalone bin/calendar reload ... workflow for testing/debugging
Summary:
Ref T11801. This makes testing/debugging a little easier.

Also fix some inconsistencies with `importAuthorPHID` handling -- it should be the import's author PHID in all cases, so we update imported events properly.

Test Plan: Imported a French holiday with `bin/calendar reload ...`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11801

Differential Revision: https://secure.phabricator.com/D16814
2016-11-07 10:55:18 -08:00
epriestley
2f93ce4c25 Don't show "Limited" or "Test" translations unless an install is in developer mode
Summary:
Ref T5267. Although translations with very few strings are already put into a "Limited Translations" group, this isn't necessarily clear and was empirically confusing to at least one user, who was surprised that selecting "Spanish" had no UI effect.

Instead, hide limited and test translations entirely unless the install is in developer mode.

Test Plan: In a non-developer-mode install, viewed translations menu. No longer saw translations with very few strings.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T5267

Differential Revision: https://secure.phabricator.com/D16807
2016-11-06 14:31:41 -08:00
epriestley
f4f3b90c87 On tasks, put Task Graph, Mocks and Mentions into a tabgroup
Summary:
Fixes T4788. This change:

  - converts the "Task Graph" into a "Related Objects" tabgroup.
  - makes "Task Graph" the first tab in the group.
  - moves "Mocks" to become a tab.
  - adds a new "Mentions" tab, which shows inbound and outbound mentions.

Primary goal of "mocks" is to give us room for a pinboard/thumbnail view after the next Pholio iteration. Might make sense to make it the default tab (if present) at that point, too, since mocks are probably more important than related tasks when they're present.

Primary goal of "mentions" is to provide a bit of general support for various freeform relationships between tasks: if you want to treat tasks as "siblings" or "related" or "following" or whatever, you can at least find them all in one place. I don't plan to formalize any of these weird one-off relationships in the upstream, although it's vaguely possible that some far-future update might just let you define arbitrary custom relationships and then you can do whatever you want.

Test Plan: {F1906974}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4788

Differential Revision: https://secure.phabricator.com/D16806
2016-11-06 09:05:14 -08:00
epriestley
17bd483207 Queue large ICS files for background import
Summary: Ref T11801. When a file is larger than 512KB, queue it for background import instead of trying to do it in the foreground, sinc we risk hitting `max_execution_time`.

Test Plan: {F1906943}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11801

Differential Revision: https://secure.phabricator.com/D16805
2016-11-06 07:46:21 -08:00