1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-24 14:30:56 +01:00
Commit graph

637 commits

Author SHA1 Message Date
epriestley
2c72c2b924 Add basic support for OpenGraph header tags for public installs
Summary: Ref T13018. This is easy to get working roughly, at least, and seems reasonable.

Test Plan: Viewed page source, saw tags. Custom header logo still worked. Pretty hard to debug against a local install since Disqus / debugger tools can't hit it, but I'll see what it looks like in production and tweak it if I got anything horribly wrong.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13018

Differential Revision: https://secure.phabricator.com/D18780
2017-11-22 11:16:56 -08:00
epriestley
821c7ac833 For backup persitsence, mark the "common ngrams" table as a data table, not an index table
Summary:
Ref T13000. Garbage collecting common ngrams is slow because MySQL isn't all that great at deleting rows quickly. See PHI96, where it looks like it's going to take a week to GC ngrams for a ~million objects at a relatively conservative 0.15 threshold.

In the event of a restore, we can reduce the impact by persisting this table so the ngrams just don't get built when the reindex happens.

Test Plan: Viewed schema in Config, saw common ngrams tables marked as "Data" instead of "Index".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18696
2017-10-11 11:07:43 -07:00
Dmitri Iouchtchenko
9bd6a37055 Fix spelling
Summary: Noticed a couple of typos in the docs, and then things got out of hand.

Test Plan:
  - Stared at the words until my eyes watered and the letters began to swim on the screen.
  - Consulted a dictionary.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18693
2017-10-09 10:48:04 -07:00
epriestley
c767c971ca Add "persistence" types (data, cache, or index) to tables, and tweak what "storage dump" dumps
Summary:
Ref T13000. This marks each table as either "data" (normal data), "cache" (automatically rebuilt, no need to ever dump) or "index" (can be manually rebuilt).

By default, `bin/storage dump` dumps data and index tables, but not cache tables.

With `--no-indexes`, it dumps only data tables. Indexes can be rebuilt after a restore with `bin/search index --all ...`.

Test Plan:
  - Ran `--no-indexes` and normal dumps with `--trace`, verified that cache and index (former case) or cache only (latter case) tables were dumped with `--no-data`.
  - Verified dump has the same number of `CREATE TABLE` statements as before the changes.
  - Reviewed persistence tags in the web UI (note Ferret engine tables are "Index"):

{F5210886}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18682
2017-10-04 12:09:33 -07:00
epriestley
1de130c9f5 Allow the Ferret engine to remove "common" ngrams from the index
Summary:
Ref T13000. This adds support for tracking "common" ngrams, which occur in too many documents to be useful as part of the ngram index.

If an ngram is listed in the "common" table, it won't be written when indexing documents, or queried for when searching for them.

In this change, nothing actually writes to the "common" table. I'll start writing to the table in a followup change.

Specifically, I plan to do this:

  - A new GC process updates the "common" table periodically, by writing ngrams which appear in more than X% of documents to it, for some value of X, if there are at least a minimum number of documents (maybe like 4,000).
  - A new GC process deletes ngrams that have been added to the common table from the existing indexes.

Hopefully, this will pare down the ngrams index to something reasonable over time without requiring any manual tuning.

Test Plan:
  - Ran some queries and indexes.
  - Manually inserted ngrams `xxx` and `yyy` into the ngrams table, searched and indexed, saw them ignored as viable ngrams for search/index.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18672
2017-10-03 13:27:42 -07:00
epriestley
36df39761e Create revisions into "Draft", publish them when builds finish
Summary:
Ref T2543. This doesn't stand alone since mail still goes out normally, but gets this piece working: new revisions start as "Draft", then after updates if there are no builds they go into "Needs Review".

This should work in general because builds update revisions when they complete, to publish a "Harbormaster finished build yada yada" transaction. So either we'll un-draft immediately, or un-draft after the last build finishes.

I'll hold this until the mail and some other stuff (like UI hints) are in slightly better shape since I think it's probably too rough on its own.

Test Plan: Created revisions locally, saw them un-draft after builds.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18628
2017-09-21 07:21:21 -07:00
epriestley
2218caee0f Reduce the amount of boilerplate that implementing FerretInterface requires
Summary:
See brief discussion in D18554. All the index tables are the same for every application (and, at this point, seem unlikely to change) and we never actually pass these objects around (they're only used internally).

In some other cases (like Transactions) not every application has the same tables (for example, Differential has extra field for inline comments), and/or we pass the objects around (lots of stuff uses `$xactions` directly).

However, in this case, and in Edges, we don't interact with any representation of the database state directly in much of the code, and it doesn't change from application to application.

Just automatically define document, field, and ngram tables for anything which implements `FerretInterface`. This makes the query and index logic a tiny bit messier but lets us delete a ton of boilerplate classes.

Test Plan: Indexed objects, searched for objects. Same results as before with much less code. Ran `bin/storage upgrade`, got a clean bill of health.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18559
2017-09-07 13:23:31 -07:00
Chad Little
af7c92f2c6 Config re-design
Summary:
This is a full UI pass at a cleaner "Config" application. The main idea is to simplify the UI, center it, and have a different feel than other UI, a sort of "manage" UI theme for objects with loads of settings. Also adds a new minimalistic "WHITE_CONFIG" box type which may get re-used in Diffusion settings. This is a 90% pass, I'll have a few follow up diffs. Specifically:

 - Build breadcrumbs as a flexible UI to go into headers.
 - One click ObjectItemView option, for hover states.
 - Sidenav doesn't always select (AphrontFilter issue)
 - Mobile touchups, though it's pretty reasonable.

Test Plan:
Click through every page here, edit options, see new navigation UI. Test a few various setup issue layouts including fatals.

{F5163228}

{F5163229}

{F5163230}

{F5163231}

{F5163232}

{F5163233}

{F5163234}

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D18519
2017-09-05 15:24:15 -07:00
Chad Little
33b4de9acf Update Setup Issue UI
Summary: Slightly cleaner layout

Test Plan: review setup issues resolved and unresolved in local config. fake a fatal.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18521
2017-09-05 10:40:48 -07:00
epriestley
68008dce60 Fix a possible database ref fatal during MySQL setup checks if a host is unreachable
Summary:
Ref T12966. See that task for a description and reproduction steps.

If you put Phabricator in a master/replica configuration and then restart it, we may fatal here if the master is unreachable. Instead, we should survive setup checks.

Test Plan: Put Phabricator in a master/replica configuration, explicitly disabled the master by misconfiguring the port, restarted Phabricator. Before: fatal; after: login screen in read-only mode.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12966

Differential Revision: https://secure.phabricator.com/D18442
2017-08-17 11:43:13 -07:00
epriestley
c9986fd5de Don't fatal in ElasticSearch setup check if no "master" database is configured
Summary:
Ref T12965. See that task for discussion, and PHI36 for context.

This sweeps the fatal under the rug by skipping it, letting things move forward for now.

Test Plan: Followed instructions in T12965, got a read-only recovery after restart instead of a fatal.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12965

Differential Revision: https://secure.phabricator.com/D18440
2017-08-17 10:39:00 -07:00
epriestley
f48f2dae9f Move Phabricator to use PhutilBinaryAnalyzer and show binary versions
Summary:
Fixes T12942.

  - Adds binary version and path information to {nav Config > Version Information}.
  - Replaces old code all over the place with new consolidated code.

Test Plan:
{F5073531}

Also faked some cases of missing binaries, bad versions, etc.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12942

Differential Revision: https://secure.phabricator.com/D18306
2017-08-01 07:14:48 -07:00
Aviv Eyal
27a243cd88 Link to docs from metamta.mail-adapter config
Test Plan: Look at new config page, click link.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D18259
2017-07-21 00:51:30 +00:00
epriestley
b46e2bb4cc Convert cluster/projects config options to newer modular structure
Summary: Ref T12845. Converts the cluster and project config options to the new stuff; this is mostly just shifting boilerplate around.

Test Plan: Edited, deleted, and mangled these options from the web UI and CLI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18166
2017-06-27 12:35:54 -07:00
epriestley
6984d239b0 Convert Maniphest custom config to new config types
Summary:
Fixes T12870. Ref T12845.

Technically, this addresses the core issue in T12845 too, but I'm going to convert the rest of the `custom:...` types before closing that.

In particular, for T12870:

  - Validates that keywords are unique across priorities.
  - Fixes missing newline in documentation.
  - Updates documentation to note that keywords are now mandatory and must be unique across priorities.

Test Plan: Edited, deleted and mangled all the Maniphest custom options (priorities, statuses, points, subtypes).

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12870, T12845

Differential Revision: https://secure.phabricator.com/D18165
2017-06-27 12:35:29 -07:00
epriestley
a14b82d4f4 Move "wild" config types to new code
Summary:
Ref T12845. This is the last of the hard-coded types.

These are mostly used for values which users don't directly edit, so it's largely OK that they aren't carefully validated. In some cases, it would be good to introduce a separate validator eventually.

Test Plan: Edited, deleted and mangled these values via the web UI and CLI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18164
2017-06-27 12:34:56 -07:00
epriestley
ec2af08625 Move 'set' config option type to new structure
Summary: Ref T12845. This move 'set' options (a set of values).

Test Plan: Set, deleted and mangled 'set' options from CLI and web UI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18160
2017-06-27 12:34:37 -07:00
epriestley
0afdabff00 Convert 'class' config options to new validation
Summary: Ref T12845. These options prompt the user to select from among concrete subclasses of some base class.

Test Plan: Set, deleted and mangled these values from the web UI and CLI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18159
2017-06-27 12:14:19 -07:00
epriestley
72119e786c Convert "bool" config values to new modular system
Summary: Ref T12845. Moves the "bool" values over.

Test Plan: Set, deleted, and mangled bool values from CLI and web UI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18158
2017-06-27 12:13:55 -07:00
epriestley
467be5e53f Convert the "list<string>" and "list<regex>" Config option types
Summary: Ref T12845. This updates the "list<string>" and "list<regex>" options.

Test Plan: Set, deleted, and mangled options of these types from the web UI and CLI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18157
2017-06-27 12:13:33 -07:00
epriestley
9d30d49cfc Convert "enum" and "string" config options to new modular option types
Summary: Ref T12845. This moves the "enum" and "string" types to the new code.

Test Plan: Set, deleted, and tried to set invalid values for various enum and string config values (header color, mail prefixes, etc) from the CLI and web.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18156
2017-06-27 12:13:15 -07:00
epriestley
03b6bdde19 Begin modularizing config options in a more modern way
Summary:
Ref T12845. Config options are "modular", but the modularity is very old, half-implemented, and doesn't use modern patterns.

Half the types are hard-coded, while half the types are semi-modular but in a weird hacky way where you prefix the type with `custom:...`.

The actual API is also weird and requires types to return a lot of `array($stuff, $thing, $other_thing, $more_stuff)` sorts of tuples.

Instead:

  - Add a new replacement layer which uses modern modularity patterns and overrides the older stuff if available, so we can migrate things one at a time.
  - New layer uses a more modern API -- no `return array($thing, $other_thing, ...)`, and more modern building blocks (like AphrontHTTPParameterType).
  - New layer allows custom types to be deleted, which will ultimately let us deal with T12845.

Then, convert the `'int'` type to use the new layer.

Test Plan:
  - Set, edited, tried-to-change-in-an-invalid-way, and deleted an `'int'` option from the web UI.
  - Same from the CLI.
  - Edited `config.json` to have an invalid value, verified that the error was detected and config was repaired.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18155
2017-06-27 12:12:37 -07:00
Chad Little
d3c464a610 Separate button CSS classes
Summary: Try to dis-ambiguate various button types and colors. Moves `simple` to `phui-button-simple` and moves colors to `button-color`.

Test Plan: Grep for buttons still inline, UIExamples, PHUIX, Herald, and Email Preferences.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18077
2017-06-05 20:14:34 +00:00
epriestley
c4e45c6c8c Detect and prevent invalid configuation of "ui.footer-items"
Summary: Fixes T12775. Currently, we do not validate this option and it's possible to configure it in an invalid way.

Test Plan: Tried to misconfigure things, was helpfully pointed toward errors.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12775

Differential Revision: https://secure.phabricator.com/D18041
2017-05-30 10:15:24 -07:00
epriestley
d0e6bf831d Add "%I" (instance name) to application log formats
Summary:
Ref T12611. Currently, the HTTP/SSH logs don't have an option to include the instance name.

Add such an option.

Leave it out of the default logs because most installs don't use this.

Test Plan: See next changes.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12611

Differential Revision: https://secure.phabricator.com/D17776
2017-04-23 11:07:19 -07:00
epriestley
95dd9dbf43 Make Applications extend LiskDAO
Summary:
Ref T11476. This is a bit hacky, but makes `Application` extend `LiskDAO` so we can apply transactions to it with an `Editor` class.

Also fixes schema stuff so builds should produce a clean bill of health again.

This might only get you slightly further, yell if you run into more trouble.

Test Plan:
  - Ran `bin/storage upgrade -f` and got no warnings.
  - Browsed around, nothing exploded?

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11476

Differential Revision: https://secure.phabricator.com/D17738
2017-04-19 16:06:14 -07:00
epriestley
88157a9442 Hold recent messages in Aphlict so they can be replayed after clients reconnect
Summary:
Ref T12563. Before broadcasting messages from the server, store them in a history buffer.

A future change will let clients retrieve them.

Test Plan:
  - Used the web frontend to look at the buffer, reloaded over time, sent messages. Saw buffer size go up as I sent messages and fall after 60 seconds.
  - Set size to 4 messages, sent a bunch of messages, saw the buffer size max out at 4 messages.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12563

Differential Revision: https://secure.phabricator.com/D17707
2017-04-17 15:53:58 -07:00
Austin McKinley
980d6cb70b Add validation for config settings of type regex
Summary: Also fixes insufficiently-escaped regex examples

Test Plan: Made several changes to http://local.phacility.com/config/edit/syntax.filemap/ and observed validation failures on malformed regexes, and success on well-formed regexes.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12532

Differential Revision: https://secure.phabricator.com/D17684
2017-04-13 13:57:03 -07:00
Austin McKinley
bfffd807d6 Change syntax highlighting for custom phabricator dot configs
Test Plan:
Created new paste with title '.arcconfig' without choosing a language; observed that the paste gets highlighted as JSON.

JSON mode:
{F4901762}

Javascript mode:
{F4901763}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11667

Differential Revision: https://secure.phabricator.com/D17682
2017-04-13 13:55:33 -07:00
epriestley
7707685733 Fix two strings with missing pht()
Summary: Fixes T12517.

Test Plan: Viewed Config application; viewed repository list.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12517

Differential Revision: https://secure.phabricator.com/D17639
2017-04-07 10:07:01 -07:00
epriestley
3d816e94df Rename "PhabricatorHash::digest()" to "weakDigest()"
Summary: Ref T12509. This encourages code to move away from HMAC+SHA1 by making the method name more obviously undesirable.

Test Plan: `grep`, browsed around.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17632
2017-04-06 15:43:33 -07:00
epriestley
d450a08890 Support HMAC+SHA256 with automatic key generation and management
Summary:
Ref T12509. This adds support for HMAC+SHA256 (instead of HMAC+SHA1). Although HMAC+SHA1 is not currently broken in any sense, SHA1 has a well-known collision and it's good to look at moving away from HMAC+SHA1.

The new mechanism also automatically generates and stores HMAC keys.

Currently, HMAC keys largely use a per-install constant defined in `security.hmac-key`. In theory this can be changed, but in practice essentially no install changes it.

We generally (in fact, always, I think?) don't use HMAC digests in a way where it matters that this key is well-known, but it's slightly better if this key is unique per class of use cases. Principally, if use cases have unique HMAC keys they are generally less vulnerable to precomputation attacks where an attacker might generate a large number of HMAC hashes of well-known values and use them in a nefarious way. The actual threat here is probably close to nonexistent, but we can harden against it without much extra effort.

Beyond that, this isn't something users should really have to think about or bother configuring.

Test Plan:
  - Added unit tests.
  - Used `bin/files integrity` to verify, strip, and recompute hashes.
  - Tampered with a generated HMAC key, verified it invalidated hashes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17630
2017-04-06 15:42:59 -07:00
epriestley
7e6f37fffb Rename "ElasticSearch" filenames to "Elasticsearch" (2/2)
Sometimes git does some odd magic on case-insensitive filesystems, try to
trick it.

Auditors: chad
2017-04-02 14:59:36 -07:00
epriestley
a9e2732a5c Spell "Elasticsearch" correctly, not "ElasticSearch"
Summary: Ref T12450. These are like 95% my fault, but Elastic appears to spell the name "Elasticsearch" consistently in their branding.

Test Plan: `grep ElasticSearch`

Reviewers: chad, 20after4

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17601
2017-04-02 14:58:59 -07:00
Mukunda Modell
699228c73b Address some New Search Configuration Errata
Summary:
  [ ] Write an "Upgrading: ..." guidance task with narrow instructions for installs that are upgrading.
  [ ] Do we need to add an indexing activity (T11932) for installs with ElasticSearch?
  [ ] We should more clearly detail exactly which versions of ElasticSearch are supported (for example, is ElasticSearch <2 no longer supported)? From T9893 it seems like we may //only// have supported ElasticSearch <2 before, so are the two regions of support totally nonoverlapping and all ElasticSearch users will need to upgrade?
  [ ] Documentation should provide stronger guidance toward MySQL and away from Elastic for the vast majority of installs, because we've historically seen users choosing Elastic when they aren't actually trying to solve any specific problem.
  [ ] When users search for fulltext results in Maniphest and hit too many documents, the current behavior is approximately silent failure (see T12443). D17384 has also lowered the ceiling for ElasticSearch, although previous changes lowered it for MySQL search. We should not fail silently, and ideally should build toward T12003.
  [ ] D17384 added a new "keywords" field, but MySQL does not search it (I think?). The behavior should be as consistent across MySQL and Elastic as we can make it. Likely cleaner is giving "Project" objects a body, with "slugs" and "description" separated by newlines?
  [ ] `PhabricatorSearchEngineTestCase` is now pointless and only detects local misconfigurations.
  [ ] It would be nice to build a practical test suite instead, where we put specific documents into the index and then search for them. The upstream test could run against MySQL, and some `bin/search test` could run against a configured engine like ElasticSearch. This would make it easier to make sure that behavior was as uniform as possible across engine implementations.
  [ ] Does every assigned task now match "user" in ElasticSearch?
  [x] `PhabricatorElasticFulltextStorageEngine` has a `json_encode()` which should be `phutil_json_encode()`.
  [ ] `PhabricatorSearchService` throws an untranslated exception.
  [ ] When a search cluster is down, we probably don't degrade with much grace (unhandled exception)?
  [ ] I haven't run bin/search init, but bin/search index doesn't warn me that I may want to. This might be worth adding. The UI does warn me.
  [ ] bin/search init warns me that the index is "incorrect". It might be more clear to distinguish between "missing" and "incorrect", since it's more comforting to users to see "everything is as we expect, doing normal first-time setup now" than "something is wrong, fixing it".
  [ ] CLI message "Initializing search service "ElasticSearch"" does not end with a period, which is inconsistent with other UI messages.
  [ ] It might be nice to let bin/search commands like init and index select a specific service (or even service + host) to act on, as bin/storage --ref ... now does. You can generally get the result you want by fiddling with config.
  [ ] When a service isn't writable, bin/search init reports "Search cluster has no hosts for role "write".". This is accurate but does not provide guidance: it might be more useful to the user to explain "This service is not writable, so we're skipping index check for it.".
  [x] Even with write off for MySQL, bin/search index --type task --trace still updates MySQL, I think? I may be misreading the trace output. But this behavior doesn't make sense if it is the actual behavior, and it seems like reindexAbstractDocument() uses "all services", not "writable services", and the MySQL engine doesn't make sure it's writable before indexing.
  [x] Searching or user fails to find task Grant users tokens when a mention is created, suggesting that stemming is not working.
  [x] Searching for users finds that task, but fails to find a task containing "per user per month" in a comment, also suggesting that stemming is not working.
  [x] Searching for maniphest fails to find task maniphest.query elephant, suggesting that tokenization in ElasticSearch is not as good as the MySQL tokenization for these words (see D17330).
  [x] The "index incorrect" warning UI uses inconsistent title case.
  [x] The "index incorrect" warning UI could format the command to be run more cleanly (with addCommand(), I think).

refs T12450

Test Plan:
* Stared blankly at the code.
* Disabled 'write' role on mysql fulltext service.
* Edited a task, ran search indexer, verified that the mysql index wasn't being updated.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T12450

Differential Revision: https://secure.phabricator.com/D17564
2017-03-28 20:19:38 +00:00
Mukunda Modell
e41c25de50 Support multiple fulltext search clusters with 'cluster.search' config
Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.

When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.

Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.

These two roles make it possible to have any combination of:

* read-only
* write-only
* read-write
* disabled

This 'roles' mechanism is extensible to add new roles should that be needed in the future.

In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).

The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)

Hopefully this is useful in the upstream as well.

Remaining TODO:

* test cases
* documentation

Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.

Tested with elasticsearch versions 2.4 and 5.2 using the following config:

```lang=json
  "cluster.search": [
    {
      "type": "elasticsearch",
      "hosts": [
        {
          "host": "localhost",
          "roles": { "read": true, "write": true }
        }
      ],
      "port": 9200,
      "protocol": "http",
      "path": "/phabricator",
      "version": 5
    },
    {
      "type": "mysql",
      "roles": { "write": true }
     }
  ]

Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Tags: #elasticsearch, #clusters, #wikimedia

Differential Revision: https://secure.phabricator.com/D17384
2017-03-26 08:16:47 +00:00
epriestley
ba2ee3a66e Make "bin/config set --database ..." resurrect deleted values
Summary:
Fixes T12409. Config entries may be marked as "deleted", and `bin/config set --database` doesn't un-delete them, so the edit doesn't do anything.

The "most correct" fix here is to swap to transactions so we run the same code, but just fix this narrowly for now since it's one line of code.

Test Plan:
  - Set `maniphest.default-priority` to `123`.
  - Deleted `maniphest.default-priority` from the web UI by deleting all the text in the box.
  - Before patch: `bin/config set --database maniphest.default-priority 789` had no effect.
  - After patch: `bin/config set --database maniphest.default-priority 789` worked.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12409

Differential Revision: https://secure.phabricator.com/D17506
2017-03-16 12:26:33 -07:00
epriestley
d6d3ad6f80 Allow administrators to get a list of users who don't have MFA configured
Summary:
Fixes T12400. Adds a "Has MFA" filter to People so you can figure out who you need to harass before turning on "require MFA".

When you run this as a non-admin, you don't currently actually hit the exception: the query just doesn't work. I think this is probably okay, but if we add more of these it might be better to make the "this didn't work" more explicit since it could be confusing in some weird edge cases (like, an administrator sending a non-administrator a link which they expect will show the non-administrator some interesting query results, but they actually just get no constraint). The exception is more of a fail-safe in case we make application changes in the future and don't remember this weird special case.

Test Plan:
  - As an administrator and non-administrator, used People and Conduit to query MFA, no-MFA, and don't-care-about-MFA. These queries worked for an admin and didn't work for a non-admin.
  - Viewed the list as an administrator, saw MFA users annotated.
  - Viewed config help, clicked link as an admin, ended up in the right place.

{F4093033}

{F4093034}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12400

Differential Revision: https://secure.phabricator.com/D17500
2017-03-15 17:49:01 -07:00
epriestley
939fb69aa6 Be less strict when detecting dead daemons
Summary:
Fixes T12306. Currently, we warn about daemons not running even if they're in normal "alive" states, particularly "waiting to restart after a failure".

This check was made more strict in D12088, back when we tried to version check running daemons. Since we implemented auto-restart-after-config-change we don't do this anymore, so it should be fine to make this more lax again.

Test Plan:
  - Faked an exception for all tasks.
  - Before patch: reloading the daemon setup error sometimes raised a false positive ("waiting" daemon detected as dead).
  - After patch: daemon setup error no longer triggers.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12306

Differential Revision: https://secure.phabricator.com/D17397
2017-02-22 14:11:28 -08:00
Jakub Vrana
a778151f28 Fix errors found by PHPStan
Test Plan: Ran `phpstan analyze -a autoload.php phabricator/src`.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D17371
2017-02-17 10:10:15 +00:00
epriestley
56b1ff833b Fix some outdated help text about "Reply All" in "metamta.one-mail-per-recipient"
Summary: Ref T12240. When you "Reply All" to a Phabricator mail, we make an effort not to send the response to recipients who you hit with the original message. This isn't perfect and we can't always get it right, but the old description implies it's a bigger problem than it should be in practice.

Test Plan: Read text.

Reviewers: chad, eadler

Reviewed By: chad

Maniphest Tasks: T12240

Differential Revision: https://secure.phabricator.com/D17331
2017-02-10 07:17:10 -08:00
epriestley
3d44208e4f Clarify that "account.editable" no longer extends to profile pictures
Summary: Fixes T12216. I'd like to remove this option eventually, but just narrow its scope in the config description for now.

Test Plan: Read config description.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12216

Differential Revision: https://secure.phabricator.com/D17317
2017-02-06 08:37:26 -08:00
Austin Seipp
ab923e0a75 Implement new reCAPTCHA interface
Summary:
Fixes T12195. For the past few years, Recaptcha (now part of Google) has supported
a new, "no captcha" one-click user interface. This new UI is stable, doesn't
require any typing or reading words, and can even work without JavaScript (if
the administrator enables it on the Recaptcha side).

Furthermore, the new Recaptcha has a completely trivial API that can be dealt
with in a few lines of code. Thus, the external `recaptcha` php library is now
gone.

This API is a complete replacement for the old one, and does not require any
upgrade path for users or Phabricator administrators - public and secret keys
for the "new" Recaptcha UI are the exact same as the "classic" Recaptcha. Any
old Recaptcha keys for a domain will continue to work.

Note that Google is currently testing Yet Another new Captcha API, called
"Invisible reCAPTCHA", that will not require user interaction at all. In fact,
the user will not even be aware there //is even a captcha form//, as far as I
understand. However, this new API is 1) in beta, 2) requires new Recaptcha keys
(so it cannot be a drop-in replacement), and 3) requires more drastic API
changes, as form submission buttons must instead invoke JavaScript code, rather
than a token being passed along with the form submission. This would require far
more extensive changes to the controllers. Maybe when it's several years old, it
can be considered.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Test Plan:
Created a brand-new Phabricator installation, saw the new Captcha UI
on administrator sign up. Logged out, made 5 invalid login attempts, and saw the
new Captcha UI. Reworked the conditional to invert the condition, etc to test
and make sure the API responded properly.

Reviewers: epriestley, #blessed_reviewers, chad

Reviewed By: epriestley, #blessed_reviewers

Subscribers: avivey, Korvin

Maniphest Tasks: T12195

Differential Revision: https://secure.phabricator.com/D17304
2017-02-03 20:06:29 +00:00
epriestley
a2cd3d9a89 Change PHP 7 setup warning to complain about 7.0 only, not 7.1+
Summary: Ref T9640. On 7.0 we had signal handling issues so we can never support it, but async signals should resolve them on 7.1 or newer.

Test Plan: On PHP 7.1, got through the setup warning.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9640

Differential Revision: https://secure.phabricator.com/D17197
2017-01-12 15:59:28 -08:00
epriestley
4a34f26a44 Don't warn about "always_populate_raw_post_data" on PHP7
Summary: Ref T9640. This option was removed in PHP7, so there's no reason to warn about it.

Test Plan: No longer saw a setup warning on PHP7.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9640

Differential Revision: https://secure.phabricator.com/D17196
2017-01-12 15:59:14 -08:00
epriestley
065d865bce In the "Version Information" panel, try to include branchpoints
Summary:
Fixes T12040. In T12039, a user running local patches followed the report instructions as far as grabbing version information, but didn't update or revert their local changes or try against a clean install before reporting.

This obviously isn't ideal for us, but it's understandable (grabbing version information is much easier than upgrading/reverting), and we can do better about making this information useful: when compiling version information, try to figure out the branchpoint from a known upstream `master` branch by listing remotes, then running `git merge-base` against them.

Additionally, explicitly document that we want upstream hashes. We have to have a fallback case in this document anyway (for when you can't get to Config) so hopefully this makes it more likely that we get useful information in initial reports.

Test Plan: {F2229574}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12040

Differential Revision: https://secure.phabricator.com/D17103
2016-12-23 11:42:20 -08:00
epriestley
39b618039f Remove a very old piece of config documentation
Summary: Ref T571. This was accidentally left behind in D12266.

Test Plan: Used {key command F} to search for "bulk".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T571

Differential Revision: https://secure.phabricator.com/D17034
2016-12-12 23:22:21 +00:00
epriestley
f3d9a0b930 Fix two cache issues (global settings; initial setup)
Summary:
  - Fixes T11995. This got moved but I missed renaming this callsite.
  - Fixes T11993. If you have valid credentials, but haven't run `storage upgrade` yet, we can hit this exception during setup. Just ignore it instead.

Test Plan:
  - Saved global settings, no more fatal.
  - Changed `storage-namespace` to junk, loaded web UI with valid database credentials.

{F2106358}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11993, T11995

Differential Revision: https://secure.phabricator.com/D17024
2016-12-11 08:28:10 -08:00
epriestley
9017bb9925 Add a setup check for installation on a burstable instance type
Summary: Fixes T11544. Attempt to detect if we're on a tiny, burstable-CPU AWS instance and complain.

Test Plan:
  - Completely faked this locally.
  - Hit the URI on an EC2 instance to check that it's correct (got back "m3.large", since that was the instance class).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11544

Differential Revision: https://secure.phabricator.com/D17014
2016-12-09 08:32:16 -08:00
epriestley
ffdc082852 Add a wide range of HTTP-request-based setup checks
Summary:
Ref T11553. With some regularity, users make various configuration mistakes which we can detect by making a request to ourselves.

I use a magical header to make this request because we want to test everything else (parameters, path).

  - Fixes T4854, probably. Tries to detect mod_pagespeed by looking for a header. This is a documentation-based "fix", I didn't actually install mod_pagespeed or formally test this.
  - Fixes T6866. We now test for parameters (e.g., user somehow lost "QSA").
  - Ref T6709. We now test that stuff is decoded exactly once (e.g., user somehow lost "B").
  - Fixes T4921. We now test that Authorization survives the request.
  - Fixes T2226. Adds a setup check to determine whether gzip is enabled on the web server, and attempts to enable it at the PHP level.
  - Fixes `<space space newline newline space><?php` in `preamble.php`.

Test Plan: Tested all of these setup warnings, although mostly by faking them.

Reviewers: joshuaspence, chad

Reviewed By: chad

Subscribers: Korvin

Maniphest Tasks: T4854, T4921, T6709, T6866, T11553, T2226

Differential Revision: https://secure.phabricator.com/D12622
2016-12-08 15:46:23 -08:00