1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-11 17:32:41 +01:00
Commit graph

366 commits

Author SHA1 Message Date
Chad Little
5d8bb61dde Add FontIcon bridge to AppIcons
Summary: Select a similar or better FontAwesome icon to represent each application

Test Plan: Visual inspection

Reviewers: epriestley, btrahan

Subscribers: hach-que, Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11489
2015-01-24 23:43:01 -08:00
Joshua Spence
c2ac63e9ad Increase visibility of PhabricatorController::buildApplicationMenu methods
Summary: Ref T6822. This method needs to be `public` because it is called from `PhabricatorApplicationSearchController::buildApplicationMenu()`.

Test Plan: I wouldn't expect //increasing// method visibility to break anything.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11416
2015-01-16 07:41:26 +11:00
Bob Trahan
923096efc8 Config - add phd.variant-config to suppress "Daemon & Web config" error message on a per key basis
Summary: Fixes T6959.

Test Plan: When I was ready to test the feature, the "Daemon & Web config" error already showed up, from having added phd.variant-config. I went meta and changed the value of phd.variant-config to have phd.variant-config. The config error disappeared. I then changed the conpherence setting about conpherence email prefix and the error showed up again. Removing the conpherence config setting made the error disappear once more.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6959

Differential Revision: https://secure.phabricator.com/D11399
2015-01-14 13:46:31 -08:00
Joshua Spence
463d094f96 Fix method visibility for PhabricatorPolicyAwareQuery subclasses
Summary: Ref T6822.

Test Plan:
`grep` for the following:

  - `->willFilterPage(`
  - `->loadPage(`
  - `->didFilterPage(`
  - `->getReversePaging(`
  - `->didFilterPage(`
  - `->willExecute(`
  - `->nextPage(`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: hach-que, Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11367
2015-01-14 07:01:16 +11:00
Joshua Spence
d6b882a804 Fix visiblity of LiskDAO::getConfiguration()
Summary: Ref T6822.

Test Plan: `grep`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: hach-que, Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11370
2015-01-14 06:54:13 +11:00
Bob Trahan
1a997fb0df Config - fix management scripts with --database parameter
Summary: Fixes T6923. Turns out we can't use the editor since we don't have a user with a phid (just some omnipotent guy).

Test Plan: ./bin/config set --database syntax.filemap '{}'; ./bin/config delete --database syntax.filemap

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6923

Differential Revision: https://secure.phabricator.com/D11301
2015-01-09 13:58:11 -08:00
epriestley
9e0f70e17d Rewrite Aphlict to use Websockets
Summary:
Fixes T6559. No more flash, use Websockets. This is less aggressive than the earlier version, and retains more server logic.

  - Support "wss".
  - Make the client work.
  - Remove "notification.user" entirely.
  - Seems ok?

Test Plan:
In Safari, Firefox and Chrome, saw the browsers connect. Made a bunch of comments/updates and saw notifications.

Notable holes in the test plan:

  - Haven't tested "wss" yet. I'll do this on secure.
  - Notifications are //too fast// now, locally. I get them after I hit submit but before the page reloads.
  - There are probably some other rough edges, this is a fairly big patch.

Reviewers: joshuaspence, btrahan

Reviewed By: joshuaspence, btrahan

Subscribers: fabe, btrahan, epriestley

Maniphest Tasks: T6713, T6559

Differential Revision: https://secure.phabricator.com/D11143
2015-01-08 10:03:00 -08:00
Joshua Spence
e7f8e79742 Fix method visibility for PhabricatorController subclasses
Summary: Ref T6822.

Test Plan: Visual inspection. These methods are only called from within `PhabricatorController` subclasses.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11241
2015-01-07 07:34:59 +11:00
Joshua Spence
2c855a8993 Fix method visibility for PhabricatorConfigStorageSchema methods
Summary: Ref T6822.

Test Plan: Visual inspection. These methods are only called from within the `PhabricatorConfigStorageSchema` class.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11247
2015-01-07 07:33:52 +11:00
epriestley
fa7bb8ff7a Add cluster.addresses and require membership before accepting cluster authentication tokens
Summary:
Ref T2783. Ref T6706.

  - Add `cluster.addresses`. This is a whitelist of CIDR blocks which define cluster hosts.
  - When we recieve a request that has a cluster-based authentication token, require the cluster to be configured and require the remote address to be a cluster member before we accept it.
    - This provides a general layer of security for these mechanisms.
    - In particular, it means they do not work by default on unconfigured hosts.
  - When cluster addresses are configured, and we receive a request //to// an address not on the list, reject it.
    - This provides a general layer of security for getting the Ops side of cluster configuration correct.
    - If cluster nodes have public IPs and are listening on them, we'll reject requests.
    - Basically, this means that any requests which bypass the LB get rejected.

Test Plan:
  - With addresses not configured, tried to make requests; rejected for using a cluster auth mechanism.
  - With addresses configred wrong, tried to make requests; rejected for sending from (or to) an address outside of the cluster.
  - With addresses configured correctly, made valid requests.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6706, T2783

Differential Revision: https://secure.phabricator.com/D11159
2015-01-02 15:13:41 -08:00
epriestley
08126d3904 Improve top-level exception handling
Summary:
Fixes T6692. Addresses two main issues:

  - The write guard would sometimes not get disposed of on exception pathways, generating an unnecessary secondary error which was just a symptom of the original root error.
    - This was generally confusing and reduced the quality of reports we received because users would report the symptomatic error sometimes instead of the real error.
    - Instead, reflow the handling so that we always dispose of the write guard if we create one.
  - If we missed the Controller-level error page generation (normally, a nice page with full CSS, etc), we'd jump straight to Startup-level error page generation (very basic plain text).
    - A large class of errors occur too early or too late to be handled by Controller-level pages, but many of these errors are not fundamental, and the plain text page is excessively severe.
    - Provide a mid-level simple HTML error page for errors which can't get full CSS, but also aren't so fundamental that we have no recourse but plain text.

Test Plan:
Mid-level errors now produce an intentional-looking error page:

{F259885}

Verified that setup errors still render properly.

@chad, feel free to tweak the exception page -- I just did a rough pass on it. Like the setup error stuff, it doesn't have Celerity, so we can't use `{$colors}` and no other CSS will be loaded.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: epriestley, chad

Maniphest Tasks: T6692

Differential Revision: https://secure.phabricator.com/D11126
2015-01-02 10:49:27 -08:00
Fabian Stelzer
00495e3a0e remove unused FeedStory object in getTitleForFeed functions
Summary:
Removes an unused PhabricatorFeedStory Parameter from all getTitleForFeed() and getApplicationTransactionTitleForFeed() functions.
Ref D11088 Ref T6545

Test Plan: ran all unit tests and viewed some dashboard feeds

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6545

Differential Revision: https://secure.phabricator.com/D11146
2015-01-02 08:45:43 -08:00
Joshua Spence
a98ec20dba Rename PhabricatorSetupCheck subclasses for consistency
Summary: Ref T5655.

Test Plan: `arc lint` and `arc unit`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T5655

Differential Revision: https://secure.phabricator.com/D11130
2015-01-02 15:27:58 +11:00
Fabian Stelzer
86eb7c0ec4 Settings History
Summary:
Shows a timeline of all modified settings Fixes T6545
Will show all settings (no pagination, should be not so difficult to add if needed but most installs won't have hundreds of settings changes)
I'm not happy by how the PhabricatorConfigTransaction object is instructed to render the config keys but i don't see any other reasonable way.
We could always show the keys though.

Test Plan: Changed settings and called the history page

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6545

Differential Revision: https://secure.phabricator.com/D11088
2015-01-01 06:52:13 -08:00
epriestley
376729b44c Don't check "repository.default-local-path" for readability in a cluster environment
Summary:
Ref T2783. When repository services are defined, skip the check for local repository storage.

In particular, in the Phacility cluster, this path won't exist on web nodes, but nothing will ever try to access it.

Test Plan: Verified new branch gets hit with cluster services defined.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2783

Differential Revision: https://secure.phabricator.com/D11100
2014-12-31 11:50:20 -08:00
epriestley
2fe553761a Increase severity of "phabricator.base-uri" warning
Summary: Fixes T6828.

Test Plan: reading

Reviewers: chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T6828

Differential Revision: https://secure.phabricator.com/D11075
2014-12-30 08:01:44 -08:00
epriestley
c8d707caa0 Improve pygments.enabled discussion
Summary: Ref T6533. Provide better instructions around installing, enabling, and troubleshooting Pygments.

Test Plan: Read documentation, viewed config option, clicked links.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, fabe

Maniphest Tasks: T6533

Differential Revision: https://secure.phabricator.com/D11053
2014-12-29 16:15:48 -08:00
Fabian Stelzer
736857d1f6 Setup check to install/enable pygments
Summary: Fixes T6533 - actively prompt the user to install and enable pygments

Test Plan: uninstalled / disabled pygments and checked setup checks

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6533

Differential Revision: https://secure.phabricator.com/D11048
2014-12-29 09:51:40 -08:00
Fabian Stelzer
2fedb6f941 Start phd daemons as the correctly configured user and refuse otherwise
Summary:
Fixes T5196
If no phd.user is configured the behaviour is unchanged besides printing a warning when run as root (Usually i would add an exit(1) here but that would break existing installs who do that).
If phd.user is set and the current user is root it will run the daemon as: su USER -c "command" (I'm not sure if this works for every platform needed)
Otherwise it will refuse to start if configured and current user mismatch.

Test Plan: Stopped & Started phd daemon with various users and different phd.user settings including root

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: vinzent, epriestley

Maniphest Tasks: T5196

Differential Revision: https://secure.phabricator.com/D11036
2014-12-23 08:15:51 -08:00
Chad Horohoe
a366f85c11 Properly create Elasticsearch index
Summary:
When the index does not exist and auto_create_index isn't
enabled, running ./bin/index results in a failure. That's
T5990

Instead create an index properly. This also allows us to do
nice things like do a proper mapping and analysis like for
substring matching like outlined by @fabe in T6552.

Test Plan:
Deleted and created index multiple times to verify
proper index creation and usage.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, manybubbles, chasemp, fabe, epriestley

Differential Revision: https://secure.phabricator.com/D10955
2014-12-22 13:10:52 -08:00
Fabian Stelzer
853ce5e927 setup warning for mysql with default boolean syntax
Summary: Ref: T6747 add a setup warning if mysql is configured with the default boolean syntax

Test Plan: Run setup checks with default mysql config and then add the desired config option, restart and check again.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Maniphest Tasks: T6747

Differential Revision: https://secure.phabricator.com/D11030
2014-12-22 12:57:15 -08:00
epriestley
2c7be52fc2 Lock phabricator.show-prototypes
Summary:
Two goals:

  - If an attacker compromises an administrator account (without compromising the host itself), they can currently take advantage of vulnerabilities in prototype applications by enabling the applications, then exploiting the vulnerability. Locking this option requires CLI access to enable prototypes, so installs which do not have prototypes enabled have no exposure to security issues in prototype applications.
  - Making this very slightly harder to enable is probably a good thing, given the state of the world and support.

Test Plan: Verified that web UI shows the value is locked and instructs the user to update via the CLI.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D10993
2014-12-15 11:00:20 -08:00
Bob Trahan
6ab3f06b6e Transactions - adding willRenderTimeline to handle tricky cases
Summary: Fixes T6693.

Test Plan:
Made a bunch of comments on a diff with differential, being sure to leave inlines here and there. This reproduced the issue in T6693. With this patch this issue no longer reproduces!

Successfully "showed older changes" in Maniphest too.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6693

Differential Revision: https://secure.phabricator.com/D10931
2014-12-04 13:58:52 -08:00
Bob Trahan
c2522a89b9 Transactions - deploy buildTransactionTimeline against Config, Pholio, and Projects
Summary: Ref T4712. These are a bit time consuming to test so might as well send off a batch now and again.

Test Plan: foreach impacted controller, made sure the timeline rendered as it did before. for project column and config, noted the "should terminate" UI was also rendered unlike before.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T4712

Differential Revision: https://secure.phabricator.com/D10923
2014-12-03 12:28:53 -08:00
lkassianik
f7aa87311a Add email preference links to email footers
Summary: Ref T1217, Add link to email preferences to email template

Test Plan: Add comment to object like Maniphest task, check that email has a footer with a link to email preferences.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Maniphest Tasks: T1217

Differential Revision: https://secure.phabricator.com/D10883
2014-11-19 17:06:33 -08:00
Bob Trahan
4e5775f1da Config / setup - check for ONLY_FULL_GROUP_BY and warn the user
Summary: Fixes T6243.

Test Plan: simulated having this on by adding 'ONLY_FULL_GROUP_BY' to the $modes array. verified the error message looked nice {F233954}.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6243

Differential Revision: https://secure.phabricator.com/D10856
2014-11-14 18:51:39 -08:00
Bob Trahan
c9c4dc1d9b Config - add setup checks for minimum and "bad" versions of VCS software.
Summary: Fixes T3046. Git has no minimum or bad versions; svn has no minimum and the bad version of 1.7.1, and mercurial has a minimum of 1.9 and bad versions 2.2 and 2.1. These error messages are specific about the problem but do not include the details as to what specifically is bad about a given version. I don't think that's really necessary - the details don't help solve the problem and its probably booooooring at best to the average user. The details about a bad version are included in the code however.

Test Plan: hardcoded a VCS array to let me test all the VCS stuff. added some phlog() calls to make sure the VCS version parsing stuff was working correctly relative to the version(s) on my system. played around with setting minimum versions and bad versions and saw good results depending on what I set.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T3046

Differential Revision: https://secure.phabricator.com/D10852
2014-11-14 14:58:18 -08:00
Scott Kroll
84f741f408 Allow config get to work when db is not functional
Summary: Fixes T6554.

Test Plan:
Run `bin/config get mysql.host` when no db is functional. Should not get an exception, but should see:
```
lang=json
{
  "config" : [
    {
      "key"       : "mysql.host",
      "source"    : "local",
      "value"     : null,
      "status"    : "unset",
      "errorInfo" : null
    },
    {
      "key"       : "mysql.host",
      "source"    : "database",
      "value"     : null,
      "status"    : "error",
      "errorInfo" : "Database source is not configured properly"
    }
  ]
}
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Maniphest Tasks: T6554

Differential Revision: https://secure.phabricator.com/D10851
2014-11-14 09:12:20 -08:00
Bob Trahan
aeb3a12e13 Config - improve lock message for option phabricator.uninstalled-applications
Summary: Fixes T6175. This particular option is only editable via the Applications application so let the user know that.

Test Plan: saw error message and clicked through to applications app. tried another locked option and saw old message

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6175

Differential Revision: https://secure.phabricator.com/D10841
2014-11-12 11:31:54 -08:00
Bob Trahan
755c518631 Config migration script bug fix
Summary: i think way back in D10490 I didn't incorporate feedback correctly. make this code right as it fatals in this codepath as is. Fixes T6508.

Test Plan: @joshuaspence to the rescue  (I remain unable to test this effectively with my baby-clean installation.)

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley, joshuaspence

Maniphest Tasks: T6508

Differential Revision: https://secure.phabricator.com/D10833
2014-11-11 15:05:57 -08:00
epriestley
000760b645 Do a better job of handling spec errors during schema adjustment
Summary:
Ref T1191. Currently if a developer forgot to specify a column type, `storage adjust` aborts explosively mid-stream. Instead:

  - Make this a formal error with an unambiugous name/description instead of something you sort of infer by seeing "<unknown>".
  - Make this error prevent generation of adjustment warnings, so we don't try to `ALTER TABLE t CHANGE COLUMN c <unknown>`, which is nonsense.
  - When schemata errors exist, surface them prominiently in `storage adjust`.

Overall:

  - Once `storage upgrade` runs `storage adjust` automatically (soon), this will make it relatively difficult to miss these errors.
  - Letting these errors slip through no longer escalates into a more severe issue.

Test Plan:
Commented out the recent `mailKey` spec and ran `storage adjust`:

```
$ ./bin/storage adjust --force
Verifying database schemata...
Found no adjustments for schemata.

Target                                            Error
phabricator2_phriction.phriction_document.mailKey Column Has No Specification

 SCHEMATA ERRORS

The schemata have serious errors (detailed above) which the adjustment
workflow can not fix.

If you are not developing Phabricator itself, report this issue to the
upstream.

If you are developing Phabricator, these errors usually indicate that your
schema specifications do not agree with the schemata your code actually
builds.
```

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10771
2014-11-04 04:42:05 -08:00
epriestley
f5c426639c Document the adjustment workflow and warn users about adjusting old MySQL
Summary: Ref T1191. Explain the adjustment workflow, how to resolve common errors, etc.

Test Plan: Read it, clicked doc links.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10758
2014-11-01 08:25:05 -07:00
epriestley
4a00e47646 Fix wrong subclass in ConfigIgnore controller
Summary: Fixes T6436. We subclass the wrong controller and miss the admin-only check.

Test Plan: Ignored / unignored set up issues.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6436

Differential Revision: https://secure.phabricator.com/D10765
2014-10-30 15:33:13 -07:00
epriestley
917da08417 Fix various MySQL version issues with new charset stuff
Summary:
Ref T1191. Notable stuff:

  - Adds `--disable-utf8mb4` to `bin/storage` to make it easier to test what things will (approximately) do on old MySQL. This isn't 100% perfect but should catch all the major stuff. It basically makes us pretend the server is an old server.
  - Require utf8mb4 to dump a quickstart.
  - Fix some issues with quickstart generation, notably special casing the FULLTEXT handling.
  - Add an `--unsafe` flag to `bin/storage adjust` to let it truncate data to fix schemata.
  - Fix some old patches which don't work if the default table charset is utf8mb4.

Test Plan:
  - Dumped a quickstart.
  - Loaded the quickstart with utf8mb4.
  - Loaded the quickstart with `--disable-utf8mb4` (verified that we get binary columns, etc).
  - Adjusted schema with `--disable-utf8mb4` (got a long adjustment with binary columns, some truncation stuff with weird edge case test data).
  - Adjusted schema with `--disable-utf8mb4 --unsafe` (got truncations and clean adjust).
  - Adjusted schema back without `--disable-utf8mb4` (got a long adjustment with utf8mb4 columns, some invalid data on truncated utf8).
  - Adjusted schema without `--disable-utf8mb4`, but with `--unsafe` (got truncations on the invalid data).

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10757
2014-10-29 15:49:29 -07:00
Bob Trahan
5b7353689e Fix a typo in an admin tool
Summary: Fixes T6358.

Test Plan: ran bin/migrate

Reviewers: epriestley, chad

Reviewed By: chad

Subscribers: Korvin, epriestley

Maniphest Tasks: T6358

Differential Revision: https://secure.phabricator.com/D10735
2014-10-21 10:11:55 -07:00
Bob Trahan
c0848bca6d Allow bin/config to affect database configuration and migrate between local and database configuration
Summary: Fixes T4018. Basically hits the bullet points in that task description except the "ideally" one.

Test Plan:
ran bin/config migrate and saw sensible output.

```
~> ./bin/config migrate
Migrating file-based config to more modern config...
Skipping config of source type PhabricatorConfigDatabaseSource...
Skipping config of source type PhabricatorConfigLocalSource...
Skipping config of source type PhabricatorConfigDefaultSource...
Done. Migrated 0 keys.
```

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: hach-que, epriestley, Korvin

Maniphest Tasks: T4018

Differential Revision: https://secure.phabricator.com/D10490
2014-10-08 16:15:05 -07:00
Joshua Spence
3cf9a5820f Minor formatting changes
Summary: Apply some autofix linter rules.

Test Plan: `arc lint` and `arc unit`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D10585
2014-10-08 08:39:49 +11:00
epriestley
0ddb187508 Add a setup warning about innodb_buffer_pool_size
Summary: Fixes T6119. This is a little fuzzy, but generally bumping up `innodb_buffer_pool_size` to something bigger than the default (which is often anemic, at `8M`) is desriable, and it seems like it will fix the specific issue a user encountered in T6119.

Test Plan: {F211855}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6119

Differential Revision: https://secure.phabricator.com/D10630
2014-10-02 14:44:36 -07:00
epriestley
d67b7f0f47 Correct column mutations for old versions of MySQL
Summary:
Ref T1191. Although I fixed some of the mutations earlier (in D10598), I missed the column mutations under old versions of MySQL. In particular, this isn't valid:

  - `ALTER TABLE ... MODIFY columnName VARCHAR(64) COLLATE binary`

Issue the permitted version of this instead, which is:

  - `ALTER TABLE ... MODIFY columnName VARBINARY(64)`

Also fixed an issue where a clean schema had the wrong nullability for a column in the draft table. Force it to the expected nullability.

The other trick here is around the one column with a FULLTEXT index on it, which needs a little massaging.

Test Plan:
  - Forced my local install to return `false` for utf8mb4 support.
  - Did a clean adjust into `binary` columns.
  - Poked around, added emoji to things.
  - Reverted the fake check and did a clean adjust into `utf8mb4` columns.
  - Emoji survived.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: fabe, epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10627
2014-10-02 14:44:22 -07:00
epriestley
8fa8415c07 Automatically build all Lisk schemata
Summary:
Ref T1191. Now that the whole database is covered, we don't need to do as much work to build expected schemata. Doing them database-by-database was helpful in converting, but is just reudndant work now.

Instead of requiring every application to build its Lisk objects, just build all Lisk objects.

I removed `harbormaster.lisk_counter` because it is unused.

It would be nice to autogenerate edge schemata, too, but that's a little trickier.

Test Plan: Database setup issues are all green.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10620
2014-10-02 09:51:20 -07:00
epriestley
3c6781b177 Differentiate between "no pygmetnize" and "nonworking pygmentize" during setup
Summary: Fixes T6210. The current messaging may be confusing if `pygmentize` is available but broken.

Test Plan: Faked the binary names and hit the errors, which seemed helpful.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6210

Differential Revision: https://secure.phabricator.com/D10626
2014-10-02 09:48:04 -07:00
epriestley
300172e799 Support AUTO_INCREMENT in bin/storage adjust
Summary:
Ref T1191. When changing the column type of an AUTO_INCREMENT column, we currently may lose the autoincrement attribute.

Instead, support it. This is a bit messy because AUTO_INCREMENT columns interact with PRIMARY KEY columns (tables may only have one AUTO_INCREMENT column, and it must be a primary key). We need to migrate in more phases to avoid this issue.

Introduce new `auto` and `auto64` types to represent autoincrement IDs.

Test Plan:
  - Saw autoincrement show up correctly in web UI.
  - Fixed an autoincrement issue on the XHProf storage table with `bin/storage adjust` safely.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10607
2014-10-01 08:24:51 -07:00
epriestley
0d7489da79 Provide bin/storage quickstart to automate generation of quickstart.sql
Summary:
Ref T1191. Currently, the `quickstart.sql` gets generated in a pretty manual fashion. This is a pain, and will become more of a pain in the world of utf8mb4.

Provide a workflow which does upgrade + adjust + dump + destroy, then massages the output to produce a workable `quickstart.sql`.

Test Plan: Inspected output; I'll test this more throughly before actually generating a new quickstart, but that's some ways away.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10603
2014-10-01 08:22:37 -07:00
epriestley
1dfa94e571 Use binary collations for most text
Summary:
Ref T1191. For most text columns, we either don't care if "a" and "A" are the same, or we expect them to be different (for example: keys, domains, secrets, etc). Default text columns to the `_bin` collation so they are compared by strict character value. This is safer in cases where we aren't sure.

For some text columns, we allow the user to sort by the column in the UI (like Maniphest task titles) or we do care that "A" and "a" are the same (for example: project names). Introduce a new class of virtual data types, the "sort..." types, to cover these columns. These are like the "text..." types but use sorting collations which treat "A" and "a" the same.

Test Plan:
  - Made an effort to identify all columns where the UI relies on database collation.
  - Ran `bin/storage adjust` and cleared all warnings.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: beng, epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10602
2014-10-01 08:18:53 -07:00
epriestley
4fcc634a99 Fix almost all remaining schemata issues
Summary:
Ref T1191. This fixes nearly every remaining blocker for utf8mb4 -- primarily, overlong keys.

Remaining issue is https://secure.phabricator.com/T1191#77467

Test Plan: I'll annotate inline.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T6099, T6129, T6133, T6134, T6150, T6148, T6147, T6146, T6105, T1191

Differential Revision: https://secure.phabricator.com/D10601
2014-10-01 08:18:36 -07:00
epriestley
22ee8432d2 Allow bin/storage adjust to correct column types and collations
Summary:
Ref T1191. Allow `bin/storage adjust` to modify columns.

  - Although `CREATE TABLE ... colname VARCHAR(64) CHARACTER SET BINARY` works fine, it's actually a trick. Adjust the binary columns for this.

Test Plan: See comment.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6130, T6128, T6135, T6137, T6138, T6149, T6151, T1191

Differential Revision: https://secure.phabricator.com/D10598
2014-10-01 08:17:45 -07:00
epriestley
ab6c6836f4 Remove the "note" database issue status
Summary:
Ref T1191. This was useful for annotating everything but we no longer need it; there are just two types of issues now:

  - Error: stuff we can't fix (missing or surplus tables/database/columns, bad column nullability).
  - Warning: stuff we can fix (column types, character sets, collations, missing or surplus keys, incorrectly defined keys, bad key uniqueness).

Test Plan: Saw 3,399 warnings and 0 errors.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10594
2014-10-01 08:00:11 -07:00
epriestley
03519c53bb Mark questionable column nullability for later
Summary:
Ref T1191. Ref T6203. While generating expected schemata, I ran into these columns which seem to have sketchy nullability.

  - Mark most of them for later resolution (T6203). They work fine today and don't need to block T1191. Changing them can break the application, so we can't autofix them.
  - Forgive a couple of them that are sort-of reasonable or going to get wiped out.

Test Plan: Saw 94 remaining warnings.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: hach-que, epriestley

Maniphest Tasks: T1191, T6203

Differential Revision: https://secure.phabricator.com/D10593
2014-10-01 07:59:44 -07:00
epriestley
4f87adc438 Ignore keys with trailing index on table primary key for now
Summary:
Ref T1191. We have several keys on `<x, y, id>`. When `id` is an auto-increment primary key, I believe this is exactly equivalent to a key on `<x, y>`, because the leaf nodes are implicitly sorted by `id`. We omit the implicit `id` elsewhere.

It would be nice to drop the `id` bit for consistency, but it's not doing any harm and this doesn't need to block the primary work of T1191.

Test Plan: Saw slightly fewer warnings.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10592
2014-10-01 07:55:09 -07:00
epriestley
943c62d1e9 Add missing expected keys and uniqueness
Summary:
Ref T1191.

  - Adds definitions for missing keys and keys with wrong uniqueness. Generally, I defined these before fixing the key query to actually pull all keys and support uniqueness.
  - Moves "key uniqueness" to note severity; this is fixable (probably?) and there are no remaining issues.
  - Moves "Missing Key" to note severity; missing keys are fixable and all remaining missing keys are really missing (either missing edge keys, or missing PHID keys):

{F210089}

  - Moves "Surplus Key" to note seveirty; surplus keys are fixable all remaining surplus keys are really surplus (duplicate key in Harbormaster, key on unused column in Worker):

{F210090}

Test Plan:
  - Vetted missing/surplus/unique messages.
  - 146 issues remaining.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10590
2014-10-01 07:53:50 -07:00