1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-21 21:10:56 +01:00
Commit graph

8815 commits

Author SHA1 Message Date
epriestley
fcd2025a85 Add Merchants to Phortune
Summary:
Ref T2787. Currently, you add payment providers (Stripe, Paypal, etc) in global configuration.

Generally, this approach is cumbersome, limiting, and often hard for users to figure out. It also doesn't provide a natural way to segment payment receivers or provide web access to administrative payment functions like issuing refunds, canceling orders, etc. I think that stuff definitely needs to be in the web UI, and the rule for access to it can't reasonably just be "all administrators" in a lot of reasonable cases.

The only real advantage is that it prevents an attacker from adjusting settings and pointing something at an account they control. But this attack can be mitigated through notifications, some sort of CLI-only merchant lock, payment accounts being relatively identifiable, etc.

So introduce "merchants", which are basically payable entities. An individual merchant will have attached Paypal, Stripe, etc., accounts, and access rules. When you buy something in an application, the merchant to pay is also specified. They also provide an umbrella for dealing with permissions down the line.

This may get a //little// cumbersome because if there are several merchants your saved card information is not shared across them. I think that will be fine in the normal case (most installs will have only one merchant). Even if it isn't and we leave providers global, I think introducing this is the right call from a web UI / permissions point of view. I'll play around with it in the next couple of diffs and figure out exactly where the line goes.

Test Plan: Listed, created, edited, viewed merchants.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10648
2014-10-07 10:55:16 -07:00
epriestley
61b1fe78c7 Modernize Phortune PHID constants
Summary:
Ref T2787. These were still stuck in the stone ages.

(The handles are pretty skeletal but most aren't used anywehre.)

Test Plan: Funded an initiative without anything breaking. Grepped for removed constants.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10647
2014-10-06 16:48:16 -07:00
epriestley
19c1c07d35 Give Balanced provider complete workflow logic in Phortune
Summary: Ref T2787. Like Stripe, this one is pretty easy to get working correctly on the "good" path and fataling out in a safe way on bad paths.

Test Plan: Funded an initiative with Balanced.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10645
2014-10-06 16:48:02 -07:00
Bob Trahan
0bbe3a6d28 Storage - fix more query errors by escaping collate_text and collate_sort
Summary: second bit of https://github.com/phacility/phabricator/pull/729

Test Plan: this is a weird pull request merge

Reviewers: chad, epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D10646
2014-10-06 15:51:42 -07:00
epriestley
31817b9097 Disable Phortune Paypal payment provider
Summary:
Ref T2787. For test charges, Paypal is putting the charge in a "payment review" state. Dealing with this state requires way more infrastructure than other providers: we're supposed to pause delivery, then poll Paypal every 6 hours to see if the review has resolved.

Since I can't seem to generate normal test charges, I can't test Paypal for now. Disable it until we have more infrastructure.

(This diff gets us further along, up to the point where I hit this issue.)

Test Plan: Read documentation, rolled eyes.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10644
2014-10-06 14:51:51 -07:00
epriestley
83e1a1ddb2 Give Stripe complete payment workflow logic in Phortune
Summary:
Ref T2787. This basically already works correctly since the hard logic is external to the provider on API providers. Tweak a couple of things.

Failures still just fail the cart completely, for now.

Test Plan: Completed a charge with Stripe.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10640
2014-10-06 14:21:11 -07:00
epriestley
4ef547f8d6 Give WePay complete payment logic in Phortune
Summary:
Ref T2787. This doesn't get all the edge cases quite correct, but is generally a safe, complete payment workflow:

  - Shares the actual charging state logic.
  - Makes it appropriately stateful with locking and transactions.
  - Gets the main flow correct.
  - Detects failure cases, just tends to blow up rather than help the user resolve them.

Test Plan:
  - Charged with WePay.
  - Charged with Infinite Free Money.
  - Resumed an abandoned cart.
  - Hit all failure states where we just dead-end the cart. Not ideal, but (seemingly) complete/safe/correct.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10639
2014-10-06 14:20:40 -07:00
epriestley
0beb8228da Give applications control over Phortune cart logic
Summary: Ref T2787. Similar to D10634, give applications more control over the cart workflow. For now this just means they get to pick exit URIs, but in the future they can manage more details of cart behavior.

Test Plan: Funded an initiative and got returned to the initiative instead of dead-ending in Phortune.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10638
2014-10-06 14:19:08 -07:00
Chad Little
ed0b23cb4d Add new Payment Icons
Summary: Fixes T6244, adds icons for payment providers. May split into different sprites down the road.

Test Plan: Photoshop

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6244

Differential Revision: https://secure.phabricator.com/D10642
2014-10-06 13:21:29 -07:00
Bob Trahan
69b1ff3032 Configuration - re-jigger how we handle bad configuration files
Summary: just explicitly check if the file doesn't exist *first*, and then do the standard include thing with the more generic error if that doesn't work. Fixes T6255.

Test Plan: re-started apache and phabricator still worked; will ask csilvers to give it a whirl too

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6255

Differential Revision: https://secure.phabricator.com/D10643
2014-10-06 13:20:56 -07:00
Bob Trahan
928b4edffb Storage - escape collation type in create database code pathway
Summary: without escapage here, creating databases fails. Fixes T6251.

Test Plan: ran the command CREATE DATABASE foo COLLATION binary and it failed; ran the command CREATE DATABASE foo2 COLLATION "binary" and it worked; trusting that the %T still works as advertised.

Reviewers: chad, epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6251

Differential Revision: https://secure.phabricator.com/D10641
2014-10-06 13:03:23 -07:00
Chad Little
923f625130 Fix incorrect maniphest.update conduit UI
Summary: Fixes T6254 and renames status as string. Though maybe this should go through `formatStringConstants`?

Test Plan: Reload Conduit page, see new text.

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6254

Differential Revision: https://secure.phabricator.com/D10637
2014-10-06 11:06:23 -07:00
epriestley
35dc510e18 Add more structure to Phortune product purchasing flow
Summary:
Ref T2787. When a user purchases a product in Phortune, transition the cart through a purchased state and invoke product callbacks so applications can respond to the workflow.

Also shore up some stuff like preventing negative amounts of funding.

Test Plan: Backed an initiative and saw it show up on the initiative after completing the purcahsing workflow.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10635
2014-10-06 10:36:43 -07:00
epriestley
e9615b74a5 Move Phortune product logic into applications
Summary: Ref T2787. `Product` is currently a fairly heavy object, but as Phortune develops it makes a lot of sense to make it a lighter object and put more product logic in applications. Convert it into a fairly lightweight reference to applications. The idea is that Phortune is mostly providing a cart flow, and applications manage the details of products.

Test Plan: Funded an initiative for $1.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10634
2014-10-06 10:30:06 -07:00
epriestley
f86f9dc512 Make Currency a more formal type
Summary:
Ref T2787. Phortune currently stores a bunch of stuff as `...inUSDCents`. This ends up being pretty cumbersome and I worry it will create a huge headache down the road (and possibly not that far off if we do Coinbase/Bitcoin soon). Even now, it's more of a pain than I figured it would be.

Instead:

  - Provide an application-level serialization mechanism.
  - Provide currency serialization.
  - Store currency in an abstract way (currently, as "1.23 USD") that can handle currencies in the future.
  - Change all `...inUSDCents` to `..asCurrency`.
  - This generally simplifies all the application code.
  - Also remove some columns which don't make sense or don't make sense anymore. Notably, `Product` is going to get more abstract and mostly be provided by applications.

Test Plan:
  - Created a new product.
  - Purchased a product.
  - Backed an initiative.
  - Ran unit tests.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10633
2014-10-06 10:26:48 -07:00
epriestley
3463ce8a51 Create new databases with appropriate collation
Summary: Ref T1191. We don't create new databases with appropriate collation yet.

Test Plan:
Created a new database and saw it issue:

```
>>> [10] <query> CREATE DATABASE IF NOT EXISTS `phabricator2_testo` COLLATE utf8mb4_bin
```

Reviewers: btrahan, hach-que

Reviewed By: hach-que

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10632
2014-10-03 06:01:21 -07:00
James Rhodes
8fbebce501 Implement storage of a host ID and a public key for authorizing Conduit between servers
Summary:
Ref T4209.  This creates storage for public keys against authorized hosts, such that servers can be authorized to make Conduit calls as the omnipotent user.

Servers are registered into this system by running the following command once:

```
bin/almanac register
```

NOTE: This doesn't implement authorization between servers, just the storage of public keys.

Placing this against Almanac seemed like the most sensible place, since I'm imagining in future that the `register` command will accept more information (like the hostname of the server so it can be found in the service directory).

Test Plan: Ran `bin/almanac register` and saw the host (and public key information) appear in the database.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4209

Differential Revision: https://secure.phabricator.com/D10400
2014-10-03 22:52:41 +10:00
epriestley
0ddb187508 Add a setup warning about innodb_buffer_pool_size
Summary: Fixes T6119. This is a little fuzzy, but generally bumping up `innodb_buffer_pool_size` to something bigger than the default (which is often anemic, at `8M`) is desriable, and it seems like it will fix the specific issue a user encountered in T6119.

Test Plan: {F211855}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6119

Differential Revision: https://secure.phabricator.com/D10630
2014-10-02 14:44:36 -07:00
epriestley
b16527d93d Remove old, confusing configuration files
Summary:
Fixes T6230. These files have not been read by default for a long time, but users are frequently confused and try to edit `default.conf.php`.

Remove the actual files. Allow `phabricator_read_config_file(...)` to continue working as though they exist so as to not break config-file-based installs.

Test Plan:
I used this script to make sure that removing `default.conf.php` won't change things for installs which are still using config files:

```
<?php

require_once 'scripts/__init_script__.php';

$file = require 'conf/default.conf.php';
$global = new PhabricatorConfigDefaultSource();
$global_values = $global->getAllKeys();

foreach ($file as $key => $value) {
  $global_value = idx($global_values, $key, (object)array());

  if ($value !== $global_value) {
    echo "{$key}\n\n";
    echo "FILE VALUE\n";
    var_dump($value);
    echo "\n";
    echo "DEFAULT VALUE\n";
    var_dump($global_value);
    return;
  }
}
```

These were the keys that had issues:

  - `log.access.format` Not specified in default.conf.php, safe to speciy.
  - `mysql.pass` Empty string in file, null in global. Same effect.
  - `metamta.default-addrress` One used `noreply@example.com`, one `noreply@phabricator.example.com`. These are just human-readable examples so it's safe to change behavior.
  - `metamta.domain` same as above, `example.com` vs `phabricator.example.com`.
  - `phpmailer.smtp-host` One used null, one empty string.
  - `phpmailer.smtp-protocol` As above.
  - `files.viewable-mime-types` File version is out of date.
  - `repository.default-local-path` Null in file, set in global. This is correct to set to a default value now.
  - `pygments.dropdown-choices` File version is out of date.
  - `environment.append-paths` File version is empty, global version adds common paths. This //could// change behavior, but the web behavior is better and more reasonable in general, and a system would need to be configured in a very bizarre way for this to be relevant.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6230

Differential Revision: https://secure.phabricator.com/D10628
2014-10-02 14:44:29 -07:00
epriestley
d67b7f0f47 Correct column mutations for old versions of MySQL
Summary:
Ref T1191. Although I fixed some of the mutations earlier (in D10598), I missed the column mutations under old versions of MySQL. In particular, this isn't valid:

  - `ALTER TABLE ... MODIFY columnName VARCHAR(64) COLLATE binary`

Issue the permitted version of this instead, which is:

  - `ALTER TABLE ... MODIFY columnName VARBINARY(64)`

Also fixed an issue where a clean schema had the wrong nullability for a column in the draft table. Force it to the expected nullability.

The other trick here is around the one column with a FULLTEXT index on it, which needs a little massaging.

Test Plan:
  - Forced my local install to return `false` for utf8mb4 support.
  - Did a clean adjust into `binary` columns.
  - Poked around, added emoji to things.
  - Reverted the fake check and did a clean adjust into `utf8mb4` columns.
  - Emoji survived.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: fabe, epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10627
2014-10-02 14:44:22 -07:00
epriestley
e333017c3e Strip email signatures from Mailbox
Summary: thanks mailbox

Test Plan: unit tests

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D10629
2014-10-02 12:33:43 -07:00
epriestley
410c2ecbfd Remove UTF8 BMP unit test and replace it with a general UTF8 test
Summary:
Ref T1191. After utf8mb4 conversion, these tests no longer pass because MySQL allows emoji and gclefs and such.

We could keep these tests running by keeping a `ut8f_bin` table around somewhere, but we have no other use cases for it and it does not seem worth the added complexity. All these BMP-only codepaths are on the way out.

Update the `%s` / `%B` test to make sure it's rejecting invalid byte sequences, which are still not permitted.

Test Plan: Tests now pass.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10621
2014-10-02 11:47:25 -07:00
epriestley
8fa8415c07 Automatically build all Lisk schemata
Summary:
Ref T1191. Now that the whole database is covered, we don't need to do as much work to build expected schemata. Doing them database-by-database was helpful in converting, but is just reudndant work now.

Instead of requiring every application to build its Lisk objects, just build all Lisk objects.

I removed `harbormaster.lisk_counter` because it is unused.

It would be nice to autogenerate edge schemata, too, but that's a little trickier.

Test Plan: Database setup issues are all green.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10620
2014-10-02 09:51:20 -07:00
epriestley
e4e5a2004f Use sort, not text, for search index table
Summary: Ref T1191. The index's case sensitivity depends on the column type. Using `text` makes the search case-sensitive, which is not desirable.

Test Plan: After adjustment, searched for "PROJECTS" and found hits against "projects".

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10619
2014-10-02 09:50:23 -07:00
epriestley
5f82805068 Make execution order of Herald rules explicit
Summary: Fixes T6211. This gives Herald rules an explicit execution order, which seems generally good. See some discussion on T6211 and inline.

Test Plan:
  - Added unit test.
  - Dry ran rules and saw rules appear in the expected order in the transcript.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6211

Differential Revision: https://secure.phabricator.com/D10624
2014-10-02 09:49:32 -07:00
epriestley
3c6781b177 Differentiate between "no pygmetnize" and "nonworking pygmentize" during setup
Summary: Fixes T6210. The current messaging may be confusing if `pygmentize` is available but broken.

Test Plan: Faked the binary names and hit the errors, which seemed helpful.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6210

Differential Revision: https://secure.phabricator.com/D10626
2014-10-02 09:48:04 -07:00
epriestley
7018909447 Fix TokenGiven stories for Asana
Summary: Ref T6201. This isn't quite perfect but should be good enough. At some point far in the future I plan to revamp feed rendering a bit. This should possibly become a real ApplicationTransaction story eventually, too.

Test Plan: {F211777}

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T6201

Differential Revision: https://secure.phabricator.com/D10625
2014-10-02 09:20:27 -07:00
epriestley
fda0b086b5 Make #🐳 work properly
Summary:
Ref T6223. Two issues:

  - We don't use `/u` mode on these regexps. Without `/u`, the `\w`/`\W`/`\s`/`\S` modifiers have bad behavior on non-ASCII bytes. Add the flag to use unicode mode, making `\w` and `\s` behave like we expect.
    - We might possibly want to do something different here eventually (for example, if the `/u` flag has some huge performance penalty) but this seems OK for now.
  - We use `\b` (word boundary) to terminate the match, but `🐳` is not a word character. Use `(?!\w)` instead ("don't match before a word character") which is what we mean.

Test Plan: {F211498}

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T6223

Differential Revision: https://secure.phabricator.com/D10618
2014-10-01 12:45:31 -07:00
epriestley
0a6473138f Purge readthrough caches before applying schema adjustments
Summary: Ref T1191. The bulk of the slowness in T1191 is copying tables. In some cases, we can't avoid this, but we have various readthrough caches which may be very large and are safe to drop, and dropping them is very quick (much less than 1 second). In particular, dropping the `changeset_parse_cache` made the process at least ~8 minutes faster on `secure.phabricator.com` (I killed it after 8 minutes, so I'm not sure what the real number is).

Test Plan: Ran `bin/storage adjust` and saw it drop caches before applying adjustments.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10616
2014-10-01 12:44:42 -07:00
epriestley
3a644cf6cc Truncate very old, overlong Maniphest mail keys
Summary:
Ref T1191. Long ago, Maniphest generated with 40-character mail keys. These prevent the migration to `bytes20`. We had about 300 of these on secure.phabricator.com from several years ago.

Just truncate them. This adjusts reply-to addresses, but it's very likely that none are relevant anymore.

Test Plan: Ran migration on `secure.phabricator.com` to truncate keys.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10615
2014-10-01 12:43:58 -07:00
epriestley
3629ebebe9 Drop very old schema_version table if it exists
Summary: Ref T1191. This predates the mdoern patch stuff and may exist on very, very old installs. By the time they apply this patch, it's guaranteed it won't matter anymore. Drop it to make the schemata consistent with expectations.

Test Plan: Ran patch on installs with and without the table.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10611
2014-10-01 12:43:20 -07:00
epriestley
4eb4399640 Convert SavedQuery / NamedQuery to text, not bytes
Summary: Ref T1191. Similar issue to D10613. This column usually has a hash exactly 12 bytes long, but sometimes stores an internal builtin query name like "open", "all", etc. It might be nice to promote those to 12-byte hashes of a consistent length eventually, but for now just make this a variable-length column.

Test Plan: Ran migration, no longer saw issues with reordering builtin saved searches.

Reviewers: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10614
2014-10-01 10:11:19 -07:00
epriestley
f881b5dd7f Convert DivinerLiveSymbol to text, not bytes
Summary:
Ref T1191. The `bytes` types are BINARY(...), which is fixed-length and zero-pads. These hashes are not 64 characters long, so migrating them to `binary` ends up with a bunch of zero-padding.

Instead, migrate them to `text` so we drop the zero padding. It would be vaguely nice to either introduce a `varbytes` type (ick) or change the hash size to a standard size (nicer) eventually, but this isn't very important.

Test Plan: Will adjust `secure.phabricator.com`.

Reviewers: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10613
2014-10-01 10:10:56 -07:00
epriestley
5ce3575fb5 Fix adjust phases for keys
Summary: Ref T1191. I renamed the phases but missed these two since I didn't have any more key issues locally.

Test Plan: Ran `bin/storage adjust` in production with key issues.

Reviewers: btrahan

Subscribers: chad, epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10612
2014-10-01 10:10:32 -07:00
epriestley
300172e799 Support AUTO_INCREMENT in bin/storage adjust
Summary:
Ref T1191. When changing the column type of an AUTO_INCREMENT column, we currently may lose the autoincrement attribute.

Instead, support it. This is a bit messy because AUTO_INCREMENT columns interact with PRIMARY KEY columns (tables may only have one AUTO_INCREMENT column, and it must be a primary key). We need to migrate in more phases to avoid this issue.

Introduce new `auto` and `auto64` types to represent autoincrement IDs.

Test Plan:
  - Saw autoincrement show up correctly in web UI.
  - Fixed an autoincrement issue on the XHProf storage table with `bin/storage adjust` safely.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10607
2014-10-01 08:24:51 -07:00
epriestley
0d7489da79 Provide bin/storage quickstart to automate generation of quickstart.sql
Summary:
Ref T1191. Currently, the `quickstart.sql` gets generated in a pretty manual fashion. This is a pain, and will become more of a pain in the world of utf8mb4.

Provide a workflow which does upgrade + adjust + dump + destroy, then massages the output to produce a workable `quickstart.sql`.

Test Plan: Inspected output; I'll test this more throughly before actually generating a new quickstart, but that's some ways away.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10603
2014-10-01 08:22:37 -07:00
epriestley
1dfa94e571 Use binary collations for most text
Summary:
Ref T1191. For most text columns, we either don't care if "a" and "A" are the same, or we expect them to be different (for example: keys, domains, secrets, etc). Default text columns to the `_bin` collation so they are compared by strict character value. This is safer in cases where we aren't sure.

For some text columns, we allow the user to sort by the column in the UI (like Maniphest task titles) or we do care that "A" and "a" are the same (for example: project names). Introduce a new class of virtual data types, the "sort..." types, to cover these columns. These are like the "text..." types but use sorting collations which treat "A" and "a" the same.

Test Plan:
  - Made an effort to identify all columns where the UI relies on database collation.
  - Ran `bin/storage adjust` and cleared all warnings.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: beng, epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10602
2014-10-01 08:18:53 -07:00
epriestley
4fcc634a99 Fix almost all remaining schemata issues
Summary:
Ref T1191. This fixes nearly every remaining blocker for utf8mb4 -- primarily, overlong keys.

Remaining issue is https://secure.phabricator.com/T1191#77467

Test Plan: I'll annotate inline.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T6099, T6129, T6133, T6134, T6150, T6148, T6147, T6146, T6105, T1191

Differential Revision: https://secure.phabricator.com/D10601
2014-10-01 08:18:36 -07:00
epriestley
a5ce56aa76 Allow bin/storage adjust to make key changes
Summary:
Ref T1191. These are a bit tricky because keys can interact with column changes, so basically we do three phases:

  1. Nuke all bad keys.
  2. Make all column (and database/table) changes.
  3. Fix all nuked keys.

Test Plan: Ran migration locally. See note for remaining issues.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10599
2014-10-01 08:18:11 -07:00
epriestley
22ee8432d2 Allow bin/storage adjust to correct column types and collations
Summary:
Ref T1191. Allow `bin/storage adjust` to modify columns.

  - Although `CREATE TABLE ... colname VARCHAR(64) CHARACTER SET BINARY` works fine, it's actually a trick. Adjust the binary columns for this.

Test Plan: See comment.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6130, T6128, T6135, T6137, T6138, T6149, T6151, T1191

Differential Revision: https://secure.phabricator.com/D10598
2014-10-01 08:17:45 -07:00
epriestley
f7ee2c7467 Add bin/storage adjust, for adjusting schemata
Summary:
Ref T1191. Adds a new workflow which can apply schema adjustments.

For now, it only performs database and table collation/charset adjustments. I believe these are extremely safe/minor, because they only affect the default values for newly created columns.

Test Plan:
  - Ran migration on various database states, database/table changes went through cleanly.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10595
2014-10-01 08:16:31 -07:00
epriestley
ab6c6836f4 Remove the "note" database issue status
Summary:
Ref T1191. This was useful for annotating everything but we no longer need it; there are just two types of issues now:

  - Error: stuff we can't fix (missing or surplus tables/database/columns, bad column nullability).
  - Warning: stuff we can fix (column types, character sets, collations, missing or surplus keys, incorrectly defined keys, bad key uniqueness).

Test Plan: Saw 3,399 warnings and 0 errors.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10594
2014-10-01 08:00:11 -07:00
epriestley
03519c53bb Mark questionable column nullability for later
Summary:
Ref T1191. Ref T6203. While generating expected schemata, I ran into these columns which seem to have sketchy nullability.

  - Mark most of them for later resolution (T6203). They work fine today and don't need to block T1191. Changing them can break the application, so we can't autofix them.
  - Forgive a couple of them that are sort-of reasonable or going to get wiped out.

Test Plan: Saw 94 remaining warnings.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: hach-que, epriestley

Maniphest Tasks: T1191, T6203

Differential Revision: https://secure.phabricator.com/D10593
2014-10-01 07:59:44 -07:00
epriestley
4f87adc438 Ignore keys with trailing index on table primary key for now
Summary:
Ref T1191. We have several keys on `<x, y, id>`. When `id` is an auto-increment primary key, I believe this is exactly equivalent to a key on `<x, y>`, because the leaf nodes are implicitly sorted by `id`. We omit the implicit `id` elsewhere.

It would be nice to drop the `id` bit for consistency, but it's not doing any harm and this doesn't need to block the primary work of T1191.

Test Plan: Saw slightly fewer warnings.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10592
2014-10-01 07:55:09 -07:00
epriestley
8bf24f53b3 Destroy surplus columns
Summary:
Ref T1191. This destroys surplus columns:

  - Pholio's transaction comments have a `mockID` column, but this is not used. The `imageID` column is used instead.
  - Phragment has an unused `description` column.
  - Releeph has an unused `summary` column.

Test Plan:
  - Grepped for usage of these columns.
  - Checked that these exist in production, too.
  - Ran upgrades.
  - Added Pholio inline comments.
  - Saw fewer warnings.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10591
2014-10-01 07:54:33 -07:00
epriestley
943c62d1e9 Add missing expected keys and uniqueness
Summary:
Ref T1191.

  - Adds definitions for missing keys and keys with wrong uniqueness. Generally, I defined these before fixing the key query to actually pull all keys and support uniqueness.
  - Moves "key uniqueness" to note severity; this is fixable (probably?) and there are no remaining issues.
  - Moves "Missing Key" to note severity; missing keys are fixable and all remaining missing keys are really missing (either missing edge keys, or missing PHID keys):

{F210089}

  - Moves "Surplus Key" to note seveirty; surplus keys are fixable all remaining surplus keys are really surplus (duplicate key in Harbormaster, key on unused column in Worker):

{F210090}

Test Plan:
  - Vetted missing/surplus/unique messages.
  - 146 issues remaining.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10590
2014-10-01 07:53:50 -07:00
epriestley
2880732a49 Generate expected schemata for Search
Summary:
Ref T1191. Notable:

  - Drops a very old saved query table. See comments inline: plan was to remove it after a year. It's been ~a year and two weeks.
  - This has our only fulltext index. I'm not supporting that formally for now, but left a note.
  - This has our only MyISAM table. I'm not supporting that explicitly for now, but it shouldn't affect anything. I may deal with this in the future.
  - These tables don't actually write directly via Lisk, so there's some fiddling to get the schemata right.

Test Plan: Down to ~250 warnings. No more surplus databases or tables.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10589
2014-10-01 07:53:35 -07:00
epriestley
dc8b2ae6d2 Generate expected schemata for Fact, Owners, Herald and Diviner
Summary:
Ref T1191. Notable:

  - `HeraldApplyTranscript` is not actually a DAO and has no table (it is serialized into HeraldTranscript).

Test Plan: Down to fewer than 300 issues.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10588
2014-10-01 07:53:12 -07:00
epriestley
cfbcd69e9b Generate expected schemata for Pholio, Phortune, Phragment, Phriction and Policy
Summary: Ref T1191. Nothing too out-of-the-ordinary here.

Test Plan: Saw fewer than 500 remaining issues.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: chad, epriestley, hach-que

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10587
2014-10-01 07:52:26 -07:00
epriestley
e7b590a1cf Generate expected schemata for Harbormaster
Summary:
Ref T1191. Nothing too notable here:

  - Allow a Lisk object to specify that there's no expectation that a table exists. We have one Harbormaster object and one Token object like this.
  - Removed BuildPlanTransactionComment because it's currently unused.

Test Plan:
  - Saw ~200 fewer warnings; just ~800 left.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10583
2014-10-01 07:40:36 -07:00