1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-30 10:42:41 +01:00
Commit graph

652 commits

Author SHA1 Message Date
Bob Trahan
1cc81b1d0a OAuthServer - hide client secret behind a "View Secret" action
Summary: ...also adds policies on who can view and who can edit an action. Fixes T6949.

Test Plan: viewed a secret through the new UI and it worked

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6949

Differential Revision: https://secure.phabricator.com/D11401
2015-01-14 17:27:45 -08:00
epriestley
a455e50e29 Build a Conpherence thread index
Summary:
Ref T3165. Builds a dedicated index for Conpherence to avoid scale/policy filtering concerns.

  - This is pretty one-off but I think it's generally OK.
  - There's no UI for it.
  - `ConpherenceFulltextQuery` is very low-level. You would need to do another query on the PHIDs it returns to actually show anything to the user.
  - The `previousTransactionPHID` is so you can load chat context efficiently. Specifically, if you want to show results like this:

> previous line of context
> **line of chat that matches the query**
> next line of context

...you can read the previous lines out of `previousTransactionPHID` directly, and the next lines by issuing one query with `WHERE previousTransactionPHID IN (...)`.

I'm not 100% sure this is useful, but it seemed like a reasonable thing to provide, since there's no way to query this efficiently otherwise and I figure a lot of chat might make way more sense with a couple of lines of context.

Test Plan:
  - Indexed a thread manually (whole thing indexed).
  - Indexed a thread by updating it (just the new comment indexed).
  - Wrote a hacky test script and got reasonable-looking query results.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T3165

Differential Revision: https://secure.phabricator.com/D11234
2015-01-06 10:24:30 -08:00
Joshua Spence
b3e196b694 Rename PhabricatorPolicyRule subclasses for consistency
Summary: Ref T5655. Fixes T6849. This is another take on D11131, which was missing the DB migration and was reverted in rP7c4de0f6be77ddaea593e1f41ae27211ec179a55.

Test Plan: Ran `./bin/storage upgrade` and verified that the classes were renamed in the `phabricator_policy.policy` table.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6849, T5655

Differential Revision: https://secure.phabricator.com/D11166
2015-01-03 23:48:55 +11:00
Joshua Spence
c7cd844a07 Remove deprecated task subscriber class
Summary: This class is no longer used after D10965.

Test Plan: `grep`

Reviewers: btrahan, epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11133
2015-01-03 11:32:14 +11:00
Joshua Spence
7c2a7d0365 Modernize remaining edge types
Summary: Modernize remaining edges to subclass `PhabricatorEdgeType`. Largely based on D11045.

Test Plan: Browsed around and performed various actions include subscribing, unsubscribing and watching.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D11116
2015-01-03 10:58:20 +11:00
Joshua Spence
f0db6e4818 Migrate Project edges to subclass PhabricatorEdgeType
Summary: Modernize Project edges to subclass `PhabricatorEdgeType`. Largely based on D11045.

Test Plan: Add a member to a project, saw new rows in the `phabricator_project.edge` and `phabricator_user.edge` tables.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11111
2015-01-02 10:10:59 +11:00
Joshua Spence
7cab903943 Migrate Differential revision edges to use modern EdgeType subclasses
Summary: Modernize Differential edges to subclass `PhabricatorEdgeType`. Largely based on D11045.

Test Plan: From previous experience, these changes are fairly trivial and safe. I poked around a little to make sure things looked reasonably okay.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, Krenair, epriestley

Differential Revision: https://secure.phabricator.com/D11074
2015-01-01 15:07:03 +11:00
Alex Monk
a84cd99893 Paste: Add edit policy
Summary: T5549

Test Plan: Set edit policy on paste, check that only users meeting the policy requirements can edit it.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Maniphest Tasks: T5549

Differential Revision: https://secure.phabricator.com/D11097
2014-12-31 08:24:57 -08:00
Alex Monk
102e431feb Migrate Maniphest task blockers to modern EdgeType classes
Summary:
Prevents "edited tasks, added: 1; removed: 1"

Fixes T6757, using D9839 as an example

Test Plan: Added and removed blockers to/from tasks, saw the expected history entries.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Maniphest Tasks: T6757

Differential Revision: https://secure.phabricator.com/D11045
2014-12-28 06:40:39 -08:00
Bob Trahan
9219645287 Daemons - add "objectPHID" to task tables.
Summary: Ref T5402. This more or less "fixes" it but there's probably some polish to do?

Test Plan:
stopped and started daemons. error logs look good.

ran bin/storage upgrade.  noted that `adjust` added the appropriate indices for active and archive task.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T5402

Differential Revision: https://secure.phabricator.com/D11044
2014-12-23 16:30:05 -08:00
Fabian Stelzer
2fedb6f941 Start phd daemons as the correctly configured user and refuse otherwise
Summary:
Fixes T5196
If no phd.user is configured the behaviour is unchanged besides printing a warning when run as root (Usually i would add an exit(1) here but that would break existing installs who do that).
If phd.user is set and the current user is root it will run the daemon as: su USER -c "command" (I'm not sure if this works for every platform needed)
Otherwise it will refuse to start if configured and current user mismatch.

Test Plan: Stopped & Started phd daemon with various users and different phd.user settings including root

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: vinzent, epriestley

Maniphest Tasks: T5196

Differential Revision: https://secure.phabricator.com/D11036
2014-12-23 08:15:51 -08:00
Bob Trahan
e76499bbbb Maniphest - kill TYPE_PROJECTS
Summary: Fixes T5245. Migrate old TYPE_PROJECTS transaction to new style edge transactions. Kill remaining rendering code.

Test Plan:
issued some fun queries to get some old-style transaction in my install:

```
// go from nothing to 1
INSERT INTO maniphest_transaction (phid, authorPHID, objectPHID, viewPolicy, editPolicy, commentVersion, transactionType, oldValue, newValue, contentSource, metadata, dateCreated, dateModified)
VALUES ('PHID-XACT-TASK-000000000000000', 'PHID-USER-zo35vxnoi4bxqak6yqhc', 'PHID-TASK-hb5wphctibxxqryo6ssi', 'users', 'users', 0, 'projects', '[]', '["PHID-PROJ-4teaxbjk5okv7mdz2qlx"]', '', '{}', 1419274578, 1419274578)

// go from 1 to 2
INSERT INTO maniphest_transaction (phid, authorPHID, objectPHID, viewPolicy, editPolicy, commentVersion, transactionType, oldValue, newValue, contentSource, metadata, dateCreated, dateModified)
VALUES ('PHID-XACT-TASK-111111111111111', 'PHID-USER-zo35vxnoi4bxqak6yqhc', 'PHID-TASK-hb5wphctibxxqryo6ssi', 'users', 'users', 0, 'projects', '["PHID-PROJ-4teaxbjk5okv7mdz2qlx"]', '["PHID-PROJ-4teaxbjk5okv7mdz2qlx", "PHID-PROJ-a7giqlyyfirqswg6gn6x"]', '', '{}', 1419274580, 1419274580)

// swap 1 for 1 with 2 in set
INSERT INTO maniphest_transaction (phid, authorPHID, objectPHID, viewPolicy, editPolicy, commentVersion, transactionType, oldValue, newValue, contentSource, metadata, dateCreated, dateModified)
VALUES ('PHID-XACT-TASK-222222222222222', 'PHID-USER-zo35vxnoi4bxqak6yqhc', 'PHID-TASK-hb5wphctibxxqryo6ssi', 'users', 'users', 0, 'projects', '["PHID-PROJ-4teaxbjk5okv7mdz2qlx", "PHID-PROJ-a7giqlyyfirqswg6gn6x"]', '["PHID-PROJ-4teaxbjk5okv7mdz2qlx", "PHID-PROJ-cety4gr55gpxzhwtrkhx"]', '', '{}', 1419274582, 1419274582)

// go from 2 to 1
INSERT INTO maniphest_transaction (phid, authorPHID, objectPHID, viewPolicy, editPolicy, commentVersion, transactionType, oldValue, newValue, contentSource, metadata, dateCreated, dateModified)
VALUES ('PHID-XACT-TASK-333333333333333', 'PHID-USER-zo35vxnoi4bxqak6yqhc', 'PHID-TASK-hb5wphctibxxqryo6ssi', 'users', 'users', 0, 'projects', '["PHID-PROJ-4teaxbjk5okv7mdz2qlx", "PHID-PROJ-cety4gr55gpxzhwtrkhx"]', '["PHID-PROJ-4teaxbjk5okv7mdz2qlx"]', '', '{}', 1419274584, 1419274584)
```

took a screenshot. ran the migration script and compared the screenshots and things looked correctly migrated...!

old style:

{F255408}

new style:

{F255407}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T5245

Differential Revision: https://secure.phabricator.com/D11032
2014-12-22 11:54:02 -08:00
Bob Trahan
9141da84c8 Maniphest - convert old CC transactions to modern SUBSCRIBER transactions
Summary: database migration + drop old view code. Fixes T5604.

Test Plan: grepped src/ for TYPE_CCS (no hits); viewed some tasks with old cc transactions and noted they still rendered correctly post data conversion

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T5604

Differential Revision: https://secure.phabricator.com/D11015
2014-12-18 14:42:46 -08:00
epriestley
d2df3064bc Allow Almanac services to be locked
Summary:
Fixes T6741. This allows Almanac services to be locked from the CLI. Locked services (and their bindings, interfaces and devices) can not be edited. This serves two similar use cases:

  - For normal installs, you can protect cluster configuration from an attacker who compromises an account (or generally harden services which are intended to be difficult to edit).
  - For Phacility, we can lock externally-managed instance cluster configuration without having to pull any spooky tricks.

Test Plan:
  - Locked and unlocked services.
  - Verified locking a service locks connected properties, bindings, binding properties, interfaces, devices, and device properties.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6741

Differential Revision: https://secure.phabricator.com/D11006
2014-12-18 14:31:36 -08:00
epriestley
c85327ca3e Give AlmanacServices a service type
Summary:
Ref T5833. This allows services to be typed, to distinguish between different kinds of services. This makes a few things easier:

  - It's easier for clients to select the services they're interested in (see note in T5873 about Phacility). This isn't a full-power solution, but gets is some of the way there.
  - It's easier to set appropriate permissions around when modifications to the Phabricator cluster are allowed. These service nodes need to be demarcated as special in some way no matter what (see T6741). This also defines a new policy for users who are permitted to create services.
  - It's easier to browse/review/understand services.
  - Future diffs will allow ServiceTypes to specify more service structure (for example, default properties) to make it easier to configure services correctly. Instead of a free-for-all, you'll get a useful list of things that consumers of the service expect to read.

The "custom" service type allows unstructured/freeform services to be created.

Test Plan:
  - Created a new service (and hit error cases).
  - Edited an existing service.
  - Saw service types on list and detail views.
  - Poked around new permission stuff.
  - Ran `almanac.queryservices` with service class specification.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10995
2014-12-17 11:10:27 -08:00
epriestley
39f2bbaeea Add Conduit Tokens to make authentication in Conduit somewhat more sane
Summary:
Ref T5955. Summary of intended changes:

**Improve Granularity of Authorization**: Currently, users have one Conduit Certificate. This isn't very flexible, and means that you can't ever generate an API token with limited permissions or IP block controls (see T6706). This moves toward a world where you can generate multiple tokens, revoke them individually, and assign disparate privileges to them.

**Standardize Token Management**: This moves Conduit to work the same way that sessions, OAuth authorizations, and temporary tokens already work, instead of being this crazy bizarre mess.

**Make Authentication Faster**: Authentication currently requires a handshake (conduit.connect) to establish a session, like the web UI. This is unnecessary from a security point of view and puts an extra round trip in front of all Conduit activity. Essentially no other API anywhere works like this.

**Make Authentication Simpler**: The handshake is complex, and involves deriving hashes. The session is also complex, and creates issues like T4377. Handshake and session management require different inputs.

**Make Token Management Simpler**: The certificate is this huge long thing right now, which is not necessary from a security perspective. There are separate Arcanist handshake tokens, but they have a different set of issues. We can move forward to a token management world where neither of these problems exist.

**Lower Protocol Barrier**: The simplest possible API client is very complex right now. It should be `curl`. Simplifying authentication is a necessary step toward this.

**Unblock T2783**: T2783 is blocked on nodes in the cluster making authenticated API calls to other nodes. This provides a simpler way forward than the handshake mess (or enormous-hack-mess) which would currently be required.

Test Plan:
  - Generated tokens.
  - Generated tokens for a bot account.
  - Terminated tokens (and for a bot account).
  - Terminated all tokens (and for a bot account).
  - Ran GC and saw it reap all the expired tokens.

NOTE: These tokens can not actually be used to authenticate yet!

{F249658}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5955

Differential Revision: https://secure.phabricator.com/D10985
2014-12-15 11:14:23 -08:00
epriestley
4505724cc4 Allow repositories to be bound to an AlmanacService
Summary:
Ref T2783. This is primarily exploratory and just figuring out what we're blocked on:

  - Allow a Repository to be bound to a Service. The Service may eventually define multiple read/write nodes, etc.
    - There's no UI to do this binding yet, you have to touch the database manually.
  - If a repository is bound to a Service, effect Conduit calls via calls to the remote service instead of executing them in-process.
    - These don't actually work yet since there's no authentication (see T5955).

Test Plan:
  - Made a nice Service with a nice Binding to a nice Interface on a nice Device.
  - Force-associated a repository with the service using a raw MySQL query.
  - Saw Phabricator try to make a remote call to the service (on localhost) and fail because of missing auth stuff.
  - Also ran `almanac.queryservices`.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2783

Differential Revision: https://secure.phabricator.com/D10982
2014-12-12 12:07:11 -08:00
Bob Trahan
7d96870570 Maniphest - use subscribers framework properly
Summary: Fixes T5604. This should fix some random bugs, lets us move forward more easily, and all that good stuff about killing code debt.

Test Plan:
- Conduit method maniphest.createtask
  - verified creating user subscribed
  - verified subscription transaction
- Conduit method maniphest.update
  - verified subscribers set as specified to ccPHIDs parameter
  - verified subscription transaction
- Herald
  - verified herald rule to add subscriber worked
  - verified no subscribers removed accidentally
- edit controller
  - test create and verify author gets added IFF they put themselves in subscribers control box
  - test update gets set to exactly what user enters
- lipsum generator'd tasks work
- bulk add subscribers works
- bulk remove subscriber works
- detail controller
  - added myself by leaving a comment
  - added another user via explicit action
  - added another user via implicit mention
- task merge via search attach controller
- mail reply handler
  - add subscriber via ./bin/mail receive-test
  - unsubscribe via ./bin/mail receive-test

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T5604

Differential Revision: https://secure.phabricator.com/D10965
2014-12-10 16:27:30 -08:00
epriestley
b5f7e9eec6 Reverse meaning of task priority column
Summary:
Ref T6615. Mixing ASC and DESC ordering on a multipart key makes it dramatically less effective (or perhaps totally ineffective).

Reverse the meaning of the `priority` column so it goes in the same direction as the `id` column (both ascending, lower values execute sooner).

Test Plan:
  - Queued 1.2M tasks with `bin/worker flood`.
  - Processed ~1 task/second with `bin/phd debug taskmaster` before patch.
  - Applied patch, took ~5 seconds for ~1.2M rows.
  - Processed ~100-200 tasks/second with `bin/phd debug taskmaster` after patch.
  - "Next in Queue" query on daemon page dropped from 1.5s to <1ms.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: aklapper, 20after4, epriestley

Maniphest Tasks: T6615

Differential Revision: https://secure.phabricator.com/D10895
2014-11-24 11:10:35 -08:00
epriestley
5e0f218fe4 Allow device SSH keys to be trusted
Summary:
Ref T6240. Some discussion in that task. In instance/cluster environments, daemons need to make Conduit calls that bypass policy checks.

We can't just let anyone add SSH keys with this capability to the web directly, because then an adminstrator could just add a key they own and start signing requests with it, bypassing policy checks.

Add a `bin/almanac trust-key --id <x>` workflow for trusting keys. Only trusted keys can sign requests.

Test Plan:
  - Generated a user key.
  - Generated a device key.
  - Trusted a device key.
  - Untrusted a device key.
  - Hit the various errors on trust/untrust.
  - Tried to edit a trusted key.

{F236010}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6240

Differential Revision: https://secure.phabricator.com/D10878
2014-11-20 17:33:30 -08:00
Bob Trahan
a414fc497f Diffusion - make projects work properly with commits
Summary: Fixes T3189. Now if you say #projects in a commit message they will associate nicely with the commit. Also we record transactions about all this project editing fun.

Test Plan: tested migration by associating some projects with commits and verifying they still showed up post migration. tested adding / removing projects by doing so from the UI, noting transactions written nicely as well

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Projects: #projects

Maniphest Tasks: T3189

Differential Revision: https://secure.phabricator.com/D10877
2014-11-19 14:43:59 -08:00
Bob Trahan
4350858628 Differential - allow setting viewPolicy from web ui during diff creation process
Summary: Fixes T6152, T6237. This introduces a viewPolicy column to the DifferentialDiff, and re-jiggers the DifferentialDiff policy implementation such that things behave as before once associated with a revision, else use the DifferentialDiff policy column value.

Test Plan: made a diff with a non-standard view policy and noted that policy was still selected in the revision step. arc lint.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6237, T6152

Differential Revision: https://secure.phabricator.com/D10875
2014-11-19 12:16:07 -08:00
Bob Trahan
ffe0765b50 Differential - make DifferentialDiffEditor into a real transaction editor.
Summary: Ref T6237. This sets us up for some future work like T6152, T6200 and generally cleaning up this workflow a bit. Tried to do as little as possible so not exposing transaction view yet. (Though that timeline is going to be a little funky in the common case of just the lone create transaction.)

Test Plan: made a diff from web ui and it worked. made a herald rule to block certain diffs then tried to make such a diff and saw UI letting me know i was blocked

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6237

Differential Revision: https://secure.phabricator.com/D10869
2014-11-18 15:32:23 -08:00
Bob Trahan
310373ebc4 Audit - delete duplicate audit requests and add unique key
Summary: Fixes T1768. This is mostly a data cleanliness issue as duplicate rows don't really do anything, but let's clear it up now.

Test Plan: made some duplicate rows by adding the same auditor multiple times. ran ./bin/storage upgrade and it worked perfectly!

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T1768

Differential Revision: https://secure.phabricator.com/D10849
2014-11-13 16:11:57 -08:00
epriestley
ac87ab275a Fix Phriction document migration after project magic removal
Summary: See <https://github.com/phacility/phabricator/issues/760>. We removed these methods in D10832 but still need the migration to be able to do project checks.

Test Plan: Ran on a test wiki with `/`, `/projects/` and `/projects/example/`. The first two pages didn't try to use project policies; the third one did.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D10836
2014-11-11 17:55:27 -08:00
epriestley
3fa8b152b6 Make the default Phriction view migration policy just "Users"
This is consistent with the old policy and more consistent with how other
objects have been migrated into policies.

Auditors: btrahan
2014-11-07 15:46:02 -08:00
epriestley
2e13a31722 Make Phriction policy migration even more robust
We have at least one project with `null` as a viewPolicy. This should get
sorted out separately, but make the migration robust against it.

Auditors: btrahan
2014-11-07 15:44:19 -08:00
epriestley
3d44941373 Minor, make Phriction migration a little more robust
On secure.phabricator.com, we have at least one page with a bad project slug
from long ago. Make the migration recover in this case.

Auditors: btrahan
2014-11-07 15:43:29 -08:00
Bob Trahan
8a3b1b9730 Phriction - add viewPolicy and editPolicy back-end data
Summary: Ref T4029. this diff makes the pertinent database changes AND adds the migration script. This is important to get the data backend straightened away before we fully ship T4029. Next diff will expose the edit controls for these policies and whatever else work is needed to get that part done right.

Test Plan: made sure the lone project page on my wiki had a project with restrictive view policy. Post migration verified correct policy applied to this lone project page AND most open policy applied to the others

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T4029

Differential Revision: https://secure.phabricator.com/D10814
2014-11-07 15:35:28 -08:00
epriestley
bf17b12daf Standardize SSH key storage
Summary:
Ref T5833. This fixes a few weird things with this table:

  - A bunch of columns were nullable for no reason.
  - We stored an MD5 hash of the key (unusual) but never used it and callers were responsible for manually populating it.
  - We didn't perform known-key-text lookups by using an index.

Test Plan:
  - Ran migrations.
  - Faked duplicate keys, saw them clean up correctly.
  - Added new keys.
  - Generated new keys.
  - Used `bin/auth-ssh` and `bin/auth-ssh-key`.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10805
2014-11-07 15:34:44 -08:00
epriestley
fbc175aa6e Force Differential draft uniqueness
Summary: Ref T1191. A couple of installs have hit issues with this table, so clean it up before adjustment adds a unique key to it.

Test Plan: Dropped key, added duplicate rows, ran patch, got cleanup, ran adjust to get the key back.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10799
2014-11-07 12:30:00 -08:00
epriestley
dbef5660fc Update the quickstart.sql
Summary:
Ref T1191. Use `storage quickstart` to regenerate `quickstart.sql` using modern schema construction statements.

This puts new installs into utf8mb4 mode immediately without requiring storage adjustment.

Test Plan:
  - Ran `arc unit --everything`, which uses quickstart.
  - Ran `bin/storage upgrade --namespace temp`, to quickstart a new namespace.
  - Ran `bin/storage upgrade --namespace temp --disable-utf8mb4`, to quickstart a new namespace without utf8mb4 support.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10797
2014-11-07 12:29:24 -08:00
epriestley
e58259b4f5 Retroactively populate Phriction mailKey column
Summary:
Fixes T6487. Ref T1191. Ref T4029. D10756 introduced, but did not populate, this column. This can cause it to fill with `"\0\0\0..."` after adjustment.

Regardless of the adjustment issue, it's nice to populate this column anyway because there's no fundamental reason an object can't have mail sent about it without being saved first, even though it may not practically be possible in the codebase today.

Test Plan:
  - Ran `storage upgrade`, saw the column populate for older documents.
  - Forced a couple of keys to bad values (too short or with "\0") and saw the migration fix them.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4029, T1191, T6487

Differential Revision: https://secure.phabricator.com/D10804
2014-11-07 09:48:15 -08:00
epriestley
e29955b48d Move SSHKey table to Auth database
Summary: Ref T5833. Since these will no longer be bound specifically to users, bring them to a more central location.

Test Plan:
  - Edited SSH keys.
  - Ran `bin/ssh-auth` and `bin/ssh-auth-key`.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10791
2014-11-06 12:37:22 -08:00
epriestley
2b495f1f03 Implement PhabricatorProjectInterface on Almanac Services and Devices
Summary:
Ref T5833. Allow services and devices to be tagged with projects.

(These fluff apply implementations are a good example of the issue discussed in T6403.)

Test Plan: {F229569}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10782
2014-11-05 15:30:00 -08:00
epriestley
2f1b5ae010 Give Almanac generic, custom-field-based properties
Summary:
Ref T5833. Currently, we have an `AlmanacDeviceProperty`, but it doesn't use CustomFields and is specific to devices. Make this more generic:

  - Reuse most of the CustomField infrastructure (so we can eventually get easy support for nice editor UIs, etc).
  - Make properties more generic so Services, Bindings and Devices can all have them.

The major difference between this implementation and existing CustomField implementations is that all other implementations are application-authoritative: the application code determines what the available list of fields is.

I want Almanac to be a bit more freeform (basically: you can write whatever properties you want, and we'll put nice UIs on them if we have a nice UI available). For example, we might have some sort of "ServiceTemplate" that says "a database binding should usually have the fields 'writable', 'active', 'credential'", which would do things like offer these as options and put a nice UI on them, but you should also be able to write whatever other properties you want and add services without building a specific service template for them.

This involves a little bit of rule bending, but ends up pretty clean. We can adjust CustomField to accommodate this a bit more gracefully later on if it makes sense.

Test Plan: {F229172}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10777
2014-11-05 15:27:16 -08:00
Bob Trahan
a9fc343d9e Phriction - start the move towards transactions and an editor
Summary:
This implements as little as possible to stick a working transactions + editor codepath in the basic create / edit flow. Aside from the transaction tables, this also required adding a mailKey to a phrictionDocument.

Future work would include adding more transactions types for things like "move" and all the pertinent support. Even future work is to add things like policies which will work easily in the transaction framework. Ref T4029.

Test Plan:
 - made a wiki doc
 - edit a wiki doc
 - had someone subscribe to a wiki doc and edited it

For all three, the edits worked, a reasonable email was sent out, and feed stories were generated.

 - made a wiki doc at a /location/like/this

document "stubs" were made as expected in /location and /location/like

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: chad, Korvin, epriestley

Maniphest Tasks: T4029

Differential Revision: https://secure.phabricator.com/D10756
2014-10-30 08:59:21 -07:00
epriestley
917da08417 Fix various MySQL version issues with new charset stuff
Summary:
Ref T1191. Notable stuff:

  - Adds `--disable-utf8mb4` to `bin/storage` to make it easier to test what things will (approximately) do on old MySQL. This isn't 100% perfect but should catch all the major stuff. It basically makes us pretend the server is an old server.
  - Require utf8mb4 to dump a quickstart.
  - Fix some issues with quickstart generation, notably special casing the FULLTEXT handling.
  - Add an `--unsafe` flag to `bin/storage adjust` to let it truncate data to fix schemata.
  - Fix some old patches which don't work if the default table charset is utf8mb4.

Test Plan:
  - Dumped a quickstart.
  - Loaded the quickstart with utf8mb4.
  - Loaded the quickstart with `--disable-utf8mb4` (verified that we get binary columns, etc).
  - Adjusted schema with `--disable-utf8mb4` (got a long adjustment with binary columns, some truncation stuff with weird edge case test data).
  - Adjusted schema with `--disable-utf8mb4 --unsafe` (got truncations and clean adjust).
  - Adjusted schema back without `--disable-utf8mb4` (got a long adjustment with utf8mb4 columns, some invalid data on truncated utf8).
  - Adjusted schema without `--disable-utf8mb4`, but with `--unsafe` (got truncations on the invalid data).

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10757
2014-10-29 15:49:29 -07:00
epriestley
d5b70e2c1c Add AlmanacBinding, to bind a service to an interface
Summary: Ref T5833. Allows you to bind a service (like `db.example.com`) to one or more interfaces (for example, to specify a pool with one read/write host and two read-only hosts). You can't configure which hosts have which properties yet, but you can add all the relevant interfaces to the service. Next diff will start supporting service, binding, and device properties like "is writable", "is active", etc., so that Almanac will be able to express operations like "change which database is writable", "disable writes", "bring a device down", etc.

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10745
2014-10-27 13:39:36 -07:00
epriestley
3e516e40f3 Build AlmanacInterface
Summary: Ref T5833. An interface is an IP (maybe v4, maybe v6) and port on a specified network (public internet, VPN, NAT block, etc).

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10718
2014-10-17 05:04:24 -07:00
epriestley
3e704f6c78 Build AlmanacNetwork
Summary: Ref T5833. This differentiates address spaces like the public internet from VPNs, so when a service is available at `192.168.0.1`, we'll know it's on some specific NAT block or whatever.

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10715
2014-10-17 05:04:02 -07:00
epriestley
247cb94d5a Build AlmanacDevice UI
Summary: Ref T5833. The "uninteresting" part of this object is virtually identical to AlmanacService.

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10714
2014-10-17 05:02:14 -07:00
epriestley
796921021b Build AlmanacService
Summary: Ref T5833. See that task for functional goals and some discussion of design.

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10713
2014-10-17 05:01:57 -07:00
epriestley
384fd24627 Send mail about cart/order changes from Phortune
Summary: Ref T2787. When order statuses change, send merchants and users email about it.

Test Plan: Used `bin/mail` to review mail.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10694
2014-10-13 11:16:44 -07:00
epriestley
214b5b7158 Add cart transactions to Phortune
Summary: Ref T2787. I mostly just want these in place so I can glue emails to them, but they're also useful on their own.

Test Plan: {F216515}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10688
2014-10-13 11:16:27 -07:00
epriestley
2d0ee77bd4 Give Phortune merchants explicit members
Summary:
Ref T2787. Make this a little more concrete with explicit membership instead of a general edit policy. In particular, we need to know who to email when orders happen, and can't reasonably do that with an edit policy.

I imagine this might eventually get more nuanced (e.g., users who can only approve orders vs users who can manage the merchant itself) but that's a long ways away.

Test Plan: {F216284}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10681
2014-10-13 11:13:50 -07:00
epriestley
1112419a97 Support feed and email in Fund
Summary: Ref T5835. Make fund stories publish to feed and send email.

Test Plan: Made edits, etc., saw them in feed and outbound email.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5835

Differential Revision: https://secure.phabricator.com/D10677
2014-10-10 11:29:42 -07:00
epriestley
1e8c314c81 Mostly implement order refunds and cancellations
Summary:
Ref T2787. This has some rough edges but basically works.

  - Users can cancel orders that are in incomplete states (or in complete states, if the application allows them to -- for example, some future application might allow cancellation of billed-but-not-shipped orders).
  - Merchant controllers can partially or fully refund orders from any state after payment.

Test Plan: This is still rough around the edges, but issued Stripe and WePay refunds.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: chad, epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10664
2014-10-08 15:33:25 -07:00
epriestley
19db3fbb60 Allow merchants to disable payment providers and show more UI info
Summary:
Ref T2787.

  - Allow merchants to disable payment providers.
  - Show more useful information about providers on the payments page.
  - Make test vs live more clear.
  - Show merchant status.
  - Add a description to merchants to flesh them out a bit -- the merchant areas of responsibilities seem to be fitting well with accounts, etc.

Test Plan: {F215109}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10662
2014-10-08 08:31:24 -07:00
epriestley
43215bf0bd Make Fund backers render a little more nicely
Summary: Ref T5835. Mostly UI smoothing. Also add a "Risks" field to initiatives.

Test Plan: {F214952}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5835

Differential Revision: https://secure.phabricator.com/D10660
2014-10-08 05:32:42 -07:00
epriestley
608465da1e Track total funding amount on Fund initiatives
Summary: Ref T5835. Show total funding amount and payable merchant on initiatives.

Test Plan: {F214936}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5835

Differential Revision: https://secure.phabricator.com/D10657
2014-10-08 05:31:48 -07:00
epriestley
9c4b8a0fb2 Adjust payment workflows to deal with merchants and configurable providers in Phortune
Summary:
Ref T2787. Builds on D10649 by rebining existing objects (carts, charges, etc) to merchantPHIDs and providerPHIDs instead of an implicit global merchant and weird global artifacts (providerType / providerKey).

Basically:

  - When you create something that users can pay for, you specify a merchant to control where the payment goes.
  - Accounts are install-wide, but payment methods are bound to merchants. This seems to do a reasonable job of balancing usability and technical concerns.
  - Replace a bunch of weird links between objects with standard PHIDs.
  - Improve "add payment method" flow.

Test Plan: Went through the Fund flow with Stripe and WePay, funding an initiative.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10652
2014-10-07 14:41:59 -07:00
epriestley
9aa5a8cb7b Make payment providers a configurable property of Merchants in Phortune
Summary:
Ref T2787. Instead of making providers global configuration, make them a thing on merchants with web configuration.

Payment methods and some of the pyament workflow needs to be retooled a bit after this, but this seemed like a reasonable cutoff point for this diff.

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10649
2014-10-07 14:41:41 -07:00
epriestley
fcd2025a85 Add Merchants to Phortune
Summary:
Ref T2787. Currently, you add payment providers (Stripe, Paypal, etc) in global configuration.

Generally, this approach is cumbersome, limiting, and often hard for users to figure out. It also doesn't provide a natural way to segment payment receivers or provide web access to administrative payment functions like issuing refunds, canceling orders, etc. I think that stuff definitely needs to be in the web UI, and the rule for access to it can't reasonably just be "all administrators" in a lot of reasonable cases.

The only real advantage is that it prevents an attacker from adjusting settings and pointing something at an account they control. But this attack can be mitigated through notifications, some sort of CLI-only merchant lock, payment accounts being relatively identifiable, etc.

So introduce "merchants", which are basically payable entities. An individual merchant will have attached Paypal, Stripe, etc., accounts, and access rules. When you buy something in an application, the merchant to pay is also specified. They also provide an umbrella for dealing with permissions down the line.

This may get a //little// cumbersome because if there are several merchants your saved card information is not shared across them. I think that will be fine in the normal case (most installs will have only one merchant). Even if it isn't and we leave providers global, I think introducing this is the right call from a web UI / permissions point of view. I'll play around with it in the next couple of diffs and figure out exactly where the line goes.

Test Plan: Listed, created, edited, viewed merchants.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10648
2014-10-07 10:55:16 -07:00
epriestley
0beb8228da Give applications control over Phortune cart logic
Summary: Ref T2787. Similar to D10634, give applications more control over the cart workflow. For now this just means they get to pick exit URIs, but in the future they can manage more details of cart behavior.

Test Plan: Funded an initiative and got returned to the initiative instead of dead-ending in Phortune.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10638
2014-10-06 14:19:08 -07:00
epriestley
e9615b74a5 Move Phortune product logic into applications
Summary: Ref T2787. `Product` is currently a fairly heavy object, but as Phortune develops it makes a lot of sense to make it a lighter object and put more product logic in applications. Convert it into a fairly lightweight reference to applications. The idea is that Phortune is mostly providing a cart flow, and applications manage the details of products.

Test Plan: Funded an initiative for $1.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10634
2014-10-06 10:30:06 -07:00
epriestley
f86f9dc512 Make Currency a more formal type
Summary:
Ref T2787. Phortune currently stores a bunch of stuff as `...inUSDCents`. This ends up being pretty cumbersome and I worry it will create a huge headache down the road (and possibly not that far off if we do Coinbase/Bitcoin soon). Even now, it's more of a pain than I figured it would be.

Instead:

  - Provide an application-level serialization mechanism.
  - Provide currency serialization.
  - Store currency in an abstract way (currently, as "1.23 USD") that can handle currencies in the future.
  - Change all `...inUSDCents` to `..asCurrency`.
  - This generally simplifies all the application code.
  - Also remove some columns which don't make sense or don't make sense anymore. Notably, `Product` is going to get more abstract and mostly be provided by applications.

Test Plan:
  - Created a new product.
  - Purchased a product.
  - Backed an initiative.
  - Ran unit tests.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2787

Differential Revision: https://secure.phabricator.com/D10633
2014-10-06 10:26:48 -07:00
James Rhodes
8fbebce501 Implement storage of a host ID and a public key for authorizing Conduit between servers
Summary:
Ref T4209.  This creates storage for public keys against authorized hosts, such that servers can be authorized to make Conduit calls as the omnipotent user.

Servers are registered into this system by running the following command once:

```
bin/almanac register
```

NOTE: This doesn't implement authorization between servers, just the storage of public keys.

Placing this against Almanac seemed like the most sensible place, since I'm imagining in future that the `register` command will accept more information (like the hostname of the server so it can be found in the service directory).

Test Plan: Ran `bin/almanac register` and saw the host (and public key information) appear in the database.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4209

Differential Revision: https://secure.phabricator.com/D10400
2014-10-03 22:52:41 +10:00
epriestley
d67b7f0f47 Correct column mutations for old versions of MySQL
Summary:
Ref T1191. Although I fixed some of the mutations earlier (in D10598), I missed the column mutations under old versions of MySQL. In particular, this isn't valid:

  - `ALTER TABLE ... MODIFY columnName VARCHAR(64) COLLATE binary`

Issue the permitted version of this instead, which is:

  - `ALTER TABLE ... MODIFY columnName VARBINARY(64)`

Also fixed an issue where a clean schema had the wrong nullability for a column in the draft table. Force it to the expected nullability.

The other trick here is around the one column with a FULLTEXT index on it, which needs a little massaging.

Test Plan:
  - Forced my local install to return `false` for utf8mb4 support.
  - Did a clean adjust into `binary` columns.
  - Poked around, added emoji to things.
  - Reverted the fake check and did a clean adjust into `utf8mb4` columns.
  - Emoji survived.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: fabe, epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10627
2014-10-02 14:44:22 -07:00
epriestley
8fa8415c07 Automatically build all Lisk schemata
Summary:
Ref T1191. Now that the whole database is covered, we don't need to do as much work to build expected schemata. Doing them database-by-database was helpful in converting, but is just reudndant work now.

Instead of requiring every application to build its Lisk objects, just build all Lisk objects.

I removed `harbormaster.lisk_counter` because it is unused.

It would be nice to autogenerate edge schemata, too, but that's a little trickier.

Test Plan: Database setup issues are all green.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10620
2014-10-02 09:51:20 -07:00
epriestley
3a644cf6cc Truncate very old, overlong Maniphest mail keys
Summary:
Ref T1191. Long ago, Maniphest generated with 40-character mail keys. These prevent the migration to `bytes20`. We had about 300 of these on secure.phabricator.com from several years ago.

Just truncate them. This adjusts reply-to addresses, but it's very likely that none are relevant anymore.

Test Plan: Ran migration on `secure.phabricator.com` to truncate keys.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10615
2014-10-01 12:43:58 -07:00
epriestley
3629ebebe9 Drop very old schema_version table if it exists
Summary: Ref T1191. This predates the mdoern patch stuff and may exist on very, very old installs. By the time they apply this patch, it's guaranteed it won't matter anymore. Drop it to make the schemata consistent with expectations.

Test Plan: Ran patch on installs with and without the table.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10611
2014-10-01 12:43:20 -07:00
epriestley
0d7489da79 Provide bin/storage quickstart to automate generation of quickstart.sql
Summary:
Ref T1191. Currently, the `quickstart.sql` gets generated in a pretty manual fashion. This is a pain, and will become more of a pain in the world of utf8mb4.

Provide a workflow which does upgrade + adjust + dump + destroy, then massages the output to produce a workable `quickstart.sql`.

Test Plan: Inspected output; I'll test this more throughly before actually generating a new quickstart, but that's some ways away.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10603
2014-10-01 08:22:37 -07:00
epriestley
8bf24f53b3 Destroy surplus columns
Summary:
Ref T1191. This destroys surplus columns:

  - Pholio's transaction comments have a `mockID` column, but this is not used. The `imageID` column is used instead.
  - Phragment has an unused `description` column.
  - Releeph has an unused `summary` column.

Test Plan:
  - Grepped for usage of these columns.
  - Checked that these exist in production, too.
  - Ran upgrades.
  - Added Pholio inline comments.
  - Saw fewer warnings.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10591
2014-10-01 07:54:33 -07:00
epriestley
2880732a49 Generate expected schemata for Search
Summary:
Ref T1191. Notable:

  - Drops a very old saved query table. See comments inline: plan was to remove it after a year. It's been ~a year and two weeks.
  - This has our only fulltext index. I'm not supporting that formally for now, but left a note.
  - This has our only MyISAM table. I'm not supporting that explicitly for now, but it shouldn't affect anything. I may deal with this in the future.
  - These tables don't actually write directly via Lisk, so there's some fiddling to get the schemata right.

Test Plan: Down to ~250 warnings. No more surplus databases or tables.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10589
2014-10-01 07:53:35 -07:00
epriestley
152a62db7a Generate expected Ponder schemata
Summary:
Ref T1191.

  - Removes ponder comment table. This was migrated a very long time ago.

Test Plan:
  - Grepped for removed table.
  - Saw ~100 fewer issues in web UI.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10582
2014-10-01 07:37:14 -07:00
epriestley
ac9182af58 Generate expected Project schemata
Summary:
Ref T1191. Notes:

  - Drops the project affiliation table. This is a very old membership table which was migrated to edges.
  - Drops the subproject table. This is a very old table for a removed feature.

Test Plan:
  - Grepped for dropped tables.
  - Saw ~100 fewer setup issues.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10581
2014-10-01 07:37:01 -07:00
epriestley
098d0d93d6 Generate expected schemata for User/People tables
Summary:
Ref T1191. Some notes here:

  - Drops the old LDAP and OAuth info tables. These were migrated to the ExternalAccount table a very long time ago.
  - Separates surplus/missing keys from other types of surplus/missing things. In the long run, my plan is to have only two notice levels:
    - Error: something we can't fix (missing database, table, or column; overlong key).
    - Warning: something we can fix (surplus anything, missing key, bad column type, bad key columns, bad uniqueness, bad collation or charset).
    - For now, retaining three levels is helpful in generating all the expected scheamta.

Test Plan:
  - Saw ~200 issues resolve, leaving ~1,300.
  - Grepped for removed tables.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10580
2014-10-01 07:36:47 -07:00
epriestley
9be2bf2119 Generate expected schemata for Releeph
Summary:
Ref T1191.

  - This drops two tables.
  - Both tables were migrated to transactions a very long time ago and no longer have readers or writers.

Test Plan: Saw ~150 fewer warnings.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10576
2014-09-28 15:12:41 -07:00
epriestley
84568eba84 Generate expected schemata for Maniphest
Summary:
Ref T1191.

  - Adds support for custom fields.
  - Adds support for partial indexes (indexes on a prefix of a column).
  - Drops old auxiliary storage table: this was moved to custom field storage about a year ago.
  - Drops old project table: this was moved to edges about two months ago.

Test Plan:
  - Viewed web UI, saw fewer issues.
  - Used `grep` to verify no readers/writers for storage or project table.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10526
2014-09-19 11:46:44 -07:00
epriestley
6bfe8b5984 Generate expected schemata for Calendar
Summary:
Ref T1191.

  - There was a varchar(50) column. I changed it to `text64`, since this length is unusual.
  - There was an int(3) column. I changed it to `int32`, since this length is unusual.

Test Plan: Ran migrations, saw warnings disappear from config tool.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10524
2014-09-19 11:46:20 -07:00
epriestley
a42e4a867e Remove SlowvoteComment and storage
Summary: Ref T1191. This was migrated to transactions a very long time ago.

Test Plan: Ran migration, grepped, left comments in Slowvote.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10523
2014-09-19 05:45:36 -07:00
epriestley
263dbe7bfe Drop old Audit tables; make markup cache binary
Summary:
Ref T1191.

  - Fixes T6096. We've migrated away from this table in T4896. The data is now in the transaction table. There have been no reads or writes to this table for some time and I haven't seen any issues from users.
  - Fixes T6097. Same deal as above. The data is now in the transaction comment table.
  - Fixes T6100. This cache is safe to wipe out, since it's purely read-through. Wiping it will make the migration faster. The column type change fixes storage of PHP serialized objects in a text column.

Test Plan:
  - Ran migrations.
  - Observed some yellow go blue on the Database Status screen.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6096, T6100, T6097, T1191

Differential Revision: https://secure.phabricator.com/D10520
2014-09-19 05:44:29 -07:00
Bob Trahan
3238f1e091 Projects - add "lock membership", which prevents people from leaving
Summary:
Fixes T5603. Puts the toggling of locking membership into the editor so we get exceptions and all that.

I think the dialogue when you try to leave a project that is locked could be a little better maybe? Right now it just says "You can't leave" and "The membership is locked" more or less; should I surface a link to the policy stuff there too?

Test Plan:
 - made a project, toggled the "lock" setting, observed stickiness and good transactions being made
 - locked a project and tried to leave as a non-editor - got a dialogue letting me know i couldn't
 - locked a project and tried to leave as an editor - left successfully

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T5603

Differential Revision: https://secure.phabricator.com/D10508
2014-09-18 11:00:50 -07:00
epriestley
298604c9d3 Rename "beta" to "prototype" and document support policy
Summary:
Fixes T6084. Changes:

  - Rename `phabricator.show-beta-applications` to `phabricator.show-prototypes`, to reinforce that these include early-development applications.
  - Migrate the config setting.
  - Add an explicit "no support" banner to the config page.
  - Rename "Beta" to "Prototype" in the UI.
  - Use "bomb" icon instead of "half star" icon.
  - Document prototype applications in more detail.
  - Explicitly document that we do not support these applications.

Test Plan:
  - Ran migration.
  - Resolved "obsolete config" issue.
  - Viewed config setting.
  - Browsed prototypes in Applications app.
  - Viewed documentation.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T6084

Differential Revision: https://secure.phabricator.com/D10493
2014-09-17 18:25:57 -07:00
epriestley
ffa017630f There are too many m's 2014-09-14 16:30:46 -07:00
epriestley
7987b4b189 [Later] Drop legacyCommentID column from DifferentialTransactionComment
Summary: Ref T2222. No callsites. Holding until we're more clearly stable.

Test Plan: `grep`

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, aran

Maniphest Tasks: T2222

Differential Revision: https://secure.phabricator.com/D8241
2014-09-14 16:29:15 -07:00
epriestley
09fb5667cc Allow users to back initiatives in Fund
Summary:
Ref T5835. This is still completely made up (no payment integration), but you can "back" an initiative, type a number in the box, and generate a database row. You can then seach for backers and things you've backed and such.

Notable changes:

  - Renamed "FundBacking" to "FundBacker". The former name was sort of because you can back things multiple times, but stuff like `$backings` was just too weird.
  - I think that's it?

Test Plan:
  - Backed an initiative.
  - Viewed that I became a backer.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5835

Differential Revision: https://secure.phabricator.com/D10486
2014-09-12 06:31:11 -07:00
epriestley
e4f399b9fa Scaffolding for Fund
Summary:
Ref T5835. This is all pretty boilerplate, and does not interact with Phortune at all yet.

You can create "Initiatives", which have a title and description, and support most of the expected infrastructure (policies, transactions, mentions, edges, appsearch, remakrup, etc).

Only notable decisions:

  - Initiatives have an explicit owner. I think it's good to have a single clearly-responsible user behind an initiative.
  - I think that's it?

Test Plan:
  - Created an initiative.
  - Edited an initiative.
  - Changed application policy defaults.
  - Searched for initiatives.
  - Subscribed to an initiative.
  - Opened/closed an initiative.
  - Used `I123` and `{I123}` in remarkup.
  - Destroyed an initiative.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5835

Differential Revision: https://secure.phabricator.com/D10481
2014-09-11 13:38:58 -07:00
Joshua Spence
d6671cbbe1 Don't use parentheses for echo unnecessarily
Summary: As established in D10122.

Test Plan: I basically ran `arc lint --everything --apply-patches`.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D10437
2014-09-08 10:03:19 +10:00
epriestley
8038af4bd5 Fix old image macros and memes for logged out users
Summary:
Fixes T6013. Old image macros/memes never had the file edge written.

We also never wrote file edges for audio.

Finally, the meme controller didn't allow public access.

Write edges for images and audio, perform a migration to populate the historic ones, and make the Editor keep them up to date going forward.

Test Plan:
  - Updated image, saw new image attach and old image detach.
  - Updated audio, saw new audio attach and old audio detach.
  - Ran migration.
  - Viewed memes as a logged-out user.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6013

Differential Revision: https://secure.phabricator.com/D10411
2014-09-04 12:50:51 -07:00
epriestley
6be8d65763 Convert two missed phutil_utf8_shorten() callsites
Summary: Fixes T6006. These didn't get caught by D10392.

Test Plan: Forced migration to re-run; ran SSH commands against Phabricator.

Auditors: btrahan
2014-08-30 07:20:35 -07:00
Bob Trahan
6f246bd351 Daemons - add a config check for out of date daemon environment
Summary: Fixes T4881.

Test Plan: made a config change, saw the issue, restarted daemons and it went away

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4881

Differential Revision: https://secure.phabricator.com/D10339
2014-08-22 14:52:36 -07:00
James Rhodes
efadfbbc97 Implement build generations in Harbormaster
Summary:
Ref T5932.  Ref T5936.  This implements build generations in Harbormaster, which provides the infrastructure required to both show users the previous states of restarted builds and to allow users to forcefully abort builds (and their targets).

You can view previous generations of a build by adding `?g=<n>` to the URI, but this isn't exposed in the UI anywhere yet.

Test Plan: Ran a build plan with a Sleep step in it.  Reconfigured it for various sleep times and viewed previous generations of the build after restarting it.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T5932, T5936

Differential Revision: https://secure.phabricator.com/D10321
2014-08-21 22:55:24 +10:00
James Rhodes
6f85c22faf Create an index on harbormaster_buildlog to reduce page load times
Summary: Resolves T5895.  This reduces page load times significantly when looking at builds.

Test Plan: Viewed a build, saw the page load a lot faster.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T5895

Differential Revision: https://secure.phabricator.com/D10286
2014-08-20 09:19:57 +10:00
James Rhodes
26f283fe21 Implement passphrase.query for querying credentials
Summary: Resolves T5868.  This implements `passphrase.query` and a mechanism for allowing Conduit access to credentials.

Test Plan: Tested locally.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: talshiri, epriestley, Korvin

Maniphest Tasks: T5868

Differential Revision: https://secure.phabricator.com/D10262
2014-08-16 22:41:03 +10:00
epriestley
607e99490b Migrate "cancdn" to "canCDN" in the database
Summary: Ref T5884. We migrated with "canCDN" and then had live writes with "cancdn". Move everything to "canCDN" for consistency.

Test Plan: Ran migration, verified DB only has "canCDN" afterward.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5884

Differential Revision: https://secure.phabricator.com/D10273
2014-08-15 11:07:40 -07:00
epriestley
e616f166ae Provide a setup warning about using the default MySQL stopword file
Summary:
Fixes T2605.

  - Add a setup warning about the stopword file.
  - Provide a simpler stopword file.

Test Plan:
  - Hit setup warning.
  - Resolved it according to instructions.
  - Added "various" to a task, then searched for it, found the task.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T2605

Differential Revision: https://secure.phabricator.com/D10258
2014-08-13 15:34:09 -07:00
epriestley
94389fcd9f Allow projects to be filtered by icon and color
Summary: Ref T5819. Implements basic icon and color filtering for projects.

Test Plan: {F189350}

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T5819

Differential Revision: https://secure.phabricator.com/D10230
2014-08-12 08:04:38 -07:00
James Rhodes
efc82c727b Measure how long build targets take in Harbormaster
Summary:
Ref T1049.  This keeps track of how long a build target takes to execute in Harbormaster and displays it in the build view page.  I'm not sure whether "Started" is really that useful once the target has completed?

Also, I change the name of the time taken depending on whether or not the target has completed; if it's still in progress it's called "Elapsed" and if it's completed then it's "Duration".  The primary reason for this is that "Duration" sounds like post tense, whereas "Elapsed" is current tense.  I'm not sure whether this is okay or not?

Test Plan: Ran a Sleep build step and saw the target dates / times appear correctly.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: talshiri, epriestley, Korvin

Maniphest Tasks: T5824, T1049

Differential Revision: https://secure.phabricator.com/D10174
2014-08-12 08:34:43 +10:00
epriestley
31e1653a4e Convert pushlog and refcursor to BLOB storage
Summary:
Fixes T5840. Some time ago I incorrectly believed that `latin1_bin` collation was synonymous with "binary". It is not, and does not permit UTF8 characters outside of BMP, among other sequences.

These two tables currently have `LONGTEXT` columns which should be `LONGBLOB`. The table design is explicilty intended to accommodate invalid/unreasonably long ref names, but the collation prevents this from working properly.

After T1191, we'll have a general system for resolving this, but a user hit an issue yesterday (T5840) with a brnach name containing Chinese characters.

Test Plan:
  - Tried emoji inserts into both tables, was rebuffed.
  - Ran migration.
  - Performed emoji inserts into both tables.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5840

Differential Revision: https://secure.phabricator.com/D10217
2014-08-11 12:29:46 -07:00
Mukunda Modell
25ae4c458d Protect file data with a one-time-token
Test Plan: currently untested work in progress

Reviewers: #blessed_reviewers, epriestley

Subscribers: rush898, aklapper, Korvin, epriestley

Projects: #wikimedia

Maniphest Tasks: T5685

Differential Revision: https://secure.phabricator.com/D10054
2014-08-11 07:32:17 -07:00
epriestley
24a6eeb8d8 Allow the workboard backlog column to be reordered
Summary:
Fixes T5677.

  - Instead of using `sequence == 0` to mean "this is the backlog column", flag the column explicitly.
  - Migrate existing sequence 0 columns to have the flag.
  - Add the flag when initializing or copying a board.
  - Remove special backlog logic when reordering columns.

Test Plan:
  - Migrated columns, viewed some boards, they looked identical.
  - Reordered the backlog column a bunch of times (first, last, middle, dragged other stuff around).
  - Added tasks to a project, saw them show up in the reordered backlog.
  - Initialized a new board and saw a backlog column show up.
  - Copied an existing board and saw the backlog column come over.
  - Tried to hide a backlog column.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5677

Differential Revision: https://secure.phabricator.com/D10189
2014-08-08 15:50:36 -07:00
epriestley
09868271bd Move board relationships to dedicated storage
Summary:
Fixes T5476. Using edges to store which objects are on which board columns ends up being pretty awkward. In particular, it makes T4807 very difficult to implement.

Introduce a dedicated `BoardColumnPosition` storage.

This doesn't affect ordering rules (T4807) yet: boards are still arranged by priority. We just read which tasks are on which columns out of a new table.

Test Plan:
  - Migrated data, then viewed some boards. Saw exactly the same data.
  - Dragged tasks from column to column.
  - Created a task directly into a column.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5476

Differential Revision: https://secure.phabricator.com/D10160
2014-08-06 15:09:09 -07:00
Joshua Spence
8fd098329b Rename AphrontQueryException subclasses
Summary: Ref T5655. Depends on D10149.

Test Plan: Ran `arc unit`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin, hach-que

Maniphest Tasks: T5655

Differential Revision: https://secure.phabricator.com/D10150
2014-08-06 07:51:21 +10:00
epriestley
89b942c183 Move Audit to proper Subscriptions
Summary:
Ref T4896. Currently, subscriptions to commits are stored as auditors with a special "CC" type.

Instead, use normal subscriptions storage, reads and writes.

Test Plan:
  - Ran migration and verified data still looked good.
  - Viewed commits in UI and saw "subscribers".
  - Saw "Automatically Subscribed", clicked Subscribe/Unsubscribe on a non-authored commit, saw subscriptions update.
  - Pushed a commit through Herald rules and saw them trigger subscriptions and auditors.
  - Used "Add CCs".
  - Added CCs with mentions.

Reviewers: btrahan, joshuaspence

Reviewed By: btrahan, joshuaspence

Subscribers: epriestley

Maniphest Tasks: T4896

Differential Revision: https://secure.phabricator.com/D10103
2014-08-02 00:06:13 -07:00
James Rhodes
dfa9b27a94 Use tabs on build targets and allow build steps to have a description
Summary:
Ref T1049. This uses tabs on build targets to hide the configuration details and variables by default, instead promoting the target name, it's status and a description of the build step.  The description is a new field on each build step.

The primary advantage of having a description on build steps is that DevOps can configure appropriate description information (including any troubleshooting information for build failures) on build steps, and developers who have builds fail against their code review can then look at this information.

Test Plan: Viewed a build plan and saw the appropriate information.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D10093
2014-08-01 08:09:15 +10:00
epriestley
fa2fcc7852 Fix missed migration constant from PHIDType rename
Summary: See D9986, we missed this one.

Test Plan: `grep`

Reviewers: joshuaspence, btrahan, bizrad6

Reviewed By: bizrad6

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D10098
2014-07-31 11:30:40 -07:00
James Rhodes
cad41ea294 Implement build simulation; convert Harbormaster to be purely dependency based
Summary:
Depends on D9806.  This implements the build simulator, which is used to calculate the order of build steps in the plan editor.  This includes a migration script to convert existing plans from sequential based to dependency based, and then drops the sequence column.

Because build plans are now dependency based, the grippable and re-order behaviour has been removed.

Test Plan: Tested the migration, saw the dependencies appear correctly.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9847
2014-07-31 11:39:49 +10:00
epriestley
f965126dc4 Migrate audit comments to transactions
Summary:
Ref T4896. Depends on D10052. This is the major/scary migration, but not really so bad. It is substantially similar to D8210, but less complex because there are fewer actions here.

This moves `PhabricatorAuditComment` storage to `PhabricatorAuditTransaction`, then reads `PhabricatorAuditComment`s as a proxy around the new objects.

Test Plan:
  - Before migrating, browsed around. Nothing appeared broken.
  - Migrated cleanly.
  - Viewed old transactions (inlines, comments, accept/reject/etc, add auditors, add ccs, implicit CCs).
  - Added all of those comment types.
  - Edited a draft.
  - Deleted a draft.
  - Spot checked the database for sanity.

Reviewers: btrahan, joshuaspence

Reviewed By: joshuaspence

Subscribers: epriestley

Maniphest Tasks: T4896

Differential Revision: https://secure.phabricator.com/D10055
2014-07-28 15:00:46 -07:00