1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-19 03:50:54 +01:00
Commit graph

446 commits

Author SHA1 Message Date
epriestley
ea954c37e4 Add dashboard panel types
Summary: Ref T3583. These will be the primary class carrying panel implementations.

Test Plan:
{F149125}

{F149126}

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T3583

Differential Revision: https://secure.phabricator.com/D8912
2014-04-30 14:28:20 -07:00
epriestley
17709bc167 Add multi-factor auth and TOTP support
Summary:
Ref T4398. This is still pretty rough and isn't exposed in the UI yet, but basically works. Some missing features / areas for improvement:

  - Rate limiting attempts (see TODO).
  - Marking tokens used after they're used once (see TODO), maybe. I can't think of ways an attacker could capture a token without also capturing a session, offhand.
  - Actually turning this on (see TODO).
  - This workflow is pretty wordy. It would be nice to calm it down a bit.
  - But also add more help/context to help users figure out what's going on here, I think it's not very obvious if you don't already know what "TOTP" is.
  - Add admin tool to strip auth factors off an account ("Help, I lost my phone and can't log in!").
  - Add admin tool to show users who don't have multi-factor auth? (so you can pester them)
  - Generate QR codes to make the transfer process easier (they're fairly complicated).
  - Make the "entering hi-sec" workflow actually check for auth factors and use them correctly.
  - Turn this on so users can use it.
  - Adding SMS as an option would be nice eventually.
  - Adding "password" as an option, maybe? TOTP feels fairly good to me.

I'll post a couple of screens...

Test Plan:
  - Added TOTP token with Google Authenticator.
  - Added TOTP token with Authy.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4398

Differential Revision: https://secure.phabricator.com/D8875
2014-04-28 09:27:11 -07:00
epriestley
f42ec84d0c Add "High Security" mode to support multi-factor auth
Summary:
Ref T4398. This is roughly a "sudo" mode, like GitHub has for accessing SSH keys, or Facebook has for managing credit cards. GitHub actually calls theirs "sudo" mode, but I think that's too technical for big parts of our audience. I've gone with "high security mode".

This doesn't actually get exposed in the UI yet (and we don't have any meaningful auth factors to prompt the user for) but the workflow works overall. I'll go through it in a comment, since I need to arrange some screenshots.

Test Plan: See guided walkthrough.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4398

Differential Revision: https://secure.phabricator.com/D8851
2014-04-27 17:31:11 -07:00
lkassianik
9a827096a7 Ability to close poll
Summary: Fixes T3566 List of poll actions should include ability to close an open poll or reopen a closed poll.

Test Plan: Poll author should be able to close/reopen poll. Non-author should get policy screen when attempting to close/reopen poll.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T3566

Differential Revision: https://secure.phabricator.com/D8846
2014-04-24 12:02:56 -07:00
epriestley
2ac8457cb9 [Later] Drop the project profile table
Summary: Going to sit on this for a bit so we can fall back to it if needbe, but this table no longer has any reads or writes in the application.

Test Plan: Applied patch locally and poked around.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: aran

Differential Revision: https://secure.phabricator.com/D8190
2014-04-24 08:15:24 -07:00
epriestley
4bbd2d5203 Make an old migration a little more robust
Summary: See https://github.com/facebook/phabricator/pull/507

Test Plan: This is hard to test since the migration no longer runs against HEAD, but pull 507 strongly implies this is the correct fix.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D8837
2014-04-23 14:22:02 -07:00
epriestley
1a3ac09975 Add "requestedObjectPHID" to ReleephRequest
Summary:
Ref T3551. Currently, ReleephRequests don't have a direct concept of the //object// being requested. You can request `D123`, but that is just a convenient way to write `rXyyyy`.

When the UI wants to display information about a revision, it deduces it by examining the commit.

This is primarily an attack on T3551, so we don't need to load <commit -> edge -> revision> (in an ad-hoc way) to get revisions. Instead, when you request a revision we keep track of it and can load it directly later.

Later, this will let us do more things: for example, if you request a branch, we can automatically update the commits (as GitHub does), etc. (Repository branches will need PHIDs first, of course.)

This adds and populates the column but doesn't use it yet. The second part of the migration could safely be run while Phabricator is up, although even for Facebook this table is probably quite small.

Test Plan:
  - Ran migration.
  - Verified existing requests associated sensibly.
  - Created a new commit request.
  - Created a new revision request.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T3551

Differential Revision: https://secure.phabricator.com/D8822
2014-04-20 11:55:18 -07:00
epriestley
4918773afe Drop nonsense buildStatus field from Buildable
Summary:
Ref T4809. Buildables currently have buildStatus and buildableStatus. Neither are used, and no one knows why we have two.

I'm going to use buildableStatus shortly, but buildStatus is meaningless; burn it.

Test Plan: `grep`, examined similar get/set calls, created a new buildable, ran storage upgrade.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4809

Differential Revision: https://secure.phabricator.com/D8796
2014-04-17 16:01:06 -07:00
epriestley
ab7d89edc8 Use better secrets in generating account tokens
Summary:
When we generate account tokens for CSRF keys and email verification, one of the inputs we use is the user's password hash. Users won't always have a password hash, so this is a weak input to key generation. This also couples CSRF weirdly with auth concerns.

Instead, give users a dedicated secret for use in token generation which is used only for this purpose.

Test Plan:
  - Ran upgrade scripts.
  - Verified all users got new secrets.
  - Created a new user.
  - Verified they got a secret.
  - Submitted CSRF'd forms, they worked.
  - Adjusted the CSRF token and submitted CSRF'd forms, verified they don't work.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D8748
2014-04-10 11:45:10 -07:00
epriestley
582ec54465 Add a checkbox to the LDAP auth configuration UI to "Always Search"
Summary: Fixes T3208. This forces us to bind+search even if there are no anonymous credentials.

Test Plan: Checked the box, saved the form. Unchecked the box, saved the form. LDAP??

Reviewers: Firehed

Reviewed By: Firehed

Subscribers: epriestley

Maniphest Tasks: T3208

Differential Revision: https://secure.phabricator.com/D8723
2014-04-08 11:36:23 -07:00
epriestley
847b7977c1 Add semi-generic rate limiting infrastructure
Summary:
This adds a system which basically keeps a record of recent actions, who took them, and how many "points" they were worth, like:

  epriestley email.add 1 1233989813
  epriestley email.add 1 1234298239
  epriestley email.add 1 1238293981

We can use this to rate-limit actions by examining how many actions the user has taken in the past hour (i.e., their total score) and comparing that to an allowed limit.

One major thing I want to use this for is to limit the amount of error email we'll send to an email address. A big concern I have with sending more error email is that we'll end up in loops. We have some protections against this in headers already, but hard-limiting the system so it won't send more than a few errors to a particular address per hour should provide a reasonable secondary layer of protection.

This use case (where the "actor" needs to be an email address) is why the table uses strings + hashes instead of PHIDs. For external users, it might be appropriate to rate limit by cookies or IPs, too.

To prove it works, I rate limited adding email addresses. This is a very, very low-risk security thing where a user with an account can enumerate addresses (by checking if they get an error) and sort of spam/annoy people (by adding their address over and over again). Limiting them to 6 actions / hour should satisfy all real users while preventing these behaviors.

Test Plan:
This dialog is uggos but I'll fix that in a sec:

{F137406}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D8683
2014-04-03 11:22:38 -07:00
epriestley
f239d81c1c Allow very long notes on flags
Summary: Fixes T4703. This is a VARCHAR(255) for no particular reason.

Test Plan: {F136160}

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T4703

Differential Revision: https://secure.phabricator.com/D8652
2014-03-30 19:51:46 -07:00
epriestley
d3dbbec88d Rename Releeph "Project" transactions to "Product"
Summary: Ref T3549. This table isn't written to yet; rename it and the DAOs and modernize the history controller.

Test Plan: Viewed history page for a product.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T3549

Differential Revision: https://secure.phabricator.com/D8633
2014-03-29 09:15:09 -07:00
Bob Trahan
655ac9927f Workboards - add column detail page
Summary: followup to D8544. This ends up creating an editor + transactions to get the job done.

Test Plan: made a column - saw a nice created transaction. edited the name - saw a nice name edit. deleted the column - saw a deleted transaction, updated "deleted" ui, and hte action change to activate. "Activated" the column and saw a transaction and updated UI. Tried to delete a column with tasks in it and got an error.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D8620
2014-03-26 14:40:47 -07:00
epriestley
543a313387 Skip a very old project reindex migration
Summary:
During migration of very old installs, this script no longer runs properly since at HEAD it can't index against older schemas.

Since it's pretty fluff, just toss it. Installs can run `bin/search index --type PROJ` after finishing migrations to achieve the same effect, if necessary.

Test Plan: eyeballed it

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D8619
2014-03-26 13:51:49 -07:00
epriestley
a5f55d506f Provide a real object ("PhabricatorRepositoryPushEvent") to represent an entire push transaction
Summary:
Ref T4677. Currently, we record individual actions in a push as PhabricatorRepositoryPushLogs, but tie them together only loosely with a `transactionKey`.

Provide a real PushEvent object, and move some of the denormalized fields to it. This primarily just gives us more robust infrastructure for building, e.g., email about pushes, for T4677, since we can act on real PHIDs rather than passing awkward identifiers around.

Test Plan:
  - Performed migration.
  - Looked at database for consistency.
  - Browsed/queried push logs.
  - Pushed a bunch of stuff.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4677

Differential Revision: https://secure.phabricator.com/D8615
2014-03-26 13:51:06 -07:00
epriestley
d6b937ca27 Allow external systems to send messages to build targets
Summary:
Ref T1049. Allows external systems to send a message to a build target. The primary intended use case is:

  - You make an HTTP request to Jenkins.
  - The build goes into a "waiting" state.
  - Later, Jenkins calls `harbormaster.sendmessage` to report that the target passed or failed.
  - The build continues as appropriate.

This is deceptively complicated because:

  - There are a lot of race concerns. We might get a message back from an external system before it even responds to the request we made. We want to make sure we process these messages no matter when we receive them.
  - These messages need to be sent to a build target (vs a build or buildable) because we'll get into trouble with parallelization later on otherwise (Jenkins is told to do 3 builds; we can't tell which ones failed or what overall state is unless the message are sent to targets).
  - I initially thought about implementing this as a separate "Wait for a response from an external system" build step. This gets a lot more complicated for users once we do parallelization, though. Particularly, in the case where you've told Jenkins to do 3 builds, the three "wait" steps need to know which target they're waiting for (and jenkins needs to know some unique identifier for each target). So this pretty much boils down to a more complicated, more error-prone version of using target PHIDs.

This makes the already-muddy Build UI a bit worse, but it needs a general clarity pass anyway (it's showing way too much uninteresting data, and should show a better summary of results instead).

Test Plan:
  - This doesn't really do anything interesting yet.
  - Used Conduit to send messages to build plans.
  - Viewed the messages on the build screen.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D8604
2014-03-25 16:11:28 -07:00
epriestley
cec8d10731 Rename concrete Harbormaster step implementations
Summary: Ref T1049. For consistency, rename these to "Harbormaster...".

Test Plan: Ran migration, ran builds, everything still works fine.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D8602
2014-03-25 16:09:51 -07:00
epriestley
a246c85c6b Use ApplicationTransactions and CustomField to implement build steps
Summary:
Ref T1049. Fixes T4602. Moves all the funky field stuff to CustomField. Uses ApplicationTransactions to apply and record edits.

This makes "artifact" fields a little less nice (but still perfectly usable). With D8599, I think they're reasonable overall. We can improve this in the future.

All other field types are better (e.g., fixes weird bugs with "bool", fixes lots of weird behavior around required fields), and this gives us access to many new field types.

Test Plan:
Made a bunch of step edits. Here's an example:

{F133694}

Note that:

  - "Required" fields work correctly.
  - the transaction record is shown at the bottom of the page.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4602, T1049

Differential Revision: https://secure.phabricator.com/D8600
2014-03-25 16:08:40 -07:00
epriestley
0a76d82a7c Use string constants, not integer constants, to represent task status internally
Summary:
Ref T1812. I think integer constants are going to be confusing and error prone for users to interact with. For example, because we use 0-5, adding a second "open" status like "needs verification" without disrupting the existing statuses would require users to define a status with, e.g., constant `6`, but order it between constants `0` and `1`. And if they later remove statuses, they need to avoid reusing existing constants.

Instead, use more manageable string constants like "open", "resolved", etc.

We must migrate three tables:

  - The task table itself, to update task status.
  - The transaction table, to update historic status changes.
  - The saved query table, to update saved queries which specify status sets.

Test Plan:
  - Saved a query with complicated status filters.
  - Ran migrations.
  - Looked at the query, at existing tasks, and at task transactions.
  - Forced migrations to run again to verify idempotentcy/safety.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1812

Differential Revision: https://secure.phabricator.com/D8583
2014-03-25 13:58:14 -07:00
Bob Trahan
809e5a0389 Workboards - let users delete columns
Summary: Fixes T4408. I had to add a "status" to colum. I think we'll need this once we get fancier anyway but for now we have "active" and deleted.

Test Plan: deleted a column. noted reloaded workboard with all those tasks back in the default colun. loaded a task and saw the initial transaction had a "Disabled" icon next to the deleted workboard. also saw the new transaction back to the default column worked.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4408

Differential Revision: https://secure.phabricator.com/D8544
2014-03-18 10:40:31 -07:00
epriestley
5b2887b69b Add "Date Updated" query fields for Maniphest
Summary:
Fixes T4637.

  - We already allow you to order by this column but don't have a key on it. Add one.
  - Expose UI for querying on ranges.

Test Plan:
  - Ran some queries, got reasonable-looking results and no table scans.

Reviewers: btrahan, bigo

Reviewed By: bigo

Subscribers: bigo, epriestley

Maniphest Tasks: T4637

Differential Revision: https://secure.phabricator.com/D8557
2014-03-17 15:53:07 -07:00
epriestley
be92bf182f Drop Maniphest legacy transaction table
Summary: Ref T2217. I'll hold this for a month or so, but once we're confident the migration didn't ruin anything we should nuke this old data -- it's just an insurance policy against discovering migration issues.

Test Plan: Will run in a month or so.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, aran

Maniphest Tasks: T2217

Differential Revision: https://secure.phabricator.com/D7104
2014-03-12 06:04:45 -07:00
epriestley
2dbfb1d5fb Remove DifferentialComment
Summary: Ref T2222. Remove this; no more callsites.

Test Plan: `grep`

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2222

Differential Revision: https://secure.phabricator.com/D8477
2014-03-11 13:02:33 -07:00
epriestley
03abde0b25 Fix Diviner links to articles by title
Summary:
Ref T988. This fixes the biggest current problem with Diviner, which is dead links to articles.

In the new Diviner, articles can have both a "name" (derived from the file name, and used in the URI) and a "title" (optional, specified explicitly). For example, we have one document with the name "feedback" and the title "Give Feedback! Get Support!".

On disk, we want to use the name for the actual file where the text lives ("feedback.diviner"). We also want to use the name in the URI, to generate a clean URI and to allow us to retitle the document slightly without breaking links to it (for example, we renamed the "Backup" document to "Backups and Migrations").

However, when displaying the article we want to use the title.

Currently, you can //only// link to the name, not the title. This is inconvenient:

  - We have a bunch of existing docs which link to titles.
  - It's natural/intuitive to link to titles.
  - Linking to titles makes it easier/cheaper to generate documentation, because we don't need to be able to resolve things at render time.

To remedy this, allow links to target either names or titles. If we miss on a name query, we'll do a title query. This is implemented with a slug hash to allow approximately correct titles (wrong case/spacing/punctuation, e.g.) and sidestep all the UTF8/column length issues.

(In the long run, atom resolution should theoretically be more sophistiated than it is now, and we should do render-time lookups on at least some documents to catch bad links. However, this is fairly complicated and a relatively advanced feature, and I think allowing links to titles is desirable no matter what.)

Test Plan: The user documentation book now has valid links to articles when the titles and names differ.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T988

Differential Revision: https://secure.phabricator.com/D8407
2014-03-05 12:07:26 -08:00
epriestley
bca0ad8fdd Make "EditPro" controller work with diff updates
Summary:
Ref T2222. Make the "EditPro" controller accommodate diff updates, and support the transaction type. This one is pretty straightforward.

Also make `revisionPHID` in the comments table nullable to fix the "Edit" action.

Test Plan:
  - Created new revision.
  - Updated revision.
  - Tried to do some invalid stuff.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2222

Differential Revision: https://secure.phabricator.com/D8376
2014-02-28 16:49:22 -08:00
epriestley
cd7171ec6e Migrate old AuxiliaryField storage to modern CustomField storage
Summary: Ref T2222. Ref T3886. Differential has a legacy storage table for auxiliary fields; move the data to modern storage.

Test Plan:
  - Ran migration.
  - Verified fields still worked properly afterward (view, edit, etc).

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T3886, T2222

Differential Revision: https://secure.phabricator.com/D8355
2014-02-26 16:52:30 -08:00
Joshua Spence
6270114767 Various linter fixes.
Summary:
- Removed trailing newlines.
- Added newline at EOF.
- Removed leading newlines.
- Trimmed trailing whitespace.
- Spelling fix.
- Added newline at EOF

Test Plan: N/A

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: hach-que, chad, Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D8344
2014-02-26 12:44:58 -08:00
epriestley
0f0673b9e5 Remove "dateCommitted" field from DifferentialRevision
Summary: Ref T2222. This is obsolete and no longer used. We could deduce it from transactions or commits in modern Phabricator if we wanted it. We may implement a more general mechanism for T4434.

Test Plan: `grep`

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2222

Differential Revision: https://secure.phabricator.com/D8330
2014-02-25 12:36:14 -08:00
epriestley
70b008d18d Add test coverage that our definition of BMP agrees with MySQL
Summary:
Ref T1191. Test that MySQL's rules match those of `phutil_is_utf8_with_only_bmp_characters()`:

  - Build a string with //every// character that we consider to be a BMP character.
  - Write it into MySQL.
  - Read it back out.
  - Make sure MySQL didn't truncate it.

Test Plan: Ran unit test. This test runs pretty quickly (50ms), the string with every character isn't all that enormous.

Reviewers: btrahan, arice

Reviewed By: arice

CC: chad, arice, aran

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D8314
2014-02-23 16:20:38 -08:00
Bob Trahan
dcd7a316d2 Differential - add DifferentialDraft to track whether revisions have draft feedback or not
Summary: ...do it somewhat generically, so we could fairly easily add this to other applications. Fixes T3496. I got a wee bit lazy and decided not to migrate existing drafts. My excuses aside from laziness are doing it this way will let us see if anyone complains, we can always do a migration later if people do complain, and there's likely to be a lot of garbage data for older / bigger installs, and the migration didn't seem worth itgiven it would also likely be expensive in these cases.

Test Plan: made a draft inline comment on DX and observed DX had a note icon on Differential home page. made a draft comment on DX and observed DX had a note icon on Differential home page. deleted a draft inline comment and noted icon disappeared from Differential homepage. Submitted a draft comment + inline comment and noted icon disappeared.

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T3496

Differential Revision: https://secure.phabricator.com/D8275
2014-02-18 16:25:16 -08:00
epriestley
b96ab5aadf Modernize VCS password storage to use shared hash infrastructure
Summary: Fixes T4443. Plug VCS passwords into the shared key stretching. They don't use any real stretching now (I anticipated doing something like T4443 eventually) so we can just migrate them into stretching all at once.

Test Plan:
  - Viewed VCS settings.
  - Used VCS password after migration.
  - Set VCS password.
  - Upgraded VCS password by using it.
  - Used VCS password some more.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4443

Differential Revision: https://secure.phabricator.com/D8272
2014-02-18 14:09:36 -08:00
epriestley
5778627e41 Provide more storage space for password hashes and migrate existing hashes to "md5:"
Summary: Ref T4443. Provide more space; remove the hack-glue.

Test Plan: Logged out, logged in, inspected database.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4443

Differential Revision: https://secure.phabricator.com/D8269
2014-02-18 14:09:36 -08:00
epriestley
f3cbc0e006 Move many task status hardcodes into ManiphestTaskStatus
Summary:
Ref T1812. This cleans up most of the easy hard-coded references to specific statuses:

  - The "fixes" language moves into ManiphestTaskStatus.
  - Add a method to list open statuses.
  - Add a method to test if a status is open.
  - Add a method to get default status for new tasks.

Test Plan: Browsed around, lint, grep, created, filtered and updated tasks.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1812

Differential Revision: https://secure.phabricator.com/D8264
2014-02-17 15:59:31 -08:00
epriestley
18938b5310 Migrate Differential comments to ApplicationTransactions
Summary:
Ref T2222. This is the big one.

This migrates each `DifferentialComment` to one or more ApplicationTransactions (action, cc, reviewers, update, comment, inlines), and makes `DifferentialComment` a double-reader for ApplicationTransactions.

The migration is pretty straightforward:

  - If a comment took an action not otherwise covered, it gets an "action" transaction. This is something like "epriestley abandoned this revision.".
  - If a comment updated the diff, it gets an "updated diff" transaction. Very old transactions of this type may not have a diff ID (probably only at Facebook).
  - If a comment added or removed reviewers, it gets a "changed reviewers" transaction.
  - If a comment added CCs, it gets a "subscribers" transaction.
  - If a comment added comment text, it gets a "comment" transaction.
  - For each inline attached to a comment, we generate an "inline" transaction.

Most comments generate a small number of transactions, but a few generate a significant number.

At HEAD, the code is basically already doing this, so comments in the last day or two already obey these rules, roughly, and will all generate only one transaction (except inlines).

Because we've already preallocated PHIDs in the comment text table, we only need to write to the transaction table.

NOTE: This significantly degrades Differential, making inline comments pretty much useless (they each get their own transaction, and don't show line numbers or files). The data is all fine, but the UI is garbage now. This needs to be fixed before we can deploy this to users, but it's easily separable since it's all just display code.

Specifically, they look like this:

{F112270}

Test Plan:
I've migrated locally and put things through their paces, but it's hard to catch sketchy stuff locally because most of my test data is nonsense and bad migrations wouldn't necessarily look out of place.
IMPORTANT: I'm planning to push this to a branch and then shift production over to the branch, and run it for a day or two before bringing it to master.

I generally feel good about this change: it's not that big since we were able to separate a lot of pieces out of it, and it's pretty straightforward. That said, it's still one of the most scary/dangerous changes we've ever made.

Reviewers: btrahan

CC: chad, aran

Maniphest Tasks: T2222

Differential Revision: https://secure.phabricator.com/D8210
2014-02-12 14:34:48 -08:00
epriestley
8f9b7f4196 Move Differential to proper subscriptions
Summary:
Ref T2222. Ref T4415. We're still writing Differential subscription stuff into this weird legacy `differential_relationship` table, which is like an edge table but extremely ancient.

Move it into a proper table.

I've removed `withSubscriptions()` from `DifferentialRevisionQuery`. It was weird, doesn't work consistently with other similar filters, and was only used by the API. Now it means "ccs", which is consistent with the ApplicationSearch UI and with Maniphest.

Test Plan:
Without migrating, added and removed subscribers via various workflows. Queried for subscribers. Everything worked as expected.

Ran the migration, verified data survived.

Reviewers: btrahan

Reviewed By: btrahan

CC: FacebookPOC, aran

Maniphest Tasks: T2222, T4415

Differential Revision: https://secure.phabricator.com/D8202
2014-02-12 08:53:40 -08:00
epriestley
305fb3fbd9 Migrate all Differential comment text into new storage
Summary:
Ref T2222. Currently, `DifferentialComment` stores both (a) the text of comments and (b) various other transaction details. This data needs to map to both Transactions and TransactionComments in the long run. This diff separates out all the data which is bound for the TransactionComment table, so that when we migrate `DifferentialComment` itself it will //only// need to migrate into the Transactions table. This is a much simpler migration than the inline comment one was, partly because it set up infrastructure and partly because the data is less complex.

Basically, I'm just proxying the read/write for the comment text into the other table. All readers already go through the Query class, and there are only three writers (preview, comment, implicit comment on diff update) which are all highly regular and straightforward to test.

We can also back out of this diff very easily: doing double writes cost only one line of code (`$this->content = $content;`) so we have proper double writes and a trivial revert path.

Test Plan:
  - Without migrating, added comments and saw them show up.
  - Migrated.
  - Saw all the old comments, and no damage to the new ones.
  - Added new comments.
  - Used comment preview.
  - Updated a revision to implicitly create an update comment and verified it looked OK.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2222

Differential Revision: https://secure.phabricator.com/D8196
2014-02-11 11:34:15 -08:00
epriestley
b1243e549c Allow unsubscription from projects
Summary:
Fixes T4379. Several changes:

  - Migrate all project members into subscribers.
  - When members are added or removed, subscribe or unsubscribe them.
  - Show sub/unsub in the UI.
  - Determine mailable membership of projects by querying subscribers.

Test Plan:
  - As `duck`, joined a project.
  - Added the project as a reviewer to a revision.
  - Commented on the revision.
  - Observed `duck` receive mail.
  - Unsubscribed as `duck`.
  - Observed no mail.
  - Resubscribed as `duck`.
  - Mail again.
  - Joined/left project, checked sub/unsub status.
  - Ran migration, looked at database.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, asherkin

Maniphest Tasks: T4379

Differential Revision: https://secure.phabricator.com/D8189
2014-02-11 07:45:56 -08:00
Bob Trahan
1830868007 Herald - make herald condition of herald rule display better
Summary: ...by the surprising step of changing how this data is stored from id to phid. Also a small fix to not allow "disabled" rules to be used as herald rule conditions, i.e. can't make a rule that depends on a disabled rule.

Test Plan: viewed existing herald rule that had a rule condition and noted nice new display using handle. made a new rule that had a rule condition and verified it worked correctly.

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D8186
2014-02-10 14:40:09 -08:00
epriestley
8c84ed61b1 Migrate project profiles onto projects, and remove ProjectProfile object
Summary:
Ref T4379. Long ago, the "Project" vs "ProjectProfile" split was intended to allow a bunch of special fields on projects without burdening the simple use cases, but CustomField handles that far better and far more generally, and doing this makes using ApplicationTransactions a pain to get right, so get rid of it.

The only remaining field is `profileImagePHID`, which we can just move to the main Project object. This is custom enough that I think it's reasonable not to express it as a custom field.

Test Plan: Created a project, set profile, edited project, viewed in typeahead, ran migration, verified database results.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4379

Differential Revision: https://secure.phabricator.com/D8183
2014-02-10 14:32:14 -08:00
epriestley
1520065ec0 Migrate project blurb/description to standard custom field storage
Summary: Ref T4379. Major goal here is to remove `ProjectProfile` so all edits use ApplicationTransactions. This also makes things more flexible, allowing users to disable this field if they don't like it.

Test Plan: Ran migration, verified data survived, edited/created projects, reordered fields.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4379

Differential Revision: https://secure.phabricator.com/D8182
2014-02-10 14:31:57 -08:00
epriestley
b13a51adeb Fix inadvertent forward dependency in task owner migration
Summary:
See <https://github.com/facebook/phabricator/issues/505>. When the status/event table moved, it broke this migration, which implicitly loads statuses by loading events.

Instead, access just the row we care about.

Test Plan: Used `--apply` to apply the new version of the patch.

Reviewers: btrahan, chad

Reviewed By: chad

CC: aran

Differential Revision: https://secure.phabricator.com/D8162
2014-02-07 09:17:11 -08:00
epriestley
eef2727ea6 Fix accidental forward dependency in old commit summary migration
Summary: See IRC. This migration inadvertently depends on the columns in the commit table, because it calls `save()`, and thus broke for installs with data after we added the `importStatus` column. Since that was ~9 months after this patch, probably not many installs are affected.

Test Plan: Ran patch locally with `--apply` on data. Had user verify fix.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D8152
2014-02-06 10:33:54 -08:00
epriestley
4c7acbafac Assign PHIDs to calendar events
Summary: Ref T4375. We're going to need these for a bunch of infrastructure to work.

Test Plan: Ran migrations, checked DB, used `phid.query`.

Reviewers: btrahan, chad

Reviewed By: chad

CC: aran

Maniphest Tasks: T4375

Differential Revision: https://secure.phabricator.com/D8151
2014-02-06 10:10:43 -08:00
epriestley
ab636f36bf Rename PhabricatorUserStatus to PhabricatorCalendarEvent
Summary: Ref T4375. We never join this table, so this is a pretty straight find/replace.

Test Plan: Browsed around Calendar, verified that nothing seemed broken. Saw my red dot in other apps.

Reviewers: btrahan, chad

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4375

Differential Revision: https://secure.phabricator.com/D8145
2014-02-06 10:07:29 -08:00
epriestley
cb605ad5ee Add edit/view plumbing for dashboards and panels
Summary: Ref T3583. This doesn't add any dashboard/panel-specific code beyond headers/titles/buttons/etc., but allows you to create and view dashboards and panel skeletons.

Test Plan: See screenshots.

Reviewers: btrahan, chad

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T3583

Differential Revision: https://secure.phabricator.com/D8131
2014-02-03 10:52:15 -08:00
epriestley
88436349b8 Add a GC for sent and received mail
Summary: Ref T4368. We don't currently GC these tables, and the sent mail table is one of the largest on `secure.phabricator.com`. There's no value in retaining this information indefinitely. Instead, retain it for 90 days, which should be plenty of time to debug/diagnose any issues.

Test Plan: Ran `phd debug garbage`, saw it clean up a reasonable amount of data from these tables.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4368

Differential Revision: https://secure.phabricator.com/D8127
2014-02-03 10:51:31 -08:00
epriestley
4b0ef353e4 Remove retry/failure mechanisms from MetaMTA
Summary:
Fixes T4202. We have old code in MetaMTA which implements gradual backoff and maximum retries.

However, we have more general code in the task queue which does this, too. We can just use the more general stuff in the task queue; it obsoletes the specific stuff in MetaMTA, which is more complex and ran into some kind of issue in T4202.

Remove `retryCount`, `nextRetry` (obsoleted by task queue retry mechanisms) and "simulated failures" (no longer in use).

Generally, modern infrastructure has replaced these mechanisms with more general ones.

Test Plan:
  - Sent mail.
  - Observed unsendable mail failing in reasonable ways in the queue.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4202

Differential Revision: https://secure.phabricator.com/D8115
2014-02-01 14:35:42 -08:00
epriestley
73924dfa18 Add initial skeleton for Dashboard application
Summary:
Ref T3583. General idea here is:

  - Users will be able to create `DashboardPanel`s, which are things like the jump nav, or a minifeed, or recent assigned tasks, or recent tokens given, or whatever else.
  - The `DashboardPanel`s can be combined into `Dashboard`s, which select specific panels and arrange them in some layout (and maybe have a few other options eventually).
  - Then, you'll be able to set a specific `Dashboard` for your home page, and maybe for project home pages. But you can also use `Dashboard`s on their own if you just like dashboards.

My plan is pretty much:

  - Put in basic infrastructure for dashboards (this diff).
  - Add basic create/edit (next few diffs).
  - Once dashboards sort of work, do the homepage integration.

This diff does very little: you can't create dashboards or panels yet, and thus there are no dashboards to look at. This is all skeleton code, pretty much.

IMPORTANT: We need an icon bwahahahahaha

Test Plan:
omg si purrfect

{F106367}

Reviewers: chad, btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T3583

Differential Revision: https://secure.phabricator.com/D8109
2014-01-30 11:43:24 -08:00
epriestley
014a873773 Update DifferentialDiff: add repositoryPHID, drop parentRevisionID
Summary:
Moves away from ArcanistProjects:

  - Adds storage for diffs to be directly associated with a repository (instead of indirectly, through arcanist projects). Not really populated yet.
  - Drops `parentRevisionID`, which is obsoleted by the "Depends On" edge. This is not exposed in the UI anywhere and doesn't do anything. Resolves TODO.

Test Plan: Ran storage upgrades, browsed around, lots of `grep`.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D8072
2014-01-26 15:29:22 -08:00