Summary: Ref T2222. I'll hold this, but there are no more reads or writes from/to this table in the application.
Test Plan: Grepped for usage, ran migration, browsed around.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley, aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8203
Summary:
Ref T4209. Unifies the local (`./bin/phd status`) and global (`./bin/phd status --all`) view into a single table. This generally makes it easy to administer daemons running across multiple hosts.
Depends on D9606.
Test Plan:
```
> sudo ./bin/phd status
ID Host PID Started Daemon Arguments
38 localhost 2282 Jun 18 2014, 7:52:56 AM PhabricatorRepositoryPullLocalDaemon
39 localhost 2289 Jun 18 2014, 7:52:57 AM PhabricatorGarbageCollectorDaemon
40 localhost 2294 Jun 18 2014, 7:52:57 AM PhabricatorTaskmasterDaemon
41 localhost 2314 Jun 18 2014, 7:52:58 AM PhabricatorTaskmasterDaemon
42 localhost 2319 Jun 18 2014, 7:52:59 AM PhabricatorTaskmasterDaemon
43 localhost 2328 Jun 18 2014, 7:53:00 AM PhabricatorTaskmasterDaemon
44 localhost 2354 Jun 18 2014, 7:53:08 AM PhabricatorRepositoryPullLocalDaemon X --not Y
```
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4209
Differential Revision: https://secure.phabricator.com/D9607
Summary:
We already have GC for daemon log events, but not for daemon logs themselves.
Collect old daemon logs which aren't still running.
Test Plan: Ran `phd debug garbage`, observed old logs get cleaned up. Started some daemons, re-ran garbage, made sure they stuck around.
Reviewers: joshuaspence
Reviewed By: joshuaspence
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D9610
Summary:
Ref T4566. Currently, mocks have a conservative (author only), immutable default edit policy.
Instead:
- Let the edit policy be changed.
- Default the edit policy to "all users", similar to other applications.
- Add an application-level setting for it.
- Migrate existing edit policies to be consistent with the old policy (just the author).
This stops short of adding a separate "owner" and letting that be changed, since Pholio doesn't really have any review/approve type features (at least, so far). We can look at doing this if we get more feedback about it, or if we make owners more meaningful (e.g., add more "review-like" process to mocks).
Test Plan:
- Ran migration scripts.
- Confirmed existing mocks retained their effective policies (author only).
- Created a new mock, saw edit policy.
- Changed edit policy.
- Changed global edit policy default.
- Tried to edit a mock I couldn't edit.
Reviewers: chad
Reviewed By: chad
Subscribers: epriestley
Maniphest Tasks: T4566
Differential Revision: https://secure.phabricator.com/D9550
Summary:
Ref T4045. Ref T5179. Hunk storage has two major issues:
- It's utf8, but actual diffs are binary.
- It's huge and can't be compressed or archived.
This introduces a second datastore which solves these problems: by recording hunk encoding, supporting compression, and supporting alternate storage. There's no actual compression or storage support yet, but there's space in the table for them.
Since nothing actually uses hunk IDs, it's fine to have these tables exist at the same time and use the same IDs. We can migrate data between the tables gradually without requiring downtime or disrupting installs.
Test Plan:
- There are no writes to the new table yet.
- The only effect this has is making us issue one extra query when looking for hunks.
- Observed the query issue, but everything else continue working fine.
- Created a new diff.
- Ran unit tests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4045, T5179
Differential Revision: https://secure.phabricator.com/D9290
Summary: Ref T5089. Adds a `security.require-multi-factor-auth` which forces all users to enroll in MFA before they can use their accounts.
Test Plan:
Config:
{F159750}
Roadblock:
{F159748}
After configuration:
{F159749}
- Required MFA, got roadblocked, added MFA, got unblocked.
- Removed MFA, got blocked again.
- Used `bin/auth strip` to strip MFA, got blocked.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5089
Differential Revision: https://secure.phabricator.com/D9285
Summary: Fixes T5090. Introduced getIcon into Handle stack which allows you to specify a per handle icon. getIcon falls back ot getTypeIcon.
Test Plan: changed the icon on a project a bunch. verified transactions showed up. verified icon showed up in typeahead. verified icon showed up in tokens that were pre-generated (not typed in). units test passed.
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T5090
Differential Revision: https://secure.phabricator.com/D9264
Summary:
Fixes T4021. Chooses to keep a "primary" slug based off the name - including all that lovely logic - and allow the user to specify "additional" slugs. Expose these as "hashtags" to the user.
Sets us up for a fun diff where we can delete all the Project => Phriction automagicalness. In terms of this diff, see the TODOs i added.
Test Plan:
added a primary slug as an additional slug - got an error. added a slug in use on another project - got an error. added multiple good slugs and they worked. removed slugs and it worked. made some remark using multiple new slugs and they all linked to the correct project
ran epriestley's case
- Create project "A".
- Give it additional slug "B".
- Try to create project "B".
and i got a nice error about hashtag collision
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4021
Differential Revision: https://secure.phabricator.com/D9250
Summary:
Ref T4398. We have several auth-related systems which require (or are improved by) the ability to hand out one-time codes which expire after a short period of time.
In particular, these are:
- SMS multi-factor: we need to be able to hand out one-time codes for this in order to prove the user has the phone.
- Password reset emails: we use a time-based rotating token right now, but we could improve this with a one-time token, so once you reset your password the link is dead.
- TOTP auth: we don't need to verify/invalidate keys, but can improve security by doing so.
This adds a generic one-time code storage table, and strengthens the TOTP enrollment process by using it. Specifically, you can no longer edit the enrollment form (the one with a QR code) to force your own key as the TOTP key: only keys Phabricator generated are accepted. This has no practical security impact, but generally helps raise the barrier potential attackers face.
Followup changes will use this for reset emails, then implement SMS multi-factor.
Test Plan:
- Enrolled in TOTP multi-factor auth.
- Submitted an error in the form, saw the same key presented.
- Edited the form with web tools to provide a different key, saw it reject and the server generate an alternate.
- Change the expiration to 5 seconds instead of 1 hour, submitted the form over and over again, saw it cycle the key after 5 seconds.
- Looked at the database and saw the tokens I expected.
- Ran the GC and saw all the 5-second expiry tokens get cleaned up.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4398
Differential Revision: https://secure.phabricator.com/D9217
Summary:
See title. Adds PhabricatorDashboardInstall data object which scopes installs to objectPHID + applicationClass. This is because we already have a collision for user home pages and user profiles. Assume only one dashboard per objectPHID + applicationClass though at the database level.
Fixes T5076.
Test Plan: From dashboard view, installed a dashboard - success! Went back to dashboard view and uninstalled it!
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T5076
Differential Revision: https://secure.phabricator.com/D9206
Summary: Fixes T4299, Add status dropdown to mock edit view
Test Plan: Edit mock, close mock, thumbnail title should read (Disabled). Default mocks list should show only open mocks.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: chad, epriestley, Korvin
Maniphest Tasks: T4299
Differential Revision: https://secure.phabricator.com/D9145
Summary:
Fixes T4898. After we increased the strictness of the `%s` conversion, most `serialize()` output is rejected from the cache.
Drop the cache, change the column type to latin1_bin, and then use `%B` to mark the data as binary during query construction.
Test Plan: Viewed Differential, saw cache fills.
Reviewers: btrahan, spicyj
Reviewed By: spicyj
Subscribers: epriestley
Maniphest Tasks: T4898
Differential Revision: https://secure.phabricator.com/D9171
Summary:
Ref T4994. This stuff works:
- You can dump a blob of coverage information into `diffusion.updatecoverage`. This wipes existing coverage information and replaces it.
- It shows up when viewing files.
- It shows up when viewing commits.
This stuff does not work:
- When viewing files, the Javascript hover interaction isn't tied in yet.
- We always show this information, even if you're behind the commit where it was generated.
- You can't do incremental updates.
- There's no aggregation at the file (this file has 90% coverage), diff (the changes in this commit are 90% covered), or directory (the code in this directory has 90% coverage) levels yet.
- This is probably not the final form of the UI, storage, or API, so you should expect occasional changes over time. I've marked the method as "Unstable" for now.
Test Plan:
- Ran `save_lint.php` to check for collateral damage; it worked fine.
- Ran `save_lint.php` on a new branch to check creation.
- Published some fake coverage information.
- Viewed an affected commit.
- Viewed an affected file.
{F151915}
{F151916}
Reviewers: chad, btrahan
Reviewed By: btrahan
Subscribers: jhurwitz, epriestley, zeeg
Maniphest Tasks: T5044, T4994
Differential Revision: https://secure.phabricator.com/D9022
Summary: This is useful when you're trying to onboard an entire office and you end up using the Google OAuth anyway.
Test Plan: tested locally. Maybe I should write some tests?
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D9150
Summary:
This gets us the ability to specify a "layout mode" and which column a panel should appear in at panel add time. Changing the layout mode from a multi column view to a single column view or vice versa will reset all panels to the left most column.
You can also drag and drop where columns appear via the "arrange" mode.
We also have a new dashboard create flow. Create dashboard -> arrange mode. (As opposed to view mode.) This could all possibly use massaging.
Fixes T4996.
Test Plan:
made a dashboard with panels in multiple columns. verified correct widths for various layout modes
re-arranged collumns like whoa.
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4996
Differential Revision: https://secure.phabricator.com/D9031
Summary:
Create transaction, editor, etc, and move command generation over to editor.
Show in a timeline in the buildable page.
Also prevent Engine from creating an empty transaction when build starts (Fixes T4885).
Fixes T4886.
Test Plan: Restart builds and buildables, look at timeline.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4885, T4886
Differential Revision: https://secure.phabricator.com/D9110
Summary:
Ref T4455. This adds a `repository_parents` table which stores `<childCommitID, parentCommitID>` relationships.
For new commits, it is populated when commits are discovered.
For older commits, there's a `bin/repository parents` script to rebuild the data.
Right now, there's no UI suggestion that you should run the script. I haven't come up with a super clean way to do this, and this table will only improve performance for now, so it's not important that we get everyone to run the script right away. I'm just leaving it for the moment, and we can figure out how to tell admins to run it later.
The ultimate goal is to solve T2683, but solving T4455 gets us some stuff anyway (for example, we can serve `diffusion.commitparentsquery` faster out of this cache).
Test Plan:
- Used `bin/repository discover` to discover new commits in Git, SVN and Mercurial repositories.
- Used `bin/repository parents` to rebuild Git and Mercurial repositories (SVN repos just exit with a message).
- Verified that the table appears to be sensible.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: jhurwitz, epriestley
Maniphest Tasks: T4455
Differential Revision: https://secure.phabricator.com/D9044
Summary:
Provides a working SMS implementation with support for Twilio.
This version doesn't really retry if we get any gruff at all. Future versions should retry.
Test Plan: used bin/sms to send messages and look at them.
Reviewers: chad, epriestley
Reviewed By: epriestley
Subscribers: aurelijus, epriestley, Korvin
Maniphest Tasks: T920
Differential Revision: https://secure.phabricator.com/D8930
Summary: Fixes T4931. Each new credential should come with the ability to lock the credential permanently, so that no one can ever edit again. Each existing credential must allow user to lock existing credential.
Test Plan: Create new credential, verify that you can lock it before saving it. Open existing unlocked credential, verify that option to lock it exists. Once credential is locked, the option to reveal it should be disabled, and editing the credential won't allow username/password updates.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4931
Differential Revision: https://secure.phabricator.com/D8947
Summary:
Ref T4749. Ref T3265. Ref T4909. Several goals here:
- Move user destruction to the CLI to limit the power of rogue admins.
- Start consolidating all "destroy named object" scripts into a single UI, to make it easier to know how to destroy things.
- Structure object destruction so we can do a better and more automatic job of cleaning up transactions, edges, search indexes, etc.
- Log when we destroy objects so there's a record if data goes missing.
Test Plan: Used `bin/remove destroy` to destroy several users.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T3265, T4749, T4909
Differential Revision: https://secure.phabricator.com/D8940
Summary:
Ref T4398. This prompts users for multi-factor auth on login.
Roughly, this introduces the idea of "partial" sessions, which we haven't finished constructing yet. In practice, this means the session has made it through primary auth but not through multi-factor auth. Add a workflow for bringing a partial session up to a full one.
Test Plan:
- Used Conduit.
- Logged in as multi-factor user.
- Logged in as no-factor user.
- Tried to do non-login-things with a partial session.
- Reviewed account activity logs.
{F149295}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4398
Differential Revision: https://secure.phabricator.com/D8922
Summary:
Ref T3583. Adds edges, query relationships, etc. Lots of debugging/temporary UI.
My general intent here is to use edges to track where panels appear, and then put additional data on the dashboard itself to control layout, positioning, etc.
Dashboards don't actually render yet so this is still pretty boring.
Test Plan:
{F149175}
{F149176}
{F149177}
Reviewers: chad, btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T3583
Differential Revision: https://secure.phabricator.com/D8916
Summary: Ref T3583. These will be the primary class carrying panel implementations.
Test Plan:
{F149125}
{F149126}
Reviewers: chad, btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T3583
Differential Revision: https://secure.phabricator.com/D8912
Summary:
Ref T4398. This is still pretty rough and isn't exposed in the UI yet, but basically works. Some missing features / areas for improvement:
- Rate limiting attempts (see TODO).
- Marking tokens used after they're used once (see TODO), maybe. I can't think of ways an attacker could capture a token without also capturing a session, offhand.
- Actually turning this on (see TODO).
- This workflow is pretty wordy. It would be nice to calm it down a bit.
- But also add more help/context to help users figure out what's going on here, I think it's not very obvious if you don't already know what "TOTP" is.
- Add admin tool to strip auth factors off an account ("Help, I lost my phone and can't log in!").
- Add admin tool to show users who don't have multi-factor auth? (so you can pester them)
- Generate QR codes to make the transfer process easier (they're fairly complicated).
- Make the "entering hi-sec" workflow actually check for auth factors and use them correctly.
- Turn this on so users can use it.
- Adding SMS as an option would be nice eventually.
- Adding "password" as an option, maybe? TOTP feels fairly good to me.
I'll post a couple of screens...
Test Plan:
- Added TOTP token with Google Authenticator.
- Added TOTP token with Authy.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4398
Differential Revision: https://secure.phabricator.com/D8875
Summary:
Ref T4398. This is roughly a "sudo" mode, like GitHub has for accessing SSH keys, or Facebook has for managing credit cards. GitHub actually calls theirs "sudo" mode, but I think that's too technical for big parts of our audience. I've gone with "high security mode".
This doesn't actually get exposed in the UI yet (and we don't have any meaningful auth factors to prompt the user for) but the workflow works overall. I'll go through it in a comment, since I need to arrange some screenshots.
Test Plan: See guided walkthrough.
Reviewers: chad, btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4398
Differential Revision: https://secure.phabricator.com/D8851
Summary: Fixes T3566 List of poll actions should include ability to close an open poll or reopen a closed poll.
Test Plan: Poll author should be able to close/reopen poll. Non-author should get policy screen when attempting to close/reopen poll.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Maniphest Tasks: T3566
Differential Revision: https://secure.phabricator.com/D8846
Summary: Going to sit on this for a bit so we can fall back to it if needbe, but this table no longer has any reads or writes in the application.
Test Plan: Applied patch locally and poked around.
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: aran
Differential Revision: https://secure.phabricator.com/D8190
Summary:
Ref T3551. Currently, ReleephRequests don't have a direct concept of the //object// being requested. You can request `D123`, but that is just a convenient way to write `rXyyyy`.
When the UI wants to display information about a revision, it deduces it by examining the commit.
This is primarily an attack on T3551, so we don't need to load <commit -> edge -> revision> (in an ad-hoc way) to get revisions. Instead, when you request a revision we keep track of it and can load it directly later.
Later, this will let us do more things: for example, if you request a branch, we can automatically update the commits (as GitHub does), etc. (Repository branches will need PHIDs first, of course.)
This adds and populates the column but doesn't use it yet. The second part of the migration could safely be run while Phabricator is up, although even for Facebook this table is probably quite small.
Test Plan:
- Ran migration.
- Verified existing requests associated sensibly.
- Created a new commit request.
- Created a new revision request.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T3551
Differential Revision: https://secure.phabricator.com/D8822
Summary:
Ref T4809. Buildables currently have buildStatus and buildableStatus. Neither are used, and no one knows why we have two.
I'm going to use buildableStatus shortly, but buildStatus is meaningless; burn it.
Test Plan: `grep`, examined similar get/set calls, created a new buildable, ran storage upgrade.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4809
Differential Revision: https://secure.phabricator.com/D8796
Summary:
When we generate account tokens for CSRF keys and email verification, one of the inputs we use is the user's password hash. Users won't always have a password hash, so this is a weak input to key generation. This also couples CSRF weirdly with auth concerns.
Instead, give users a dedicated secret for use in token generation which is used only for this purpose.
Test Plan:
- Ran upgrade scripts.
- Verified all users got new secrets.
- Created a new user.
- Verified they got a secret.
- Submitted CSRF'd forms, they worked.
- Adjusted the CSRF token and submitted CSRF'd forms, verified they don't work.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D8748
Summary:
This adds a system which basically keeps a record of recent actions, who took them, and how many "points" they were worth, like:
epriestley email.add 1 1233989813
epriestley email.add 1 1234298239
epriestley email.add 1 1238293981
We can use this to rate-limit actions by examining how many actions the user has taken in the past hour (i.e., their total score) and comparing that to an allowed limit.
One major thing I want to use this for is to limit the amount of error email we'll send to an email address. A big concern I have with sending more error email is that we'll end up in loops. We have some protections against this in headers already, but hard-limiting the system so it won't send more than a few errors to a particular address per hour should provide a reasonable secondary layer of protection.
This use case (where the "actor" needs to be an email address) is why the table uses strings + hashes instead of PHIDs. For external users, it might be appropriate to rate limit by cookies or IPs, too.
To prove it works, I rate limited adding email addresses. This is a very, very low-risk security thing where a user with an account can enumerate addresses (by checking if they get an error) and sort of spam/annoy people (by adding their address over and over again). Limiting them to 6 actions / hour should satisfy all real users while preventing these behaviors.
Test Plan:
This dialog is uggos but I'll fix that in a sec:
{F137406}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D8683
Summary: Fixes T4703. This is a VARCHAR(255) for no particular reason.
Test Plan: {F136160}
Reviewers: btrahan, chad
Reviewed By: chad
Subscribers: epriestley
Maniphest Tasks: T4703
Differential Revision: https://secure.phabricator.com/D8652
Summary: Ref T3549. This table isn't written to yet; rename it and the DAOs and modernize the history controller.
Test Plan: Viewed history page for a product.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T3549
Differential Revision: https://secure.phabricator.com/D8633
Summary: followup to D8544. This ends up creating an editor + transactions to get the job done.
Test Plan: made a column - saw a nice created transaction. edited the name - saw a nice name edit. deleted the column - saw a deleted transaction, updated "deleted" ui, and hte action change to activate. "Activated" the column and saw a transaction and updated UI. Tried to delete a column with tasks in it and got an error.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D8620
Summary:
Ref T4677. Currently, we record individual actions in a push as PhabricatorRepositoryPushLogs, but tie them together only loosely with a `transactionKey`.
Provide a real PushEvent object, and move some of the denormalized fields to it. This primarily just gives us more robust infrastructure for building, e.g., email about pushes, for T4677, since we can act on real PHIDs rather than passing awkward identifiers around.
Test Plan:
- Performed migration.
- Looked at database for consistency.
- Browsed/queried push logs.
- Pushed a bunch of stuff.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4677
Differential Revision: https://secure.phabricator.com/D8615
Summary:
Ref T1049. Allows external systems to send a message to a build target. The primary intended use case is:
- You make an HTTP request to Jenkins.
- The build goes into a "waiting" state.
- Later, Jenkins calls `harbormaster.sendmessage` to report that the target passed or failed.
- The build continues as appropriate.
This is deceptively complicated because:
- There are a lot of race concerns. We might get a message back from an external system before it even responds to the request we made. We want to make sure we process these messages no matter when we receive them.
- These messages need to be sent to a build target (vs a build or buildable) because we'll get into trouble with parallelization later on otherwise (Jenkins is told to do 3 builds; we can't tell which ones failed or what overall state is unless the message are sent to targets).
- I initially thought about implementing this as a separate "Wait for a response from an external system" build step. This gets a lot more complicated for users once we do parallelization, though. Particularly, in the case where you've told Jenkins to do 3 builds, the three "wait" steps need to know which target they're waiting for (and jenkins needs to know some unique identifier for each target). So this pretty much boils down to a more complicated, more error-prone version of using target PHIDs.
This makes the already-muddy Build UI a bit worse, but it needs a general clarity pass anyway (it's showing way too much uninteresting data, and should show a better summary of results instead).
Test Plan:
- This doesn't really do anything interesting yet.
- Used Conduit to send messages to build plans.
- Viewed the messages on the build screen.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T1049
Differential Revision: https://secure.phabricator.com/D8604
Summary: Ref T1049. For consistency, rename these to "Harbormaster...".
Test Plan: Ran migration, ran builds, everything still works fine.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T1049
Differential Revision: https://secure.phabricator.com/D8602
Summary:
Ref T1049. Fixes T4602. Moves all the funky field stuff to CustomField. Uses ApplicationTransactions to apply and record edits.
This makes "artifact" fields a little less nice (but still perfectly usable). With D8599, I think they're reasonable overall. We can improve this in the future.
All other field types are better (e.g., fixes weird bugs with "bool", fixes lots of weird behavior around required fields), and this gives us access to many new field types.
Test Plan:
Made a bunch of step edits. Here's an example:
{F133694}
Note that:
- "Required" fields work correctly.
- the transaction record is shown at the bottom of the page.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4602, T1049
Differential Revision: https://secure.phabricator.com/D8600
Summary:
Ref T1812. I think integer constants are going to be confusing and error prone for users to interact with. For example, because we use 0-5, adding a second "open" status like "needs verification" without disrupting the existing statuses would require users to define a status with, e.g., constant `6`, but order it between constants `0` and `1`. And if they later remove statuses, they need to avoid reusing existing constants.
Instead, use more manageable string constants like "open", "resolved", etc.
We must migrate three tables:
- The task table itself, to update task status.
- The transaction table, to update historic status changes.
- The saved query table, to update saved queries which specify status sets.
Test Plan:
- Saved a query with complicated status filters.
- Ran migrations.
- Looked at the query, at existing tasks, and at task transactions.
- Forced migrations to run again to verify idempotentcy/safety.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T1812
Differential Revision: https://secure.phabricator.com/D8583
Summary: Fixes T4408. I had to add a "status" to colum. I think we'll need this once we get fancier anyway but for now we have "active" and deleted.
Test Plan: deleted a column. noted reloaded workboard with all those tasks back in the default colun. loaded a task and saw the initial transaction had a "Disabled" icon next to the deleted workboard. also saw the new transaction back to the default column worked.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4408
Differential Revision: https://secure.phabricator.com/D8544
Summary:
Fixes T4637.
- We already allow you to order by this column but don't have a key on it. Add one.
- Expose UI for querying on ranges.
Test Plan:
- Ran some queries, got reasonable-looking results and no table scans.
Reviewers: btrahan, bigo
Reviewed By: bigo
Subscribers: bigo, epriestley
Maniphest Tasks: T4637
Differential Revision: https://secure.phabricator.com/D8557
Summary: Ref T2217. I'll hold this for a month or so, but once we're confident the migration didn't ruin anything we should nuke this old data -- it's just an insurance policy against discovering migration issues.
Test Plan: Will run in a month or so.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley, aran
Maniphest Tasks: T2217
Differential Revision: https://secure.phabricator.com/D7104
Summary:
Ref T988. This fixes the biggest current problem with Diviner, which is dead links to articles.
In the new Diviner, articles can have both a "name" (derived from the file name, and used in the URI) and a "title" (optional, specified explicitly). For example, we have one document with the name "feedback" and the title "Give Feedback! Get Support!".
On disk, we want to use the name for the actual file where the text lives ("feedback.diviner"). We also want to use the name in the URI, to generate a clean URI and to allow us to retitle the document slightly without breaking links to it (for example, we renamed the "Backup" document to "Backups and Migrations").
However, when displaying the article we want to use the title.
Currently, you can //only// link to the name, not the title. This is inconvenient:
- We have a bunch of existing docs which link to titles.
- It's natural/intuitive to link to titles.
- Linking to titles makes it easier/cheaper to generate documentation, because we don't need to be able to resolve things at render time.
To remedy this, allow links to target either names or titles. If we miss on a name query, we'll do a title query. This is implemented with a slug hash to allow approximately correct titles (wrong case/spacing/punctuation, e.g.) and sidestep all the UTF8/column length issues.
(In the long run, atom resolution should theoretically be more sophistiated than it is now, and we should do render-time lookups on at least some documents to catch bad links. However, this is fairly complicated and a relatively advanced feature, and I think allowing links to titles is desirable no matter what.)
Test Plan: The user documentation book now has valid links to articles when the titles and names differ.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D8407
Summary:
Ref T2222. Make the "EditPro" controller accommodate diff updates, and support the transaction type. This one is pretty straightforward.
Also make `revisionPHID` in the comments table nullable to fix the "Edit" action.
Test Plan:
- Created new revision.
- Updated revision.
- Tried to do some invalid stuff.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8376
Summary: Ref T2222. Ref T3886. Differential has a legacy storage table for auxiliary fields; move the data to modern storage.
Test Plan:
- Ran migration.
- Verified fields still worked properly afterward (view, edit, etc).
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3886, T2222
Differential Revision: https://secure.phabricator.com/D8355
Summary: Ref T2222. This is obsolete and no longer used. We could deduce it from transactions or commits in modern Phabricator if we wanted it. We may implement a more general mechanism for T4434.
Test Plan: `grep`
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8330
Summary:
Ref T1191. Test that MySQL's rules match those of `phutil_is_utf8_with_only_bmp_characters()`:
- Build a string with //every// character that we consider to be a BMP character.
- Write it into MySQL.
- Read it back out.
- Make sure MySQL didn't truncate it.
Test Plan: Ran unit test. This test runs pretty quickly (50ms), the string with every character isn't all that enormous.
Reviewers: btrahan, arice
Reviewed By: arice
CC: chad, arice, aran
Maniphest Tasks: T1191
Differential Revision: https://secure.phabricator.com/D8314
Summary: ...do it somewhat generically, so we could fairly easily add this to other applications. Fixes T3496. I got a wee bit lazy and decided not to migrate existing drafts. My excuses aside from laziness are doing it this way will let us see if anyone complains, we can always do a migration later if people do complain, and there's likely to be a lot of garbage data for older / bigger installs, and the migration didn't seem worth itgiven it would also likely be expensive in these cases.
Test Plan: made a draft inline comment on DX and observed DX had a note icon on Differential home page. made a draft comment on DX and observed DX had a note icon on Differential home page. deleted a draft inline comment and noted icon disappeared from Differential homepage. Submitted a draft comment + inline comment and noted icon disappeared.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Maniphest Tasks: T3496
Differential Revision: https://secure.phabricator.com/D8275
Summary: Fixes T4443. Plug VCS passwords into the shared key stretching. They don't use any real stretching now (I anticipated doing something like T4443 eventually) so we can just migrate them into stretching all at once.
Test Plan:
- Viewed VCS settings.
- Used VCS password after migration.
- Set VCS password.
- Upgraded VCS password by using it.
- Used VCS password some more.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4443
Differential Revision: https://secure.phabricator.com/D8272
Summary:
Ref T2222. This is the big one.
This migrates each `DifferentialComment` to one or more ApplicationTransactions (action, cc, reviewers, update, comment, inlines), and makes `DifferentialComment` a double-reader for ApplicationTransactions.
The migration is pretty straightforward:
- If a comment took an action not otherwise covered, it gets an "action" transaction. This is something like "epriestley abandoned this revision.".
- If a comment updated the diff, it gets an "updated diff" transaction. Very old transactions of this type may not have a diff ID (probably only at Facebook).
- If a comment added or removed reviewers, it gets a "changed reviewers" transaction.
- If a comment added CCs, it gets a "subscribers" transaction.
- If a comment added comment text, it gets a "comment" transaction.
- For each inline attached to a comment, we generate an "inline" transaction.
Most comments generate a small number of transactions, but a few generate a significant number.
At HEAD, the code is basically already doing this, so comments in the last day or two already obey these rules, roughly, and will all generate only one transaction (except inlines).
Because we've already preallocated PHIDs in the comment text table, we only need to write to the transaction table.
NOTE: This significantly degrades Differential, making inline comments pretty much useless (they each get their own transaction, and don't show line numbers or files). The data is all fine, but the UI is garbage now. This needs to be fixed before we can deploy this to users, but it's easily separable since it's all just display code.
Specifically, they look like this:
{F112270}
Test Plan:
I've migrated locally and put things through their paces, but it's hard to catch sketchy stuff locally because most of my test data is nonsense and bad migrations wouldn't necessarily look out of place.
IMPORTANT: I'm planning to push this to a branch and then shift production over to the branch, and run it for a day or two before bringing it to master.
I generally feel good about this change: it's not that big since we were able to separate a lot of pieces out of it, and it's pretty straightforward. That said, it's still one of the most scary/dangerous changes we've ever made.
Reviewers: btrahan
CC: chad, aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8210
Summary:
Ref T2222. Ref T4415. We're still writing Differential subscription stuff into this weird legacy `differential_relationship` table, which is like an edge table but extremely ancient.
Move it into a proper table.
I've removed `withSubscriptions()` from `DifferentialRevisionQuery`. It was weird, doesn't work consistently with other similar filters, and was only used by the API. Now it means "ccs", which is consistent with the ApplicationSearch UI and with Maniphest.
Test Plan:
Without migrating, added and removed subscribers via various workflows. Queried for subscribers. Everything worked as expected.
Ran the migration, verified data survived.
Reviewers: btrahan
Reviewed By: btrahan
CC: FacebookPOC, aran
Maniphest Tasks: T2222, T4415
Differential Revision: https://secure.phabricator.com/D8202
Summary:
Ref T2222. Currently, `DifferentialComment` stores both (a) the text of comments and (b) various other transaction details. This data needs to map to both Transactions and TransactionComments in the long run. This diff separates out all the data which is bound for the TransactionComment table, so that when we migrate `DifferentialComment` itself it will //only// need to migrate into the Transactions table. This is a much simpler migration than the inline comment one was, partly because it set up infrastructure and partly because the data is less complex.
Basically, I'm just proxying the read/write for the comment text into the other table. All readers already go through the Query class, and there are only three writers (preview, comment, implicit comment on diff update) which are all highly regular and straightforward to test.
We can also back out of this diff very easily: doing double writes cost only one line of code (`$this->content = $content;`) so we have proper double writes and a trivial revert path.
Test Plan:
- Without migrating, added comments and saw them show up.
- Migrated.
- Saw all the old comments, and no damage to the new ones.
- Added new comments.
- Used comment preview.
- Updated a revision to implicitly create an update comment and verified it looked OK.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8196
Summary:
Fixes T4379. Several changes:
- Migrate all project members into subscribers.
- When members are added or removed, subscribe or unsubscribe them.
- Show sub/unsub in the UI.
- Determine mailable membership of projects by querying subscribers.
Test Plan:
- As `duck`, joined a project.
- Added the project as a reviewer to a revision.
- Commented on the revision.
- Observed `duck` receive mail.
- Unsubscribed as `duck`.
- Observed no mail.
- Resubscribed as `duck`.
- Mail again.
- Joined/left project, checked sub/unsub status.
- Ran migration, looked at database.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, asherkin
Maniphest Tasks: T4379
Differential Revision: https://secure.phabricator.com/D8189
Summary: ...by the surprising step of changing how this data is stored from id to phid. Also a small fix to not allow "disabled" rules to be used as herald rule conditions, i.e. can't make a rule that depends on a disabled rule.
Test Plan: viewed existing herald rule that had a rule condition and noted nice new display using handle. made a new rule that had a rule condition and verified it worked correctly.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D8186
Summary:
Ref T4379. Long ago, the "Project" vs "ProjectProfile" split was intended to allow a bunch of special fields on projects without burdening the simple use cases, but CustomField handles that far better and far more generally, and doing this makes using ApplicationTransactions a pain to get right, so get rid of it.
The only remaining field is `profileImagePHID`, which we can just move to the main Project object. This is custom enough that I think it's reasonable not to express it as a custom field.
Test Plan: Created a project, set profile, edited project, viewed in typeahead, ran migration, verified database results.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4379
Differential Revision: https://secure.phabricator.com/D8183
Summary: Ref T4379. Major goal here is to remove `ProjectProfile` so all edits use ApplicationTransactions. This also makes things more flexible, allowing users to disable this field if they don't like it.
Test Plan: Ran migration, verified data survived, edited/created projects, reordered fields.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4379
Differential Revision: https://secure.phabricator.com/D8182
Summary: Ref T4375. We're going to need these for a bunch of infrastructure to work.
Test Plan: Ran migrations, checked DB, used `phid.query`.
Reviewers: btrahan, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T4375
Differential Revision: https://secure.phabricator.com/D8151
Summary: Ref T4375. We never join this table, so this is a pretty straight find/replace.
Test Plan: Browsed around Calendar, verified that nothing seemed broken. Saw my red dot in other apps.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4375
Differential Revision: https://secure.phabricator.com/D8145
Summary: Ref T3583. This doesn't add any dashboard/panel-specific code beyond headers/titles/buttons/etc., but allows you to create and view dashboards and panel skeletons.
Test Plan: See screenshots.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3583
Differential Revision: https://secure.phabricator.com/D8131
Summary: Ref T4368. We don't currently GC these tables, and the sent mail table is one of the largest on `secure.phabricator.com`. There's no value in retaining this information indefinitely. Instead, retain it for 90 days, which should be plenty of time to debug/diagnose any issues.
Test Plan: Ran `phd debug garbage`, saw it clean up a reasonable amount of data from these tables.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4368
Differential Revision: https://secure.phabricator.com/D8127
Summary:
Fixes T4202. We have old code in MetaMTA which implements gradual backoff and maximum retries.
However, we have more general code in the task queue which does this, too. We can just use the more general stuff in the task queue; it obsoletes the specific stuff in MetaMTA, which is more complex and ran into some kind of issue in T4202.
Remove `retryCount`, `nextRetry` (obsoleted by task queue retry mechanisms) and "simulated failures" (no longer in use).
Generally, modern infrastructure has replaced these mechanisms with more general ones.
Test Plan:
- Sent mail.
- Observed unsendable mail failing in reasonable ways in the queue.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4202
Differential Revision: https://secure.phabricator.com/D8115
Summary:
Ref T3583. General idea here is:
- Users will be able to create `DashboardPanel`s, which are things like the jump nav, or a minifeed, or recent assigned tasks, or recent tokens given, or whatever else.
- The `DashboardPanel`s can be combined into `Dashboard`s, which select specific panels and arrange them in some layout (and maybe have a few other options eventually).
- Then, you'll be able to set a specific `Dashboard` for your home page, and maybe for project home pages. But you can also use `Dashboard`s on their own if you just like dashboards.
My plan is pretty much:
- Put in basic infrastructure for dashboards (this diff).
- Add basic create/edit (next few diffs).
- Once dashboards sort of work, do the homepage integration.
This diff does very little: you can't create dashboards or panels yet, and thus there are no dashboards to look at. This is all skeleton code, pretty much.
IMPORTANT: We need an icon bwahahahahaha
Test Plan:
omg si purrfect
{F106367}
Reviewers: chad, btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3583
Differential Revision: https://secure.phabricator.com/D8109
Summary:
Moves away from ArcanistProjects:
- Adds storage for diffs to be directly associated with a repository (instead of indirectly, through arcanist projects). Not really populated yet.
- Drops `parentRevisionID`, which is obsoleted by the "Depends On" edge. This is not exposed in the UI anywhere and doesn't do anything. Resolves TODO.
Test Plan: Ran storage upgrades, browsed around, lots of `grep`.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D8072
Summary:
Ref T4327. I want to make change parsing testable; one thing which is blocking this is that the Git discovery process is still part of `PullLocal` daemon instead of being part of `DiscoveryEngine`. The unit test stuff which I want to use for change parsing relies on `DiscoveryEngine` to discover repositories during unit tests.
The major reason git discovery isn't part of `DiscoveryEngine` is that it relies on the messy "autoclose" logic, which we never implemented for Mercurial. Generally, I don't like how autoclose was implemented: it's complicated and gross and too hard to figure out and extend.
Instead, I want to do something more similar to what we do for pushes, which is cleaner overall. Basically this means remembering the old branch heads from the last time we parsed a repository, and figuring out what's new by comparing the old and new branch heads. This should give us several advantages:
- It should be simpler to understand than the autoclose stuff, which is pretty mind-numbing, at least for me.
- It will let us satisfy branch and tag queries cheaply (from the database) instead of having to go to the repository. We could also satisfy some ref-resolve queries from the database.
- It should be easier to extend to Mercurial.
This implements the basics -- pretty much a table to store the cursors, which we update only for Git for now.
Test Plan:
- Ran migration.
- Ran `bin/repository discover X --trace --verbose` on various repositories with branches and tags, before and after modifying pushes.
- Pushed commits to a git repo.
- Looked at database tables.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D7982
Summary:
Ref T4310. Fixes T3720. This change:
- Removes concurrent session limits. Instead, unused sessions are GC'd after a while.
- Collapses all existing "web-1", "web-2", etc., sessions into "web" sessions.
- Dramatically simplifies the code for establishing a session (like omg).
Test Plan: Ran migration, checked Sessions panel and database for sanity. Used existing session. Logged out, logged in. Ran Conduit commands.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4310, T3720
Differential Revision: https://secure.phabricator.com/D7978
Summary:
Ref T3116. This creates a policy rule where you can require a signature on a given legalpad document.
NOTE: signatures must be for the *latest* document version.
Test Plan: made a task have a custom policy requiring a legalpad signature. verified non-signers were locked out.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Maniphest Tasks: T3116
Differential Revision: https://secure.phabricator.com/D7977
Summary:
Ref T3720. Ref T4310. Currently, we limit the maximum number of concurrent sessions of each type. This is primarily because sessions predate garbage collection and we had no way to prevent the session table from growing fairly quickly and without bound unless we did this.
Now that we have GC (and it's modular!) we can just expire unused sessions after a while and throw them away:
- Add a `sessionExpires` column to the table, with a key.
- Add a GC for old sessions.
- When we establish a session, set `sessionExpires` to the current time plus the session TTL.
- When a user uses a session and has used up more than 20% of the time on it, extend the session.
In addition to this, we could also rotate sessions, but I think that provides very little value. If we do want to implement it, we should hold it until after T3720 / T4310.
Test Plan:
- Ran schema changes.
- Looked at database.
- Tested GC:
- Started GC.
- Set expires on one row to the past.
- Restarted GC.
- Verified GC nuked the session.
- Logged in.
- Logged out.
- Ran Conduit method.
- Tested refresh:
- Set threshold to 0.0001% instead of 20%.
- Loaded page.
- Saw a session extension ever few page loads.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4310, T3720
Differential Revision: https://secure.phabricator.com/D7976
Summary:
Ref T4310. Ref T3720. Two major things are going on here:
- I'm making this table work more like a standard table, which, e.g., makes `delete()` simpler to implement.
- Currently, the primary key is `(userPHID, type)`. I want to get rid of this, issue unlimited sessions, and GC old sessions. This means we can't have a unique key on `(userPHID, type)` anymore. This removes it as the primary key and adds it as a normal key instead. There's no functional change -- the code to generate sessions guarantees that it will never write duplicate rows or write additional rows -- but allows us to drop the `-1`, `-2` qualifiers in the future.
- Also of note, our task is made far simpler here because MySQL will automatically assign values to new `AUTO_INCREMENT` columns, so we don't have to migrate to get real IDs.
Test Plan: Ran migrations, verified table looked sane. Logged out, logged in.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3720, T4310
Differential Revision: https://secure.phabricator.com/D7975
Summary: Adds "verified" and "secretKey" to Legalpad document signatures. For logged in users using an email address they own, things are verified right away. Otherwise, the email is sent a verification letter. When the user clicks the link the signature is marked verified.
Test Plan: signed the document with a bogus email address not logged in. verified the email that would be sent looked good from command line. followed link and successfully verified bogus email address
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, epriestley, aran, asherkin
Maniphest Tasks: T4283
Differential Revision: https://secure.phabricator.com/D7930
Summary: This adds in the create flow for the Project board columns on the super secret board page which totally doesn't do anything right now.
Test Plan:
1. Apply diff.
2. Go to super secret page.
3. Click link close to top with a way too long name.
4. Enter a name for the column.
5. Enjoy a new column briefly before realising you cannot remove it.
6. Stay happy!
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: tmaroschik, Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7925
Summary:
Ref T2015. Adds human-readable names to Drydock blueprints.
Also the new patches stuff is so much nicer.
Test Plan: Edited, created, and reviewed migrated blueprints.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2015
Differential Revision: https://secure.phabricator.com/D7918
Summary:
Ref T1049. Currently you can cancel a build, but now that we're tracking a lot more state we can stop, resume, and restart builds.
When the user issues a command against a build, I'm writing it into an auxiliary queue (`HarbormasterBuildCommand`) and then reading them out in the worker. This is mostly to avoid race messes where we try to `save()` the object in multiple places: basically, the BuildEngine is the //only// thing that writes to Build objects, and it holds a lock while it does it.
Test Plan:
- Created a plan which runs "sleep 2" a bunch of times in a row.
- Stopped, resumed, and restarted it.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, chad
Maniphest Tasks: T1049
Differential Revision: https://secure.phabricator.com/D7892
Summary:
If you have private replies on and a Macro reply handler set, we try to access `getMailKey()` and fail. See P1039 for a trace.
(Thanks to @Korvin for picking this up.)
Test Plan: Set configuration, repro'd the exception, applied the patch, then disabled/enabled a macro.
Reviewers: btrahan
Reviewed By: btrahan
CC: Korvin, aran
Differential Revision: https://secure.phabricator.com/D7896
