Summary: Ref T1049. For consistency, rename these to "Harbormaster...".
Test Plan: Ran migration, ran builds, everything still works fine.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T1049
Differential Revision: https://secure.phabricator.com/D8602
Summary:
Ref T1049. Fixes T4602. Moves all the funky field stuff to CustomField. Uses ApplicationTransactions to apply and record edits.
This makes "artifact" fields a little less nice (but still perfectly usable). With D8599, I think they're reasonable overall. We can improve this in the future.
All other field types are better (e.g., fixes weird bugs with "bool", fixes lots of weird behavior around required fields), and this gives us access to many new field types.
Test Plan:
Made a bunch of step edits. Here's an example:
{F133694}
Note that:
- "Required" fields work correctly.
- the transaction record is shown at the bottom of the page.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4602, T1049
Differential Revision: https://secure.phabricator.com/D8600
Summary:
Ref T1812. I think integer constants are going to be confusing and error prone for users to interact with. For example, because we use 0-5, adding a second "open" status like "needs verification" without disrupting the existing statuses would require users to define a status with, e.g., constant `6`, but order it between constants `0` and `1`. And if they later remove statuses, they need to avoid reusing existing constants.
Instead, use more manageable string constants like "open", "resolved", etc.
We must migrate three tables:
- The task table itself, to update task status.
- The transaction table, to update historic status changes.
- The saved query table, to update saved queries which specify status sets.
Test Plan:
- Saved a query with complicated status filters.
- Ran migrations.
- Looked at the query, at existing tasks, and at task transactions.
- Forced migrations to run again to verify idempotentcy/safety.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T1812
Differential Revision: https://secure.phabricator.com/D8583
Summary: Fixes T4408. I had to add a "status" to colum. I think we'll need this once we get fancier anyway but for now we have "active" and deleted.
Test Plan: deleted a column. noted reloaded workboard with all those tasks back in the default colun. loaded a task and saw the initial transaction had a "Disabled" icon next to the deleted workboard. also saw the new transaction back to the default column worked.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4408
Differential Revision: https://secure.phabricator.com/D8544
Summary:
Fixes T4637.
- We already allow you to order by this column but don't have a key on it. Add one.
- Expose UI for querying on ranges.
Test Plan:
- Ran some queries, got reasonable-looking results and no table scans.
Reviewers: btrahan, bigo
Reviewed By: bigo
Subscribers: bigo, epriestley
Maniphest Tasks: T4637
Differential Revision: https://secure.phabricator.com/D8557
Summary: Ref T2217. I'll hold this for a month or so, but once we're confident the migration didn't ruin anything we should nuke this old data -- it's just an insurance policy against discovering migration issues.
Test Plan: Will run in a month or so.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley, aran
Maniphest Tasks: T2217
Differential Revision: https://secure.phabricator.com/D7104
Summary:
Ref T988. This fixes the biggest current problem with Diviner, which is dead links to articles.
In the new Diviner, articles can have both a "name" (derived from the file name, and used in the URI) and a "title" (optional, specified explicitly). For example, we have one document with the name "feedback" and the title "Give Feedback! Get Support!".
On disk, we want to use the name for the actual file where the text lives ("feedback.diviner"). We also want to use the name in the URI, to generate a clean URI and to allow us to retitle the document slightly without breaking links to it (for example, we renamed the "Backup" document to "Backups and Migrations").
However, when displaying the article we want to use the title.
Currently, you can //only// link to the name, not the title. This is inconvenient:
- We have a bunch of existing docs which link to titles.
- It's natural/intuitive to link to titles.
- Linking to titles makes it easier/cheaper to generate documentation, because we don't need to be able to resolve things at render time.
To remedy this, allow links to target either names or titles. If we miss on a name query, we'll do a title query. This is implemented with a slug hash to allow approximately correct titles (wrong case/spacing/punctuation, e.g.) and sidestep all the UTF8/column length issues.
(In the long run, atom resolution should theoretically be more sophistiated than it is now, and we should do render-time lookups on at least some documents to catch bad links. However, this is fairly complicated and a relatively advanced feature, and I think allowing links to titles is desirable no matter what.)
Test Plan: The user documentation book now has valid links to articles when the titles and names differ.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D8407
Summary:
Ref T2222. Make the "EditPro" controller accommodate diff updates, and support the transaction type. This one is pretty straightforward.
Also make `revisionPHID` in the comments table nullable to fix the "Edit" action.
Test Plan:
- Created new revision.
- Updated revision.
- Tried to do some invalid stuff.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8376
Summary: Ref T2222. Ref T3886. Differential has a legacy storage table for auxiliary fields; move the data to modern storage.
Test Plan:
- Ran migration.
- Verified fields still worked properly afterward (view, edit, etc).
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3886, T2222
Differential Revision: https://secure.phabricator.com/D8355
Summary: Ref T2222. This is obsolete and no longer used. We could deduce it from transactions or commits in modern Phabricator if we wanted it. We may implement a more general mechanism for T4434.
Test Plan: `grep`
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8330
Summary:
Ref T1191. Test that MySQL's rules match those of `phutil_is_utf8_with_only_bmp_characters()`:
- Build a string with //every// character that we consider to be a BMP character.
- Write it into MySQL.
- Read it back out.
- Make sure MySQL didn't truncate it.
Test Plan: Ran unit test. This test runs pretty quickly (50ms), the string with every character isn't all that enormous.
Reviewers: btrahan, arice
Reviewed By: arice
CC: chad, arice, aran
Maniphest Tasks: T1191
Differential Revision: https://secure.phabricator.com/D8314
Summary: ...do it somewhat generically, so we could fairly easily add this to other applications. Fixes T3496. I got a wee bit lazy and decided not to migrate existing drafts. My excuses aside from laziness are doing it this way will let us see if anyone complains, we can always do a migration later if people do complain, and there's likely to be a lot of garbage data for older / bigger installs, and the migration didn't seem worth itgiven it would also likely be expensive in these cases.
Test Plan: made a draft inline comment on DX and observed DX had a note icon on Differential home page. made a draft comment on DX and observed DX had a note icon on Differential home page. deleted a draft inline comment and noted icon disappeared from Differential homepage. Submitted a draft comment + inline comment and noted icon disappeared.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Maniphest Tasks: T3496
Differential Revision: https://secure.phabricator.com/D8275
Summary: Fixes T4443. Plug VCS passwords into the shared key stretching. They don't use any real stretching now (I anticipated doing something like T4443 eventually) so we can just migrate them into stretching all at once.
Test Plan:
- Viewed VCS settings.
- Used VCS password after migration.
- Set VCS password.
- Upgraded VCS password by using it.
- Used VCS password some more.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4443
Differential Revision: https://secure.phabricator.com/D8272
Summary:
Ref T2222. This is the big one.
This migrates each `DifferentialComment` to one or more ApplicationTransactions (action, cc, reviewers, update, comment, inlines), and makes `DifferentialComment` a double-reader for ApplicationTransactions.
The migration is pretty straightforward:
- If a comment took an action not otherwise covered, it gets an "action" transaction. This is something like "epriestley abandoned this revision.".
- If a comment updated the diff, it gets an "updated diff" transaction. Very old transactions of this type may not have a diff ID (probably only at Facebook).
- If a comment added or removed reviewers, it gets a "changed reviewers" transaction.
- If a comment added CCs, it gets a "subscribers" transaction.
- If a comment added comment text, it gets a "comment" transaction.
- For each inline attached to a comment, we generate an "inline" transaction.
Most comments generate a small number of transactions, but a few generate a significant number.
At HEAD, the code is basically already doing this, so comments in the last day or two already obey these rules, roughly, and will all generate only one transaction (except inlines).
Because we've already preallocated PHIDs in the comment text table, we only need to write to the transaction table.
NOTE: This significantly degrades Differential, making inline comments pretty much useless (they each get their own transaction, and don't show line numbers or files). The data is all fine, but the UI is garbage now. This needs to be fixed before we can deploy this to users, but it's easily separable since it's all just display code.
Specifically, they look like this:
{F112270}
Test Plan:
I've migrated locally and put things through their paces, but it's hard to catch sketchy stuff locally because most of my test data is nonsense and bad migrations wouldn't necessarily look out of place.
IMPORTANT: I'm planning to push this to a branch and then shift production over to the branch, and run it for a day or two before bringing it to master.
I generally feel good about this change: it's not that big since we were able to separate a lot of pieces out of it, and it's pretty straightforward. That said, it's still one of the most scary/dangerous changes we've ever made.
Reviewers: btrahan
CC: chad, aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8210
Summary:
Ref T2222. Ref T4415. We're still writing Differential subscription stuff into this weird legacy `differential_relationship` table, which is like an edge table but extremely ancient.
Move it into a proper table.
I've removed `withSubscriptions()` from `DifferentialRevisionQuery`. It was weird, doesn't work consistently with other similar filters, and was only used by the API. Now it means "ccs", which is consistent with the ApplicationSearch UI and with Maniphest.
Test Plan:
Without migrating, added and removed subscribers via various workflows. Queried for subscribers. Everything worked as expected.
Ran the migration, verified data survived.
Reviewers: btrahan
Reviewed By: btrahan
CC: FacebookPOC, aran
Maniphest Tasks: T2222, T4415
Differential Revision: https://secure.phabricator.com/D8202
Summary:
Ref T2222. Currently, `DifferentialComment` stores both (a) the text of comments and (b) various other transaction details. This data needs to map to both Transactions and TransactionComments in the long run. This diff separates out all the data which is bound for the TransactionComment table, so that when we migrate `DifferentialComment` itself it will //only// need to migrate into the Transactions table. This is a much simpler migration than the inline comment one was, partly because it set up infrastructure and partly because the data is less complex.
Basically, I'm just proxying the read/write for the comment text into the other table. All readers already go through the Query class, and there are only three writers (preview, comment, implicit comment on diff update) which are all highly regular and straightforward to test.
We can also back out of this diff very easily: doing double writes cost only one line of code (`$this->content = $content;`) so we have proper double writes and a trivial revert path.
Test Plan:
- Without migrating, added comments and saw them show up.
- Migrated.
- Saw all the old comments, and no damage to the new ones.
- Added new comments.
- Used comment preview.
- Updated a revision to implicitly create an update comment and verified it looked OK.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8196
Summary:
Fixes T4379. Several changes:
- Migrate all project members into subscribers.
- When members are added or removed, subscribe or unsubscribe them.
- Show sub/unsub in the UI.
- Determine mailable membership of projects by querying subscribers.
Test Plan:
- As `duck`, joined a project.
- Added the project as a reviewer to a revision.
- Commented on the revision.
- Observed `duck` receive mail.
- Unsubscribed as `duck`.
- Observed no mail.
- Resubscribed as `duck`.
- Mail again.
- Joined/left project, checked sub/unsub status.
- Ran migration, looked at database.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, asherkin
Maniphest Tasks: T4379
Differential Revision: https://secure.phabricator.com/D8189
Summary: ...by the surprising step of changing how this data is stored from id to phid. Also a small fix to not allow "disabled" rules to be used as herald rule conditions, i.e. can't make a rule that depends on a disabled rule.
Test Plan: viewed existing herald rule that had a rule condition and noted nice new display using handle. made a new rule that had a rule condition and verified it worked correctly.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D8186
Summary:
Ref T4379. Long ago, the "Project" vs "ProjectProfile" split was intended to allow a bunch of special fields on projects without burdening the simple use cases, but CustomField handles that far better and far more generally, and doing this makes using ApplicationTransactions a pain to get right, so get rid of it.
The only remaining field is `profileImagePHID`, which we can just move to the main Project object. This is custom enough that I think it's reasonable not to express it as a custom field.
Test Plan: Created a project, set profile, edited project, viewed in typeahead, ran migration, verified database results.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4379
Differential Revision: https://secure.phabricator.com/D8183
Summary: Ref T4379. Major goal here is to remove `ProjectProfile` so all edits use ApplicationTransactions. This also makes things more flexible, allowing users to disable this field if they don't like it.
Test Plan: Ran migration, verified data survived, edited/created projects, reordered fields.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4379
Differential Revision: https://secure.phabricator.com/D8182
Summary: Ref T4375. We're going to need these for a bunch of infrastructure to work.
Test Plan: Ran migrations, checked DB, used `phid.query`.
Reviewers: btrahan, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T4375
Differential Revision: https://secure.phabricator.com/D8151
Summary: Ref T4375. We never join this table, so this is a pretty straight find/replace.
Test Plan: Browsed around Calendar, verified that nothing seemed broken. Saw my red dot in other apps.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4375
Differential Revision: https://secure.phabricator.com/D8145
Summary: Ref T3583. This doesn't add any dashboard/panel-specific code beyond headers/titles/buttons/etc., but allows you to create and view dashboards and panel skeletons.
Test Plan: See screenshots.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3583
Differential Revision: https://secure.phabricator.com/D8131
Summary: Ref T4368. We don't currently GC these tables, and the sent mail table is one of the largest on `secure.phabricator.com`. There's no value in retaining this information indefinitely. Instead, retain it for 90 days, which should be plenty of time to debug/diagnose any issues.
Test Plan: Ran `phd debug garbage`, saw it clean up a reasonable amount of data from these tables.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4368
Differential Revision: https://secure.phabricator.com/D8127
Summary:
Fixes T4202. We have old code in MetaMTA which implements gradual backoff and maximum retries.
However, we have more general code in the task queue which does this, too. We can just use the more general stuff in the task queue; it obsoletes the specific stuff in MetaMTA, which is more complex and ran into some kind of issue in T4202.
Remove `retryCount`, `nextRetry` (obsoleted by task queue retry mechanisms) and "simulated failures" (no longer in use).
Generally, modern infrastructure has replaced these mechanisms with more general ones.
Test Plan:
- Sent mail.
- Observed unsendable mail failing in reasonable ways in the queue.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4202
Differential Revision: https://secure.phabricator.com/D8115
Summary:
Ref T3583. General idea here is:
- Users will be able to create `DashboardPanel`s, which are things like the jump nav, or a minifeed, or recent assigned tasks, or recent tokens given, or whatever else.
- The `DashboardPanel`s can be combined into `Dashboard`s, which select specific panels and arrange them in some layout (and maybe have a few other options eventually).
- Then, you'll be able to set a specific `Dashboard` for your home page, and maybe for project home pages. But you can also use `Dashboard`s on their own if you just like dashboards.
My plan is pretty much:
- Put in basic infrastructure for dashboards (this diff).
- Add basic create/edit (next few diffs).
- Once dashboards sort of work, do the homepage integration.
This diff does very little: you can't create dashboards or panels yet, and thus there are no dashboards to look at. This is all skeleton code, pretty much.
IMPORTANT: We need an icon bwahahahahaha
Test Plan:
omg si purrfect
{F106367}
Reviewers: chad, btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3583
Differential Revision: https://secure.phabricator.com/D8109
Summary:
Moves away from ArcanistProjects:
- Adds storage for diffs to be directly associated with a repository (instead of indirectly, through arcanist projects). Not really populated yet.
- Drops `parentRevisionID`, which is obsoleted by the "Depends On" edge. This is not exposed in the UI anywhere and doesn't do anything. Resolves TODO.
Test Plan: Ran storage upgrades, browsed around, lots of `grep`.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D8072
Summary:
Ref T4327. I want to make change parsing testable; one thing which is blocking this is that the Git discovery process is still part of `PullLocal` daemon instead of being part of `DiscoveryEngine`. The unit test stuff which I want to use for change parsing relies on `DiscoveryEngine` to discover repositories during unit tests.
The major reason git discovery isn't part of `DiscoveryEngine` is that it relies on the messy "autoclose" logic, which we never implemented for Mercurial. Generally, I don't like how autoclose was implemented: it's complicated and gross and too hard to figure out and extend.
Instead, I want to do something more similar to what we do for pushes, which is cleaner overall. Basically this means remembering the old branch heads from the last time we parsed a repository, and figuring out what's new by comparing the old and new branch heads. This should give us several advantages:
- It should be simpler to understand than the autoclose stuff, which is pretty mind-numbing, at least for me.
- It will let us satisfy branch and tag queries cheaply (from the database) instead of having to go to the repository. We could also satisfy some ref-resolve queries from the database.
- It should be easier to extend to Mercurial.
This implements the basics -- pretty much a table to store the cursors, which we update only for Git for now.
Test Plan:
- Ran migration.
- Ran `bin/repository discover X --trace --verbose` on various repositories with branches and tags, before and after modifying pushes.
- Pushed commits to a git repo.
- Looked at database tables.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D7982
Summary:
Ref T4310. Fixes T3720. This change:
- Removes concurrent session limits. Instead, unused sessions are GC'd after a while.
- Collapses all existing "web-1", "web-2", etc., sessions into "web" sessions.
- Dramatically simplifies the code for establishing a session (like omg).
Test Plan: Ran migration, checked Sessions panel and database for sanity. Used existing session. Logged out, logged in. Ran Conduit commands.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4310, T3720
Differential Revision: https://secure.phabricator.com/D7978
Summary:
Ref T3116. This creates a policy rule where you can require a signature on a given legalpad document.
NOTE: signatures must be for the *latest* document version.
Test Plan: made a task have a custom policy requiring a legalpad signature. verified non-signers were locked out.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Maniphest Tasks: T3116
Differential Revision: https://secure.phabricator.com/D7977
Summary:
Ref T3720. Ref T4310. Currently, we limit the maximum number of concurrent sessions of each type. This is primarily because sessions predate garbage collection and we had no way to prevent the session table from growing fairly quickly and without bound unless we did this.
Now that we have GC (and it's modular!) we can just expire unused sessions after a while and throw them away:
- Add a `sessionExpires` column to the table, with a key.
- Add a GC for old sessions.
- When we establish a session, set `sessionExpires` to the current time plus the session TTL.
- When a user uses a session and has used up more than 20% of the time on it, extend the session.
In addition to this, we could also rotate sessions, but I think that provides very little value. If we do want to implement it, we should hold it until after T3720 / T4310.
Test Plan:
- Ran schema changes.
- Looked at database.
- Tested GC:
- Started GC.
- Set expires on one row to the past.
- Restarted GC.
- Verified GC nuked the session.
- Logged in.
- Logged out.
- Ran Conduit method.
- Tested refresh:
- Set threshold to 0.0001% instead of 20%.
- Loaded page.
- Saw a session extension ever few page loads.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4310, T3720
Differential Revision: https://secure.phabricator.com/D7976
Summary:
Ref T4310. Ref T3720. Two major things are going on here:
- I'm making this table work more like a standard table, which, e.g., makes `delete()` simpler to implement.
- Currently, the primary key is `(userPHID, type)`. I want to get rid of this, issue unlimited sessions, and GC old sessions. This means we can't have a unique key on `(userPHID, type)` anymore. This removes it as the primary key and adds it as a normal key instead. There's no functional change -- the code to generate sessions guarantees that it will never write duplicate rows or write additional rows -- but allows us to drop the `-1`, `-2` qualifiers in the future.
- Also of note, our task is made far simpler here because MySQL will automatically assign values to new `AUTO_INCREMENT` columns, so we don't have to migrate to get real IDs.
Test Plan: Ran migrations, verified table looked sane. Logged out, logged in.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3720, T4310
Differential Revision: https://secure.phabricator.com/D7975
Summary: Adds "verified" and "secretKey" to Legalpad document signatures. For logged in users using an email address they own, things are verified right away. Otherwise, the email is sent a verification letter. When the user clicks the link the signature is marked verified.
Test Plan: signed the document with a bogus email address not logged in. verified the email that would be sent looked good from command line. followed link and successfully verified bogus email address
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, epriestley, aran, asherkin
Maniphest Tasks: T4283
Differential Revision: https://secure.phabricator.com/D7930
Summary: This adds in the create flow for the Project board columns on the super secret board page which totally doesn't do anything right now.
Test Plan:
1. Apply diff.
2. Go to super secret page.
3. Click link close to top with a way too long name.
4. Enter a name for the column.
5. Enjoy a new column briefly before realising you cannot remove it.
6. Stay happy!
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: tmaroschik, Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7925
Summary:
Ref T2015. Adds human-readable names to Drydock blueprints.
Also the new patches stuff is so much nicer.
Test Plan: Edited, created, and reviewed migrated blueprints.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2015
Differential Revision: https://secure.phabricator.com/D7918
Summary:
Ref T1049. Currently you can cancel a build, but now that we're tracking a lot more state we can stop, resume, and restart builds.
When the user issues a command against a build, I'm writing it into an auxiliary queue (`HarbormasterBuildCommand`) and then reading them out in the worker. This is mostly to avoid race messes where we try to `save()` the object in multiple places: basically, the BuildEngine is the //only// thing that writes to Build objects, and it holds a lock while it does it.
Test Plan:
- Created a plan which runs "sleep 2" a bunch of times in a row.
- Stopped, resumed, and restarted it.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, chad
Maniphest Tasks: T1049
Differential Revision: https://secure.phabricator.com/D7892
Summary:
If you have private replies on and a Macro reply handler set, we try to access `getMailKey()` and fail. See P1039 for a trace.
(Thanks to @Korvin for picking this up.)
Test Plan: Set configuration, repro'd the exception, applied the patch, then disabled/enabled a macro.
Reviewers: btrahan
Reviewed By: btrahan
CC: Korvin, aran
Differential Revision: https://secure.phabricator.com/D7896
