Summary: See PHI1319. Ref T13291. Bump the remarkup cache version, since the old JIRA / Asana rules may exist in the partial cached representation of remarkup blocks from older versions.
Test Plan: Typed some comments with various formatting, saw remarkup work fine.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291
Differential Revision: https://secure.phabricator.com/D20619
Summary:
Ref T13328. Currently, we read from `mysqldump` something like this:
```
until (done) {
for (100 ms) {
mysqldump > in-memory-buffer;
}
in-memory-buffer > disk;
}
```
This general structure isn't great. In this use case, where we're streaming a large amount of data from a source to a sink, we'd prefer to have a "select()"-like way to interact with futures, so our code is called after every read (or maybe once some small buffer fills up, if we want to do the writes in larger chunks).
We don't currently have this (`FutureIterator` can wake up every X milliseconds, or on future exit, but, today, can not wake for readable futures), so we may buffer an arbitrary amount of data into memory (however much data `mysqldump` can write in 100ms).
Reduce the update frequency from 100ms to 10ms, and limit the buffer size to 32MB. This effectively imposes an artificial 3,200MB/sec limit on throughput, but hopefully that's fast enough that we'll have a "wake on readable" mechanism by the time it's a problem.
Test Plan:
- Replaced `mysqldump` with `cat /dev/zero` as the source command, to get fast input.
- Ran `bin/storage dump` with `var_dump()` on the buffer size.
- Before change: saw arbitrarily large buffers (300MB+).
- After change: saw consistent maximum buffer size of 32MB.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13328
Differential Revision: https://secure.phabricator.com/D20617
Summary:
Fixes T13326. In D20571, I slightly generalized construction of an iterator over a set of files, but missed some code in other "bin/files ..." commands which was also affected.
Today, basically all of these workflows define their own "--all" and "names" flags. Pull these definitions up and implement them more consistently.
Test Plan: Ran multiple different `bin/files` commands with different combinations of arguments, saw consistent handling of iterator construction.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13326
Differential Revision: https://secure.phabricator.com/D20614
Summary:
Fixes T13327. Currently, when we try to bill an account and all members are disabled, we fail temporarily and the task retries forever.
At least for now, just treat this as a permanent failure.
Test Plan:
- Used `bin/phortune invoice` to generate a normal invoice for a regular subscription.
- Disabled all the account members, then tried again. Got a helpful permanent failure:
```
$ ./bin/phortune invoice --subscription PHID-PSUB-kbedwt5cyepoc6tohjq5 --auto-range
Set current time to Mon, Jun 24, 2:47 PM.
Preparing to invoice subscription "localb.phacility.com" from Fri, May 31, 10:14 AM to Sun, Jun 30, 10:14 AM.
WARNING
Manually invoicing will double bill payment accounts if the range overlaps an
existing or future invoice. This script is intended for testing and
development, and should not be part of routine billing operations. If you
continue, you may incorrectly overcharge customers.
Really invoice this subscription? [y/N] y
[2019-06-24 14:47:57] EXCEPTION: (PhabricatorWorkerPermanentFailureException) All members of the account ("PHID-ACNT-qp54y3unedoaxgkkjpj4") for this subscription ("PHID-PSUB-kbedwt5cyepoc6tohjq5") are disabled. at [<phabricator>/src/applications/phortune/worker/PhortuneSubscriptionWorker.php:88]
arcanist(head=experimental, ref.master=d92fa96366c0, ref.experimental=db4cd55d4673), corgi(head=master, ref.master=6371578c9d32), instances(head=stable, ref.master=ba9e4a19df1c, ref.stable=37fb1f4917c7), libcore(), phabricator(head=master, ref.master=65bc481c91de, custom=11), phutil(head=master, ref.master=7adfe4e4f4a3), services(head=master, ref.master=5424383159ac)
#0 PhortuneSubscriptionWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:124]
#1 PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:163]
#2 PhabricatorWorker::scheduleTask(string, array, array) called at [<phabricator>/src/applications/phortune/management/PhabricatorPhortuneManagementInvoiceWorkflow.php:169]
#3 PhabricatorPhortuneManagementInvoiceWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:457]
#4 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:349]
#5 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/setup/manage_phortune.php:21]
$
```
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13327
Differential Revision: https://secure.phabricator.com/D20613
Summary: Ref T13321. The daemons no longer write PID files, so we no longer need to pass any of this stuff to them.
Test Plan: Grepped for affected symbols.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13321
Differential Revision: https://secure.phabricator.com/D20608
Summary:
Ref T13321. This gets rid of the last pidfile readers in Phabricator; we just use the process list instead.
These commands always only work on the current instance since they don't make much sense otherwise.
Test Plan: Ran `bin/phd start` and `bin/phd reload` with and without daemons running.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13321
Differential Revision: https://secure.phabricator.com/D20606
Summary:
Ref T13321. Fixes T11037. Realign "bin/phd status" to just mean "show daemon processes on this host".
The value of `bin/phd status` as a mixed remote/local command isn't clear, and the current output is a confusing mess (see T11037).
This also continues letting us move away from PID files.
Test Plan: Ran `bin/phd status`, saw sensible local process status.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13321, T11037
Differential Revision: https://secure.phabricator.com/D20604
Summary:
Ref T13321. Previously, the behavior was:
- `bin/phd stop --gently`: Stop all daemons with PID files that belong to the current instance.
- `bin/phd stop`: Stop all daemons with PID files that belong to the current instance. Complain if there are more processes.
- `bin/phd stop --force`: Stop all processes that look like daemons, ignoring instances.
The new behavior is:
- `bin/phd stop`: Stop all processes that look like daemons and belong to the current instance.
- `bin/phd stop --force`: Stop all processes that look like daemons, period.
Test Plan: Grep / documentation only.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13321
Differential Revision: https://secure.phabricator.com/D20602
Summary:
Ref T13321. Depends on D20600. Make `bin/phd stop` mean:
- `bin/phd stop`: Stop all processes which have daemon process titles. If we're instanced, only stop daemons for the current instance.
- `bin/phd stop --force`: Stop all processes which have deamon process titles for any instance.
We no longer read or care about PID files on disk, and this moves us away from PID files.
This makes unusual flag `--gently` do nothing. A followup will update the documentation and flags to reflect actual usage/behavior.
This also removes the ability to stop specific PIDs. This was somewhat useful long, long ago when you might explicitly run different copies of the `PullLocal` daemon with flags to control which repositories they updated, but with the advent of clustering it's no longer valid to run custom daemon loadouts.
Test Plan: Ran `bin/phd start`, then `bin/phd stop`. Saw instance daemons stop. Ran `bin/phd stop --force`, saw all daemons stop.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13321
Differential Revision: https://secure.phabricator.com/D20601
Summary:
See <https://discourse.phabricator-community.org/t/cannot-audit-a-git-commit/2848>. In D20581, I made some audit behavior dependent upon identities, but the actual edit flow doesn't load them. This can cause us to raise an "attach identities first" exception in the bowels of the edit workflow and trigger unexpected behavior at top level.
Load identities when editing a commit so that the transaction flows have access to identity information and can use it to figure out if a user is an author, etc.
Test Plan:
- As an auditor, applied an "Accept Commit" action to an open audit after D20581.
- Before patch: accept no-ops internally since the preconditions throw.
- After patch: accept works properly.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20612
Summary:
Ref T13291. See PHI1312. Currently, if you link to a JIRA or Asana issue with an anchor (`#asdf`) or query parameters (`?a=b`), we:
- treat the link as an external object reference and attempt a lookup on it;
- if the lookup succeeds, we discard the fragment or parameters when re-rendering the rich link (with the issue/task title).
Particularly, the re-rendering part uses the canonical URI of the object, and can discard these parameters/fragments, which is broken/bad.
As a first pass at improving this, just don't apply special behavior for links with anchors or parameters -- simply treat them as links.
In some future change, we could specialize this behavior and permit certain known parameters or anchors or something, but these use cases are likely fairly marginal.
Test Plan:
Before:
{F6516392}
After:
{F6516393}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291
Differential Revision: https://secure.phabricator.com/D20592
Summary: Ref T13319. Ref PHI1302. Migrate `PhabricatorEditEngineConfigurationTransaction` to modular transactions and add some additional transaction rendering to make these edits less opaque.
Test Plan: Hit all the form edit controllers, viewed resulting transaction timeline.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T13319
Differential Revision: https://secure.phabricator.com/D20595
Summary:
Fixes T13324. Ref PHI1288. Currently, if you edit an Owners package that has some paths with no trailing slashes (like `README.md`) so their internal names and display names differ (`/README.md` display, vs `/README.md/` internal), the "Show Details" in the transaction log shows the path as re-normalized even if you didn't touch it.
Instead, be more careful about handling display paths vs internal paths.
(This code on the whole is significantly less clear than it probably could be, but this issue is so minor that I'm hesitant to start ripping things out.)
Test Plan:
- In a package with some paths like `/src/` and some paths like `/src`:
- Added new paths.
- Removed paths.
- Changed paths from `/src/` to `/src`.
- Changed paths from `/src` to `/src/`.
In all cases, the "paths" list and the transaction record identically reflected the edit in the way I expected them to.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13324
Differential Revision: https://secure.phabricator.com/D20596
Summary:
Ref T13319. Currently, transactions about changes to a default form value use a raw internal key for the affected field and don't show the actual value change.
An ideal implementation will likely require us to specialize a great deal of rendering, but we can do much better than we currently do without too much work:
- Try to pull the actual `EditField` object for the key so we can `getLabel()` it and get a human-readable label (like `Visible To` instead of `policy.view`).
- Add a "(Show Changes)" action that dumps the raw values as more-or-less JSON, so you can at least figure out what happened if you're sophisticated enough.
Test Plan:
Before:
{F6516640}
After:
{F6516642}
The quality of "Show Details" varies a lot. For some fields, like "Description", it's pretty good:
{F6516645}
For others, like "Assigned To", it's better than nothing but pretty technical:
{F6516647}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13319
Differential Revision: https://secure.phabricator.com/D20594
Summary:
Fixes T13317. On `admin.phacility.com`, an enterprising user added `noreply@admin.phacility.com` to their account. This caused them to become CC'd on several support issues over the last year, because we send mail "From" this address and it can get CC'd via reply/reply all/whatever else.
The original driving goal here is that if I reply to a task email and CC you on my reply, that should count as a CC in Phabricator, since this aligns with user intent and keeps them in the loop.
This misfire on `noreply@` is ultimately harmless (being CC'd does not grant the user access permission, see T4411), but confusing and undesirable. Instead:
- Don't allow reserved addresses ("noreply@", "ssladmin@", etc) to trigger this subscribe-via-CC behavior.
- Only count verified addresses as legitimate user recipients.
Test Plan:
- Added a `bin/mail receive-test --cc ...` flag to make this easier to test.
- Sent mail as `bin/mail receive-test --to X --as alice --cc bailey@verified.com`. Bailey was CC'd both before and after the change.
- Sent mail as `bin/mail receive-test --to X --as alice --cc unverified@imaginary.com`, an address which Bailey has added to her account but not verified.
- Before change: Bailey was CC'd on the task anyway.
- After change: Bailey is not CC'd on the task.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13317
Differential Revision: https://secure.phabricator.com/D20593
Summary:
Fixes T13312. Currently, {nav Manage > Branches} has a list of branches on the same page. This has a few minor issues:
- Pager is at the top (see T13312), which is weird.
- "Default" icon is mystery meat.
- Table is kind of pointless/redundant in general?
Previously, this table had more information about technical status of each branch (autoclose/track/publish) but most of these details have been simplified/eliminated, and the main "Branches" view now has more information than it did before.
Get rid of this and just link to the main view.
Test Plan: Viewed "Branches" in UI, saw a link to the main view instead of a weird table.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13312
Differential Revision: https://secure.phabricator.com/D20584
Summary:
Fixes T13309. If you void the warranty on a repository on disk and turn it into a shallow clone, Phabricator currently can't serve it.
We don't support hosting shallow working copies, but we should still parse and proxy the protocol rather than breaking in a mysterious way.
Test Plan:
- Created a shallow working copy with `mv X X.full; git clone --depth Y file://.../X.full X` in the storage directory on disk.
- Cloned it with `git clone <uri>`.
- Deleted all the refs inside it so the wire only has "shallow" frames; cloned it.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13309
Differential Revision: https://secure.phabricator.com/D20577
Summary:
See <https://discourse.phabricator-community.org/t/view-task-from-maniphest-e-mail-doesnt-have-url/2827>.
I added "View Task" / "View Commit" buttons recently but the logic for generating URIs isn't quite right. Fix it up.
Test Plan:
- Commented on a task.
- Used `bin/mail show-outbound --id ... --dump-html > out.html` to dump the HTML.
- Previewed the HTML in a browser.
- This time, actually clicked the button to go to the task.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20586
Summary:
Fixes T13304. Shell pipes and redirects do not have robust behavior when errors occur. We provide "--compress" and "--output" flags as robust alternatives, but do not currently recommend their use.
- Recommend their use, since their error handling behavior is more robust in the face of issues like full disks.
- If "--compress" is provided but won't work because the "zlib" extension is missing, raise an explicit error. I believe this extension is very common and this error should be rare. If that turns out to be untrue, we could take another look at this.
- Also, verify some flag usage sooner so we can exit with an error faster if you mistype a "bin/storage dump" command.
Test Plan: Read documentation, hit affected error cases, did a dump and spot-checked that it came out sane looking.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13304
Differential Revision: https://secure.phabricator.com/D20572
Summary:
See <https://discourse.phabricator-community.org/t/unhandled-exception-when-logging-in-with-mfa/2828>. The recent changes to turn `msort()` on a vector an error have smoked out a few more of these mistakes.
These cases do not meaningfully rely on sort stability so there's no real bug being fixed, but we'd still prefer `msortv()`.
Test Plan: Viewed MFA and External Account settings panels. Did a `git grep 'msort(' | grep -i vector` for any more obvious callsites, but none turned up.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20587
Summary:
Fixes T13315. See that task for discussion.
Without `--background`, we currently treat this as a catastrophic failure, but it's relatively routine for some repository states. We can safely continue reparsing other steps.
Test Plan: Ran `bin/repository reparse --all X --message` with commits faked to all be unreachable. Got warnings instead of a hard failure on first problem.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13315
Differential Revision: https://secure.phabricator.com/D20588
Summary:
Depends on D20580. Fixes T13311. When we choose which actions to show a user, we can either show them "auditor" actions (like "raise concern") or "author" actions (like "request verification").
Currently, we don't show "author" actions if you're the author of the commit via an identity mapping, but we should. Use identity mappings where they exist.
(Because I've implemented `getEffectiveAuthorPHID()` in a way that requires `$data` be attached, it's possible this will make something throw a "DataNotAttached" exception, but: probably it won't?; and that's easy to fix if it happens.)
Test Plan:
See D20580. As `@alice`, viewed the commit in the UI.
- Before: got auditor actions presented to me.
- After: got author actions.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13311
Differential Revision: https://secure.phabricator.com/D20581
Summary:
Ref T13311. We currently don't use committer identity mappings when triggering audits, so if a user is only associated with an identity via manual mapping we won't treat them as the author.
Instead, use the identity and manual mapping if they're available.
Test Plan:
- Pushed a commit as `xyz <xyz@example.org>`, an address with no corresponding user.
- In the UI, manually associated that identity with user `@alice`.
- Ran `bin/repository reparse --publish <hash>` to trigger audits and publishing for the commit.
- Before: observed the `$author_phid` was `null`.
- After: observed the `$author_phid` is Alice.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13311
Differential Revision: https://secure.phabricator.com/D20580
Summary:
Fixes T13310. Use cases in the form "users with no access to any spaces can not <do things>" are generally unsupported (that is, we consider this to mean that the install is misconfigured), but "log out" is a somewhat more reasonable sort of thing to do and easy to support.
Drop the requirement that users be logged in to access the Logout controller. This skips the check for access to any Spaces and allows users with no Spaces to log out.
For users who are already logged out, this just redirects home with no effect.
Test Plan:
- As a user with access to no Spaces, logged out. (Before: error; after: worked).
- As a logged-out user, logged out (was redirected).
- As a normal user, logged out (normal logout).
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13310
Differential Revision: https://secure.phabricator.com/D20578
Summary:
Fixes T13313. The "Download Raw Diff" workflow in Differential currently uses an older way of interacting with Files that doesn't engage the chunk engine and can't handle 8MB+ files.
Update to `IteratorFileUploadSource` -- we're still passing in a single giant blob, but this approach can be chunked.
This will still break somewhere north of 8MB (it will break at 2GB with the PHP string limit if nowhere sooner, since we're putting the entire raw diff in `$raw_diff` rather than using a rope/stream) but will likely survive diffs in the hundreds-of-megabytes range for now.
Test Plan:
- Added `str_repeat('x', 1024 * 1024 * 9)` to the `$raw_diff` to create a 9MB+ diff.
- Configured file storage with no engine explicitly configured for >8MB chunks (i.e., "reasonably").
- Clicked "Download Raw Diff".
- Before: misleading file storage engine error ("no engine can store this file").
- After: large, raw diff response.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13313
Differential Revision: https://secure.phabricator.com/D20579
Summary: Ref T13303. See B22967. This should be "msortv()" but didn't get updated properly.
Test Plan: The system works!
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13303
Differential Revision: https://secure.phabricator.com/D20585
Summary: Ref T13303. I upgraded this to a vector-based sort but forgot to type a "v", which means the sort has different stability under PHP 5.5. See D20582 for a root cause fix.
Test Plan: Locally, on PHP7, not much changes. I expect this to fix the odd selection of title stories in mail and notification stories on `secure`, which is running PHP 5.5.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13303
Differential Revision: https://secure.phabricator.com/D20583
Summary:
Fixes T13307. We currently require "CAN_EDIT" to sign actions, but it's fine to sign a comment with only "CAN_INTERACT".
Since the actions like "Accept Revision" already work like this, the fix is one line.
Test Plan: {F6488135}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13307
Differential Revision: https://secure.phabricator.com/D20574
Summary:
Ref T13306. Currently, there's no easy way to import a third-party local-disk file dump into a Phacility instance.
Add some more options to `bin/files migrate` to support this. In particular, this enables:
```
$ ./bin/files --from-engine local-disk --engine amazon-s3 --local-disk-source path/to/backup
```
...to import these files into S3 directly.
These are general-purpose options and theoretically useful in other use cases, although realistically those cases are probably very rare.
Test Plan: Used `bin/files` with the new options to move files in and out of local disk storage in an arbitrary backup directory. Got clean exports/imports.
Reviewers: amckinley
Maniphest Tasks: T13306
Differential Revision: https://secure.phabricator.com/D20571
Summary:
Ref T13298. Add a simple profiler as a starting point to catch any egregiously expensive rules or conditions.
This doesn't profile rule actions, so if "Add subscriber" (or whatever) is outrageously expensive it won't show up on the profile. Right now, actions get evaluated inside the Adapter so they're hard to profile. A future change could likely dig them out without too much trouble. I generally expect actions to be less expensive than conditions.
This also can't pin down a //specific// condition being expensive, but if you see that `H123` takes 20s to evaluate you can probably guess that the giant complicated regex is the expensive part.
Test Plan: {F6473407}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13298
Differential Revision: https://secure.phabricator.com/D20566
Summary:
Depends on D20566. Ref T13298. See PHI1280. Currently, there's no clean way to disable problematic personal rules. This comes up occasionally and sometimes isn't really the best approach to solving a problem, but is a generally reasonable capability to provide.
Allow Herald rules (including personal rules) to be disabled/enabled via `bin/herald rule ... --disable/--enable`.
Test Plan: Used the CLI to disable and enable a personal rule.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: jmeador
Maniphest Tasks: T13298
Differential Revision: https://secure.phabricator.com/D20567
Summary:
Fixes T13300. Currently, if you create a revision and then immediately land it (either using `--draft` or just beating Harbormaster to the punch) it can be stuck in "Draft" forever.
Instead, count landing changes like this as a publishing action.
Test Plan:
- Used `arc diff --hold` to create a revision, then pushed the commit immediately.
- Before change: revision closed, but was stuck in draft.
- After change: revision closed and was promoted out of draft.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13300
Differential Revision: https://secure.phabricator.com/D20565
Summary:
See PHI1118. That issue may describe more than one bug, but the recent ordering changes to the import pipeline likely make this at least part of the problem.
Previously, commits would always close associated revisions before we made it to the "publish" step. This is no longer true, so we might be triggering audits on a commit before the associated revision actually closes.
Accommodate this by counting a revision in either "Accepted" or "Published (Was Previously Accepted)" as "reviewed".
Test Plan:
- With commit C affecting paths in package P with "Audit Unreviewed Commits and Commits With No Owner Involvement", associated with revision R, with both R and C authored by the same user, and "R" in the state "Accepted", used `bin/repository reparse --publish <hash>` to republish the commit.
- Before change: audit by package P triggered.
- After change: audit by package P no longer triggered.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20564
Summary:
Ref T13303. In D20525 I fixed an issue where transaction rendering could use cached values with the wrong viewer by reloading transactions.
However, reloading transactions may also reorder them as a side effect, since `withPHIDs(...)` does not imply an order. This can make transaction rendering order in mail wrong/inconsistent.
Instead, reorder the transactions before continuing so mail transaction order is consistent.
Test Plan: Applied a group of transactions to a task, saw a more consistent rendering order in mail after the change.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13303
Differential Revision: https://secure.phabricator.com/D20563
Summary:
See downstream <https://phabricator.wikimedia.org/T1050>. Some time ago, we added a "View Revision" button to Differential mail. This hasn't created any problems and generally seems good / desirable.
It isn't trivial to just add everywhere since we need a translation string in each case, but at least add it to Maniphest for now. Going forward, we can fill in more applications as they come up.
Test Plan:
Used `bin/mail show-outbound --id <x> --dump-html`:
{F6470461}
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20561
Summary:
See downstream <https://phabricator.wikimedia.org/T88655>. This is very marginal, but we currently allow comments consisting of //only// whitespace.
These are probably always mistakes, so treat them like completely empty comments.
(We intentionally do not trim leading or trailing whitespace from comments when posting them becuase leading spaces can be used to trigger codeblock formatting.)
Test Plan:
- Posted empty, nonempty, and whitespace-only comments.
- Whitespace-only comments now have the same behavior as truly empty comments (e.g., do not actually generate a transaction).
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20562
Summary: These instructions are fairly old and can be a little fancier and more clear in the context of modern Phabricator. Drop the reference to "HPHP", link the actual timezone list, wordsmith a little.
Test Plan: d( O_o )b
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20560
Summary:
See downstream <https://phabricator.wikimedia.org/T902>. Currently, timezones are rendered with their raw internal names (like `America/Los_Angeles`) which include underscores.
Replacing underscores with spaces is a more human-readable (and perhaps meaningfully better for things like screen readers, although this is pure speculation).
There's some vague argument against this, like "administrators may need to set a raw internal value in `phabricator.timezone` and this could mislead them", but we already give a pretty good error message if you do this and could improve hinting if necessary.
Test Plan: Viewed timezone list in {nav Settings} and the timezone "reconcile" dialog, saw a more-readable "Los Angeles".
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20559
Summary:
Ref T13289. See D20551. In D20551, I implemented some "CAN_INTERACT" checks against certain edits, but these checks end up testing "CAN_INTERACT" against objects like Conpherence threads which do not support a distinct "CAN_INTERACT" permission. I misrembered how the "CAN_INTERACT" fallback to "CAN_VIEW" actually works: it's not fully automatic, and needs some explicit "interact, or view if interact is not available" checks.
Use the "interact" wrappers to test these policies so they fall back to "CAN_VIEW" if an object does not support "CAN_INTERACT". Generally, objects which have a "locked" state have a separate "CAN_INTERACT" permission; objects which don't have a "locked" state do not.
Test Plan: Created and edited comments in Conpherence (or most applications other than Maniphest).
Reviewers: amckinley
Maniphest Tasks: T13289
Differential Revision: https://secure.phabricator.com/D20558
Summary:
Ref T13289. If you do this:
- Subscribe to a task (so we don't generate a subscribe side-effect later).
- Prepare a transaction group: sign with MFA, change projects (don't make any changes), add a comment.
- Submit the transaction group.
...you'll get prompted "Some actions don't have any effect (the non-change to projects), apply remaining effects?".
If you confirm, you get MFA'd, but the MFA flow loses the "continue" confirmation, so you get trapped in a workflow loop of confirming and MFA'ing.
Instead, retain the "continue" bit through the MFA.
Also, don't show "You can't sign an empty transaction group" if there's a comment.
See also T13295, since the amount of magic here can probably be reduced. There's likely little reason for "continue" or "hisec" to be magic nowadays.
Test Plan:
- Went through the workflow above.
- Before: looping workflow.
- After: "Continue" carries through the MFA gate.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13289
Differential Revision: https://secure.phabricator.com/D20552
Summary: Ref T11741. See PHI1276. After the switch to "Ferret", this table has no remaining readers or writers.
Test Plan:
- Ran `bin/storage upgrade -f`, no warnings.
- Grepped for class name, table name, `stemmedCorpus` column; got no relevant hits.
- Did a fulltext search.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T11741
Differential Revision: https://secure.phabricator.com/D20549
Summary:
Ref T13289. This tightens up a couple of corner cases around locked threads.
Locking is primarily motivated by two use cases: stopping nonproductive conversations on open source installs (similar to GitHub's feature); and freezing object state for audit/record-keeping purposes.
Currently, you can edit or remove comments on a locked thread, but neither use case is well-served by allowing this. Require "CAN_INTERACT" to edit or remove a comment.
Administrators can still remove comments from a locked thread to serve "lock a flamewar, then clean it up", since "Remove Comment" on a comment you don't own is fairly unambiguously an administrative action.
Test Plan:
- On a locked task, tried to edit and remove my comments as a non-administrator. Saw appropriate disabled UI state and error dialogs (actions were disallowed).
- On a locked task, tried to remove another user's comments as an administrator. This works.
- On a normal task, edited comments normally.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13289
Differential Revision: https://secure.phabricator.com/D20551
Summary:
Ref T13289. See <https://discourse.phabricator-community.org/t/fatal-error-in-pagination-in-drydock-resources-host-logs-all-logs/2735>.
`bin/drydock lease` and the web UI for reviewing all object logs when there is more than one page of logs didn't get fully updated to the new cursors.
- Use a cursor pager in `bin/drydock lease`.
- Implement `withIDs()` in `LeaseQuery` so the default paging works properly.
Test Plan:
- Ran `bin/drydock lease`, got a lease with log output along the way.
- Set page size to 2, viewed host logs with multiple pages, paged to page 2.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13289
Differential Revision: https://secure.phabricator.com/D20553
Summary:
Depends on D20546. Ref T13283. Currently, if you do something (transactions "A", "B") and Herald does some things in response (transaction "C"), Herald acts only on the things you did ("A", "B") since the thing it did ("C") didn't exist yet, until it ran.
However, if you use the test console to test rules against the object we'll pick up all three transactions since they're all part of the same group. This isn't ideal.
To fix this, skip transactions which Herald applied, since it obviously didn't consider them when it was evaluating.
Test Plan:
- Created a revision, in the presence of a Herald rule that adds reviewers.
- Then, ran the revision through the test console.
- Before: saw the "Herald added reviewers: ..." transaction in the transaction group Herald evaluated.
- After: saw only authentic human transactions.
{F6464064}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13283
Differential Revision: https://secure.phabricator.com/D20547
Summary:
Ref T13283. Currently, each Editor sets its own group ID, so if you create a revision and then Herald does some stuff, the two groups of transactions get different group IDs.
This means the test console is slightly misleading (it will only pick up the Herald transactions). It's going to be misleading anyway (Herald obviously can't evaluate Herald transactions) but this is at least a little closer to reality and stops Herald actions from masking non-Herald actions.
Test Plan:
- Created a revision. Herald applied one transaction.
- Used the test console.
- Before: The test console only picked up the single most recent Herald transaction.
- After: The test console picked up the whole transaction group.
{F6464059}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13283
Differential Revision: https://secure.phabricator.com/D20546
Summary:
Ref T13289. When you create a Phriction document, you currently get an email with the whole new content as a "diff".
You also get extra transactions in the email and on the page.
This is because Phriction isn't on EditEngine and doesn't mark "create" transactions in a modern way. Get them marked properly to fix these obviously-broken behaviors. This can all go away once Phriction switches to EditEngine, although I don't have any particular plans to do that in the immediate future.
Test Plan:
- Created a new document, viewed email, no longer saw redundant "edited content" transaction or "CHANGES TO CONTENT" diff.
- Updated a document, viewed email, got interdiff.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13289
Differential Revision: https://secure.phabricator.com/D20548
Summary:
Ref T13290. Prior to recent changes, if we parsed some commit C which was associated with a revision R, but R was already closed, we'd skip the whole set of updates because the "close the revision" transaction would fail and we'd throw because we did not `setContinueOnNoEffect()`.
We now continue on no effect so we can get the edge ("commit has revision" / "revision has commit"), since we want it in all cases, but this means we may also apply an extra "Updated revision to reflect committed changes" transaction and new diff. This can happen even if we're careful about not trying to apply this transaction to closed revisions, since two workers may race. (Today, we aren't too careful about this.)
To fix this, just make this transaction no-op itself if the revision is already closed by the time it tries to apply.
This happened on D20451 because a merge commit with the same hash as the last diff was pushed, but it's easiest to reproduce by just running `bin/repository reparse --message <commit>`, which updates related revisions with a new diff every time.
Test Plan:
- Ran `bin/repository reparse --messsage <commit>` several times, on a commit with an associated revision.
- Before: each run attached a new diff and created a new "updated to reflect committed changes" transaction.
- After: repeated runs had no effects.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13290
Differential Revision: https://secure.phabricator.com/D20545
Summary:
Ref T13290. Ref T13291. Now that a full URI is a "mention", the full URI in "Differential Revision: ..." also triggers a mention.
Stop it from doing that, since these mentions are silly/redundant/unintended.
The API here is also slightly odd; simplify it a little bit to get rid of doing "append" with "get + append + set".
Test Plan: Used `bin/repository reparse --publish` to republish commits with "Differential Revision: ..." and verified that the revision PHID was properly dropped from the mention list.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291, T13290
Differential Revision: https://secure.phabricator.com/D20544
Summary:
See PHI1222. When we publish several transactions to feed at once, we sort them by "action strength" to figure out which one gets to be the title story.
This sort currently uses `msort()`, which uses `asort()`, which is not a stable sort and has inconsistent behavior across PHP versions:
{F6463721}
Switch to `msortv()`, which is a stable sort. Previously, see also T6861.
If all transactions have the same strength, we'll now consistently pick the first one.
This probably (?) does not impact anything in the upstream, but is good from a consistency point of view.
Test Plan:
Top story was published after this change and uses the chronologically first transaction as the title story.
Bottom story was published before this change and uses the chronologically second transaction as the title story.
Both stories have two transactions with the same strength ("create" + "add reviewer").
{F6463722}
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20540
Summary:
Ref T13279. See that task for some discussion.
The accumulations of some of the datasets may be negative (e.g., if more tasks are moved out of a project than into it) which can lead to negative area in the stacked chart.
Introduce `min(...)` and `max(...)` to separate a function into points above or below some line, then mangle the areas to pick the negative and positive regions apart so they at least have a plausible physical interpretation and none of the areas are negative.
This is presumably not a final version, I'm just trying to produce a chart that isn't a sequence of overlapping regions with negative areas that is "technically" correct but not really possible to interpret.
Test Plan: {F6439195}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20506
Summary:
Ref T13279. Currently, we store a fairly low-level description of functions and datasets in a chart. This will create problems with (for example) translating function labels.
If you view a chart someone links you, it should say "El Charto" if you speak Spanish, not "The Chart" if the original viewer speaks English.
To support this, store a slightly higher level version of the chart: the chart engine key, plus configuration parameters. This is very similar to how SearchEngine works.
For example, the burndown chart now stores a list of project PHIDs, instead of a list of `[accumulate [sum [fact task.open <project-phid>]]]` functions.
(This leaves some serialization code with no callsites, but we may eventually have a "CustomChartEngine" which stores raw functions, so I'm leaving it for now.)
As a result, function labels provided by the chart engine are now translatable.
(Note that the actual chart is meaningless since the underlying facts can't be stacked like they're being stacked, as some are negative in some areas of their accumulation.)
Test Plan: {F6439121}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20504
Summary:
Ref T13279. Replace the hard-coded default range with a range computed by examining the chart data.
Instead of having a "Dataset" return a blob of wire data, "Dataset" now returns a structure with raw wire data plus a range. I expect to add more structured data here in future changes (tooltip/hover event data, maybe function labels).
Test Plan: {F6439101}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20503
Summary: Ref T13279. Slightly simplify domain handling by putting all the "[x, y]" stuff in an Interval class. I'm planning to do something similar for ranges next, so this should make that easierr.
Test Plan: Viewed chart, saw same chart as before.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20502
Summary: Ref T13279. Makes charts incrementally more useful by allowing the server to provide labels and colors for functions.
Test Plan: {F6438872}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20501
Summary:
Ref T13279. Adds client-side support for rendering function labels on charts, then labels every function as important data.
Works okay on mobile, although I'm not planning to target mobile terribly heavily for v0.
Test Plan: {F6438860}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20500
Summary:
Ref T13279. This adds support for:
- Datasets can have types, like "stacked area".
- Datasets can have multiple functions.
- Charts can store dataset types and datasets with multiple functions.
- Adds a "stacked area" dataset.
- Makes D3 actually draw a stacked area chart.
Lots of rough edges here still, but the result looks slightly more like it's supposed to look.
D3 can do some of this logic itself, like adding up the area stacks on top of one another with `d3.stack()`. I'm doing it in PHP instead because I think it's a bit easier to debug, and it gives us more options for things like caching or "export to CSV" or "export to API" or rendering a data table under the chart or whatever.
Test Plan: {F6427780}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20498
Summary:
Ref T13279. For now, we need to render burndowns from both Maniphest (legacy) and Projects (new prototype).
Consolidate this logic into a "BurndownChartEngine". I plan to expand this to work a bit like a "SearchEngine", and serve as a UI layer on top of the raw chart features.
The old "ChartEngine" is now "ChartRenderingEngine".
Test Plan:
- Viewed burndowns ("burnups") in Maniphest.
- Viewed burndowns in Projects.
- Saw the same chart.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20496
Summary:
Ref T13279. Since the use cases that have made it upstream are all for relatively complex charts (e.g., requiring aggregation and composition of multiple data series in nontrivial ways) I'm currently looking at an overall approach like this:
- At least for now, Charts provides a low-level internal-only API for composing charts from raw datasets.
- This is exposed to users through pre-built `SearchEngine`-like interfaces that provide a small number of more manageable controls (show chart from date X to date Y, show projects A, B, C), but not the full set of composition features (`compose(scale(2), cos())` and such).
- Eventually, we may put more UI on the raw chart composition stuff and let you build your own fully custom charts by gluing together datasets and functions.
- Or we may add this stuff in piecemeal to the higher-level UI as tools like "add goal line" or "add trend line" or whatever.
This will let the low-level API mature/evolve a bit before users get hold of it directly, if they ever do. Most requests today are likely satisfiable with a small number of chart engines plus raw API data access, so maybe UI access to flexible charting is far away.
Step toward this by adding a "Reports" section to projects. For now, this just renders a basic burnup for the current project. Followups will add an "Engine" layer above this and make the chart it produces more useful.
Test Plan: {F6426984}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20495
Summary:
Depends on D20488. Ref T13279. When installs run `bin/phd start`, start the fact daemon alongside other daemons.
Since "Reports" in Maniphest now relies on Facts data, populate it.
Test Plan: Ran `bin/phd start`, saw the Fact daemon start.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20489
Summary: Depends on D20487. If you `min(1, 2, null)`, you get `null`. We want `1`.
Test Plan: Viewed a "burnup for project X" chart where one dataseries had no datapoints. Saw a sensible domain selected automatically.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Differential Revision: https://secure.phabricator.com/D20488
Summary:
Depends on D20485. Ref T13279. This removes the ad-hoc charting in Maniphest and replaces it with a Facts-based chart.
(To do this, we build a dashboard panel inline and render it.)
Test Plan: {F6412720}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20486
Summary:
Depends on D20484. Ref T13279. Allows a chart to render as a panel.
Configuring these is currently quite low-level (you have to manually copy/paste a chart key in), but works well enough.
Test Plan: {F6412708}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20485
Summary:
Ref T13279. This changes the chart controller:
- if we have no arguments, build a demo chart and redirect to it;
- otherwise, load the specified chart from storage and render it.
This mostly prepares for "Chart" panels on dashboards.
Test Plan: Visited `/fact/chart/`, got redirected to a chart from storage.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20483
Summary: Depends on D20538. Ref T13291. We now recognize full source URIs, but encoding full URIs isn't super human-friendly and we can't do stuff like relative links with them. Add `{src ...}` as a way to get to this behavior that supports options and more flexible syntax.
Test Plan: {F6463607}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291
Differential Revision: https://secure.phabricator.com/D20539
Summary:
Depends on D20530. Ref T13291. When users paste links to files in Diffusion into remarkup contexts, identify them and specialize the rendering.
When the URIs are embedded with `{...}`, parse them in more detail.
This is a lead-up to a `{src ...}` rule which will use the same `View` but give users more options to customize presentation.
Test Plan: {F6463580}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291
Differential Revision: https://secure.phabricator.com/D20538
Summary:
Ref T13289. In Maniphest, you can currently search for "Owner: none()" to find tasks with no owner, but there's no way to search for "Reviewers: none()" in Differential right now.
Add support for this, since it's consistent and reasonable and doesn't seem too weird or niche.
Test Plan: Searched for "Reviewers: none()", found revisions with no reviewers. Searched for "Reviewers: alice, none()", "Reviewers: alice", and "Reviewers: <no constraint>" and got sensible results.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13289
Differential Revision: https://secure.phabricator.com/D20537
Summary: Depends on D20534. Ref T13294. Add export support so you can dump these out, print them on paper, notarize them, and store them in a box under a tree or whatever.
Test Plan: Exported transactions to a flat file, read the file.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13294
Differential Revision: https://secure.phabricator.com/D20535
Summary: Depends on D20533. Allow querying for transactions of a specific object type, so you can run queries like "Show all edits to Herald rules between date X and Y".
Test Plan: {F6463478}
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20534
Summary: Depends on D20531. Ref T13294. Enable finding raw transactions in particular date ranges or with particular authors.
Test Plan: {F6463471}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13294
Differential Revision: https://secure.phabricator.com/D20533
Summary:
Ref T13294. An install is interested in a way to easily answer audit-focused questions like "what edits were made to any Herald rule in Q1 2019?".
We can answer this kind of question with a more granular version of feed that focuses on being exhaustive rather than being human-readable.
This starts a rough version of it and deals with the two major tricky pieces: transactions are in a lot of different tables; and paging across them is not trivial.
To solve "lots of tables", we just query every table. There's a little bit of sleight-of-hand to get this working, but nothing too awful.
To solve "paging is hard", we order by "<dateCreated, phid>". The "phid" part of this order doesn't have much meaning, but it lets us put every transaction in a single, stable, global order and identify a place in that ordering given only one transaction PHID.
Test Plan: {F6463076}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13294
Differential Revision: https://secure.phabricator.com/D20531
Summary:
See PHI1268. We currently do some weird width handling when rendering Diffusion readmes in a document directory view.
I think this came from D12330, which used `PHUIDocumentViewPro` to change the font, but we later reverted the font and were left with the `DocumentView`. Other changes after that modified `DocumentView` to have fixed-width behavior, but it doesn't make much sense here since the content panel is clearly rendered full-width.
Today, the `DocumentView` is a more structural element with methods like `setCurtain()`. Just get rid of it to simplify things, at least as a first step.
Test Plan:
Before:
{F6463493}
After:
{F6463492}
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20536
Summary:
Depends on D20528. Ref T13291. Ref T13285. Currently, we don't put a timeout on external service calls when enriching URIs for external Asana/JIRA tasks.
Add a 15-second timeout so we'll do something reasonable-ish in the face of a downed service provider. Later, I plan to healthcheck Asana/JIRA providers in a generic way (see T13287) so we can stop making calls if they time out / fail too frequently.
Test Plan:
- Linked to JIRA and Asana tasks in comments.
- Set timeout to 0.0001 seconds, saw requests time out.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291, T13285
Differential Revision: https://secure.phabricator.com/D20530
Summary:
Depends on D20527. Ref T13291. Now that we have more flexible support for URI rewriting, use it for Doorkeeper URIs.
These are used when you set up Asana or JIRA and include the URI to an Asana task or a JIRA issue in a comment.
Test Plan:
- Linked up to Asana and JIRA.
- Put Asana and JIRA URIs in comments.
- Saw the UI update to pull task titles from Asana / JIRA using my OAuth credentials.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291
Differential Revision: https://secure.phabricator.com/D20528
Summary:
Ref T13291. Currently, `T123` is a mention and adds an "alice mentioned this on Txxx." to `T123`, but `https://install.com/T123` is not a mention.
Make the full URI a mention.
Test Plan: Commented a full URI, saw the target object get a mention story in its timeline.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291
Differential Revision: https://secure.phabricator.com/D20527
Summary:
See <https://phabricator.wikimedia.org/T179591>. Some time ago, all handle rendering preloaded handles: things emitted a list of PHIDs they'd need handles for, then later used only those PHIDs.
Later, we introduced `HandlePool` and lazy/on-demand handle loading. Modern transactions mostly use this to render object PHIDs.
When we build mail, many newer transactions use an on-demand load to fetch handles to render transactions. This on-demand load may use the original viewer (the acting user) instead of the correct viewer (the mail recipient): we fetch and reset handles using the correct viewer, but do not overwrite the active viewer for on-demand loading. This could cause mail to leak the titles of related objects to users who don't have permission to see them.
Instead, just reload the transactions with the correct viewer when building mail instead of playing a bunch of `setViewer()` and `clone` games. Until we're 100% on modular transactions, several pieces of the stack cache viewer or state information.
Test Plan:
- Created task A (public) with subtask B (private).
- Closed subtask B as a user with access to it.
- Viewed mail sent to subscribers of task A who can not see subtask B.
- Before change: mail discloses title of subtask B.
- After change: mail properly labels subtask B as "Restricted Task".
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20525
Summary:
Depends on D20446. Currently, chart functions are both configured through arguments and evaluated through arguments. This sort of conflates things and makes some logic more difficult than it should be.
Instead:
- Function arguments are used to configure function behavior. For example, `scale(2)` configures a function which does `f(x) => 2 * x`.
- Evaluation is now separate, after configuration.
We can get rid of "sourceFunction" (which was basically marking one argument as "this is the thing that gets piped in" in a weird magical way) and "canEvaluate()" and "impulse".
Sequences of functions are achieved with `compose(u, v, w)`, which configures a function `f(x) => w(v(u(x)))` (note order is left-to right, like piping `x | u | v | w` to produce `y`).
The new flow is:
- Every chartable function is `compose(...)` at top level, and composes one or more functions. `compose(x)` is longhand for `id(x)`. This just gives us a root/anchor node.
- Figure out a domain, through various means.
- Ask the function for a list of good input X values in that domain. This lets function chains which include a "fact" with distinct datapoints tell us that we should evaluate those datapoints.
- Pipe those X values through the function.
- We get Y values out.
- Draw those points.
Also:
- Adds `accumluate()`.
- Adds `sum()`, which is now easy to implement.
- Adds `compose()`.
- All functions can now always evaluate everywhere, they just return `null` if they are not defined at a given X.
- Adds repeatable arguments for `compose(f, g, ...)` and `sum(f, g, ...)`.
Test Plan: {F6409890}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Differential Revision: https://secure.phabricator.com/D20454
Summary: Ref T13272. Since the move to EditEngine, these methods have no callsites.
Test Plan: `grep`
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13272
Differential Revision: https://secure.phabricator.com/D20484
Summary:
Ref T13283. See PHI1202. See D20519. When we apply a group of transactions, label all of them with the same "group ID".
This allows other things, notably Herald, to figure out which transactions applied together in a faithful way rather than by guessing, even though the guess was probably pretty good most of the time.
Also expose this to `transaction.search` in case callers want to do something similar. They get a list of transaction IDs from webhooks already anyway, but some callers use `transaction.search` outside of webhooks and this information may be useful.
Test Plan:
- Ran Herald Test Console, saw faithful selection of recent transactions.
- Changed hard limit from 1000 to 1, saw exception. Users should be very hard-pressed to hit this normally (they'd have to add 990-ish custom fields, then edit every field at once, I think) so I'm just fataling rather than processing some subset of the transaction set.
- Called `transaction.search`, saw group ID information available.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13283
Differential Revision: https://secure.phabricator.com/D20524
Summary:
See PHI1210. For certain large inputs, we spend more time than we need to replacing tabs with spaces. Add some fast paths:
- When a line only has tabs at the beginning of the line, we don't need to do as much work parsing the rest of the line.
- When a line has no unicode characters, we don't need to vectorize it to get the right result.
Test Plan:
- Added test coverage.
- Profiled this, got a ~60x performance increase on a 36,000 line 3MB text file.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20477
Summary:
See PHI1225. Ref T13277. In Diffusion, show "default", "permanent", or "not permanent" when looking at branches.
For repositories with 100 or fewer branches, put default and permanent branches on top.
Test Plan: {F6426814}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: leoluk
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20493
Summary:
Depends on D20507. See PHI1232. Previously, see T13255 and D20209.
Since nothing seems to have exploded after "projects" was exposed, give "subscribers" the same treatment.
Test Plan: Added, removed, and modified subscribers. Queried transactions with "transaction.search", saw sensible "type" and data.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20508
Summary:
Depends on D20510. Ref T5378. When remarkup includes a hyperlink to the current install in the form "/X123" (which is common), load the corresponding object and specialize the rendering.
This doesn't cover everything (notably, no handling for Diffusion paths yet), but does cover a lot of the most common cases.
The "uri" form preserves the URI as written, but adds an icon, tag, and hovercard.
The "{uri}" form is more similar to `{T123}` and shows the object name.
Test Plan: {F6440367}
Reviewers: amckinley, joshuaspence
Reviewed By: joshuaspence
Subscribers: joshuaspence
Maniphest Tasks: T5378
Differential Revision: https://secure.phabricator.com/D20512
Summary:
Depends on D20509. See PHI1224. Ref T5378. With some frequency, I paste URIs into the global search input (I am dumb).
When I do this dumb thing, redirect to the URI as though the global search was a URI bar.
Maybe only I am dumb like this, but I don't think it'll hurt anything.
Test Plan: pasted a URI and hit return; tried to eat a rock
Reviewers: amckinley, joshuaspence
Reviewed By: joshuaspence
Maniphest Tasks: T5378
Differential Revision: https://secure.phabricator.com/D20510
Summary: Ref T5378. This class was renamed more than a year ago, in D19087. Remove the leftover compatiblity layer.
Test Plan: `grep`
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T5378
Differential Revision: https://secure.phabricator.com/D20509
Summary:
See PHI1232, which describes a reasonable use case for wanting information about the "draft" ("Hold as Draft / Do Not Auto-Promote") flag.
Also, flesh out "testPlan" and "summary". It's possible these "blob of remarkup" fields might have metadata some day (e.g., a rendered version or a list of PHIDs or something), but we could add more keys, and we already have some other transactions which work like this.
Test Plan: Used "transaction.search" to fetch these transaction types, saw type information and metadata.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20507
Summary:
See PHI1229. An install has a somewhat duct-taped registration flow which can dump users on the "Wait for Approval" screen without clear guidance. The desired guidance is something like "this is totally normal, just wait a bit for a bot to approve you".
Adding guidance here is generally reasonable and consistent with the intent of this feature.
Test Plan: {F6426583}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: kylec
Differential Revision: https://secure.phabricator.com/D20492
Summary: Depends on D20519. Ref T13283. See PHI1202. Add a new rule which triggers when the current/most-recent transaction group includes a "content" or "publish" transaction, which means the published document content has changed.
Test Plan:
- Wrote a Herald rule using this field.
- Created a document (rule matched).
- Edited a document (rule matched).
- Edited a document, saving as a draft (no match).
- Edited a draft, updating it (no match).
- Published a draft docuemnt (rule matched).
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13283
Differential Revision: https://secure.phabricator.com/D20520
Summary:
Depends on D20518. Ref T13283. When you use the "Test Console" in Herald to test rules, pass the most recent group of transactions to the Adapter.
This will make it easier to test rules that depend on edit state, since you can make the type of edit you're trying to test and then use the Test Console to actually test if it matches in the way you expect.
The transactions we select may not be exactly the group of transactions that most recently applied. For example, if you make edits A, B, and C in rapid succession and then run the Test Console on the object, it may select "A + B + C" as a transaction group. But we'll show you what we selected and this is basically sane/reasonable and should be fine.
(Eventually, we could include a separate "transaction group ID" on transactions if we want to get this selection to match exactly.)
Test Plan: Ran the Test Console on various objects, saw sensible transaction lists in the transcripts.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13283
Differential Revision: https://secure.phabricator.com/D20519
Summary:
Ref T13283. Since D14575, we already pass applied transactions to Herald, but they exist only as a backwards compatibility layer and have no upstream callsites.
Save the applied transaction PHIDs as part of the object transcript, and show them in the UI.
Test Plan:
- Viewed a modern transcript, saw a list of transactions.
- Viewed an older transcript, saw nothing (since there were no transactions in the transcript).
{F6456431}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13283
Differential Revision: https://secure.phabricator.com/D20518
Summary:
Depends on D20516. See PHI1247. In D20331, I made the crumbs on workboards point at ancestor workboards.
However, this isn't a great destination if an ancestor doesn't actually have a workboard. In this case, point at the normal profile URI instead.
Test Plan:
- Viewed a milestone workboard with a parent that had no workboard. Saw a profile link instead of a workboard link (new behavior).
- Viewed a milestone workboard with a parent that also had a workboard. Saw a workboard link (existing old behavior still works).
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20517
Summary:
Ref T13276. Previously, these edges were added directly with an `EdgeEditor`, so they did not generate transaction stories.
Now, they're added properly, but they aren't terribly useful in feed/mail. Hide them in those contexts, like we already do with other types of similar edges.
Test Plan: Will verify behavior on `secure`.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20491
Summary:
See PHI1209. When a task is in "Hard Lock" mode, it's still possible to apply some changes to it. Notably:
- You can subscribe/unsubscribe.
- You can mention it on another object.
- You can add a relationship from some other object to it (e.g., select it as a "Parent Task" for some other task).
Currently, these types of edits will show a "Lock Overridden" timeline emblem icon. However, they should not: you didn't override a lock to make these changes, they just bypass locks.
For now, special case these cases (self subscribe/unsubscribe + inverse edge edits) so they don't get the little icon, since I think this list is exhaustive today.
Some day we should modularize this, but we'd need code like this anyway (since TYPE_SUBSCRIBE is not modular yet), and this seems unlikely to cause problems even if it's a bit rough.
Test Plan:
- Hard-locked a task.
- Subscribed/unsubscribed, mentioned, relationship'd it as a non-author. No timeline emblems.
- Soft-locked a task.
- Subscribed/unsubscribed, mentioned, relationship'd it, no timeline emblems.
- Clicked "Edit", answered "yes" to the override prompt, edited it. Got a timeline emblem.
- Added some comments and stuff to a normal non-locked task, no emblems.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20513
Summary:
See PHI1247. If you remove the Workboard from a project profile menu, then navigate to the URI, we currently fatal when trying to select the "Workboard" item.
Instead, only try to select the item if some matching item is present.
Test Plan:
- Disabled the workboard on a project, navigated to `/board/`, got a sensible page with no navigation item selected instead of a fatal.
- Viewed a normal workboard, saw the correct selection.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20516
Summary: Ref PHI1166. I'm documenting our dependencies, and we have approximately 5,000 lines of external code to support WePay as a Phortune provider. We don't use it, I'm almost certain it doesn't work, and we have no plans to use it in the near future. If we did pursue it, I'd probably just wrap the API in a 100-line `WePayFuture` anyway since 5K lines of dependencies to make a couple method calls is ridiculous.
Test Plan: Grepped for `wepay`, `httpful`, `restful`.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: aurelijus
Differential Revision: https://secure.phabricator.com/D20521
Summary: Ref T13276. This edge is pointed the wrong way. Point it the right way.
Test Plan: Will verify production works better.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20490
Summary: Depends on D20474. Ref T13272. Provide an easy way to rearrange tabs on a tab panel, by moving them left or right from the context menu.
Test Plan: Moved tabs left and right. Tried to move them off the end of the tab list, no luck.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13272
Differential Revision: https://secure.phabricator.com/D20475
Summary:
Depends on D20473. Ref T13272. Fixes T7216. If you want to tweak the query a panel uses, you currently have to complete 7 Great Labors.
Instead, add a "Customize Query" action which lets you update the query inline.
Test Plan: {F6402171}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13272, T7216
Differential Revision: https://secure.phabricator.com/D20474
Summary:
Depends on D20469. Ref T13276. See PHI1159. See PHI953. See PHI901.
Allow Herald to detect when "arc land" would (or did) warn users about failed or ongoing builds. This respects the "Warn on Landing" build plan behavior.
To accomplish this:
- When we close a revision, set a "wrong build state" flag if it lands in the wrong build state.
- If the revision is closed when we hit Herald, look for the flag.
- If not (common for push rules, can happen for commit rules if we race against the revision update worker), hit Harbormaster ourselves and check the current state.
Test Plan:
- Wrote a "Require Green" rule.
- Ran it against various commits with various build states (good, not good).
- Fiddled with "Warn on Landing" and saw the effect in rule evaluation.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20470
Summary:
Depends on D20468. Ref T13276. See PHI1008.
When a commit or revision "reverts <hash>", and that's the hash of a commit which has a revision, also write a "reverts" edge to the revision.
Test Plan:
Created "reverts X" objects for:
- a revision;
- a commit;
- a commit with a revision (both got marked properly).
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20469
Summary:
Depends on D20467. Ref T13277. Currently, the "MessageParserWorker" writes this property on commits, then Herald and Audit both read it.
Make them share code so this property has one writer and one reader. This property isn't great, but at least now the badness is hidden.
Currently, we can't just use edges because they may not have been written yet. I am likely to just do this, soon:
- Just write the edges (in "MessageParserWorker").
- Hide the edges from mail.
However, we'll sort-of lose the "revisionMatchData" explanation thing if I do that. Maybe this is fine? But when commits match because hashes match, it legitimately isn't obvious.
For now, just reduce the amount of harm/badness here.
Test Plan:
- Ran `bin/repository reparse --publish ...`.
- Ran a Herald "Audit" rule using the "Accepted Differential revision" field.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20468
Summary: See PHI1220. Ref T13272. I accidentally left the ability to set a query limit behind when updating this.
Test Plan: Edited a query panel, set/removed the limit, tried to set an invalid limit.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13272
Differential Revision: https://secure.phabricator.com/D20472
Summary:
Depends on D20472. Ref T13272. Currently, when you edit a panel from a dashboard, we try to add the panel to the dashboard. This always works since dashboards no longer enforce panel uniqueness, and you can end up with duplicate panels.
Instead, only add panels if we're creating them.
Test Plan:
- Edited an existing panel, no duplication.
- Created a new panel, saw it added to the dashboard.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13272
Differential Revision: https://secure.phabricator.com/D20473
Summary:
If you edit an auth message in Auth > Customize Messages, then click "Show Details" in the transaction record, the resulting dialog uses the object's handle's URI to generate a "cancel" button.
Since these handles currently have no URI, the dialog currently has no cancel/done button to close it.
Test Plan: Edited an auth message, clicked "Show Details", was now able to click "Done" to close the dialog.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20471
Summary:
See PHI985. I think we pretty much need to start applying language-specific rules, but we can apply at least one more relatively language-agnostic rule: don't match lines which are indented 3+ levels.
In C++, we may have symbols like this:
```
class X {
public:
int m() { ... }
}
```
..but I believe no mainstream language puts symbol definitions 3+ levels deep.
Also clean up some of the tab handling very slightly.
Test Plan: Tests pass, looked at some C++ code and got slightly better (but still not great) matches.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20479
Summary:
See PHI1218. When rendering "A vs B", we currently show the properties of diff A without modification.
Instead, take properties from the same place we're taking change details.
See T12664 for a followup.
Test Plan:
- In diff A, removed "+x" from a file.
- In diff B, changed the file but did not remove "+x" from it.
- Diffed B vs A.
- Before change: UI incorrectly shows "+x" removed (both sides incorrect, just showing the change from diff A).
- After change: UI shows 100644 -> null, which is half right.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20478
Summary:
See <https://discourse.phabricator-community.org/t/unable-to-reload-object-that-hasnt-been-loaded/2677>.
When editing "Config" objects, they currently get a PHID set outside of the TransactionEditor. They probably should not, but fixing that is likely an involved change.
This causes us to incorrectly fail to detect `$is_new` correctly and try to `reload()` and object with no ID.
To work around this, test for new objects with `getID()` instead of `getPHID()`.
Test Plan: Edited any config value with the web UI.
Reviewers: amckinley
Differential Revision: https://secure.phabricator.com/D20482
Summary:
Depends on D20466. Ref T13277. Currently:
- The "Owners" worker writes ownership relationships (e.g., commit X affects package Y, because it touches a path in package Y) -- these are just edges.
- It also triggers audits.
- Then it queues a "Herald" worker.
- This formally publishes the commit and triggers Herald.
These aren't really separate steps and can happen more easily in one shot. Merge them.
Test Plan:
- Ran `bin/repository reparse --publish` to republish various commits, got sensible behavior.
- Grepped for "IMPORTED_OWNERS", "IMPORTED_HERALD", "--herald", "--owners", and "--force-local" flags.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20467
Summary:
Depends on D20465. Ref T13277. Currently, when a commit is unpublished, we put a single line about it on the "Edit Commit" page. This is pretty much impossible to find.
Move it to the main page. This treatment is more big/bold than I'd probably like to end up, but we should probably overshoot on the explanatory text until users get used to this behavior.
Also, allow searching for only published / unpublished commits.
Test Plan: {F6395705}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20466
Summary:
Depends on D20464. Ref T13277. Broadly:
- Move all the "should publish X" and "why aren't we publishing X" stuff to a separate class (`PhabricatorRepositoryPublisher`).
- Rename things to be more consistent with modern terminology ("Publish", "Permanent Refs").
Test Plan:
This could use some trial-by-fire on `secure`, but:
- Grepped for all symbols.
- Viewed various commits.
- Reparsed commits.
- Here's a commit with an explanation:
{F6394569}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20465
Summary: Depends on D20463. Ref T13277. This flag was added some time before 2015 and I don't think I've ever used it. Just get rid of it.
Test Plan: Grepped for `force-autoclose`, `forceAutoclose`, `AUTOCLOSE_FORCED`.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20464
Summary:
Depends on D20462. Ref T13276. Currently, the "Message" parser also updates related tasks and revisions when a commit is published.
For PHI1165, which ran into a race with message parsing, I originally believed we needed to separate this logic and lock + yield to avoid the race. D20462 provides what is probably a better approach for avoiding the race.
Still, I think separating these "update related revisions" and "updated related tasks" chunks into separate workers is a net improvement. There may still be some value in doing lock + yield in the future to deal with other issues, and when we occasionally run into problems with pulling a diff out of the repository to update the revision (usually because the diff is too big) this isolates the problem better and allows the commit to import.
I think the only thing to watch out for here is that Herald may now run before the revision and commit are attached to one another. This is fine for all current Herald rules, we just need to be mindful in implementing new rules.
Test Plan: Used `bin/repository reparse --message` on various commits, including commits that close revisions and close tasks.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20463
Summary:
Depends on D20461. Ref T13276. Ref T13054.
Currently, we acquire the transaction read lock after populating "old values" in transactions and filtering transactions with no effect.
This isn't early enough to prevent all weird chaotic races: if two processes try to apply a "close revision" transaction at the same time, this can happen:
```
PROCESS A PROCESS B
Old Value = Open Old Value = Open
Transaction OK: Yes Transaction OK: Yes
Acquire Read Lock Acquire Read Lock
Got Read Lock! Wait...
Apply Transactions Wait...
New Value = Closed Wait...
Release Lock Wait...
Got Read Lock!
Apply Transactions
New Value = Closed
Release Lock
```
That's not great: both processes apply an "Open -> Closed" transaction since this was a valid transaction from the viewpoint of each process when it did the checks.
We actually want this:
```
PROCESS A PROCESS B
Acquire Read Lock Acquire Read Lock
Got Read Lock! Wait...
Old Value = Open Wait...
Transaction OK: Yes Wait...
Apply Transactions Wait...
New Value = Closed Wait...
Release Lock Wait...
Got Read Lock!
>>> Old Value = Closed
>>> Transaction Has No Effect!
>>> Do Nothing / Abort
Release Lock
```
Move the "lock" part up so we do that.
This may cause some kind of weird second-order effects, but T13054 went through pretty cleanly and we have to do this to get correct behavior, so we can survive those if/when they arise.
Test Plan:
- Added a `sleep(10)` before the lock.
- Ran `bin/repository message --reparse X` in two console windows, where X is a commit that closes revision Y and Y is open.
- Before patch: both windows closed the revision and added duplicate transactions.
- After patch: only one of the processes had an effect.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: jmeador
Maniphest Tasks: T13276, T13054
Differential Revision: https://secure.phabricator.com/D20462
Summary: Depends on D20459. Ref T13276. I'll file a followup to actually destroy the table.
Test Plan:
- Grepped for `TABLE_COMMIT`.
- Ran `bin/storage upgrade -f`, got a clean bill of health.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20461
Summary:
Depends on D20458. Ref T13276. Although I'm not thrilled about "needCommitPHIDs()", it has a few callers, including custom fields. Allow "need + attach + get" to survive for now since they're reasonably modern, at least.
However, use edges instead of "TABLE_COMMIT" and require `need...()` + `get...()`, removing the direct `load...()`.
Also remove `RevisionQuery->withCommitPHIDs(...)`, which has no callers.
Test Plan:
- Grepped for `loadCommitPHIDs` (only two hits, the private `RevisionQuery` method).
- Called "differential.getrevision", got commits.
- Viewed a revision, saw "Commits: ...".
- Grepped for `withCommitPHIDs()`, no callers on `RevisionQuery` (some other query classes have methods with this name).
- Called "differential.query", got commits.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20459
Summary: Depends on D20457. Ref T13276. Kill all remaining callers to this method and delete it.
Test Plan:
- Grepped for `loadIDsByCommitPHIDs`.
- Viewed blame again to make sure I didn't break it.
- Viewed "History" view for commits with revisions.
- Viewed "Graph" view for commits with revisions.
- Viewed "Merged Commits" table for commits with revisions.
- Viewed "Compare" table for commits with revisions.
- Viewed "Repository" main page history table for commits with revisions.
- Grepped for `linkRevision`.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20458
Summary:
Ref T13276. Differential has a pre-edge "TABLE_COMMIT" with about a half-dozen weird callers I'd like to get rid of.
Move blame to use edges instead. (Bonus: this makes blame respect edge edits in the UI.)
Since there are some more callers to clean up this code may move into some "RelatedObjectQueryThing" class or something, but I'm taking it one step at a time for now.
Test Plan: {F6394106}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20457
Summary:
Depends on D20445. Ref T13279. I'm not sure what the class tree of functions actually looks like, and I suspect it isn't really a tree, so I'm hesitant to start subclassing. Instead, try adding some `isSomethingSomething()` methods.
We have some different types of functions:
# Some functions can be evaluated anywhere, like "constant(3)", which always evaluates to 3.
# Some functions can't be evaluated anywhere, but have values everywhere in some domain. This is most interesting functions, like "number of open tasks". These functions also usually have a distinct set of interesting points, and are constant between those points (any count of anything, like "open points in project" or "tasks closed by alice", etc).
# Some functions can be evaluated almost nowhere and have only discrete values. This is most of the data we actually store, which is just "+1" when a task is opened and "-1" when a task is closed.
Soon, I'd like to be able to show ("all tasks" - "open tasks") and draw a chart of closed tasks. This is somewhat tricky because the two datasets are of the second class of function (straight lines connecting dots) but their "interesting" x values won't be the same (users don't open and close tasks every second, or at the same time).
The "subtract X Y" function will need to be able to know that `subtract "all tasks" 3` and `subtract "all tasks" "closed tasks"` evaluate slightly differently.
To make this worse, the data we actually //store// is of the third class of function (just the "derivative" of the line chart), then we accumulate it in the application after we pull it out of the database. So the code will need to know that `subtract "derivative of all tasks" "derivative of closed tasks"` is meaningless, or the UI needs to make that clear, or it needs to interpret it to mean "accumulate the derivative into a line first".
Anyway, I'll sort that out in future changes. For now, simplify the easy case of functions in class (1), where they're just actual functions.
Add "shift(function, number)" and "scale(function, number)". These are probably like "mul" and "add" but they can't take two functions -- the second value must always be a constant. Maybe these will go away in the future and become `add(function, constant(3))` or something?
Test Plan: {F6382885}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20446
Summary:
Depends on D20444. Ref T13279. Instead of ad-hoc parsing and messages, formalize chart function arguments.
Also, add a whole lot of extra type checking.
Test Plan: Built and charted various functions with various valid and invalid argument lists, got sensible-seeming errors and results.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20445
Summary:
See PHI1213. If we don't have identities for "revision X closed by commit Y" stories, just do the plain non-attribution rendering rather than trying to fall back. Falling back won't work since we don't load the data, which should be OK now since identities seem like they're in generally good shape.
(We could probably just throw out the fallback behavior instead, but we can always clean things up later.)
Test Plan: Forced no commit identity on a revision, loaded it, saw reasonable story rendering.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20460
Summary:
Ref T13266. See <https://discourse.phabricator-community.org/t/notification-page-throws-unrecoverable-fatal-error/2651/>.
The "notifications" query currently uses offset paging for no apparent reason (just a legacy issue?), so some of the paging code is only reachable internally.
- Stop it from using offset paging, since modern cursor paging is fine here (and Feed has used cursor paging for a long time).
- Fix the non-offset paging to work like Feed.
Also:
- Remove a couple of stub methods with no callsites after cursor refactoring.
Test Plan:
- Set things up so I had more than 100 notifications and some in the first 100 were policy filtered, to reproduce the issue (I just made `FeedStory` return `NO_ONE` as a visibility policy).
- Applied the patch, notifications now page cleanly.
- Verified that "Next Page" took me to the right place in the result list.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: hskiba
Maniphest Tasks: T13266
Differential Revision: https://secure.phabricator.com/D20455
Summary:
Ref T13280. In D20421, I changed our observe strategy to `git fetch <uri>` in all cases.
This doesn't work in an ancient, non-bare repository if `master` is checked out and `master` is also fetch: `git` refuses to overwrite the local ref unless we pass `--update-head-ok`. Pass this flag.
Also, remove some code which examines branches and tags in a special way for non-bare working copies. The old `git fetch <origin>` code without explicit revsets meant that `refs/remotes/orgin/heads/xyz` got updated instead of `refs/heads/xyz`. We now update our local refs in all cases (bare and non-bare) so we can throw away this special casing.
Test Plan:
- Replaced a modern bare working copy with a non-bare working copy by explicitly using `git clone` without `--bare`.
- Ran `bin/repository update`, hit the `--update-head-ok` error. Applied the patch, got a clean update.
- Used the "repository.branchquery" API method...
- ...with "contains" to trigger the "git branch" case. Got identical results after removing the special casing.
- ...without "contains" to trigger the "low level ref" case. Got identical results after removing the special casing.
- Grepped for `isWorkingCopyBare()`. The only remaining callsites deal with hook paths, and genuinely need to be specialized.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13280
Differential Revision: https://secure.phabricator.com/D20450
Summary:
Ref T13279. Charting changes alter how the "line-chart" behavior works, but the "Burnup Chart" still relies on the old behavior.
Although I'm intending to remove "Maniphest > Reports" once Facts is a minimally sufficient replacement, copy this behavior to keep it working until we're ready to pull the trigger.
Also fix a leftover typo from D20435.
Test Plan: Viewed a legacy Maniphest burnup rate report.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20449
Summary: This always bugs me when I'm going through `secure` looking for the spiciest macros.
Test Plan: Forced a date to be null, saw reasonable text.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D20453
Summary:
Depends on D20443. Ref T13279. This is probably not terribly useful on its own, but is mostly a function which takes another function as an argument, and a step toward more useful functions like arithmetic and drawing a picture of an owl.
The only structural change here is that functions now read data parameters (domain, sample limit) using a more tailored "ChartDataQuery" instead of reading the actual axis. Mostly, I want a more cohesive representation of query state that can be easily passed to sub-functions, as here.
Test Plan: {F6382432}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20444
Summary:
Depends on D20442. Ref T13279. Add basic support for drawing chart functions that are not based on Facts first-party ETL datasets. Some general goals:
- This might be useful to draw a line like "goal" or "profitability".
- This might be useful to pull data from an external source.
- For composable functions like "add" or "subtract", which are useful in manipulating ETL datasets, these value functions will make testing easier.
Test Plan:
Added a `constant(256)` function:
{F6382408}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20443
Summary:
Depends on D20441. Ref T13279. Currently, we pull all the data, then decide what the X-axis should look like.
Since users will reasonably want to do stuff like "show me march-april 2018" in the future, we need to move toward flipping this around so that we can support cases where the domain is specified by the user.
For actual chart functions (like "constant(3)" or "cos(x)"), we must also know the domain before we pull data, since there are an infinite number of places where we can evaluate the function "constant(3)".
See note in T13279 about continunity.
Test Plan: {F6382356}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20442
Summary:
Depends on D20440. Ref T13279. Create a class to represent a chartable function: something we can get some data points out of.
Then, make the chart chart two functions.
For now, the only supported function is "fact(key)", which pulls data from the Facts ETL pipeline, identified by "key", and takes no other arguments.
In future changes, I plan to support things like "fact(tasks.open.project, PHID-PROJ-xyz)", "constant(1000)" (e.g. to draw a goal line), "sum(fact(...), fact(...))" (to combine data from several projects), and so on.
The UI may not expose this level of power for a while (or maybe ever) but until we get close enough to the UI that these features are a ton of extra work I'm going to try to keep things fairly flexible/modular.
Test Plan: {F6382286}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20441
Summary:
Depends on D20439. Ref T13279. Some day, charts will probably need to reload themselves or do a bunch of defer/request-shaping magic when they're on a dashboard with 900 other charts.
Give the controller separate "HTML placeholder" and "actual data" modes, and make the placeholder fetch the data in a separate request.
Then, make the chart redraw if you resize the window instead of staying at whatever size it started as.
Test Plan:
- Loaded a chart, saw it load data asynchronously.
- Resized the window, saw the chart resize.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20440
Summary:
Ref T13279. I think I'm going to fling some stuff at the wall for a bit here and hope most of it sticks, so this series of changes may not be terribly cohesive or focused. Here:
The range of the chart is locked to "[0, 105% of max]". This is trying to make a pleasing extra margin above the maximum value, but currently just breaks charts with negative values. Later:
- I'll probably let users customize this.
- We should likely select 0 as the automatic minimum for charts with no negative values.
- For charts with positive values, it would be nice to automatically pick a pleasantly round number (25, 100, 1000) as a maximum by default.
We don't have any storage for charts yet. Add some. This works like queries, where every possible configuration gets a short URL slug. Nothing writes or reads this yet.
Rename `fn()` to `css_function()`. This builds CSS functions for D3. The JS is likely to get substantial structural rewrites later on, `fn()` was just particularly offensive.
Test Plan: Viewed a fact series with negative values. Ran `bin/storage upgrade`.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20438
Summary:
Depends on D20434. Fixes T5963. Broadly, the issue here is that when:
- You create a new, empty repository.
- Then, you work on some branch other than `master`, without ever creating `master`.
...you get a warning on `git clone`:
> warning: remote HEAD refers to nonexistent ref, unable to checkout
To fix this, point the symbolic-ref HEAD at `refs/heads/<default-branch>` after installing commit hooks.
This fixes the warning, and also means that `git clone` will check out the repository default branch by default, which is nice.
There are a few caveats about this behavior (see T5963 for discussion) but nothing too substantial.
The only real issue is that Git prevents deletion of the default branch without a config setting. Just set that settting.
Test Plan:
See T5963.
In a repository, set `HEAD` to point somewhere invalid. Ran `bin/repository update ...`. Saw HEAD pointed back at the repository default branch.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T5963
Differential Revision: https://secure.phabricator.com/D20435
Summary:
Depends on D20433. Ref T13277. Since "Autoclose" no longer exists, update the documentation.
Currently, this documentation focuses a lot on troubleshooting because users historically had a lot of trouble with figuring out why things were or were not autoclosing. I haven't seen any real confusion about this in years, so I suspect we may have improved the import pipeline and/or UI to make this less of a problem.
It's also possible that this document "fixed" the problem, but usually I expect a documentation fix to not affect the frequency of reports, just make them easier to resolve, so I doubt it.
If unclear things remain //and// documentation really did fix it, maybe we can fix the issues. Or we can just put the troubleshooting documentation back.
Test Plan: Read documentation.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20434
Summary:
Depends on D20432. Ref T13277. Fixes T12967. Removes some "Track Only" hints and warns that the feature is deprecated in favor of "Permanent Refs" and "Fetch Only".
(This "fixes" T12967 by mooting it.)
Test Plan: Viewed "branches" sectino of the manage UI, edited "braches" section of a repository.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277, T12967
Differential Revision: https://secure.phabricator.com/D20433
Summary:
Depends on D20427. Ref T13277. As an optimization, when we discover that a commit which was previously only on a non-permanent ref ("tmp-epriestley-123") is now reachable from a permanent ref ("master"), we currently queue only a new "message" parse step.
This is an optimization because these commits previously got the full treatment (feed, publish, audit, etc) as soon as they were discovered. Now, those steps only happen once a commit is reachable from a permanent ref, so we need to run everything.
Test Plan:
- Pushed local "tmp-123" branch to remote tag "tmp-123".
- Updated repository with "bin/repository update", saw commit import as a "not on any permanent ref" commit, with no herald/audit/etc.
- Merged "tmp-123" tag into "master".
- Pushed new "master".
- Updated repository with "bin/repository refs ... --trace --verbose", saw commit detected as now reachable from a permament ref.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20428
Summary:
Depends on D20426. Ref T13277. The new behavior is to fire Herald only once a commit becomes reachable from a permanent ref (previously, an "Autoclose" branch).
That means that every "Commit" Herald rule implicitly has this field as a condition, and it no longer does anything.
Test Plan: Wrote a Herald rule, saw this as an option in the "Deprecated" section.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20427
Summary:
Depends on D20425. Ref T13277. See PHI1067. There's currently no way to retrieve branch/ref rules over the API, which makes some management operations against a large number of repositories difficult.
Expose these rules to the API.
Test Plan: Called `diffusion.repository.search`, got rules in the result set.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20426
Summary:
Depends on D20424. Ref T13277. Now that the "Actions" panel only has one item ("Publishing"), just move it to the "Basics" panel.
Update the UI to show active/publishing status more clearly and relate them to one another and importing state.
Test Plan: {F6378087}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20425
Summary:
Depends on D20423. Ref T13277. Repositories currently have separate toggles for "Autoclose" and "Publishing".
Merge the "Autoclose" toggle into the "Publishing" toggle. I'm unaware of any valid use case for enabling one but not the other.
(This doesn't fix all the documentation, yet.)
Test Plan: Edited a repository, saw only one publishing option.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20424
Summary:
Depends on D20422. Ref T13277. Currently, "track only", "publish", and "autoclose" are three separate ideas. I'd like to generally merge them into a more natural idea called "permanent refs".
Since "Autoclose" effectively now controls both "autoclose" and "publish", rename it.
This doesn't rename all the methods or internals, and the documentation needs an update, but it renames most of the UI-facing stuff.
(You also can only specify branches as "Permanent Refs" today, but we may let you specify tags and other arbitrary refs in the future.)
Test Plan: Grepped, poked around the UI, saw UI show "Permanent" / "Permanent Refs" more often and "Autoclose" less.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20423
Summary:
Depends on D20421. Ref T13277. I'd generally like to move away from "Track Only".
Some of the use cases for "Track Only" (or adjacent to "Track Only") are better resolved with "Fetch Rules" -- basically, rules to fetch only some subset of refs from the observed remote.
Add configurable "Fetch Rules" for Git repositories. For example, if you only want to fetch `master`, you can now speify:
```
refs/heads/master
```
If you only want to fetch branches and tags, you can use:
```
refs/heads/*
refs/tags/*
```
In theory, this is slightly less powerful in the general case than "Track Only", but gives us better behavior in some cases (e.g., when the remote has 50K random temporary branches). In practice, I think this and a better "Autoclose Only" will let us move away from "Track Only", get default behavior which is better aligned with what users actually expect, and dodge all the "track tags/refs" questions.
Test Plan: Configured repositories with "Fetch Refs" rules, used `bin/repository pull --verbose --trace ...` to run pulls, saw expected pull/fetch behavior.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20422
Summary:
Depends on D20420. Ref T13277. We currently spend substantial effort trying to detect and correct the URL of the "origin" remote in Git repositories.
I believe this is unnecessary, and we can always `git fetch <url> ...` to get the desired result instead of `git muck-with-origin + git fetch origin ...`. We already do this in the more recent parts of the codebase (e.g., intracluster sync) and it works correctly in every case I'm aware of.
Test Plan:
- Grepped for `origin`, ` origin `.
- Ran `bin/repository update ...` to fetch a mirrored repository.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20421
Summary:
Depends on D20418. Ref T13277. Fixes T11314.
Currently, when you push commits to some arbitrary ref or tag (like `refs/pull/123` on GitHub, `refs/tags/phabricator/diff/123` on Phabricator, or `refs/changes/whatever` on Gerrit), we do not "autoclose" related objects. This means that we don't process `Ref T123` to create links to tasks, and don't process `Differential Revision: xyz` to close revisions.
However, we //do// still publish these commits. "Publish" means: trigger audits, publish feed stories, and run Herald rules.
- Stop publishing these commits.
- In the UI, show these commits as "Not Permanent" with a note that they are "Not [on any permanent branch]."
These commits will publish and autoclose if they ever become reachable from an "autoclose" ref (most commonly, if they are later merged to "master").
Test Plan:
- Pushed a commit to `refs/tags/quack`.
- Before: got a feed story.
- After: no feed story, UI shows commit as "Not Permanent".
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277, T11314
Differential Revision: https://secure.phabricator.com/D20419
Summary:
Depends on D20381. Ref T8093. This makes minor improvements to the protocol proxy to handle cases where we add, remove, or replace refs and may need to move the "capabilities" section.
Rather than invoking a callback on every ref: parse the whole ref list into a data structure, mutate it if necessary (in a future diff), then dump it back into wire format.
This allows us to shift the capabilities data (which needs to be coupled with the first ref) around if we modify the first ref, and reorder the reflist alphabetically like git does.
When the server has no refs, Git sends no capabilities data. This is easy to emulate, just surprising.
Test Plan:
Tested the cases not covered by D20381:
- Fetching where the fetch actually fetches data.
- `ls-remote` when we hide the first ref (capabilities data properly moves to the first visible ref).
- `ls-remote` when the remote is empty (we just drop the capabilities frame completely).
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T8093
Differential Revision: https://secure.phabricator.com/D20436
Summary:
Depends on D20380. Ref T8093. When prototypes are enabled, inject a (hopefully?) no-op proxy into the Git wire protocol.
This proxy decodes "git upload-pack" and allows the list of references to be rewritten, in a similar way to how we already proxy the Subversion protocol to rewrite URIs and proxy the Mercurial protocol to distinguish between read and write operations.
The piece we care about comes at the beginning, and looks like this:
```
<frame-length><ref-hash> <ref-name>\0<server-capabilities>\n
<frame-length><ref-hash> <ref-name>\n
<frame-length><ref-hash> <ref-name>\n
...
<0000>
```
We can add, remove, or modify this section to make it appear that the server has different refs than the refs that exist on disk.
Things I have tried:
- `git ls-remote`
- `git ls-remote` where the server hides some refs.
- `git fetch` where the fetch is a no-op.
Things I have not tried:
- `git fetch` where the fetch is not a no-op.
- Tricking things into doing protocol v2. Or: I tried this, I wasn't successful. In v2, additional "\0" tricks are used to hide data in the capabilities, I think?
- `git ls-remote` where we rewrite/hide the first ref in the list, and need to move the capabilities frame elsewhere.
- `git ls-remote` where the server has no refs at all, or we remove every ref.
So the "interesting" piece of this works, but it almost certainly needs some cleanup to survive interaction with the real world.
Test Plan: See above.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T8093
Differential Revision: https://secure.phabricator.com/D20381
Summary:
Ref T8093. Support dumping the protocol bytes to a side channel logfile, as a precursor to parsing the protocol and rewriting protocol frames to virtualize refs.
The protocol itself is mostly ASCII text so the raw protocol bytes are pretty comprehensible.
Test Plan:
{F6363221}
{F6363222}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T8093
Differential Revision: https://secure.phabricator.com/D20380
