Summary:
Report from Asana. In some unclear circumstances, we my attempt to resolve duplicate refs which currently ends up hitting a duplicate key error.
Instead, reference the same external object if we happen to be handed duplicate refs.
Test Plan:
Used this script to reproduce the issue. Applied the fix; issue went away:
#!/usr/bin/env php
<?php
require_once 'scripts/__init_script__.php';
$args = new PhutilArgumentParser($argv);
$args->parseStandardArguments();
$ref = id(new DoorkeeperObjectRef())
->setApplicationType(DoorkeeperBridgeAsana::APPTYPE_ASANA)
->setApplicationDomain(DoorkeeperBridgeAsana::APPDOMAIN_ASANA)
->setObjectType(DoorkeeperBridgeAsana::OBJTYPE_TASK)
->setObjectID(7253737283629); // Use a new task ID which we've never pulled.
$refs = array(clone $ref, clone $ref);
$asana_user = id(new PhabricatorPeopleQuery())
->setViewer(PhabricatorUser::getOmnipotentUser())
->withUsernames(array('asana'))
->executeOne();
$resolved_refs = id(new DoorkeeperImportEngine())
->setViewer($asana_user)
->setRefs($refs)
->execute();
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D7709
Summary: Request from Asana. Adds an option for adding tasks to projects.
Test Plan: Used `bin/feed republish` to create and update Asana tasks with projects configured. Saw them end up in the right projects.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D7655
Summary: Fixes T3687. Instead of rendering "JIRA Issues" in Differential using plain links, render them using Doorkeeper tags so they get the nice "enhance with object name" effect.
Test Plan: {F84886}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3687
Differential Revision: https://secure.phabricator.com/D7648
Summary:
While we mostly have reasonable effective object accessibility when you lock a user out of an application, it's primarily enforced at the controller level. Users can still, e.g., load the handles of objects they can't actually see. Instead, lock the queries to the applications so that you can, e.g., never load a revision if you don't have access to Differential.
This has several parts:
- For PolicyAware queries, provide an application class name method.
- If the query specifies a class name and the user doesn't have permission to use it, fail the entire query unconditionally.
- For handles, simplify query construction and count all the PHIDs as "restricted" so we get a UI full of "restricted" instead of "unknown" handles.
Test Plan:
- Added a unit test to verify I got all the class names right.
- Browsed around, logged in/out as a normal user with public policies on and off.
- Browsed around, logged in/out as a restricted user with public policies on and off. With restrictions, saw all traces of restricted apps removed or restricted.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D7367
Summary:
Ref T603. Move toward stamping out all the Project / ProjectProfile query irregularities with respect to policies.
- Fixes a bug with Asana publishing when the remote task is deleted.
- Fixes an issue with Herald commit rules.
Test Plan:
- Viewed projects;
- edited projects;
- added and removed members from projects;
- republished Asana-bridged feed stories about commits.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D7251
Summary:
Ref T603. Adds clarifying text which expands on policies and explains exceptions and rules. The goal is to provide an easy way for users to learn about special policy rules, like "task owners can always see a task".
This presentation might be a little aggressive. That's probably OK as we introduce policies, but something a little more tempered might be better down the road.
Test Plan: See screenshot.
Reviewers: btrahan, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D7150
Summary:
Ref T3687. JIRA is able to piggyback on a fair amount of Asana infrastructure, but the voicing we use on Asana tasks (which are always about one object) isn't very good for JIRA issues (which may have many linked objects). Specifically, we publish stories like this to Asana:
alincoln accepted this revision.
This is meaningless in JIRA since you have no idea what it's talking about. Instead, publish like this:
alincoln accepted D999: Put a bird on it
Additionally, supplement it with a URI, so the total story text we publish is:
alincoln accepted D999: Put a bird on it
https://phabricator.whitehouse.gov/D999
Signifcantly less useless!
Test Plan: {F57523} {F57524}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3687
Differential Revision: https://secure.phabricator.com/D6907
Summary:
Ref T3687. See some discussion in D6892. The JIRA doorkeeper publisher shares a reasonable amount of code with the Asana publisher. Remedy this:
- Create `DoorkeeperFeedWorker`, where shared functionality lives (mostly related to building story context objects).
- Push responsibility for enabling/disabling a worker into this new layer, via `isEnabled()`. This allows `FeedPublisherWorker` to dynamically find and schedule doorkeeper publishers, so third parties can add additional doorkeeper publishers.
- Some general cleanup/documentation.
Test Plan: Used `bin/feed republish` to republish stories about objects with JIRA and Asana links. Verified that doorkeeper publishers activated properly, made calls, and published events into the remote systems.
Reviewers: btrahan, akopanev22
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3687
Differential Revision: https://secure.phabricator.com/D6906
Summary:
Ref T3687. Publish stories into JIRA.
These need some voicing fixes, which maybe involves straightening out the feed code. For example, they're voiced in-context ("updated this revision") when they should be voiced out-of-context ("updated D123").
Generally, this is similar to the Asana stuff but a lot simpler since we don't need to do any state management.
Test Plan: {F57366}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3687
Differential Revision: https://secure.phabricator.com/D6892
Summary:
Ref T3687. Adds a Doorkeeper bridge for JIRA issues, plus remarkup support. In particular:
- The Asana and JIRA remarkup rules shared most of their implementation, so I refactored what I could into a base class.
- Actual bridge implementation is straightforward and similar to Asana, although probably not similar enough to really justify refactoring.
Test Plan:
- When logged in as a JIRA-connected user, pasted a JIRA issue link and saw it enriched at rendering time.
- Logged in and out with JIRA.
- Tested an Asana link, too (seems I haven't broken anything).
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3687
Differential Revision: https://secure.phabricator.com/D6878
Summary: Ref T2852. Asana is launching some kind of silent follow thing today; I don't know what the API is but it's probably something like this. I'll update this to actually make the right call once the call exists, this is mostly just a placeholder so I don't forget about it.
Test Plan: None yet, this API isn't documented or live and doesn't work yet so it can't be tested.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, moskov
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6740
Summary: Ref T2852. Currently, we publish commits with no audit requests and reviews with no CCs or reviewers into Asana. This creates undesired notifications, so drop events which would publish an object that doesn't exist yet and has no followers or respible users.
Test Plan: Used `bin/feed republish` to publish a story about an object with no related users, saw the publish abort with the new message. Added a CC, published again, got a publish.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6727
Summary: Ref T2852. Asana adds the actor as a follower when they create a task, so subtasks currently have up to two followers (the actor and the reviewer) when they should have only one (the reviewer). Simply removing the actor is an effective remedy for this because unfollowing tasks occurs with sneaky ninja stealth in Asana and doesn't generate notifications or even transaction activity.
Test Plan: Synchronized a revision without this patch, saw two followers on the subtask. Synchronized a revision after this patch, saw the "removeFollowers" fire and only one follower on the subtask, with no record of the removal in notifications or the transaction log.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6700
Summary:
Ref T2852. Two issues:
- Embeds (`T12`, `{T12}`) have some handle issues because handles run afoul of visibility checks under some configs. Make handles unconditionally visible.
- Asana links don't render correctly into text mode. Give them a valid text mode rendering so they don't flip out.
Test Plan: Made comments with `T12` and `http://app.asana.com/...` and published them to Asana.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6696
Summary: Ref T2852. After some discussion, Asana doesn't want "close" stories either.
Test Plan: Used `bin/feed republish` to publish close and non-close stories from Differential and Diffusion. Verified comments were synchronized in the expected cases.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6697
Summary: Ref T2852. Current code works fine, but although we want to drop creation stories, we really only want to drop the story text, not the other effects of the creation story. Also generalize this mechanism so we don't have Asana-specific code in the publishers.
Test Plan: Used `bin/feed republish` to publish creation and non-creation stories. Verified creation story published no text.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6639
Summary: Ref T2852. Pulls the Differential-specific aspects of the Asana sync out of the worker. Next diff will add a publisher for Audit/Diffusion.
Test Plan: Published events, including state changes. Saw them reflected correctly in Asana.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6569
Summary: Ref T2852. When the parent task is actionable (needs revision, accepted) give it an "Upcoming" status; otherwise give it a "Later" status.
Test Plan: Sync'd "Needs Revision" and "Needs Review" tasks and saw them both bucket correctly
Reviewers: btrahan, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6551
Summary: `FeedStory->renderText()` is garbage and I don't want to fix it in general until after T2222 / T2217. Provide an Asana-specific alternative for higher-quality feed stories (notably, including comment text).
Test Plan: {F51035}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6521
Summary:
Ref T2852. We need to distinguish between an API call which worked but got back nothing (404) and an API call which failed.
In particular, Asana hit a sync issue which was likely the result of treating a 500 (or some other error) as a 404.
Also clean up a couple small things.
Test Plan: Ran syncs against deleted tasks and saw successful syncs of non-tasks, and simulated random failures and saw them get handled correctly.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6470
Summary: We attempt to choose the most reasonable actor when synchronizing to Asana, but Asana is seeing the sync choose a less-reasonable actor. I spotted two places where the order may get disrupted; make sure we retain order. This is somewhat tricky to repro locally (it depends on things like native account order) but I think this is the right fix. If not, I'll add more logging. Ref T2852.
Test Plan: Used `bin/feed republish` to sync Asana events.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6443
Summary:
Ref T2852.
- Respect the existing setting for `"[Differential]"`.
- Show `[Request, X lines]` to make this more similar to the email.
Test Plan:
Sync'd to asana and got a task with the right subject:
{F49950}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6439
Summary:
Ref T2852. It's a little tricky to figure out Asana workspace IDs right now. If the viewer has a linked account, just pull their workspaces and show them which IDs are available.
(In theory, we could use a `<select>`, but it would have more edge cases; this seems like a pretty solid fix.)
Test Plan: {F49938}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6437
Summary: Ref T2852. When a Differential revision is linked to an Asana task, show the related task in Differential.
Test Plan: {F49234}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6387
Summary:
Ref T2852. Asana sync tasks currently have a standard retry/backoff schedule, but the defaults are quite aggressive (retry every 60s forever). Instead, retry at increasing intervals and stop retrying after a few tries.
- Retry at intervals and stop retrying after a few iterations.
- Modernize some interfaces.
- Add better information about retry behaviors to the web UI.
Test Plan: {F49194}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6381
Summary:
Depends on D6329. This fixes `http://www.example.com/D123`, which currently gets the "D123" rendered, after addition of the Asana rule. It also removes a hack for object refernces.
Basically, the "hyperlink" rule needs to happen after rules which specialize hyperlinks (Youtube, Asana) but before rules which apply to general text (like the Differential and Maniphest rules). Allow these rules to specify that they have higher or lower priority.
Test Plan: Asana rules, Differential rules and Diffusion rules now all markup correctly.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D6330
Summary: Ref T2852. Setting followers (like CCs) is a separate API call, but we don't need to do anything complicated.
Test Plan: Synchronized revisions and verified the parent task got followers.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6308
Summary:
Ref T2852.
Before trying related users, try using the feed story's actor. This is the most correct voice to act in.
Test Plan: Ran `feed/republish`.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6305
Summary:
Ref T2852.
The parent task is open unless the revision is in the states "closed" or "abandoned". If it's in "needs review", it remains open. This last bit is slightly unlike Differential, but consistent with the Google Doc and generally seems like a better fit. There's no way to put the task in a "Waiting on Others" state in Asana like we can in Differential.
The subtasks are closed unless the revision is in the state "needs review". This is generally consistent with Differential.
Test Plan:
Made a series of changes to a revision and synchronized it repeatedly:
- requested changes
- commandeered
- requested review
- abandoned
Verified task and subtasks synchronized states correctly in Asana.
{F47554}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6304
Summary:
Ref T2852. Depends on D6302. This now creates, destroys, and synchronizes subtasks.
- After finishing the parent task stuff, we pull a list of all known subtasks.
- We load all those subtasks.
- If we fail to load any, we delete their objects and edges on the Phabricator side.
- Of the remaining subtasks, we find subtasks for users who aren't related to the object any more and delete them in Asana and locally (for example, if alincoln is removed as a reviewer, we delete his subtask).
- For all the related users, we either synchronize their existing task or create a new one for them.
- Then we write edges for any new tasks we added.
This doesn't handle a few weird edge cases in any specific way:
- If a subtask is moved under a different parent, we ignore it.
- If a new subtask is created that we don't know about, we ignore it.
- If a subtask we know about is deleted, we just respawn it. This is consistent with "DON'T EDIT THESE". You can force sync to stop by deleting the parent.
Addititionally:
- Make the "don't edit" warning more compelling and visceral.
Test Plan:
- Kind of ran it a bit.
- There are like 3,000 edge cases here so this is hard to test exhaustively.
- Forced a few of the edge cases to happen.
- Nothing seems immediately broken in an obvious way?
{F47551}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6303
Summary:
Ref T2852. This is highly incomplete but seems structurally sound. Some additional context is available in the Google doc.
- Add a workspace ID configuration. Without it, nothing else activates.
- Add a worker which reacts to feed stories.
- Feed stories about things which aren't Differential objects are ignored.
- We load the revision, or fail permanently if we can't.
- We get all the related user PHIDs (author, reviewers, CCs).
- We check if any of them have linked Asana accounts, or fail permanently if they don't.
- We check for an "ASANATASK" edge from the revision.
- If we do not find one, we create a new task.
- If we do find one, we load the task.
- If we succeed, we check the chronological key of the most recent synchronized feed story ("cursor").
- If this story is the same or newer, we update the task to synchronize it to the current state of the revision.
- If we fail to load the task, we fail permanently ("asana task has been deleted").
- We then publish the actual story text to the task.
Not in yet:
- Updating followers requires separate API calls which we don't do yet.
- No subtasks yet.
- No sync of open/closed state.
Test Plan: {F47546}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6302
Summary: Ref T2852. Reduce the number of magical strings in use, and prepare the Asana bridge for eventual workspace/project support (a little bit).
Test Plan: Verified enriched links still work properly.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6301
Summary:
Ref T2852.
- Broadly, we support "I have a Ref, I need a PHID" well but not "I have a PHID, I need a Ref".
- Add DoorkeeperExternalObjectQuery, and use it to query ExternalObjects.
- Allow external objects to be imported by their internal PHIDs. Basically, if we have an edge pointing at an ExternalObject, we can say "load all the data about this" from just the PHID and have it hit all the same code.
- Allow construction of Refs from ExternalObjects. This makes the "I have a PHID, I need a Ref" easier.
Test Plan:
- Verified Asana links still enrich properly at display time.
- Used in future revision.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6294
Summary:
Ref T2852. Give OAuth providers a formal method so you can ask them for tokens; they issue a refresh request if necessary.
We could automatically refresh these tokens in daemons as they near expiry to improve performance; refreshes are blocking in-process round trip requests. If we do this for all tokens, it's a lot of requests (say, 20k users * 2 auth mechanisms * 1-hour tokens ~= a million requests a day). We could do it selectively for tokens that are actually in use (i.e., if we refresh a token in response to a user request, we keep refreshing it for 24 hours automatically). For now, I'm not pursuing any of this.
If we fail to refresh a token, we don't have a great way to communicate it to the user right now. The remedy is "log out and log in again", but there's no way for them to figure this out. The major issue is that a lot of OAuth integrations should not throw if they fail, or can't reasonably be rasied to the user (e.g., activity in daemons, loading profile pictures, enriching links, etc). For now, this shouldn't really happen. In future diffs, I plan to make the "External Accounts" settings page provide some information about tokens again, and possibly push some flag to accounts like "you should refresh your X link", but we'll see if issues crop up.
Test Plan: Used `bin/auth refresh` to verify refreshes. I'll wait an hour and reload a page with an Asana link to verify the auto-refresh part.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6280
Summary: Ref T2852. Provide a script for inspecting/debugging OAuth token refresh.
Test Plan: Ran `bin/auth refresh` with various arguments, saw token refreshes.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6276
Summary:
Ref T2852. Primarily, this expands API access to Asana. As a user-visible effect, it links Asana tasks in Remarkup.
When a user enters an Asana URI, we register an onload behavior to make an Ajax call for the lookup. This respects privacy imposed by the API without creating a significant performance impact.
Test Plan: {F47183}
Reviewers: btrahan
Reviewed By: btrahan
CC: chad, aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6274
Summary:
- `DoorkeeperObjectRef` is a convenience object to keep track of `<applicationType, applicationDomain, objectType, objectID>` tuples.
- `DoorkeeperBridge` provides pull/push between Phabricator and external systems.
- `DoorkeeperBridgeAsana` is a bridge to Asana.
Test Plan:
Ran this snippet and got a task from Asana:
{P871}
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D6273
Summary:
Ref T2852. This table holds data about external objects and allows us to write edges to them.
Objects are identified with an `<applicationType, applicationDomain, objectType, objectID>` tuple. For example, Asana tasks will be, e.g., `<asana, asana.com, asana:task, 93829279873>` or similar.
Test Plan: Ran storage upgrade.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2852
Differential Revision: https://secure.phabricator.com/D6271
