Quicksand, an ignoble successor to Quickling

Summary:
Ref T2086. Ref T7014. With the persistent column, there is significant value in retaining chrome state through navigation events, because the user may have a lot of state in the chat window (scroll position, text selection, room juggling, partially entered text, etc). We can do this by capturing navigation events and faking them with Javascript.

(This can also improve performance, albeit slightly, and I believe there are better approaches to tackle performance any problems which exist with the chrome in many cases).

At Facebook, this system was "Photostream" in photos and then "Quickling" in general, and the technical cost of the system was //staggering//. I am loathe to pursue it again. However:

  - Browsers are less junky now, and we target a smaller set of browsers. A large part of the technical cost of Quickling was the high complexity of emulating nagivation events in IE, where we needed to navigate a hidden iframe to make history entries. All desktop browsers which we might want to use this system on support the History API (although this prototype does not yet implement it).
  - Javelin and Phabricator's architecture are much cleaner than Facebook's was. A large part of the technical cost of Quickling was inconsistency, inlined `onclick` handlers, and general lack of coordination and abstraction. We will have //some// of this, but "correctly written" behaviors are mostly immune to it by design, and many of Javelin's architectural decisions were influenced by desire to avoid issues we encountered building this stuff for Facebook.
  - Some of the primitives which Quickling required (like loading resources over Ajax) have existed in a stable state in our codebase for a year or more, and adoption of these primitives was trivial and uneventful (vs a huge production at Facebook).
  - My hubris is bolstered by recent success with WebSockets and JX.Scrollbar, both of which I would have assessed as infeasibly complex to develop in this project a few years ago.

To these points, the developer cost to prototype Photostream was several weeks; the developer cost to prototype this was a bit less than an hour. It is plausible to me that implementing and maintaining this system really will be hundreds of times less complex than it was at Facebook.

Test Plan:
My plan for this and D11497 is:

  - Get them in master.
  - Some secret key / relatively-hidden preference activates the column.
  - Quicksand activates //only// when the column is open.
  - We can use column + quicksand for a long period of time (i.e., over the course of Conpherence v2 development) and hammer out the long tail of issues.
  - When it derps up, you just hide the column and you're good to go.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T2086, T7014

Differential Revision: https://secure.phabricator.com/D11507

resources/celerity/map.php

@@ -8,7 +8,7 @@
 return array(
   'names' => array(
     'core.pkg.css' => '3d6955ad',
-    'core.pkg.js' => 'df35fdde',
+    'core.pkg.js' => '7a54aa14',
     'darkconsole.pkg.js' => '8ab24e01',
     'differential.pkg.css' => '8af45893',
     'differential.pkg.js' => '5c1f3896',
@@ -189,10 +189,11 @@ return array(
     'rsrc/externals/javelin/ext/view/__tests__/ViewRenderer.js' => '6ea96ac9',
     'rsrc/externals/javelin/lib/Cookie.js' => '62dfea03',
     'rsrc/externals/javelin/lib/DOM.js' => '2d66f6ec',
-    'rsrc/externals/javelin/lib/History.js' => 'c60f4327',
+    'rsrc/externals/javelin/lib/History.js' => '2e0148bc',
     'rsrc/externals/javelin/lib/JSON.js' => '69adf288',
     'rsrc/externals/javelin/lib/Leader.js' => '331b1611',
     'rsrc/externals/javelin/lib/Mask.js' => '8a41885b',
+    'rsrc/externals/javelin/lib/Quicksand.js' => 'f960d43d',
     'rsrc/externals/javelin/lib/Request.js' => '94b750d2',
     'rsrc/externals/javelin/lib/Resource.js' => '44959b73',
     'rsrc/externals/javelin/lib/Routable.js' => 'b3e7d692',
@@ -342,7 +343,7 @@ return array(
     'rsrc/js/application/aphlict/behavior-aphlict-status.js' => 'ea681761',
     'rsrc/js/application/auth/behavior-persona-login.js' => '9414ff18',
     'rsrc/js/application/config/behavior-reorder-fields.js' => '14a827de',
-    'rsrc/js/application/conpherence/behavior-durable-column.js' => 'acad7376',
+    'rsrc/js/application/conpherence/behavior-durable-column.js' => '0c404426',
     'rsrc/js/application/conpherence/behavior-menu.js' => 'f0a41b9f',
     'rsrc/js/application/conpherence/behavior-pontificate.js' => '2f6efe18',
     'rsrc/js/application/conpherence/behavior-widget-pane.js' => '40b1ff90',
@@ -575,7 +576,7 @@ return array(
     'javelin-behavior-diffusion-locate-file' => '6d3e1947',
     'javelin-behavior-diffusion-pull-lastmodified' => '2b228192',
     'javelin-behavior-doorkeeper-tag' => 'e5822781',
-    'javelin-behavior-durable-column' => 'acad7376',
+    'javelin-behavior-durable-column' => '0c404426',
     'javelin-behavior-error-log' => '6882e80a',
     'javelin-behavior-fancy-datepicker' => 'c51ae228',
     'javelin-behavior-global-drag-and-drop' => '07f199d8',
@@ -654,12 +655,13 @@ return array(
     'javelin-dynval' => 'f6555212',
     'javelin-event' => '85ea0626',
     'javelin-fx' => '54b612ba',
-    'javelin-history' => 'c60f4327',
+    'javelin-history' => '2e0148bc',
     'javelin-install' => '05270951',
     'javelin-json' => '69adf288',
     'javelin-leader' => '331b1611',
     'javelin-magical-init' => '4df97779',
     'javelin-mask' => '8a41885b',
+    'javelin-quicksand' => 'f960d43d',
     'javelin-reactor' => '2b8de964',
     'javelin-reactor-dom' => 'c90a04fc',
     'javelin-reactor-node-calmer' => '76f4ebed',
@@ -862,6 +864,14 @@ return array(
       'javelin-dom',
       'javelin-router',
     ),
+    '0c404426' => array(
+      'javelin-behavior',
+      'javelin-dom',
+      'javelin-stratcom',
+      'javelin-scrollbar',
+      'javelin-quicksand',
+      'phabricator-keyboard-shortcut',
+    ),
     '0c6946e7' => array(
       'javelin-install',
       'javelin-dom',
@@ -989,6 +999,12 @@ return array(
       'javelin-vector',
       'javelin-stratcom',
     ),
+    '2e0148bc' => array(
+      'javelin-stratcom',
+      'javelin-install',
+      'javelin-uri',
+      'javelin-util',
+    ),
     '2f6efe18' => array(
       'javelin-behavior',
       'javelin-dom',
@@ -1603,13 +1619,6 @@ return array(
       'javelin-util',
       'phabricator-prefab',
     ),
-    'acad7376' => array(
-      'javelin-behavior',
-      'javelin-dom',
-      'javelin-stratcom',
-      'javelin-scrollbar',
-      'phabricator-keyboard-shortcut',
-    ),
     'b07b009f' => array(
       'javelin-behavior',
       'javelin-dom',
@@ -1705,12 +1714,6 @@ return array(
       'javelin-stratcom',
       'javelin-vector',
     ),
-    'c60f4327' => array(
-      'javelin-stratcom',
-      'javelin-install',
-      'javelin-uri',
-      'javelin-util',
-    ),
     'c90a04fc' => array(
       'javelin-dom',
       'javelin-dynval',
@@ -1963,6 +1966,9 @@ return array(
       'javelin-stratcom',
       'javelin-uri',
     ),
+    'f960d43d' => array(
+      'javelin-install',
+    ),
     'fa0f4fc2' => array(
       'javelin-behavior',
       'javelin-dom',

src/aphront/AphrontRequest.php

@@ -19,6 +19,7 @@ final class AphrontRequest {
   const TYPE_CONTINUE = '__continue__';
   const TYPE_PREVIEW = '__preview__';
   const TYPE_HISEC = '__hisec__';
+  const TYPE_QUICKSAND = '__quicksand__';
 
   private $host;
   private $path;
@@ -194,10 +195,14 @@ final class AphrontRequest {
     return $this->getExists(self::TYPE_AJAX);
   }
 
-  final public function isJavelinWorkflow() {
+  final public function isWorkflow() {
     return $this->getExists(self::TYPE_WORKFLOW);
   }
 
+  final public function isQuicksand() {
+    return $this->getExists(self::TYPE_QUICKSAND);
+  }
+
   final public function isConduit() {
     return $this->getExists(self::TYPE_CONDUIT);
   }

src/aphront/configuration/AphrontDefaultApplicationConfiguration.php

@@ -65,7 +65,7 @@ class AphrontDefaultApplicationConfiguration
     }
 
     // For non-workflow requests, return a Ajax response.
-    if ($request->isAjax() && !$request->isJavelinWorkflow()) {
+    if ($request->isAjax() && !$request->isWorkflow()) {
       // Log these; they don't get shown on the client and can be difficult
       // to debug.
       phlog($ex);

src/applications/base/controller/PhabricatorController.php

@@ -242,8 +242,18 @@ abstract class PhabricatorController extends AphrontController {
   public function buildStandardPageResponse($view, array $data) {
     $page = $this->buildStandardPageView();
     $page->appendChild($view);
-    $response = new AphrontWebpageResponse();
+    return $this->buildPageResponse($page);
-    $response->setContent($page->render());
+  }
+
+  private function buildPageResponse($page) {
+    if ($this->getRequest()->isQuicksand()) {
+      $response = id(new AphrontAjaxResponse())
+        ->setContent($page->renderForQuicksand());
+    } else {
+      $response = id(new AphrontWebpageResponse())
+        ->setContent($page->render());
+    }
+
     return $response;
   }
 
@@ -303,8 +313,7 @@ abstract class PhabricatorController extends AphrontController {
       $page->setApplicationMenu($application_menu);
     }
 
-    $response = new AphrontWebpageResponse();
+    return $this->buildPageResponse($page);
-    return $response->setContent($page->render());
   }
 
   public function didProcessRequest($response) {
@@ -331,7 +340,7 @@ abstract class PhabricatorController extends AphrontController {
     }
 
     if ($response instanceof AphrontDialogResponse) {
-      if (!$request->isAjax()) {
+      if (!$request->isAjax() && !$request->isQuicksand()) {
         $dialog = $response->getDialog();
 
         $title = $dialog->getTitle();

src/view/page/PhabricatorStandardPageView.php

@@ -311,8 +311,6 @@ final class PhabricatorStandardPageView extends PhabricatorBarePageView {
   }
 
   protected function getBody() {
-    $console = $this->getConsole();
-
     $user = null;
     $request = $this->getRequest();
     if ($request) {
@@ -367,11 +365,13 @@ final class PhabricatorStandardPageView extends PhabricatorBarePageView {
         $developer_warning,
         $setup_warning,
         $header_chrome,
-        phutil_tag_div('phabricator-standard-page-body', array(
+        phutil_tag(
-          ($console ? hsprintf('<darkconsole />') : null),
+          'div',
-          parent::getBody(),
+          array(
-          $this->renderFooter(),
+            'id' => 'phabricator-standard-page-body',
-        )),
+            'class' => 'phabricator-standard-page-body',
+          ),
+          $this->renderPageBodyContent()),
       ));
 
     $durable_column = null;
@@ -390,6 +390,16 @@ final class PhabricatorStandardPageView extends PhabricatorBarePageView {
       ));
   }
 
+  private function renderPageBodyContent() {
+    $console = $this->getConsole();
+
+    return array(
+      ($console ? hsprintf('<darkconsole />') : null),
+      parent::getBody(),
+      $this->renderFooter(),
+    );
+  }
+
   protected function getTail() {
     $request = $this->getRequest();
     $user = $request->getUser();
@@ -529,4 +539,15 @@ final class PhabricatorStandardPageView extends PhabricatorBarePageView {
       $foot);
   }
 
+  public function renderForQuicksand() {
+    // TODO: We could run a lighter version of this and skip some work. In
+    // particular, we end up including many redundant resources.
+    $this->willRenderPage();
+    $response = $this->renderPageBodyContent();
+    $response = $this->willSendResponse($response);
+
+    return array(
+      'content' => hsprintf('%s', $response),
+    );
+  }
 }

webroot/rsrc/externals/javelin/lib/History.js

@@ -115,9 +115,10 @@ JX.install('History', {
      * Pushes a path onto the history stack.
      *
      * @param string Path.
+     * @param wild State object for History API.
      * @return void
      */
-    push : function(path) {
+    push : function(path, state) {
       if (__DEV__) {
         if (!JX.History._installed) {
           JX.$E(
@@ -129,7 +130,7 @@ JX.install('History', {
         if (JX.History._initialPath && JX.History._initialPath !== path) {
           JX.History._initialPath = null;
         }
-        history.pushState(null, null, path);
+        history.pushState(state || null, null, path);
         JX.History._fire(path);
       } else {
         location.hash = JX.History._composeFragment(path);
@@ -167,13 +168,15 @@ JX.install('History', {
       }
     },
 
-    _handleChange : function() {
+    _handleChange : function(e) {
       var path = JX.History.getPath();
+      var state = (e && e.getRawEvent().state);
+
       if (JX.History.getMechanism() === JX.History.PUSHSTATE) {
         if (path === JX.History._initialPath) {
           JX.History._initialPath = null;
         } else {
-          JX.History._fire(path);
+          JX.History._fire(path, state);
         }
       } else {
         if (path !== JX.History._hash) {
@@ -183,9 +186,10 @@ JX.install('History', {
       }
     },
 
-    _fire : function(path) {
+    _fire : function(path, state) {
       JX.Stratcom.invoke('history:change', null, {
-        path: JX.History._getBasePath(path)
+        path: JX.History._getBasePath(path),
+        state: state
       });
     },
 

webroot/rsrc/externals/javelin/lib/Quicksand.js

@@ -0,0 +1,282 @@
+/**
+ * @requires javelin-install
+ * @provides javelin-quicksand
+ * @javelin
+ */
+
+/**
+ * Sink into a hopeless, cold mire of limitless depth from which there is
+ * no escape.
+ *
+ * Captures navigation events (like clicking links and using the back button)
+ * and expresses them in Javascript instead, emulating complex native browser
+ * behaviors in a language and context ill-suited to the task.
+ *
+ * By doing this, you abandon all hope and retreat to a world devoid of light
+ * or goodness. However, it allows you to have persistent UI elements which are
+ * not disrupted by navigation. A tempting trade, surely?
+ *
+ * To cast your soul into the darkness, use:
+ *
+ *   JX.Quicksand
+ *     .setFrame(node)
+ *     .start();
+ */
+JX.install('Quicksand', {
+
+  statics: {
+    _id: 0,
+    _onpage: 0,
+    _cursor: 0,
+    _current: 0,
+    _content: {},
+    _history: [],
+    _started: false,
+    _frameNode: null,
+    _contentNode: null,
+
+    /**
+     * Start Quicksand, accepting a fate of eternal torment.
+     */
+    start: function() {
+      var self = JX.Quicksand;
+      if (self._started) {
+        return;
+      }
+
+      JX.Stratcom.listen('click', 'tag:a', self._onclick);
+      JX.Stratcom.listen('history:change', null, self._onchange);
+
+      self._started = true;
+      self._history.push({
+        id: 0,
+        path: self._getRelativeURI(window.location)
+      });
+    },
+
+
+    /**
+     * Set the frame node which Quicksand controls content for.
+     */
+    setFrame: function(frame) {
+      var self = JX.Quicksand;
+      self._frameNode = frame;
+      return self;
+    },
+
+
+    /**
+     * Respond to the user clicking a link.
+     *
+     * After a long list of checks, we may capture and simulate the resulting
+     * navigation.
+     */
+    _onclick: function(e) {
+      var self = JX.Quicksand;
+
+      if (!self._frameNode) {
+        // If Quicksand has no frame, bail.
+        return;
+      }
+
+      if (JX.Stratcom.pass()) {
+        // If something else handled the event, bail.
+        return;
+      }
+
+      if (!e.isNormalClick()) {
+        // If this is a right-click, control click, etc., bail.
+        return;
+      }
+
+      if (e.getNode('workflow')) {
+        // Because JX.Workflow also passes these events, it might still want
+        // the event. Don't trigger if there's a workflow node in the stack.
+        return;
+      }
+
+      var a = e.getNode('tag:a');
+      var href = a.href;
+      if (!href || !href.length) {
+        // If the <a /> the user clicked has no href, or the href is empty,
+        // bail.
+        return;
+      }
+
+      if (href[0] == '#') {
+        // If this is an anchor on the current page, bail.
+        return;
+      }
+
+      var uri = new JX.$U(href);
+      var here = new JX.$U(window.location);
+      if (uri.getDomain() != here.getDomain()) {
+        // If the link is off-domain, bail.
+        return;
+      }
+
+      if (uri.getFragment() && uri.getPath() == here.getPath()) {
+        // If the link has an anchor but points at the current path, bail.
+        // This is presumably a long-form anchor on the current page.
+
+        // TODO: This technically gets links which change query parameters
+        // wrong: they are navigation events but we won't Quicksand them.
+        return;
+      }
+
+      // The fate of this action is sealed. Suck it into the depths.
+      e.kill();
+
+      // If we're somewhere in history (that is, the user has pressed the
+      // back button one or more times, putting us in a state where pressing
+      // the forward button would do something) and we're navigating forward,
+      // all the stuff ahead of us is about to become unreachable when we
+      // navigate. Throw it away.
+      var discard = (self._history.length - self._cursor) - 1;
+      for (var ii = 0; ii < discard; ii++) {
+        var obsolete = self._history.pop();
+        self._content[obsolete.id] = false;
+      }
+
+      // Set up the new state and fire a request to fetch the page content.
+      var path = self._getRelativeURI(uri);
+      var id = ++self._id;
+
+      JX.History.push(path, id);
+
+      self._history.push({path: path, id: id});
+      self._cursor = (self._history.length - 1);
+      self._content[id] = null;
+      self._current = id;
+
+      new JX.Workflow(href, {__quicksand__: true})
+        .setHandler(JX.bind(null, self._onresponse, id))
+        .start();
+    },
+
+
+    /**
+     * Receive a response from the server with page content.
+     *
+     * Usually we'll dump it into the page, but if the user clicked very fast
+     * it might already be out of date.
+     */
+    _onresponse: function(id, r) {
+      var self = JX.Quicksand;
+
+      // Before possibly updating the document, check if this response is still
+      // relevant.
+
+      // We don't save the new content if the user has already destroyed
+      // the navigation. They can do this by pressing back, then clicking
+      // another link before the content can load.
+      if (self._content[id] === false) {
+        return;
+      }
+
+      // Otherwise, this data is still relevant (either data on the current
+      // page, or data for a page that's still somewhere in history), so we
+      // save it.
+      var new_content = JX.$H(r.content).getFragment();
+      self._content[id] = new_content;
+
+      // If it's the current page, draw it into the browser. It might not be
+      // the current page if the user already clicked another link.
+      if (self._current == id) {
+        self._draw();
+      }
+    },
+
+
+    /**
+     * Draw the current page.
+     *
+     * After a navigation event or the arrival of page content, we paint it
+     * onto the page.
+     */
+    _draw: function() {
+      var self = JX.Quicksand;
+
+      if (self._onpage == self._current) {
+        // Don't bother redrawing if we're already on the current page.
+        return;
+      }
+
+      if (!self._content[self._current]) {
+        // If we don't have this page yet, we can't draw it. We'll draw it
+        // when it arrives.
+        return;
+      }
+
+      // Otherwise, we're going to replace the page content. First, save the
+      // current page content. Modern computers have lots and lots of RAM, so
+      // there is no way this could ever create a problem.
+      var old = window.document.createDocumentFragment();
+      while (self._frameNode.firstChild) {
+        JX.DOM.appendContent(old, self._frameNode.firstChild);
+      }
+      self._content[self._onpage] = old;
+
+      // Now, replace it with the new content.
+      JX.DOM.setContent(self._frameNode, self._content[self._current]);
+      self._onpage = self._current;
+
+      // Scroll to the top of the page and trigger any layout adjustments.
+
+      // TODO: Maybe store the scroll position?
+      JX.DOM.scrollToPosition(0, 0);
+      JX.Stratcom.invoke('resize');
+    },
+
+
+    /**
+     * Handle navigation events.
+     *
+     * In general, we're going to pull the content out of our history and dump
+     * it into the document.
+     */
+    _onchange: function(e) {
+      var self = JX.Quicksand;
+
+      var data = e.getData();
+      data.state = data.state || null;
+
+      // Check if we're going back to the first page we started Quicksand on.
+      // We don't have a state value, but can look at the path.
+      if (data.state === null) {
+        if (JX.$U(window.location).getPath() == self._history[0].path) {
+          data.state = 0;
+        }
+      }
+
+      // Figure out where in history the user jumped to.
+      if (data.state !== null) {
+        self._current = data.state;
+
+        // Point the cursor at the right place in history.
+        for (var ii = 0; ii < self._history.length; ii++) {
+          if (self._history[ii].id == self._current) {
+            self._cursor = ii;
+            break;
+          }
+        }
+
+        // Redraw the page.
+        self._draw();
+      }
+    },
+
+
+    /**
+     * Get just the relative part of a URI, for History operations.
+     */
+    _getRelativeURI: function(uri) {
+      return JX.$U(uri)
+        .setProtocol(null)
+        .setPort(null)
+        .setDomain(null)
+        .toString();
+    }
+  }
+
+});

webroot/rsrc/js/application/conpherence/behavior-durable-column.js

@@ -4,12 +4,14 @@
  *           javelin-dom
  *           javelin-stratcom
  *           javelin-scrollbar
+ *           javelin-quicksand
  *           phabricator-keyboard-shortcut
  */
 
 JX.behavior('durable-column', function() {
 
   var frame = JX.$('phabricator-standard-page');
+  var quick = JX.$('phabricator-standard-page-body');
   var show = false;
 
   new JX.KeyboardShortcut('\\', 'Toggle Column (Prototype)')
@@ -18,10 +20,12 @@ JX.behavior('durable-column', function() {
       JX.DOM.alterClass(frame, 'with-durable-column', show);
       JX.$('durable-column').style.display = (show ? 'block' : 'none');
       JX.Stratcom.invoke('resize');
+      JX.Quicksand.setFrame(show ? quick : null);
     })
     .register();
 
   new JX.Scrollbar(JX.$('phui-durable-column-content'));
 
+  JX.Quicksand.start();
 
 });