Manually prune Git working copy refs instead of using "--prune", to improve "Fetch Refs" behavior

Summary:
Ref T13448. When "Fetch Refs" is configured:

  - We switch to a narrow mode when running "ls-remote" against the local working copy. This excludes surplus refs, so we'll incorrectly detect that the local and remote working copies are identical in cases where the local working copy really has surplus refs.
  - We rely on "--prune" to remove surplus local refs, but it only prunes refs matching the refspecs we pass "git fetch". Since these refspecs are very narrow under "Fetch Only", the pruning behavior is also very narrow.

Instead:

  - When listing local refs, always list everything. If we have too much stuff locally, we want to get rid of it.
  - When we identify surplus local refs, explicitly delete them instead of relying on "--prune". We can just do this in all cases so we don't have separate "--prune" and "manual" cases.

Test Plan:
  - Created a new repository, observed from a GitHub repository, with many tags/refs/branches. Pulled it.
  - Observed lots of refs in `git for-each-ref`.
  - Changed "Fetch Refs" to "refs/heads/master".
  - Ran `bin/repository pull X --trace --verbose`.

On deciding to do something:

  - Before: since "master" did not change, the pull declined to act.
  - After: the pull detected surplus refs and deleted them. Since the states then matched, it declined further action.

On pruning:

  - Before: if the pull was forced to act, it ran "fetch --prune" with a narrow refspec, which did not prune the working copy.
  - After: saw working copy pruned explicitly with "update-ref -d" commands.

Also, set "Fetch Refs" back to the default (empty) and pulled, saw everything pull.

Maniphest Tasks: T13448

Differential Revision: https://secure.phabricator.com/D20892

src/applications/repository/engine/PhabricatorRepositoryPullEngine.php

@@ -353,13 +353,56 @@ final class PhabricatorRepositoryPullEngine
     // Load the refs we're planning to fetch from the remote repository.
     $remote_refs = $this->loadGitRemoteRefs(
       $repository,
-      $repository->getRemoteURIEnvelope());
+      $repository->getRemoteURIEnvelope(),
+      $is_local = false);
 
     // Load the refs we're planning to fetch from the local repository, by
     // using the local working copy path as the "remote" repository URI.
     $local_refs = $this->loadGitRemoteRefs(
       $repository,
-      new PhutilOpaqueEnvelope($path));
+      new PhutilOpaqueEnvelope($path),
+      $is_local = true);
+
+    // See T13448. The "git fetch --prune ..." flag only prunes local refs
+    // matching the refspecs we pass it. If "Fetch Refs" is configured, we'll
+    // pass it a very narrow list of refspecs, and it won't prune older refs
+    // that aren't currently subject to fetching.
+
+    // Since we want to prune everything that isn't (a) on the fetch list and
+    // (b) in the remote, handle pruning of any surplus leftover refs ourselves
+    // before we fetch anything.
+
+    // (We don't have to do this if "Fetch Refs" isn't set up, since "--prune"
+    // will work in that case, but it's a little simpler to always go down the
+    // same code path.)
+
+    $surplus_refs = array();
+    foreach ($local_refs as $local_ref => $local_hash) {
+      $remote_hash = idx($remote_refs, $local_ref);
+      if ($remote_hash === null) {
+        $surplus_refs[] = $local_ref;
+      }
+    }
+
+    if ($surplus_refs) {
+      $this->log(
+        pht(
+          'Found %s surplus local ref(s) to delete.',
+          phutil_count($surplus_refs)));
+      foreach ($surplus_refs as $surplus_ref) {
+        $this->log(
+          pht(
+            'Deleting surplus local ref "%s" ("%s").',
+            $surplus_ref,
+            $local_refs[$surplus_ref]));
+
+        $repository->execLocalCommand(
+          'update-ref -d %R --',
+          $surplus_ref);
+
+        unset($local_refs[$surplus_ref]);
+      }
+    }
 
     if ($remote_refs === $local_refs) {
       $this->log(
@@ -378,7 +421,7 @@ final class PhabricatorRepositoryPullEngine
     // checked out. See T13280.
 
     $future = $repository->getRemoteCommandFuture(
-      'fetch --prune --update-head-ok -- %P %Ls',
+      'fetch --update-head-ok -- %P %Ls',
       $repository->getRemoteURIEnvelope(),
       $fetch_rules);
 
@@ -474,8 +517,18 @@ final class PhabricatorRepositoryPullEngine
 
   private function loadGitRemoteRefs(
     PhabricatorRepository $repository,
-    PhutilOpaqueEnvelope $remote_envelope) {
+    PhutilOpaqueEnvelope $remote_envelope,
+    $is_local) {
 
+    // See T13448. When listing local remotes, we want to list everything,
+    // not just refs we expect to fetch. This allows us to detect that we have
+    // undesirable refs (which have been deleted in the remote, but are still
+    // present locally) so we can update our state to reflect the correct
+    // remote state.
+
+    if ($is_local) {
+      $ref_rules = array();
+    } else {
       $ref_rules = $this->getGitRefRules($repository);
 
       // NOTE: "git ls-remote" does not support "--" until circa January 2016.
@@ -491,6 +544,7 @@ final class PhabricatorRepositoryPullEngine
               $ref_rule));
         }
       }
+    }
 
     list($stdout) = $repository->execxRemoteCommand(
       'ls-remote %P %Ls',