Summary: Ref T6160. Ref T7100. Mercurial branch heads can be closed; track this state so we can be smarter about it.
Test Plan: Closed a branch, run `repository update`, saw it close in the cursor table.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6160, T7100
Differential Revision: https://secure.phabricator.com/D12550
Summary: Ref T5896. See that task and inline for a description.
Test Plan:
- Ran `bin/repository refs HGTESTX --trace`, saw sensible commands and a reasonable result.
- Faked the heads, set chunk size to 2, got this correct output from the algorithm:
> (((((((((1 or 2) or (3 or 4)) or ((5 or 6) or (7 or 8))) or (((9 or 10) or (11 or 12)) or ((13 or 14) or (15 or 16)))) or ((((17 or 18) or (19 or 20)) or ((21 or 22) or (23 or 24))) or (((25 or 26) or (27 or 28)) or ((29 or 30) or (31 or 32))))) or (((((33 or 34) or (35 or 36)) or ((37 or 38) or (39 or 40))) or (((41 or 42) or (43 or 44)) or ((45 or 46) or (47 or 48)))) or ((((49 or 50) or (51 or 52)) or ((53 or 54) or (55 or 56))) or (((57 or 58) or (59 or 60)) or ((61 or 62) or (63 or 64)))))) or ((((((65 or 66) or (67 or 68)) or ((69 or 70) or (71 or 72))) or (((73 or 74) or (75 or 76)) or ((77 or 78) or (79 or 80)))) or ((((81 or 82) or (83 or 84)) or ((85 or 86) or (87 or 88))) or (((89 or 90) or (91 or 92)) or ((93 or 94) or (95 or 96))))) or (((((97 or 98) or (99 or 100)) or ((101 or 102) or (103 or 104))) or (((105 or 106) or (107 or 108)) or ((109 or 110) or (111 or 112)))) or ((((113 or 114) or (115 or 116)) or ((117 or 118) or (119 or 120))) or (((121 or 122) or (123 or 124)) or ((125 or 126) or (127 or 128))))))) or (((((((129 or 130) or (131 or 132)) or ((133 or 134) or (135 or 136))) or (((137 or 138) or (139 or 140)) or ((141 or 142) or (143 or 144)))) or ((((145 or 146) or (147 or 148)) or ((149 or 150) or (151 or 152))) or (((153 or 154) or (155 or 156)) or ((157 or 158) or (159 or 160))))) or (((((161 or 162) or (163 or 164)) or ((165 or 166) or (167 or 168))) or (((169 or 170) or (171 or 172)) or ((173 or 174) or (175 or 176)))) or ((((177 or 178) or (179 or 180)) or ((181 or 182) or (183 or 184))) or (((185 or 186) or (187 or 188)) or ((189 or 190) or (191 or 192)))))) or ((((((193 or 194) or (195 or 196)) or ((197 or 198) or (199 or 200))) or (((201 or 202) or (203 or 204)) or ((205 or 206) or (207 or 208)))) or ((((209 or 210) or (211 or 212)) or ((213 or 214) or (215 or 216))) or (((217 or 218) or (219 or 220)) or ((221 or 222) or (223 or 224))))) or (((((225 or 226) or (227 or 228)) or ((229 or 230) or (231 or 232))) or (((233 or 234) or (235 or 236)) or ((237 or 238) or (239 or 240)))) or ((((241 or 242) or (243 or 244)) or ((245 or 246) or (247 or 248))) or (((249 or 250) or (251 or 252)) or ((253 or 254) or (255 or 256)))))))) or ((((((((257 or 258) or (259 or 260)) or ((261 or 262) or (263 or 264))) or (((265 or 266) or (267 or 268)) or ((269 or 270) or (271 or 272)))) or ((((273 or 274) or (275 or 276)) or ((277 or 278) or (279 or 280))) or (((281 or 282) or (283 or 284)) or ((285 or 286) or (287 or 288))))) or (((((289 or 290) or (291 or 292)) or ((293 or 294) or (295 or 296))) or (((297 or 298) or (299 or 300)) or ((301 or 302) or (303 or 304)))) or ((((305 or 306) or (307 or 308)) or ((309 or 310) or (311 or 312))) or (((313 or 314) or (315 or 316)) or ((317 or 318) or (319 or 320)))))) or ((((((321 or 322) or (323 or 324)) or ((325 or 326) or (327 or 328))) or (((329 or 330) or (331 or 332)) or ((333 or 334) or (335 or 336)))) or ((((337 or 338) or (339 or 340)) or ((341 or 342) or (343 or 344))) or (((345 or 346) or (347 or 348)) or ((349 or 350) or (351 or 352))))) or (((((353 or 354) or (355 or 356)) or ((357 or 358) or (359 or 360))) or (((361 or 362) or (363 or 364)) or ((365 or 366) or (367 or 368)))) or ((((369 or 370) or (371 or 372)) or ((373 or 374) or (375 or 376))) or (((377 or 378) or (379 or 380)) or ((381 or 382) or (383 or 384))))))) or (((((((385 or 386) or (387 or 388)) or ((389 or 390) or (391 or 392))) or (((393 or 394) or (395 or 396)) or ((397 or 398) or (399 or 400)))) or ((((401 or 402) or (403 or 404)) or ((405 or 406) or (407 or 408))) or (((409 or 410) or (411 or 412)) or ((413 or 414) or (415 or 416))))) or (((((417 or 418) or (419 or 420)) or ((421 or 422) or (423 or 424))) or (((425 or 426) or (427 or 428)) or ((429 or 430) or (431 or 432)))) or ((((433 or 434) or (435 or 436)) or ((437 or 438) or (439 or 440))) or (((441 or 442) or (443 or 444)) or ((445 or 446) or (447 or 448)))))) or ((((((449 or 450) or (451 or 452)) or ((453 or 454) or (455 or 456))) or (((457 or 458) or (459 or 460)) or ((461 or 462) or (463 or 464)))) or ((((465 or 466) or (467 or 468)) or ((469 or 470) or (471 or 472))) or (((473 or 474) or (475 or 476)) or ((477 or 478) or (479 or 480))))) or (((((481 or 482) or (483 or 484)) or ((485 or 486) or (487 or 488))) or (((489 or 490) or (491 or 492)) or ((493 or 494) or (495 or 496)))) or ((((497 or 498) or (499 or 500)) or ((501 or 502) or (503 or 504))) or (((505 or 506) or (507 or 508)) or ((509 or 510) or (511 or 512)))))))))
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: chad, epriestley
Maniphest Tasks: T5896
Differential Revision: https://secure.phabricator.com/D12549
Summary:
Ref T6755. In Git and Subversion, running `git clone http://google.com/` or `svn checkout http://google.com/` does not echo the response body.
In Mercurial, it does. Censor it from the output of `hg pull` and `hg clone`. This prevents an attacker from:
- Creating a Mercurial remote repository with URI `http://10.0.0.1/secrets/`; and
- reading the secrets out of the error message after the clone fails.
Test Plan: Set a Mercurial remote URI to a non-Mercurial repository, ran `repository update`, saw censored error message.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6755
Differential Revision: https://secure.phabricator.com/D12170
Summary:
Ref T7522. This is mostly useful in the cluster, but could be useful for external installs too.
If you want to import an instance into a test/dry-run state (in the cluster, to test an import; in the general case, to do something like test new hardware or configuration), you currently risk spamming users with a lot of duplicate notifications. In particular, if Phabricator tracks remotes, both instances will continue importing commits and sending email about them. Both instances will try to publish to mirrors, too, which could be bad news, and both instances will try to update linked services.
Instead, provide a flag to let an instance run in "silent mode", which disables all outbound messaging and data.
We need to remember to support this flag on any new outbound channels, but we add about one of those per year so I think that's reasonable.
Test Plan:
- Flipped config.
- Saw it void email, feed and mirroring.
- Didn't test SMS since it's not really in use yet and not convenient to test.
- (Can you think of any publishing I missed?)
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7522
Differential Revision: https://secure.phabricator.com/D12109
Summary:
Fixes T7019. In a cluster environment, pushes currently fail because the commit hook can't identify the instance.
For web processes, the hostname identifies the instance -- but we don't have a hostname in the hook.
For CLI processes, the environment identifies the instance -- but we don't have an environment in the hook under SVN.
Promote the instance identifier into the upstream and pack/unpack it explicitly for hooks. This is probably not useful for anyone but us, but the amount of special-purpose code we're introducing is very small.
I poked at trying to do this in a more general way, but:
- We MUST know this BEFORE we run code, so the normal subclassing stuff is useless.
- I couldn't come up with any other parameter which might ever be useful to pass in.
Test Plan: Used `git push` to push code through proxied HTTP, got a clean push.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7019
Differential Revision: https://secure.phabricator.com/D11495
Summary: Fix a few minor lint issues.
Test Plan: Ran `arc lint`.
Reviewers: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11059
Summary: I don't have `hg` yet on my new laptop; we should just skip tests if the user is missing binaries. Add a convenience method to do this.
Test Plan: Got clean `arc unit --everything` with no `hg` installed.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D11051
Summary: Fixes T4946. Theoretically.
Test Plan:
iiam
also unit tests.
also
```
cd /var/repo/X
git remote remove origin # simulates origin-missing clone under 1.7.1
cd /path/to/phabricator
./bin/repository pull X
```
and observed no errors
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T4946, T5938
Differential Revision: https://secure.phabricator.com/D10855
Summary:
Fixes T5839. If a repository has been force pushed and garbage collected, we might have a ref cursor in the database which still points at the old commit (which no longer exists).
We'll then run a command like `git log <new hash> --not <old hash>` to figure out which commits are newly pushed, and this will bomb out because `<old hash>` is invalid.
Instead, validate all the `<old hash>` values before we try to make use of them.
Test Plan:
- Forced a repository into a bad state by mucking with the datbase, generating a reproducible failure similar to the one in T5839.
- Applied patch.
- `bin/repository update <callsign> --trace` filtered the bad commit and put the repository into the right state.
- Saw new commits recognized correctly.
- Ran `bin/repository update <callsign>` for a Mercurial and SVN repo as a sanity check.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5839
Differential Revision: https://secure.phabricator.com/D10226
Summary: Fix for T4990, using export TERM directly in pre receive hook, tested for git
Test Plan:
pushing into repository over ssh will now not cause remote warning
No entry for terminal type "unknown";
using dumb terminal settings.
Tested with git
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Projects: #diffusion, #repositories
Maniphest Tasks: T4990
Differential Revision: https://secure.phabricator.com/D9744
Summary: Ran `arc lint --apply-patches --everything` over rP, mainly to change double quotes to single quotes where appropriate. These changes also validate that the `ArcanistXHPASTLinter::LINT_DOUBLE_QUOTE` rule is working as expected.
Test Plan: Eyeballed it.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin, hach-que
Differential Revision: https://secure.phabricator.com/D9431
Summary: Point everything at the new canonical URI.
Test Plan: `grep`
Reviewers: chad
Reviewed By: chad
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D9328
Summary:
Ref T2683. This is a refinement and simplification of D5257. In particular:
- D5257 only cached the commit chain, not path changes. This meant that we had to go issue an awkward query (which was slow on Facebook's install) periodically while reading the cache. This was reasonable locally but killed performance at FB scale. Instead, we can include path information in the cache. It is very rare that this is large except in Subversion, and we do not need to use this cache in Subversion. In other VCSes, the scale of this data is quite small (a handful of bytes per commit on average).
- D5257 required a large, slow offline computation step. This relies on D9044 to populate parent data so we can build the cache online at will, and let it expire with normal LRU/LFU/whatever semantics. We need this parent data for other reasons anyway.
- D5257 separated graph chunks per-repository. This change assumes we'll be able to pull stuff from APC most of the time and that the cost of switching chunks is not very large, so we can just build one chunk cache across all repositories. This allows the cache to be simpler.
- D5257 needed an offline cache, and used a unique cache structure. Since this one can be built online it can mostly use normal cache code.
- This also supports online appends to the cache.
- Finally, this has a timeout to guarantee a ceiling on the worst case: the worst case is something like a query for a file that has never existed, in a repository which receives exactly 1 commit every time other repositories receive 4095 commits, on a cold cache. If we hit cases like this we can bail after warming the cache up a bit and fall back to asking the VCS for an answer.
This cache isn't perfect, but I believe it will give us substantial gains in the average case. It can often satisfy "average-looking" queries in 4-8ms, and pathological-ish queries in 20ms on my machine; `hg` usually can't even start up in less than 100ms. The major thing that's attractive about this approach is that it does not require anything external or complicated, and will "just work", even producing reasonble improvements for users without APC.
In followups, I'll modify queries to use this cache and see if it holds up in more realistic workloads.
Test Plan:
- Used `bin/repository cache` to examine the behavior of this cache.
- Did some profiling/testing from the web UI using `debug.php`.
- This //appears// to provide a reasonable fast way to issue this query very quickly in the average case, without the various issues that plagued D5257.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley, jhurwitz
Maniphest Tasks: T2683
Differential Revision: https://secure.phabricator.com/D9045
Summary:
Ref T4455. This adds a `repository_parents` table which stores `<childCommitID, parentCommitID>` relationships.
For new commits, it is populated when commits are discovered.
For older commits, there's a `bin/repository parents` script to rebuild the data.
Right now, there's no UI suggestion that you should run the script. I haven't come up with a super clean way to do this, and this table will only improve performance for now, so it's not important that we get everyone to run the script right away. I'm just leaving it for the moment, and we can figure out how to tell admins to run it later.
The ultimate goal is to solve T2683, but solving T4455 gets us some stuff anyway (for example, we can serve `diffusion.commitparentsquery` faster out of this cache).
Test Plan:
- Used `bin/repository discover` to discover new commits in Git, SVN and Mercurial repositories.
- Used `bin/repository parents` to rebuild Git and Mercurial repositories (SVN repos just exit with a message).
- Verified that the table appears to be sensible.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: jhurwitz, epriestley
Maniphest Tasks: T4455
Differential Revision: https://secure.phabricator.com/D9044
Summary:
Ref T4605. Before discovering branches, try to prefill the cache in bulk. For repositories with large numbers of branches, this allows us to issue dramatically fewer queries.
(Before D8780, this cache was usually held across discovery events, so being able to fill it cheaply was not as relevant.)
Test Plan: Ran discovery on Git, Mercurial and SVN repositories. Observed fewer queries for Git/Mercurial.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T4605
Differential Revision: https://secure.phabricator.com/D8781
Summary:
There are quite a few tests in Arcanist, libphutil and Phabricator that do something similar to `$this->assertEqual(false, ...)` or `$this->assertEqual(true, ...)`.
This is unnecessarily verbose and it would be cleaner if we had `assertFalse` and `assertTrue` methods.
Test Plan: I contemplated adding a unit test for the `getCallerInfo` method but wasn't sure if it was required / where it should live.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D8460
Summary:
Fixes T4414. Currently, when we discover a new repository, we do something like this:
foreach (branch) {
foreach (commit on this branch) {
do_something();
}
}
In cases where there are a lot of branches which mostly just branch `master`, this leads to us doing roughly `O(branches * commits)` work.
We have a `commitCache` to prevent this, but it has two problems:
- It only fills out of the DB, and we do this whole thing before writing to the DB, which is the entire point.
- It has a fixed size (2048) and on initial discovery we're usually dealing with way more commits than that.
Instead, also stop doing work if we hit a commit which is known already.
Test Plan:
- Added `print` on the number of discovered refs and number of unique refs.
- Ran `bin/repository discover --repair X` on a repo with several branches.
- Before the patch, got 397 refs and 135 unique refs.
- After the patch, got 135 refs and 135 unique refs.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4414
Differential Revision: https://secure.phabricator.com/D8374
Summary:
See IRC. I'm having trouble figuring out what's going on with b4taylor's report, but fix two possible issues:
# The commit query is missing a `repositoryID`, which could cause issues if you import two copies of the same repository.
# I think we may try to close commits on untracked branches right now, as long as they aren't excluded by other autoclose rules.
Test Plan: Ran `bin/repository refs` on a few repos.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, brennantaylor
Differential Revision: https://secure.phabricator.com/D8373
Summary:
@mbishopim3 reported an issue in IRC:
> mbishopim3: epriestley: "Error updating working copy: Commit "" has not been discovered yet! Run discovery before updating refs." any ideas?
I can't reproduce it and it went away for him, but one theory is that we're getting here and git/hg are spitting out nothing, which we incorrectly parse as `array("")` when we intend `array()`.
Test Plan:
Pushed some new commits, ran `bin/repositoy refs X`, got expected results.
I can't actually reproduce the bug, but this might fix it and appears to make the code more correct.
Reviewers: btrahan
Reviewed By: btrahan
CC: mbishopim3, aran
Differential Revision: https://secure.phabricator.com/D8326
Summary:
Error:
Fatal error: Call to a member function getRefType() on a non-object in /opt/phabricator/phabricator/src/applications/repository/engine/PhabricatorRepositoryRefEngine.php on line 197
Test Plan: No more error in daemon.log afterwards
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D8278
Summary: See <https://github.com/facebook/phabricator/issues/501>. I think the issue here is that we created a foreign stub for commit `X-1`, probably because commit `X` was created by running `svn cp y x`.
Test Plan: I'll write a separate test for this before I land it. Huge pain to test.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D8133
Summary:
Ref T4338. Mercurial exits with exit code 1 and "no changes found" in stdout
when there's no changes. I've split up the `pushRepositoryToMirror` to make
the code a tad more readable.
It isn't perfect, but it works for me.
Test Plan:
pushed some changes to my hosted repo. Saw them appearing in the
mirrored repo
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Maniphest Tasks: T4338
Differential Revision: https://secure.phabricator.com/D8107
Summary: Ref T4338. Currently, if you have several mirrors and the first one fails, we won't try the other mirrors (since we'll throw and that will take us out of the mirroring process). Instead, try each mirror even if one fails, and then throw an AggregateException with all the failures.
Test Plan:
- Ran `bin/repository mirror` normally.
- Faked an exception, ran again, got the AggregateException I expected.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4338
Differential Revision: https://secure.phabricator.com/D8067
Summary:
Ref T4338. Currently, there's no diagnostic command to execute mirroring (so I can't give users an easy command to run), and it's roughly the last piece of real logic left in the PullLocal daemon.
Separate mirroring out, and provide `bin/repository mirror`.
Test Plan:
- Ran `bin/repository mirror` to mirror a repository.
- Ran PullLocalDaemon and verified it also continued mirroring normally.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4338
Differential Revision: https://secure.phabricator.com/D8066
Summary: See IRC. Currently, if a fetch fails, we may repair the remote URI incorrectly, replacing it with one without any credentials. Instead, retain credentials.
Test Plan: Faked it locally, got le-boom to verify on IRC.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D8055
Summary: Fixes T3238. Ref T4327. Although the instructions are fairly clear on this, it's easy to miss them. Make sure the root the user enters matches the real root.
Test Plan: Added unit tests. Used `bin/repository discover` to hit the check explicitly.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T3238, T4327
Differential Revision: https://secure.phabricator.com/D8020
Summary: Ref T4327. This adds a subpath/partial repository where we import only a subdirectory, and adds tests for it.
Test Plan: Ran unit tests.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D8019
Summary:
Ref T4327. Adds additional tests:
- Property changes on the root directory (which has special handling).
- Copying a file from a previous revision (this is `svn cp svn://server.com/root/some_file@123 some_file`).
- "Multicopying" a file: this is where a file is copied to several places and moved in the same commit.
Test Plan: Ran unit tests.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D8018
Summary:
Ref T4327. Adds additional SVN tests to cover:
- replacing a file with a file;
- replacing a file with a directory;
- removing a directory with files in it;
- adding a directory with files;
- copying a directory and adding and deleting files inside it.
Test Plan: Ran unit tests. One of these has a slightly irregular parse, see inline.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D8017
Summary:
Ref T4327. Adds the same basic battery of tests to the SVN parser as the other parsers have.
cowsay {{{ THIS IS AN AMAZING DIFF }}}
Test Plan:
Ran unit tests against SVN change parsing.
bwahaha
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D8016
Summary: Ref T4327. This adds a set of test cases for the Mercurial parser. These tests are nearly identical to the git cases, except that the Mercurial parser can't figure out symlinks right now.
Test Plan: Ran unit tests.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D8014
Summary: Ref T4327. Adds some basic tests to the Git parser for a set of common operations (add, change, move, copy, directory, symlink, propchange).
Test Plan: Ran unit tests.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D8012
Summary:
Currently, we can sit in a sleep() for up to 15 seconds (or longer, in some cases), even if there's a parse request.
Try polling the DB every second to see if we should wake up instead. This might or might not produce significant improvements, but seems OK to try and see.
Also a small fix for logging branch names with `%` in them, etc.
Test Plan: Ran `phd debug pulllocal` and pushed commits, saw them parse within a second.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, dctrwatson
Differential Revision: https://secure.phabricator.com/D7998
Summary:
Ref T4327. This moves the last pieces of discovery responsibility out of the PullLocal daemon and into the DiscoveryEngine.
(This makes it easier to discover repositories in unit tests in the future, since we don't need to build a PullLocal daemon and can just build a DiscoveryEngine.)
Test Plan: Ran `phd debug pulllocal`. Ran `repostory discover`.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D7997
Summary:
Ref T4327. Consolidates and simplifies infrastructure:
- Moves Git discovery into DiscoveryEngine.
- Collapses a bunch of the Git and Mercurial code related to stream discovery.
- Removes all cach code from PullLocal daemon (it's no longer called).
- Adds basic unit tests for Git and Mercurial discovery.
Various cleanup:
- Makes GitStream and MercurialStream extend a common base.
- Improves performance of MercurialStream in some cases, by requiring fewer commits be output and parsed.
- Makes mirroring exceptions easier to debug.
- Fixes discovery of Mercurial repositories with multiple branch heads.
- Adds some missing `pht()`.
Test Plan:
I tested this fairly throughly because I think this phase is complete:
- Made new repositories in multiple VCSes and did full imports.
- Particularly, I reimported Arcanist to make sure that TODO was resolved. I think it was related to the toposort stuff.
- Pushed commits to multiple VCSes.
- Pushed commits to a non-close branch, then pushed a merge commit. Observed commits import initially as non-close, then get flagged for close.
- Started full daemons and resolved various minor issues that showed up in the daemon log until everything ran cleanly.
- Basically spent about 30 minutes banging on this in every way I could think of to try to break it. I found and fixed some minor stuff, but it seems solid.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D7987
Summary:
Ref T4327. This provides a more general RefCursor-based way to identify closeable commits, and moves away from the messy `seenOnBranches` stuff. Basically:
- When a closeable ref (like the branch "master") is updated, query the VCS for all commits which are ancestors of the new ref, but not ancestors of the existing closeable heads we previously knew about. This is basically all the commits which have been merged or moved onto the closeable ref.
- Take these commits and set the "closeable" flag on the ones which don't have it yet, then queue new tasks to reprocess them.
I haven't removed the old stuff yet, but will do that shortly.
Test Plan:
- Ran `bin/repository discover` and `bin/repository refs` on a bunch of different VCSes and VCS states. The effects seemed correct (e.g., new commits were marked as closeable).
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D7984
Summary:
Ref T4327. I want to make change parsing testable; one thing which is blocking this is that the Git discovery process is still part of `PullLocal` daemon instead of being part of `DiscoveryEngine`. The unit test stuff which I want to use for change parsing relies on `DiscoveryEngine` to discover repositories during unit tests.
The major reason git discovery isn't part of `DiscoveryEngine` is that it relies on the messy "autoclose" logic, which we never implemented for Mercurial. Generally, I don't like how autoclose was implemented: it's complicated and gross and too hard to figure out and extend.
Instead, I want to do something more similar to what we do for pushes, which is cleaner overall. Basically this means remembering the old branch heads from the last time we parsed a repository, and figuring out what's new by comparing the old and new branch heads. This should give us several advantages:
- It should be simpler to understand than the autoclose stuff, which is pretty mind-numbing, at least for me.
- It will let us satisfy branch and tag queries cheaply (from the database) instead of having to go to the repository. We could also satisfy some ref-resolve queries from the database.
- It should be easier to extend to Mercurial.
This implements the basics -- pretty much a table to store the cursors, which we update only for Git for now.
Test Plan:
- Ran migration.
- Ran `bin/repository discover X --trace --verbose` on various repositories with branches and tags, before and after modifying pushes.
- Pushed commits to a git repo.
- Looked at database tables.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D7982
Summary: Ref T4327. Moves us one small step forward toward testable change parsers by separating out this unrelated logic from the PullLocal daemon. We will also probably want to run this logic so we can do remote path lookups to limit the role of Arcanist Projects in the future, which is why I made the URI type (here, only "git") a parameter rather than calling this a `GitURINormalizer` or something.
Test Plan:
- Ran unit tests.
- Ran `repository discover` on a correctly-configured remote repository.
- Ran `repository discover` on an incorrectly-configured remote, got an error message.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4327
Differential Revision: https://secure.phabricator.com/D7981
Summary:
Fixes T4189. Ref T4151. Allows repositories to have additional custom hooks for operations which can't be expressed with Herald (one such operation is lint).
This adds only local hook directories, since they're easier to use with existing hooks than global directories. I might add global directories eventually.
This doesn't support Mercurial since we have no demand for it and it's more complicated (we lose compatibility and power by just dropping a `hooks.d/` somewhere).
Test Plan:
- Pulled hosted SVN and Git repos to verify the hook directories generate correctly.
- Added a variety of hooks to the hook directories (echo + pass, fail).
- Pushed commits and verified the hooks fired (output expected info, or failed).
- Verified push log reflected the correct error code ("3", external) and detail ("nope.sh") when rejecting.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4151, T4189
Differential Revision: https://secure.phabricator.com/D7884
Summary:
~~Set PATH for repository's hook, so the environment.append-paths can used~~
repository's hook may can't find php path if user's profile like bash_profile is not loaded.
Test Plan: check the hook generated is contain the right path
Reviewers: epriestley, #blessed_reviewers
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7743
Summary:
Ref T4257. Currently, the pull logic looks like this:
if (new) {
create();
} else {
if (hosted) {
install_hooks();
} else {
update();
}
}
This means that the first time you run `repository pull`, hooks aren't installed, which makes debugging trickier. Instead, reorganize the logic:
if (new) {
create();
} else {
if (!hosted) {
update();
}
}
if (hosted) {
install_hooks();
}
Test Plan: Ran `bin/repository pull` on a new `hg` repo and got hooks installed immediately.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4257
Differential Revision: https://secure.phabricator.com/D7818
Summary: Ref T4195. This doesn't actually work like I thought it did: it only fires locally, when you run `hg tag`. Mercurial tags are also weird and basically don't make any sense and everyone should use bookmarks instead. We could implement some flavor of this eventually, but I'd like to see users request it first. They can implement their own with content-based hooks once those work, anyway.
Test Plan: This code didn't do anything.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4195
Differential Revision: https://secure.phabricator.com/D7765
Summary:
See <https://github.com/facebook/phabricator/issues/467>. @dctrwatson also ran into an issue where we were trying to `setPass()` a GitURI.
- For Git and Mercurial, properly generate credential URIs where relevant.
- Don't try to `setPass()` on Git-style URIs.
This isn't perfect but should clean things up a bit.
Test Plan: Added unit tests. Lots of `grep`.
Reviewers: btrahan
Reviewed By: btrahan
CC: dctrwatson, aran
Differential Revision: https://secure.phabricator.com/D7759
