Summary: These are "local" commands, but need remote credentials. If the daemon
runs as a user who does not have credentials, the initial clone will work but
subsequent updates will fail.
Test Plan:
- Nuked a local copy of a Git repo.
- Ran "phd debug fetch <phid>" as root (or any other user with no natural SSH
keys). Verified initial clone worked (since it passes credentials to the command
correctly).
- Killed daemon, re-ran, verified "fetch" failed (no credentials passed).
- Applied this patch.
- Re-ran "phd debug fetch <phid>", verified it passed credentials and
succeeded.
- Did all these steps for a Mercurial repo.
Reviewers: btrahan, jungejason
Reviewed By: btrahan
CC: aran, btrahan
Maniphest Tasks: T686
Differential Revision: 1236
Summary:
Discover commits then return; useful when initializing new repositories
in unit tests.
By which I mean "when initializing a new repository in my unit test that
I'm working on".
Test Plan: Using this in a PhabricatorTestCase.
Reviewers: epriestley, aran
Reviewed By: epriestley
CC: aran, edward, epriestley
Differential Revision: 948
Summary:
We need to issue all commands as $repository->junk() so we can pick up
credentials. Some of this stuff predates that change landing.
(I removed the "https" vs "svn+ssh" fallback code since it's specific to
Facebook, affected a tiny number of commits, is basically an SVN bug with UTF-8
handling and HTTP support, and doesn't make sense in the general case. The user
has the tools they need to force it via "reparse.php" if it's really an issue.)
Test Plan: Created new authenticated-remote mercurial and git repositories and
pulled/discovered them with credentials.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 970
Summary:
We currently rely on "remote_hooks_enabled" in .arcconfig to determine whether
commands like "arc amend" and "arc merge" should imply "arc mark-committed".
However, this is a historical artifact that is now bad for a bunch of reasons:
- The option name is confusing, it really means 'repository is tracked'.
- The option is hard to discover and generally sucks.
- We can empirically determine the right answer since we now know if a project
is in a tracked repository.
Add a call which arcanist can make on these workflows to figure out if it is
interacting with a project in a tracked repository or not.
Also added an "isTracked()" convenience method to reduce the number of magic
strings all over the place.
Test Plan: Ran "arcanist.projectinfo" for nonexistent, untracked and tracked
projects.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, epriestley, Makinde
Differential Revision: 945
Summary:
See D943, this is the second parse stage. This will mark Differential revisions
as "Committed" among other things.
Almost all the logic here is shared between VCSes so the implementation itself
is straightforward.
Test Plan: Parsed all messages for the official Mercurial repository.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 944
Summary:
Repository import has three major steps:
- Commit discovery (serial)
- Message parsing (parallel, mostly VCS independent)
- Change parsing (parallel, highly VCS dependent)
This implements commit discovery for Mercurial, similar to git's parsing:
- List the heads of all the branches.
- If we haven't already discovered them, follow them back to their roots (or
the first commit we have discovered).
- Import all the newly discovered commits, oldest first.
This is a little complicated but it ensures we discover commits in depth order,
so the discovery process is robust against interruption/failure. If we just
inserted commits as we went, we might read the tip, insert it, and then crash.
When we ran again, we'd think we had already discovered commits older than HEAD.
This also allows later stages to rely on being able to find Phabricator commit
IDs which correspond to parent commits.
NOTE: This importer is fairly slow because "hg" has a large startup time
(compare "hg --version" to "git --version" and "svn --version"; on my machine,
hg has 60ms of overhead for any command) and we need to run many commands (see
the whole "hg id" mess). You can expect something like 10,000 per hour, which
means you may need to run overnight to discover a large repository (IIRC, the
svn/git discovery processes are both about an order of magnitude faster). We
could improve this with batching, but I want to keep it as simple as possible
for now.
Test Plan: Discovered all the commits in the main Mercurial repository,
http://selenic.com/repo/hg.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 943
Summary: No actual parsing/import yet, but now you can define and pull Mercurial
repositories. I merged most of the local pull code so we can share it between
hg/git.
Test Plan:
- Created a new Mercurial repository to track Codeigniter off Bitbucket
- Edited / saved / etc.
- Launched the mercurial pull daemon, it pulled the repo. Killed and
relaunched, it updated the repo.
- Launched the git fetch deamon, it still works correctly.
Reviewers: Makinde, aran, jungejason, tuomaspelkonen
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 793
- Removed irrelevant csprintf(..)
- Updated code to use $repository->getRemoteURI()
- Updated code to use getRemoteCommandFuture(..) in Diffusion code
- Updated code to use $repository->getRemoteURI()
Summary: If a user partially discovers a repository and then deletes it, the
timeline will have events from the old repository which this daemon won't be
able to parse.
Test Plan: @ajtrichards, can you apply this locally and restart your daemons
(##phd stop##, then relaunch them) and let me know if it fixes the issue?
Reviewers: ajtrichards, jungejason, tuomaspelkonen, aran
Reviewed By: ajtrichards
CC: aran, epriestley, ajtrichards
Differential Revision: 845
Summary:
See T325. While this is a touch hacky it ends up being fairly clean, and we can
now do initial imports much more quickly and this actually cleaned up some of
the code. I also made the repository edit interface a little less foreboding.
@tuomaspelkonen, did you get anywhere with that bug you were chasing down a
couple days ago? We can hold this if it throws a wrench into stuff you're
working on.
Test Plan:
- Imported a subdirectory of a midsized SVN project (jQuery UI).
- Commit discovery for ~3500/4500 commits took just a few seconds.
- Commit discovery correctly ignored commits which didn't affect this
directory.
- Commit discovery correctly stopped at commit 13.
- Browse interface shows an incomplete listing, but that's fine, and
everything is otherwise functionally correct. We can add a note or something
later ("this is a view of commits affecting a subdirectory, some paths aren't
available"), but this behavior probably won't be too startling to users.
- Edited Git and SVN repositories to test form logic.
Reviewed By: jungejason
Reviewers: tuomaspelkonen, jungejason, aran, Girish
Commenters: tuomaspelkonen
CC: jcleveley, aran, jungejason, tuomaspelkonen
Differential Revision: 696
Summary:
there are several places we open an 'r' connection but use it
for writing. Fix them.
Test Plan:
ran parse_one_commit.php against one revision which executes
the code with problem. It used to throw exception. Now it works fine.
Reviewed By: Girish
Reviewers: tuomaspelkonen, Girish
Commenters: aran
CC: aran, Girish
Differential Revision: 213
Summary: Basic scaffolding for repository tracking, plus daemon infrastructure
(Timelines, Cursors) and some fixes (memory usage, mysql_connect() junk).
Test Plan: parsed Javelin git commit history via daemon
Reviewers:
CC:
