Summary:
Repository import has three major steps:
- Commit discovery (serial)
- Message parsing (parallel, mostly VCS independent)
- Change parsing (parallel, highly VCS dependent)
This implements commit discovery for Mercurial, similar to git's parsing:
- List the heads of all the branches.
- If we haven't already discovered them, follow them back to their roots (or
the first commit we have discovered).
- Import all the newly discovered commits, oldest first.
This is a little complicated but it ensures we discover commits in depth order,
so the discovery process is robust against interruption/failure. If we just
inserted commits as we went, we might read the tip, insert it, and then crash.
When we ran again, we'd think we had already discovered commits older than HEAD.
This also allows later stages to rely on being able to find Phabricator commit
IDs which correspond to parent commits.
NOTE: This importer is fairly slow because "hg" has a large startup time
(compare "hg --version" to "git --version" and "svn --version"; on my machine,
hg has 60ms of overhead for any command) and we need to run many commands (see
the whole "hg id" mess). You can expect something like 10,000 per hour, which
means you may need to run overnight to discover a large repository (IIRC, the
svn/git discovery processes are both about an order of magnitude faster). We
could improve this with batching, but I want to keep it as simple as possible
for now.
Test Plan: Discovered all the commits in the main Mercurial repository,
http://selenic.com/repo/hg.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 943
Summary: See T262. This creates the index on the Differential side which we need in order to execute this query efficiently on the Diffusion side.
Also renames "DiffusionGitPathIDQuery" to "DiffusionPathIDQuery", this query object has nothing to do with git.
Test Plan: Attached top-level and sub-level diffs to revisions and verified they populated the table with sensible data.
Reviewers: bmaurer, aravindn, fmoo, jungejason, nh, tuomaspelkonen, aran
CC:
Differential Revision: 931
Summary: No actual parsing/import yet, but now you can define and pull Mercurial
repositories. I merged most of the local pull code so we can share it between
hg/git.
Test Plan:
- Created a new Mercurial repository to track Codeigniter off Bitbucket
- Edited / saved / etc.
- Launched the mercurial pull daemon, it pulled the repo. Killed and
relaunched, it updated the repo.
- Launched the git fetch deamon, it still works correctly.
Reviewers: Makinde, aran, jungejason, tuomaspelkonen
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 793
Summary: See T315 for an extensive description of this feature. Adds the
descibed storage table.
Test Plan: Used phpsh to read/write symbol objects.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: tuomaspelkonen
CC: aran, epriestley, tuomaspelkonen
Differential Revision: 897
Summary: See D902. As @abdul notes, a password input is probably more
appropraite here.
Test Plan: Mashed stuff into it, got bullets instead of text.
Reviewers: abdul, jungejason, nh, tuomaspelkonen, aran
Reviewed By: nh
CC: aran, nh
Differential Revision: 913
- Removed irrelevant csprintf(..)
- Updated code to use $repository->getRemoteURI()
- Updated code to use getRemoteCommandFuture(..) in Diffusion code
- Updated code to use $repository->getRemoteURI()
Summary: I still need to go through all the daemon and Diffusion code and change
the bare execx() calls to $repository->execxXXX() to actually make this work,
but we're getting close.
Test Plan: Configured repositories with various HTTP / SVN setups and ran the
test_connection.php script to verify keys were located and added and
username/password information was supplied.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: nh
CC: aran, nh, jungejason
Differential Revision: 902
Summary:
Move toward storing credentials in configuration so it's easier to get the
daemons working. This should eventually solve all the key juggling junk you have
to do right now.
This only gets us part of the way to actually using these credentials in the
daemons since I have to go swap everything for $repository->execBlah().
I tried to write a web "Test Connection" button but it was too much of a mess to
get git to work since git doesn't give you access to its SSH command and SSH has
a bunch of interactive prompts which you can't really do anything about without
it or a bunch of ~/.ssh/config editing. This is what Git recommends:
https://git.wiki.kernel.org/index.php/GitFaq#How_do_I_specify_what_ssh_key_git_should_use.3F
..but it's not a great match for this use case.
Test Plan:
- Only partial.
- Ran "test_connection.php" on a Git repo with and without SSH, and with and
without valid credentials. This part works properly.
- Ran "test_connection.php" on a public SVN repo, but I don't have private or
WEBDAV repos set up at the moment.
- Mercurial doesn't work yet.
- Daemons haven't been converted yet.
Reviewers: jungejason, tuomaspelkonen, aran
Reviewed By: jungejason
CC: aran, abdul, nmalcolm, epriestley, jungejason
Differential Revision: 888
references
Summary:
See T325. We tentatively support doing partial subdirectory parses in
Phabricator for Subversion, so you can elect to import only "trunk/local/" or
similar. We do this by importing only some of the commits (those commits which
affected that directory).
In Subversion, you can also "svn cp
svn+ssh://example.com/svnroot/trunk/foreign/example.c@13 local.c". This means
that commits which reference "trunk/local/" may themselves reference foreign
commits.
Currently, we break in this case and can't find the commit reference. Instead,
generate a foreign commit stub so we can at least point at some reasonable
object.
Test Plan: Successfully imported trunk/a/ of the test repo in T325 without
errors. Verified commit 3 in that repo is imported as a foreign stub.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: jungejason
CC: aran, jungejason, epriestley
Differential Revision: 892
repositories
Summary:
This query isn't scoped correctly to the repository ID, so we may identify
commits from other repositories.
This causes a somewhat subtle issue since we only use it to manage file
copies/moves, so you end up with a file "copied from" the same revision in
another repository. I think the UI probably even renders correctly.
Once I finish T325 and better understand what's going on here, I'll see how much
work is involved in writing an SQL patch to fix this.
Test Plan: Parsed the test repo from T325 with the expected error.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: jungejason
CC: aran, jungejason
Differential Revision: 891
Summary: If a user partially discovers a repository and then deletes it, the
timeline will have events from the old repository which this daemon won't be
able to parse.
Test Plan: @ajtrichards, can you apply this locally and restart your daemons
(##phd stop##, then relaunch them) and let me know if it fixes the issue?
Reviewers: ajtrichards, jungejason, tuomaspelkonen, aran
Reviewed By: ajtrichards
CC: aran, epriestley, ajtrichards
Differential Revision: 845
Summary:
See T372. Always render commit messages on one display line, so the table
doesn't jump around as they AJAX in on browse views.
The goal here is to have the cell choose a size naturally and for its content to
render with "overflow: hidden" if the natural size isn't large enough to contain
the content. "white-space: pre" or "white-space: nowrap" would prevent wrapping
but potentially make the table exceed the display width when a better behavior
is to hide some of the commit message.
Also use utf8-aware shortening, now that we have a function for it.
Casting a wide net in case anyone has a better way to do the CSS here. It's kind
of nasty that we have to use so many DOM nodes.
Test Plan:
- Resized window while viewing browse and history views in Safari, Chrome and
Firefox. Table exhibited described behavior.
- Verified summaries render sensibly and are properly truncated to 100
characters.
Reviewed By: aran
Reviewers: aran, jungejason, tuomaspelkonen, tomo, mroch, cpojer
CC: aran, epriestley
Differential Revision: 750
interface
Summary:
See T58. When I originally wrote this interface you couldn't make a form that
looked like a form but wasn't really a form, so I made it a real form. That
meant you could hit "return" in the text input and submit the form, which didn't
do anything but is vaguely silly.
After D422 there's a layout-only form element, so switch to that.
Test Plan: Loaded the tab, hit return in the textarea, it didn't do a form
submit.
Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, jungejason, epriestley
Differential Revision: 719
Summary:
See T325. While this is a touch hacky it ends up being fairly clean, and we can
now do initial imports much more quickly and this actually cleaned up some of
the code. I also made the repository edit interface a little less foreboding.
@tuomaspelkonen, did you get anywhere with that bug you were chasing down a
couple days ago? We can hold this if it throws a wrench into stuff you're
working on.
Test Plan:
- Imported a subdirectory of a midsized SVN project (jQuery UI).
- Commit discovery for ~3500/4500 commits took just a few seconds.
- Commit discovery correctly ignored commits which didn't affect this
directory.
- Commit discovery correctly stopped at commit 13.
- Browse interface shows an incomplete listing, but that's fine, and
everything is otherwise functionally correct. We can add a note or something
later ("this is a view of commits affecting a subdirectory, some paths aren't
available"), but this behavior probably won't be too startling to users.
- Edited Git and SVN repositories to test form logic.
Reviewed By: jungejason
Reviewers: tuomaspelkonen, jungejason, aran, Girish
Commenters: tuomaspelkonen
CC: jcleveley, aran, jungejason, tuomaspelkonen
Differential Revision: 696
Summary:
Make it prettier, paginate, add user pictures, show document types, clean some
stuff up a little. Plenty of room for improvement but this should make it a lot
more useful.
Test Plan:
Here's what the new one looks like:
https://secure.phabricator.com/file/view/PHID-FILE-edce2b83c2e3a121c2b7/
Reviewed By: jungejason
Reviewers: tomo, jungejason, aran, tuomaspelkonen, mroch
Commenters: tomo
CC: aran, tomo, jungejason, epriestley
Differential Revision: 545
Summary:
create the indexer for commit
Test Plan:
run the reindex_one_commit.php against one existing commit
and it is returned in search result on the webpage; run parse_one_commit
against another commit; modified reindex_everything.php to let it only
parse one commit after loading all commits.
Reviewed By: epriestley
Reviewers: epriestley, aran
CC: aran, jungejason, epriestley, debow
Differential Revision: 490
Summary:
Everything breaks if this isn't true, and it's easy to get subtly wrong right
now. There are other more magical ways we could do this (automatically add a "/"
in this form or at runtime) but I think making it explicit is the easiest and
most robust approach. See T67.
Test Plan:
Tried to save a URI without a trailing slash.
Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, jungejason
Differential Revision: 305
Summary:
Someone has "defaced" secure.phabricator.com with a helpful suggestion that I
actually do this; fair enough. :P
Test Plan:
Logged in as myself, unable to edit directory information. Logged out, logged in
as admin, was able to edit directory information.
I need to fix some more CSS stuff since some of these tabs render out hideous in
the admin background, but I can followup with that.
Reviewed By: tuomaspelkonen
Reviewers: aran, jungejason, tuomaspelkonen
Commenters: aran
CC: aran, tuomaspelkonen, epriestley
Differential Revision: 296
Summary:
I never actually wrote this controller.
Test Plan:
Deleted a repository via web UI.
Reviewed By: tuomaspelkonen
Reviewers: tuomaspelkonen, jungejason, aran
Commenters: aran
CC: aran, tuomaspelkonen, epriestley
Differential Revision: 231
Summary:
there are several places we open an 'r' connection but use it
for writing. Fix them.
Test Plan:
ran parse_one_commit.php against one revision which executes
the code with problem. It used to throw exception. Now it works fine.
Reviewed By: Girish
Reviewers: tuomaspelkonen, Girish
Commenters: aran
CC: aran, Girish
Differential Revision: 213
Summary:
I pretty shortsightedly made sending a side effect of save() in the case that a
server is configured for immediate sending. Move this out, make it explicit, and
get rid of all the tangles surrounding it.
The web tool now ignores the server setting and only repsects the checkbox,
which makes far more sense.
Test Plan:
Sent mails from Maniphest, Differential, and the web console. Also ran all the
unit tests. Verified headers from Maniphest.
Reviewed By: rm
Reviewers: aran, rm
CC: tuomaspelkonen, rm, jungejason, aran
Differential Revision: 200
Summary:
See T129, some older git doesn't have %B and we can reasonably fake it with %s
and %b.
Test Plan:
Reparsed all of the Phabricator repository with this worker, commit messages
look fine.
Reviewed By: rm
Reviewers: rm, aran, jungejason, tuomaspelkonen
CC: aran, rm
Differential Revision: 209
Summary:
the Repository tool is missing the standard application link
at the top. The reason is that PhabricatorRepositoryListController and
several other classes are inheriting from a wrong class.
Test Plan:
play with the application to make sure it is not breaking
other stuff.
Reviewed By: epriestley
Reviewers: epriestley
CC: jungejason, epriestley
Differential Revision: 70
Summary: Basic scaffolding for repository tracking, plus daemon infrastructure
(Timelines, Cursors) and some fixes (memory usage, mysql_connect() junk).
Test Plan: parsed Javelin git commit history via daemon
Reviewers:
CC:
