Summary:
We need to issue all commands as $repository->junk() so we can pick up
credentials. Some of this stuff predates that change landing.
(I removed the "https" vs "svn+ssh" fallback code since it's specific to
Facebook, affected a tiny number of commits, is basically an SVN bug with UTF-8
handling and HTTP support, and doesn't make sense in the general case. The user
has the tools they need to force it via "reparse.php" if it's really an issue.)
Test Plan: Created new authenticated-remote mercurial and git repositories and
pulled/discovered them with credentials.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 970
Summary: Change import script plus almost all the view stuff. Still some rough
edges but this seems to mostly work. Blame is currently unsupported but I think
everything else works properly.
Test Plan:
Imported the hg repository itself. It doesn't immediately seem completely
broken. Here are some screens:
https://secure.phabricator.com/file/view/PHID-FILE-1438b71cc7c4a2eb4569/https://secure.phabricator.com/file/view/PHID-FILE-3cec4f72f39e7de2d041/https://secure.phabricator.com/file/view/PHID-FILE-2ea4883f160e8e5098f9/https://secure.phabricator.com/file/view/PHID-FILE-35f751a36ebf65399ade/
All the parsers were able to churn through it without errors.
Ran the new "reparse.php" script in various one-commit and repository modes.
Browsed/imported some git repos for good measure.
NOTE: The hg repository is only 15,000 commits and around 1,000 files.
Performance is okay but hg doesn't provide performant, native APIs to get some
data efficiently so we have to do some dumb stuff. If some of these interfaces
are cripplingly slow or whatever, let me know and we can start bundling some
Mercurial extensions with Arcanist.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde, epriestley
Differential Revision: 960
Summary: See D943, this is the second parse stage. This will mark Differential revisions as "Committed" among other things.
Almost all the logic here is shared between VCSes so the implementation itself is straightforward.
Test Plan: Parsed all messages for the official Mercurial repository.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
CC:
Differential Revision: 944
Summary:
Splitting up D960 a bit, see that for context.
We currently have two scripts, "parse_one_commit.php" and
"reparse_all_commit_messages.php", but they're sort of silly and you can't do
certain things with them. Replace them with one script which is more flexible
and can do specific reparse steps on individual commits or entire repos.
I left the old scripts as stubs since I think there are some FB wiki docs and
stuff that mention them. I'll delete them in a month or whenever I remember or
something.
Test Plan: Ran "reparse.php" with various arguments, including vs-one-commit,
vs-repository, with --trace, and against different types of repos.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: jungejason
CC: aran, jungejason
Differential Revision: 964
Summary:
Postponed unit tests are not unit tests with problems. The results
just haven't arrived yet.
Test Plan: Tested accepting a diff with unit status 1, 3, 5 (ok, errors,
postponed)
Reviewers: jungejason, epriestley
Reviewed By: epriestley
CC: aran, epriestley, tuomaspelkonen
Differential Revision: 969
Summary:
Just breaking D960 into some smaller parts, this is a standalone method used in
Mercurial parsing.
(There's a bad version of this function in the SVN stuff but I'll get rid of it
the next time I'm in there.)
Test Plan: See D960.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 965
Summary:
When we discover a new commit and it has a known local commit or tree hash, mark
it committed.
This supports Mercurial and Git-Immutable workflows, and improves
hybrid-Git-Mutable workflows and covers some cases where poeple just make
mistakes or whatever.
Test Plan: Parsed Mercurial, Git and SVN commits.
Reviewers: Makinde
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 963
Summary: See T502. Under some VCS setups, we get full email addresses instead of
usernames or real names. Try harder to find matches, by falling back to email
address parsing if we don't get hits on the straight-up token parsing.
Test Plan:
This is difficult to test because it depends on the account state and repository
state, and hard to pull out so it's more testable without better mocking
facilities. I just dumped this into the parser to verify the behavior:
foreach (array(
'epriestley',
'Evan Priestley',
'epriestley@epriestley.com',
'derp <epriestley@epriestley.com>',
'"Evan Priestley" <derpderpderp@derpderpderp.com>',
'quackderp <derpderpderp@derpderpderp.com>',
) as $email) {
echo "{$email} = ".$this->resolveUserPHID($email)."\n";
}
die();
Running PhabricatorRepositoryGitCommitMessageParserWorker...
epriestley = PHID-USER-79f25616ea2635089a31
Evan Priestley = PHID-USER-79f25616ea2635089a31
epriestley@epriestley.com = PHID-USER-1bec59b91be6223f07fd
derp <epriestley@epriestley.com> = PHID-USER-1bec59b91be6223f07fd
"Evan Priestley" <derpderpderp@derpderpderp.com> =
PHID-USER-79f25616ea2635089a31
quackderp <derpderpderp@derpderpderp.com> =
This is expected (all variations of my identity parsed correctly, and the bogus
one failed). There are two different user PHIDs in the result set because I have
like 30 different similar accounts on my local, including one called "derp" and
another one with address "derp@derp.com", which prevented an earlier version of
this test case from working correctly.
Reviewers: zachallia, aran, Makinde, jungejason, nh, tuomaspelkonen
Reviewed By: jungejason
CC: aran, jungejason
Differential Revision: 968
Summary:
- We throw on a missing date right now, in the DateTime constructor. This can
happen in reasonable cases and this is display code, so handle it more
gracefully (see T520).
- This stuff is a little slow and we sometimes render many hundreds of dates
per page. I've been seeing it in profiles on and off. Memoize timezones to
improve performance.
- Some minor code duplication that would have become less-minor with the
constructor change, consolidate the logic.
- Add some unit tests and a little documentation.
Test Plan:
- Ran unit tests.
- Profiled 1,000 calls to phabricator_datetime(), cost dropped from ~49ms to
~19ms with addition of memoization. This is still slower than I'd like but I
don't think there's an easy way to squeeze it down further.
Reviewers: ajtrichards, jungejason, nh, tuomaspelkonen, aran
Reviewed By: ajtrichards
CC: aran, ajtrichards, epriestley
Differential Revision: 966
datasources
Summary:
The open source Phabricator has like 3,500 user accounts now and it takes a
while to pull/render them. Add an option to switch to ondemand for large
installs.
I'll follow up with a patch at some point to address a couple of name things:
- Denormalize last names into a keyed column (although this evidences some
bias toward the western world).
- Force all usernames to lowercase (sorry Girish, Makinde).
Also this patch is so clean it's crazy.
Didn't bother with other object types for now, I'm planning to dedicate a few
days to Projects at some point and I'll flesh out some auxiliary features like
this when I do that.
Test Plan: Switched to ondemand, verified data was queried dynamically. Switched
back, verified data was preloaded.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: nh
CC: aran, epriestley, nh
Differential Revision: 923
Summary:
We currently rely on "remote_hooks_enabled" in .arcconfig to determine whether
commands like "arc amend" and "arc merge" should imply "arc mark-committed".
However, this is a historical artifact that is now bad for a bunch of reasons:
- The option name is confusing, it really means 'repository is tracked'.
- The option is hard to discover and generally sucks.
- We can empirically determine the right answer since we now know if a project
is in a tracked repository.
Add a call which arcanist can make on these workflows to figure out if it is
interacting with a project in a tracked repository or not.
Also added an "isTracked()" convenience method to reduce the number of magic
strings all over the place.
Test Plan: Ran "arcanist.projectinfo" for nonexistent, untracked and tracked
projects.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, epriestley, Makinde
Differential Revision: 945
Summary:
See D943, this is the second parse stage. This will mark Differential revisions
as "Committed" among other things.
Almost all the logic here is shared between VCSes so the implementation itself
is straightforward.
Test Plan: Parsed all messages for the official Mercurial repository.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 944
Summary:
Repository import has three major steps:
- Commit discovery (serial)
- Message parsing (parallel, mostly VCS independent)
- Change parsing (parallel, highly VCS dependent)
This implements commit discovery for Mercurial, similar to git's parsing:
- List the heads of all the branches.
- If we haven't already discovered them, follow them back to their roots (or
the first commit we have discovered).
- Import all the newly discovered commits, oldest first.
This is a little complicated but it ensures we discover commits in depth order,
so the discovery process is robust against interruption/failure. If we just
inserted commits as we went, we might read the tip, insert it, and then crash.
When we ran again, we'd think we had already discovered commits older than HEAD.
This also allows later stages to rely on being able to find Phabricator commit
IDs which correspond to parent commits.
NOTE: This importer is fairly slow because "hg" has a large startup time
(compare "hg --version" to "git --version" and "svn --version"; on my machine,
hg has 60ms of overhead for any command) and we need to run many commands (see
the whole "hg id" mess). You can expect something like 10,000 per hour, which
means you may need to run overnight to discover a large repository (IIRC, the
svn/git discovery processes are both about an order of magnitude faster). We
could improve this with batching, but I want to keep it as simple as possible
for now.
Test Plan: Discovered all the commits in the main Mercurial repository,
http://selenic.com/repo/hg.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 943
Summary: See D939. Regardless of what we do there, these will break, and they're
pretty silly anyway (see the giant caveat comments in the second one).
Test Plan: Clicked a direct-jump comment link, did save/cancel for inline
comments.
Reviewers: phil, cpojer, tomo, mroch
Reviewed By: phil
CC: aran, phil
Differential Revision: 940
Summary: Explains how to use the immutable history doctrine and mercurial.
Recommends "one idea is one commit".
Test Plan: Read documentation.
Reviewers: fratrik, Makinde, aran, jungejason, tuomaspelkonen, cpiro
Reviewed By: cpiro
CC: aran, cpiro, epriestley, ide
Differential Revision: 861
Summary:
dirname('x') returns '.', not '/'; this caused some issues for repositories with
files at the root.
There are some cases in the parsers where I should probably swap this out too
but I'll wait until I'm doing some more rigorous testing since that stuff is a
bit fragile and this fixes an immediate issue.
Test Plan: Ran unit tests. Viewed a file at root level in a test repository.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: nh
CC: aran, nh
Differential Revision: 932
Summary: See T262. This creates the index on the Differential side which we need in order to execute this query efficiently on the Diffusion side.
Also renames "DiffusionGitPathIDQuery" to "DiffusionPathIDQuery", this query object has nothing to do with git.
Test Plan: Attached top-level and sub-level diffs to revisions and verified they populated the table with sensible data.
Reviewers: bmaurer, aravindn, fmoo, jungejason, nh, tuomaspelkonen, aran
CC:
Differential Revision: 931
Summary:
we use to only add X-Frame-Options for AphrontWebpageResponse.
There some security concern about it. Example of a drag-drop attack:
http://sites.google.com/site/tentacoloviola/. The fix is to add it to
all AphrontResponse.
Test Plan:
View page which disalble this option still works (like the
xhpast tree page); verify that the AphrontAjaxResponse contains the
X-Frame-Options in the header.
Reviewers: epriestley, benmathews
Reviewed By: epriestley
CC: nh, aran, jungejason, epriestley
Differential Revision: 926
Summary: Feedback from @makinde. These are easy (and necessary) to configure so
we might as well give the user a heads up.
Test Plan: Regenerated the documentation and read "Configuration Guide".
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 929
Summary: For reasons explained in the config I've omitted this from the default
action set, but it's trivial to support it. See D916.
Test Plan: Commented on a revision, was informed I could "!accept" in the email.
Used "!accept" to accept the revision.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 928
Summary: This method relies on 64-bit math being available, which isn't a safe
assumption. Use the builtin bc functions instead for arbitrarily large integers.
Test Plan: @skrul, can you apply this locally and let me know if it works?
Reviewers: skrul, hunterbridges, jungejason, nh, tuomaspelkonen, aran
Reviewed By: skrul
CC: aran, skrul, epriestley
Differential Revision: 912
Summary:
This is pretty straightforward, except:
- We need to request read/write access to the address book to get the account
ID (which we MUST have) and real name, email and account name (which we'd like
to have). This is way more access than we should need, but there's apparently no
"get_loggedin_user_basic_information" type of call in the Google API suite (or,
at least, I couldn't find one).
- We can't get the profile picture or profile URI since there's no Plus API
access and Google users don't have meaningful public pages otherwise.
- Google doesn't save the fact that you've authorized the app, so every time
you want to login you need to reaffirm that you want to give us silly amounts of
access. Phabricator sessions are pretty long-duration though so this shouldn't
be a major issue.
Test Plan:
- Registered, logged out, and logged in with Google.
- Registered, logged out, and logged in with Facebook / Github to make sure I
didn't break anything.
- Linked / unlinked Google accounts.
Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran
Reviewed By: aran
CC: aran, epriestley, Makinde
Differential Revision: 916
Summary: No actual parsing/import yet, but now you can define and pull Mercurial
repositories. I merged most of the local pull code so we can share it between
hg/git.
Test Plan:
- Created a new Mercurial repository to track Codeigniter off Bitbucket
- Edited / saved / etc.
- Launched the mercurial pull daemon, it pulled the repo. Killed and
relaunched, it updated the repo.
- Launched the git fetch deamon, it still works correctly.
Reviewers: Makinde, aran, jungejason, tuomaspelkonen
Reviewed By: Makinde
CC: aran, Makinde
Differential Revision: 793
Summary:
Changed the documentation to describe the project-subproject join table
instead of the task-project join table.
Test Plan:
none
Reviewers:
epriestley, cadamo
CC:
Differential Revision: 927
Summary:
The CSRF changes meant that we can't generate a file URI with just its PHID
anymore, and converted a mathematical function into a service call.
Unfortunately, this caused massive perf problems in some parts of the
application, critically handles, where loading N users became N single gets.
Derp derp derp. Remedy this by doing a single multiget. This substantially
improves performance of many interfaces, particularly the Maniphest task list.
I need to go through the rest of the PhabricatorFileURI callsites and get rid of
them, but I think this is the most substantive one.
Test Plan: Profiled Maniphest task list, queries went from >100 to a handful.
Explosion of multiderp. :/ Looked at some views with profile photos to verify
they still render accurately.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: aran
CC: aran
Differential Revision: 921
Summary: This will get fancier, but here's a basic interface for doing symbol
lookups. Still all pretty tentative.
Test Plan: Looked up various things, got some sensible results.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: tuomaspelkonen
CC: aran, tuomaspelkonen
Differential Revision: 900
Summary: See T315 for an extensive description of this feature. Adds the
descibed storage table.
Test Plan: Used phpsh to read/write symbol objects.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: tuomaspelkonen
CC: aran, epriestley, tuomaspelkonen
Differential Revision: 897
Summary: This got caught in the CSRF filter but is a safe write.
Test Plan: Pasted the URI for a picture of a goat into a diff, saw a goat.
Reviewers: aran, jungejason
Reviewed By: aran
CC: aran
Differential Revision: 910
Summary: See D902. As @abdul notes, a password input is probably more
appropraite here.
Test Plan: Mashed stuff into it, got bullets instead of text.
Reviewers: abdul, jungejason, nh, tuomaspelkonen, aran
Reviewed By: nh
CC: aran, nh
Differential Revision: 913
Summary: phriction.edit allows you to omit the content string, meaning "don't
edit content". If you're also creating the page, we currently break in a
terrible, horrible, no-good, very-bad way because 'content' can't be null.
Default to empty string instead so phriction.edit creates an empty page instead
of a broken mess.
Test Plan: Called phriction.edit on a new page with no content.
Reviewers: skrul, jungejason, nh, tuomaspelkonen, aran
Reviewed By: skrul
CC: aran, skrul
Differential Revision: 920
Summary:
When Diffusion encounters an image file, it displays it as an
image, but when it encounters a PDF file, it currently shows only some
gibberish. This fixes that.
Test Plan:
I tried it. Embedding a large PDF in a data URL is a little
bit slow, but it works.
Reviewers: tuomaspelkonen, epriestley, gc3, waltermundt, jungejason, nh
Reviewed By: epriestley
CC: aran, tuomaspelkonen, epriestley, jaapweel
Differential Revision: 915
Summary:
We need to query the unit status in order to determine if
there are postponed unit tests to update after running "arc unit"
Test Plan:
1) set my conduit uri to a server running the new code
2) ensured unitStatus existed when retrieving a diff
Reviewers: epriestley
Reviewed By: epriestley
CC: dpepper, aran, epriestley
Differential Revision: 918
Summary:
@tomo ran into an issue where he had some non-SSL-only cookie or whatever, so
"Logout" had no apparent effect. Make sure "Logout" really works by destroying
the session.
I originally kept the sessions around to be able to debug session stuff, but we
have a fairly good session log now and no reprorted session bugs except for all
the cookie stuff. It's also slightly more secure to actually destroy sessions,
since it means "logout" breaks any cookies that attackers somehow stole (e.g.,
by reading your requests off a public wifi network).
Test Plan: Commented out the cookie clear and logged out. I was logged out and
given a useful error message about clearing my cookies.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: aran
CC: tomo, aran, epriestley
Differential Revision: 911
- Removed irrelevant csprintf(..)
- Updated code to use $repository->getRemoteURI()
- Updated code to use getRemoteCommandFuture(..) in Diffusion code
- Updated code to use $repository->getRemoteURI()
Summary: I still need to go through all the daemon and Diffusion code and change
the bare execx() calls to $repository->execxXXX() to actually make this work,
but we're getting close.
Test Plan: Configured repositories with various HTTP / SVN setups and ran the
test_connection.php script to verify keys were located and added and
username/password information was supplied.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: nh
CC: aran, nh, jungejason
Differential Revision: 902
setup
Summary: See T481. We'll fail the pcntl test if we don't have this, in a
potentially confusing way. Test and detect missing 'php' explicitly before we
try the pcntl test, so we can give the user a better error message.
Test Plan: In setup mode, did a good run and then faked it to execute 'phpx'
instead to get a failure.
Reviewers: johnduhart, jungejason, tuomaspelkonen, aran
Reviewed By: tuomaspelkonen
CC: aran, epriestley, tuomaspelkonen
Differential Revision: 878
Summary: Added line number 1 for each image and added code to display the
comments for each image.
Test Plan: Adding an image in my local directory and create a revision for it.
Click line number 1, and the comment window prompts. Adding and save the
comment. The comment shows in the differential comment list and in the inline
comment. Submit the comment. Create more comments for the image and the
"Previous" and "Next" buttons all work well.
Reviewers: epriestley, jungejason
CC:
Differential Revision: 901
Summary: Previously, this code accidentally did not use the best URI. Instead,
use the best URI. It's the best, obviously.
Test Plan: Uploaded a binary file and then clicked the preview.
Reviewers: hunterbridges, jungejason, nh, tuomaspelkonen, aran
Reviewed By: tuomaspelkonen
CC: aran, tuomaspelkonen
Differential Revision: 905
Summary:
Move toward storing credentials in configuration so it's easier to get the
daemons working. This should eventually solve all the key juggling junk you have
to do right now.
This only gets us part of the way to actually using these credentials in the
daemons since I have to go swap everything for $repository->execBlah().
I tried to write a web "Test Connection" button but it was too much of a mess to
get git to work since git doesn't give you access to its SSH command and SSH has
a bunch of interactive prompts which you can't really do anything about without
it or a bunch of ~/.ssh/config editing. This is what Git recommends:
https://git.wiki.kernel.org/index.php/GitFaq#How_do_I_specify_what_ssh_key_git_should_use.3F
..but it's not a great match for this use case.
Test Plan:
- Only partial.
- Ran "test_connection.php" on a Git repo with and without SSH, and with and
without valid credentials. This part works properly.
- Ran "test_connection.php" on a public SVN repo, but I don't have private or
WEBDAV repos set up at the moment.
- Mercurial doesn't work yet.
- Daemons haven't been converted yet.
Reviewers: jungejason, tuomaspelkonen, aran
Reviewed By: jungejason
CC: aran, abdul, nmalcolm, epriestley, jungejason
Differential Revision: 888
Summary:
These fields use auxiliary storage now. Migrate the data and get rid of the
columns in the main table.
- This might take a little while to run, although there are <500k rows so
probably not too long.
- Maybe grab a backup of the table first, if I screwed something up this will
delete the data in these fields.
Test Plan:
- Ran migration locally.
- Browsed Differential.
- Grepped for "revertPlan" and "blameRevision".
Reviewers: jungejason, tuomaspelkonen, aran
Reviewed By: jungejason
CC: aran, jungejason, epriestley
Differential Revision: 832
Summary:
Give users better errors and UI:
- For subpath SVN repositories, default the path to the subdirectory, not to
"/". This makes the home screen useful and things generally less confusing.
- For unparsed commits, show a more descriptive error message without the
"blah blah" silliness.
- For paths outside of the subpath parse tree, short circuit into an
appropriate error message.
- For foreign SVN stub commits (see D892), show an explicit message.
Test Plan: Looked at unparsed commits, subpath repositories, foreign stub
commits, and paths outside of the subpath parse tree. Received sensible error
messages.
Reviewers: jungejason, nh, tuomaspelkonen, aran
Reviewed By: jungejason
CC: aran, jungejason
Differential Revision: 894
