1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-21 13:00:56 +01:00
Commit graph

688 commits

Author SHA1 Message Date
epriestley
30ae22bfcf Fix many encoding and architecture problems in Diffusion request and URI handling
Summary:
Diffusion request/uri handling is currently a big, hastily ported mess. In particular, it has:

  - Tons and tons of duplicated code.
  - Bugs with handling unusual branch and file names.
  - An excessively large (and yet insufficiently expressive) API on DiffusionRequest, including a nonsensical concrete base class.
  - Other tools were doing hacky things like passing ":" branch names.

This diff attempts to fix these issues.

  - Make the base class abstract (it was concrete ONLY for "/diffusion/").
  - Move all URI generation to DiffusionRequest. Make the core static. Add unit tests.
  - Delete the 300 copies of URI generation code throughout Diffusion.
  - Move all URI parsing to DiffusionRequest. Make the core static. Add unit tests.
  - Add an appropriate static initializer for other callers.
  - Convert all code calling `newFromAphrontRequestDictionary` outside of Diffusion to the new `newFromDictionary` API.
  - Refactor static initializers to be sensibly-sized.
  - Refactor derived DiffusionRequest classes to remove duplicated code.
  - Properly encode branch names (fixes branches with "/", see <https://github.com/facebook/phabricator/issues/100>).
  - Properly encode path names (fixes issues in D1742).
  - Properly escape delimiter characters ";" and "$" in path names so files like "$100" are not interpreted as "line 100".
  - Fix a couple warnings.
  - Fix a couple lint issues.
  - Fix a bug where we would not parse filenames with spaces in them correctly in the Git browse query.
  - Fix a bug where Git change queries would fail unnecessarily.
  - Provide or improve some documentation.

This thing is pretty gigantic but also kind of hard to split up. If it's unreasonably difficult to review, let me know and I can take a stab at it though.

This supplants D1742.

Test Plan:
  - Used home, repository, branch, browse, change, history, diff (ajax), lastmodified (ajax) views of Diffusion.
  - Used Owners typeaheads and search.
  - Used diffusion.getrecentcommitsbypath method.
  - Pushed a change to an absurdly-named file on an absurdly-named branch, everything worked properly.

{F9185}

Reviewers: nh, vrana, btrahan

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1921
2012-03-19 19:52:14 -07:00
epriestley
ec736f9c50 Handle "hg pull" return code change between 2.1 and 2.1.1 more gracefully
Summary:
See <https://github.com/facebook/phabricator/issues/102>. Between Feb 1 and Mar 1, the hg released changed the exit code behavior of "hg pull". This broke us mildly (and a bunch of other applications more severely, which is why it was reverted).

Detect the common case of this (english) and don't fail.

Test Plan: @killermonk, can you try applying this? I'll try to do an upgrade to 2.1 and see if I can also do a proper test.

Reviewers: Makinde, btrahan, killermonk

Reviewed By: btrahan

CC: killermonk, aran, epriestley

Differential Revision: https://secure.phabricator.com/D1948
2012-03-19 19:19:48 -07:00
Hafsteinn Baldvinsson
fffc1e51d0 Inset view controller for inset elements of forms.
Summary:
T937 suggests 'inset' could have its own view controller.

It has the following methods:
 - setTitle         for title
 - setRightbutton   if you have to place something (preferably a button)
                    on the right side of the form
 - setDescription   if you want to describe what it does
 - setContent       for the main content
 - addDivAttributes REALLY not sure about this one but it had to be included
                    because of a single controller (see owners/controller/edit/PhabricatorOwnersEditController.php:238)
 - appendChild      works as usual if your form is complex but you still want to remove
                    ->appendChild('<div class..') ->appendChild('</div>');

It might be an overkill so maybe some could be dropped:
 - addDivAttributes() and just rewrite how PhabricatorOwnersEditController.php works
 - setContent() and use appendChild for the main content?

Test Plan:
 - Looked at the controllers in phabricator
 - Changed the controller
 - Opened the page in another tab
 - If something didnd't look the same I fixed it.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1926
2012-03-15 17:10:38 -07:00
epriestley
d0af617818 Add "final" to (almost) everything else
Summary: Last of the big final patches. Left a few debatable classes (12 out of about 400) that I'll deal with individually eventually.

Test Plan: Ran testEverythingImplemented.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T795

Differential Revision: https://secure.phabricator.com/D1881
2012-03-13 16:21:04 -07:00
Nick Harper
9b48384415 [svn1.7] Fix matching xml tag when parsing svn commits
Summary:
svn 1.7 changed their xml format slightly, they now have a
 ##<?xml version="1.0" encoding="UTF-8"?>## tag instead of
 ##<?xml version="1.0"?>##. This relaxes matching this tag.

Test Plan: ./scripts/repository/reparse.php rE521979 --change

Reviewers: epriestley, jungejason

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1866
2012-03-12 16:32:30 -07:00
epriestley
def19bb8de Add additional protections against local repository misconfigurations
Summary: We've hit a couple of these in the wild, raise better error messages when the local repo is toast / broken / nonsense.

Test Plan: Broke my local repo in all of the different ways we test for, verified I got an error message in each case.

Reviewers: btrahan, abirchall

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T964, T924

Differential Revision: https://secure.phabricator.com/D1855
2012-03-12 10:34:37 -07:00
vrana
d5bf30bb48 Prepare database for UTF-8
Summary: D1830#8

Test Plan:
`scripts/sql/upgrade_schema.php`
Try adding duplicate SSH Public Key - failed.
Try adding new SSH Public Key - succeeded.

Reviewers: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1852
2012-03-09 18:56:22 -08:00
epriestley
b2890eeb0e Add "final" to all Phabricator "Controller" classes
Summary:
These are all unambiguously unextensible. Issues I hit:

  - Maniphest Change/Diff controllers, just consolidated them.
  - Some search controllers incorrectly extend from "Search" but should extend from "SearchBase". This has no runtime effects.
  - D1836 introduced a closure, which we don't handle correctly (somewhat on purpose; we target PHP 5.2). See T962.

Test Plan: Ran "testEverythingImplemented" unit test to identify classes extending from `final` classes. Resolved issues.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T795

Differential Revision: https://secure.phabricator.com/D1843
2012-03-09 15:46:25 -08:00
Edward Speyer
226d0321af Revert "PhabricatorRepository: temporarily change the localPath"
Summary:
Reverts an unwanted commit (579941b866)
that got pushed by mistake.
2012-03-09 10:54:24 -08:00
Edward Speyer
579941b866 PhabricatorRepository: temporarily change the localPath
Summary:
I'm on a host where I have the PhabricatorRepositoryPullLocalDaemons
tracking a remote repo.  In my case, these end up as local git repos in
/var/repo/$name.

I'm working on another daemon that is going to automatically make
changes and commit them back upstream.  I figured it would be best to do
this in a new local repo.  I'll put these in /var/repo-clones/$name.

It's nice to use the exec*() functions in PhabricatorRepository, so the
approach I thought of was to load the PhabricatorRepository object from
the database, then change its localPath to point at the
/var/repo-clones/$name directory instead.

I didn't really want to change the local-path detail with setDetail(),
as that risks committing the change upstream.  It's nice to use the
repo's execLocalCommand() methods though, hence wanting to change the
local path.

Test Plan: None yet.

Reviewers: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1821
2012-03-08 15:15:13 -08:00
epriestley
89dac1cf19 When updating git repositories, use --prune to prune old branches
Summary: We'll keep deleted branches around right now because Git's behavior is not to remove them without --prune.

Test Plan: Ran "git fetch --all --prune" to make sure it at least ostensibly works.

Reviewers: btrahan, 20after4

Reviewed By: 20after4

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1833
2012-03-08 13:24:25 -08:00
epriestley
1c80a4eb58 Add a "properties" table to the Repository view in Diffusion
Summary: Notably, expose the clone URI / remote URI, after stripping credentials.

Test Plan: Looked at a repository.

Reviewers: btrahan, nh

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T91

Differential Revision: https://secure.phabricator.com/D1811
2012-03-08 12:46:19 -08:00
Edward Speyer
17d801a50e GitFetch daemon: more verbose
Summary: A smidgen more messaging about what's going on.

Test Plan:
Ran it, saw this:

  2012-03-07 5:37:30 PM [STDO] >>> [0] <exec> $ /data/phabricator/bin/list_db_services --tier_name 'cdb.phabricator'
  <<< [0] <exec> 47,101 us
  ...
  >>> [9] <exec> $ mkdir -p '/var/repo'
  <<< [9] <exec> 41,374 us
  Creating new directory /var/repo/fbcode for repo FBCode
  >>> [10] <exec> $ git clone --origin origin 'ssh://fbcode.git.vip.facebook.com/data/gitrepos/fbcode.git' '/var/repo/fbcode'

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1819
2012-03-07 19:24:06 -08:00
epriestley
8dfe8e84f0 Improve Diffusion parsing of submoudule changes
Summary: We currently parse these as directory changes and discard them. Instead, parse them as a new "SUBMODULE" type of change.

Test Plan:
  - Reparsed a commit which changes submodules and verified it parses correctly.
  - Reparsed a commit which adds submodules and verified it parses correctly.

Reviewers: btrahan, kdeggelman

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1815
2012-03-07 14:24:43 -08:00
epriestley
76fd9a2d28 Reduce laziness for "Mark Committed"
Summary:
  - Enforce proper workflow rules.
  - Fix a derp-bug with patches.

Test Plan:
  - Tried to mark a revision I didn't own.
  - Tried to mark a revision already marked committed.
  - Tried to mark a revision otherwise not accepted.
  - Verified daemon can override workflow rules and mark from arbitrary states.

Reviewers: btrahan, Makinde

Reviewed By: Makinde

CC: aran, epriestley

Maniphest Tasks: T948

Differential Revision: https://secure.phabricator.com/D1809
2012-03-07 10:20:17 -08:00
epriestley
a95c9873aa Add an "Auditors" field to commit messages which pushes audit requests when present
Summary:
Adds an optional "Auditors" field (like "Reviewers") to commit messages which gives installs a zero-config method for making audit requests.

This field does not appear on templates unless set, and is mostly ignored (but validated and preserved) by Differential.

It is then parsed by the daemons if present, and audit requests are pushed to valid users.

Test Plan: Made an "Auditors" commit and verified it was retained with "arc amend --show". Pushed it and verified the audit was triggered.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T904, T880

Differential Revision: https://secure.phabricator.com/D1793
2012-03-06 15:10:35 -08:00
epriestley
d94129b739 Migrate "PhabricatorOwnersPackageCommitRelationship" to "PhabricatorRepositoryAuditRequest"
Summary:
  - Move table to Repository, since we have no Owners joins in the application anymore but would like to do a Repository join.
  - Rename "packagePHID" to "auditorPHID", since this column may contain package, project, or user PHIDs.

Test Plan:
  - Browsed Owners, Audit, and Differential interfaces to the Audit tool.
  - Made comments and state changes.
  - Ran "reparse.php --herald --owners" on several commits.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran, epriestley, nh, vrana

Maniphest Tasks: T904

Differential Revision: https://secure.phabricator.com/D1787
2012-03-05 13:17:29 -08:00
epriestley
5590515007 Make Diffusion Herald emails thread with Audit emails
Summary:
  - Users may elect to receive an initial notification about a commit; allow it to be replied to in order to interact with the object.
  - Share thread headers between emails.
  - Add the "REPLY HANDLER ACTIONS" section to both emails.

Test Plan:
  - Used "reparse.php --herald" to trigger herald emails, verified reply-to and email body.
  - Made audit comments, verified body.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T904

Differential Revision: https://secure.phabricator.com/D1762
2012-03-05 09:54:45 -08:00
epriestley
d7bb686a47 Minor, actually update commit author field when it is parsed by the worker. 2012-02-28 21:25:15 -08:00
epriestley
8a0a00f118 Make PhabricatorRepositoryCommmit schema changes for audit
Summary:
  - Add a proper mailKey field to make these things mailable. Backfill all
existing objects.
  - Denormalize authorPHID to the commit object so we can query by it
efficiently in a future diff. We currently use the search engine to drive
"commits by author" but that's not so good for audit, which needs more
constraints.
  - Add an overall audit status field so we can efficiently query "commits that
needs your attention".
  - Add enough code to convince myself that these fields are basically
reasonable and work correctly.

Test Plan:
  - Ran schema upgrades. Checked database state afterward.
  - Ran "reparse.php --owners --herald" to verify worker changes.
  - Looked at a commit, altered aggregate status via audits / reparse.php,
verified it responded correctly.

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran, epriestley, nh

Maniphest Tasks: T904

Differential Revision: https://secure.phabricator.com/D1706
2012-02-28 21:06:34 -08:00
epriestley
37a1db6fe1 Use HGPLAIN for local hg commands in Phabricator
Summary: See D1707 -- just in case the Phabricator server is configured
suspiciously.

Test Plan: Cursory inspection.

Reviewers: Makinde, btrahan

Reviewed By: Makinde

CC: aran, epriestley

Maniphest Tasks: T922

Differential Revision: https://secure.phabricator.com/D1708
2012-02-27 19:21:03 -08:00
epriestley
d7a7bca85c Enable email for audits
Summary:
When users submit an audit, send email to relevant parties informing them.

Allow email to be replied to. Just basic support so far; no "!raise" stuff and
no threading with the Herald commit notification.

Test Plan: Made comments, got email. Replied to email, got comments.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T904

Differential Revision: https://secure.phabricator.com/D1698
2012-02-27 12:57:57 -08:00
epriestley
25fade5008 Add audits to search
Summary: Add audit information to the commit search index.

Test Plan: Updated a commit, searched for terms in its comments, got hits.

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran, epriestley

Maniphest Tasks: T904

Differential Revision: https://secure.phabricator.com/D1696
2012-02-27 09:51:00 -08:00
epriestley
1094527072 Allow Herald to trigger audits for users or projects
Summary:
Allows you to write a commit rule that triggers an audit by a user (personal
rules) or a project (global rules).

Mostly this is trying to make auditing more lightweight and accessible in
environments where setting up Owners packages doesn't make sense.

For instance, Disqus wants a rule like "trigger an audit for everything that
didn't have a Differential revision". While not necessarily scalable, this is a
perfectly reasonable rule for a small company, but a lot of work to implement
with Owners (and you'll get a lot of collateral damage if you don't make every
committer a project owner).

Instead, they can create a project called 'Unreviewed Commits' and write a rule
like:

	- When: Differential revision does not exist
 	- Action: Trigger an Audit for project: "Unreviewed Commits"

Then whoever cares can join that project and they'll see those audits in their
queue, and when they approve/raise on commits their actions will affect the
project audit.

Similarly, if I want to look at all commits that match some other rule (say,
XSS) but only want to do it like once a month, I can just set up an audit rule
and go through the queue when I feel like it.

NOTE: This abuses the 'packagePHID' field to also store user and project PHIDs.
Through the magic of handles, this (apparently) works fine for now; I'll do a
big schema patch soon but have several other edits I want to make at the same
time.

Also:

	- Adds an "active" fiew for /audit/, eventually this will be like the
Differential "active" view (stuff that is relevant to you right now).
	- On commits, highlight triggered audits you are responsible for.

Test Plan: Added personal and global audit triggers to Herald, reparsed some
commits with --herald, got audits. Browsed all audit interfaces to make sure
nothing exploded. Viewed a commit where I was responsible for only some audits.
Performed audits and made sure the triggers I am supposed to be responsible for
updated properly.

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran, epriestley

Maniphest Tasks: T904

Differential Revision: https://secure.phabricator.com/D1690
2012-02-27 09:36:30 -08:00
epriestley
a15130b47c Add a maintenance script for reconciling repositories to disk state
Summary:
@rguerin ran into an issue in his install where Phabricator appears to have
discovered commits which no longer exist, and thus is failing to proceed with
its repository import.

It's not clear how we got into this state. Previously, it was possible by, e.g.,
parsing a different repository's working copy and then switching them back, but
there are now safeguards against that.

I'm taking a three-pronged approach to try to sort this out:

  - Provide a script to get out of this state (this script) and reconcile
Phabricator's view of a repository with an authoritative copy of it. This
basically "un-discovers" any discovered commits which don't actually exist (any
queued tasks to parse them will fail permanently when they fail to load the
commit object).
  - Add more logging to the discovery daemon so we can figure out where commits
came from.
  - Improve Diffusion's UI when stuff is partially discovered (T776).

(This script should also clean up some nonsense on secure.phabricator.com from a
botched Diviner import.)

Test Plan: Ran "reconcile.php" with bogus commits and bogus differential/commit
links, had them expunged. Will work with @rguerin to see if this resolves
things.

Reviewers: btrahan, rguerin

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1552
2012-02-02 16:03:50 -08:00
Jason Ge
b4053190ec Fix two issues in owners worker
Summary:
Fix two issues in PhabricatorRepositoryCommitOwnersWorker:

- if a commit was reviewed by some owner of the package, it should not be
  marked as needing audit
- do not run herald worker when it is not needed (for example, when the
  worker is executed from reparse.php)

Test Plan:
reparse a commit which is reviewed by the owner of a package
and verify that it is not marked as needing audit, and herald is not
executed.

Reviewers: nh, epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1496
2012-01-26 11:02:42 -08:00
epriestley
5b463e634c Write fewer "applied" rows and clean up excess historical rows
Summary:
  - Only write the <ruleID, phid> row if the rule is a one-time rule.
  - Delete all the rows for rules which aren't one-time.

NOTE: This is probably like several million rows for Facebook and could take a
while.

Test Plan:
Added some one-time and every-time rules, ran them against objects, verified
only relevant rows were inserted.
Ran upgrade script against a database with one-time and every-time "ruleapplied"
rows, got the irrelevant rows removed.

Reviewers: nh, btrahan, jungejason

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1484
2012-01-25 11:53:39 -08:00
epriestley
3142fe4419 Remove massive "rule applied" query
Summary:
Herald rules may be marked as "one-time". We track this by writing a row with
<ruleID, phid> when we apply a rule.

However, the current test for rule application involves loading every <ruleID,
*> pair. We also always write this row even for rules which are not one-time, so
if there are 100 rules, we'll load 1,000,000 rows after processing 10,000
objects.

Instead, load only the <phid, *> pairs, which are guaranteed to be bounded to at
most the number of rules.

I'll follow up with a diff that causes us to write rows only for one-time rules,
and deletes all historic rows which are not associated with one-time rules.

Test Plan:
Grepped for callsites to loadAllByContentTypeWithFullData(). Ran
rules in test console.

Reviewers: nh, btrahan, jungejason

Reviewed By: nh

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1483
2012-01-24 19:29:54 -08:00
epriestley
b43eb5aa7c Add transaction-oriented editing to projects
Summary:
  - Make some editing operations transaction-oriented, like Maniphest. (This
seems to be a good model, particularly for extensibility.) I'll move the rest of
the editing operations to transactions in future diffs.
  - Make transaction-oriented operations publish feed stories.

Test Plan:
  - Created a new project.
  - Edited an existing project.
  - Created a new project via quick create flow from Maniphest.
  - Verified feed stories publish correctly.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T681

Differential Revision: https://secure.phabricator.com/D1477
2012-01-24 09:44:35 -08:00
epriestley
27f52efd37 Minor, fix spelling issues detected by linter. 2012-01-20 07:39:55 -08:00
jungejason
4faab06c3c Enable herald rule for commits that need auditing
Summary:
enable herald commit rules to have access to auditing info.

Note that the new herald condition I added contains info for the
packages. I thought about using a simpler herald condition like
"Requires audit is true or false" and let it work together with the
existing "Affected package contains any of the package". It doesn't work
because we need the info about the package to decide if the commit
requires audit, but the herald conditions work separately.

Test Plan:
- A commit requiring auditing was detected by a herald rule that checks
  the auditing status
- A commit not requiring auditing was not detected by a herald rule
  which checks auditing status, but was detected by a rule which doesn't
  check the auditing status

Reviewers: epriestley, nh

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1399
2012-01-17 09:13:07 -08:00
epriestley
b10fa8023f Support Git implicit file:// URIs
Summary: @s reported an issue with implicit file:// URIs in Git, see P270.
Recognize and handle URIs in this format. For URIs we don't understand, raise an
exception.

Test Plan:
  - Added failing tests.
  - Fixed code.
  - Tests pass.

Reviewers: btrahan, jungejason, s

Reviewed By: s

CC: aran, epriestley, s

Differential Revision: https://secure.phabricator.com/D1362
2012-01-11 09:00:18 -08:00
Bob Trahan
2a29a51080 Deploy new ArcanistManyWordsAboutDifferentialConstants class from D1328 into
Phabricator

Summary: ...this breaks without D1328.   Used good ole "codemod" to do this
work, with lots of manual edits around 80 chars.

Test Plan: clicked around phabricator tool suite, particular differential, a
bunch

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1351
2012-01-10 11:49:20 -08:00
epriestley
1903bb80bb Add a link from Differential to Diffusion
Summary:
Provide an easy way to jump to Diffusion from Differential if we have
the data we need to connect them.

Test Plan: Tested menu in linked and unlinked diffs. Used menu item.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran, epriestley, btrahan

Maniphest Tasks: T309

Differential Revision: https://secure.phabricator.com/D1326
2012-01-05 18:03:08 -08:00
epriestley
9c5a6601d6 Remove some Mercurial and Google caveats
Summary: These seem to work relatively reasonably and don't have any known
deal-breaking failures.

Test Plan: shrug~

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran, btrahan

Differential Revision: https://secure.phabricator.com/D1324
2012-01-05 14:14:06 -08:00
Bob Trahan
e478c0074c Add support for querying by commit hashes to DifferentialRevisionQuery class and
corresponding ConduitAPI

Summary: reasonable title...  also made this new functionality used by the
repository worker for parsing diffs

Test Plan:
- looked at the conduit console and queried for various types of hashes,
including hashes with no match.  got correct results.
- identified a reasonable diff from a local git repo.  set the revision status
to 2 (ACCEPTED) in the database.  augmented the worker parser code to var_dump
and die after finding revision id.   ran scripts/repository/reparse.php
--message rX and verified my var_dumps.  removed var_dumps and die and ran
reparse.php again with same paramters.  verified revision looked good in
diffusion and there were no errors.
- repeated the above reparse.php jonx for a mercurial repo.  note svn isn't in
this hash game so that test was particularly exciting no-op'dness i did not
bother with

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, btrahan, epriestley

Differential Revision: https://secure.phabricator.com/D1315
2012-01-05 13:51:51 -08:00
epriestley
99231ebba4 Allow Git repositories to track only some branches
Summary:
Some installs use Git as the backbone of a CI framework or use a Git remote to
share patches. The tracker scripts currently recognize associated revisions as
"Committed" when they appear in any branch, even if that branch is
"alincoln-personal-development_test_hack" or whatever.

To address the broadest need here, allow Git repositories to be configured to
track only certain branches instead of all branches.

This doesn't allow you to import a branch into Diffusion but ignore it in
Differential. Supporting that is somewhat technically complicated because the
parser currently goes like this:

  - Look at HEAD of all branches.
  - For any commits we haven't seen before, follow them back to something we
have seen (or the root).
  - "Discover" everything new.

Since this doesn't track <branch, commit> pairs, we currently don't have enough
information to tell when a commit appears in a branch for the first time, so we
don't have anywhere we can put a test for whether that branch is tracked and do
the Differential hook only if it is.

However, I think this cruder patch satisfies most of the need and is simple and
obvious in its implementation.

See also D1263.

Test Plan:
  - Updated a Git repository with various filters: "", "master, remote", "derp",
"  ,,, master   ,,,,,"
  - Edited SVN and Mercurial repositories to verify they didn't get caught in
the crossfire.
  - Ran daemon in debug mode on libphutil with filter "derp", got exception
about no tracked branches. Ran with filter "master", got tracking. Ran with no
filter, got tracking.
  - Looked at Diffusion with "derp" and "master", saw no branches and "master"
respectively.
  - Added unit tests to cover filtering logic.

Reviewers: btrahan, jungejason, nh, fratrik

Reviewed By: fratrik

CC: aran, fratrik, epriestley

Maniphest Tasks: T270

Differential Revision: https://secure.phabricator.com/D1290
2012-01-04 10:20:34 -08:00
epriestley
ec1df21bef Add getStrList() to AphrontRequest
Summary:
  - We have a few places where we do some kind of ad-hoc comma list tokenizing,
and I'm adding another one in D1290. Add a helper to the request object.
  - Add some unit tests.

Test Plan:
  - Ran unit tests.
  - Used PHID manager, Maniphest custom view, and Repository project editor.

Reviewers: btrahan, fratrik, jungejason

Reviewed By: btrahan

CC: aran, btrahan, epriestley

Differential Revision: https://secure.phabricator.com/D1302
2012-01-04 10:18:46 -08:00
epriestley
efb0fa739f Make tracked git repositories use an implicit 'origin' remote
Summary:
See T624. I originally wrote this to require an explicit remote, but this
creates an ugly "origin:" in all the URIs and makes T270 more difficult.

Treat all branch names as implying 'origin/'.

Test Plan:
  - Pulled and imported a fresh copy of libphutil without issues.
  - Browsed various git repositories.
  - Browsed Javelin's various branches.
  - Ran upgrade script, got a bunch of clean 'origin/master' -> 'master'
conversions.
  - Tried to specify an explicit remote in a default branch name.
  - Unit tests.

Reviewers: nh, jungejason, btrahan

Reviewed By: btrahan

CC: aran, btrahan

Maniphest Tasks: T624

Differential Revision: https://secure.phabricator.com/D1269
2011-12-29 08:35:32 -08:00
epriestley
abd8efc358 Further relax same origin checks
Summary: Allow paths to match even if they differ by trailing slashes and
".git".

Test Plan: Ran unit tests.

Reviewers: jungejason, btrahan

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T710

Differential Revision: https://secure.phabricator.com/D1286
2011-12-26 12:01:59 -08:00
epriestley
bdbe9df65e Remove support for GitHub post-receive notifications
Summary:
  - These never actually did anything.
  - I don't even really remember why I built them, maybe the Open Source team
was pushing for more GitHub integration or something? I really have no idea.
  - Anyway, repository tailers do everything these could do (and much more).

Test Plan:
  - Ran tailers off GitHub for many months without needing post-receive hooks.
  - Grepped for relevant strings, couldn't find any references.
  - Used "Repository" edit interface for a Git repository.

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T706

Differential Revision: https://secure.phabricator.com/D1273
2011-12-24 09:00:08 -08:00
epriestley
e786c44b6f Relax the same-origin check in the Git commit discovery daemon
Summary:
Git accepts either "git@x:/path" or "ssh://git@x.com/path" URIs to mean the
exact same thing, which is causing some false positives and confusion,
particularly because we sometimes mutate URIs.

Since this is just a sanity check, we don't really care about the username,
domain or credentials -- matching the paths is good enough. We're just trying to
make it hard to shoot yourself in the foot by copy-pasting the same local path
into two repositories and forgetting to change one, like I did. :P

Relax the check to only verify the paths are the same.

Test Plan:
  - Ran unit tests, which should fully cover things.
  - Ran commit discovery daemon in debug mode on incorrectly and correctly
configured repositories.

Reviewers: ajtrichards, jungejason, btrahan

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T710

Differential Revision: https://secure.phabricator.com/D1279
2011-12-24 08:54:44 -08:00
epriestley
3b65864ee1 Fix two minor bugs with recent patches:
- Use the computed remote URI (which may have an explicit 'ssh://' under Git in some cases).
  - Use '$id' correctly rather than casting the URI to an int in the message parser.
2011-12-22 06:57:04 -08:00
epriestley
b258095124 Expose symbol information over Conduit
Summary: I want to add a command like "where is ArcanistUnitTestEngine" to
phabot. I also want to add a symbol typeahead to Diffusion and generally finish
up that feature since it's useful but only half-implemented. Consolidate the
query logic and expose the data over Conduit.

Test Plan: Used /symbol/ and Conduit to lookup symbols.

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T315

Differential Revision: https://secure.phabricator.com/D1260
2011-12-22 06:44:55 -08:00
epriestley
9d8b5481ae Emit full URIs to identify Differential revisions
Summary:
  - Previously, used IDs like "33" to match a commit to a Differential revision.
This has a namespacing problem because we now have an arbitrarily large number
of Phabricator installs in the world, and they may want to track commits from
other installs.
  - In Differential, parse raw IDs or full URIs. Emit only full URIs.
  - In Repositories, parse only full URIs.
  - This might cause a few commits to not be picked up in rare circumstances.
Users can fix them with "arc mark-committed". This should be exceedingly rare
because of hash matching.
  - There are some caveats for reparsing older repositories, see comments
inline. I don't think there's much broad impact here.

Test Plan:
  - Created a new revision, got a full URI.
  - Updated revision, worked correctly.
  - Ran unit tests.
  - Monkeyed with "Differential Revision" field.
  - Reviewers: btrahan, jungejason

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran, epriestley, jungejason

Maniphest Tasks: T54, T692

Differential Revision: 1250
2011-12-20 19:58:22 -08:00
epriestley
92f9741779 Add a safety check to commit discovery daemon
Summary:
Although I couldn't repro the issue in T692, I did manage to point the "Diviner"
repository at the "Phabricator" working copy and screw some stuff up on
secure.phabricator.com.

Before discovering commits in a repository, ensure the 'origin' remote points at
the configured URI. This prevents issues where the working copy gets configured
to point at an existing (but incorrect) checkout.

Test Plan:
  - Ran gitcommitdiscovery daemon normally under "phd debug", saw it execute the
"remote show -n" command and then start working.
  - Intentionally botched the config, got an exception:

  (Exception) Working copy '/INSECURE/repos/phabricator' has origin URL
  'ssh://git@github.com/facebook/phabricator.git', but the configured URL
  'git://github.com/facebook/diviner.git' is expected. Refusing to proceed.

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T692

Differential Revision: 1253
2011-12-20 19:52:08 -08:00
jungejason
c80d1480d5 Add Basic Auditing Functionalities
Summary:
add basic auditing functionalities. For the related commits for a
package, we detect the following conditions which might be suspicious to the
owners of the package:

* no revision specified
* revision not found
* author not match
* reviewedby not match
* owners not involved
* commit author not recognized

The owners of the package can change the status of the audit entries by
accepting it or specify concern.

The owner can turn on/off the auditing for a package.

Test Plan:
*  verified that non-owner cannot see the details of the audit and cannot modify
it
*  verified that all the audit reasons can be detected
*  tested dropdown filtering and package search
*  verified really normal change not detected
*  verified accept/concern a commit
*  tested enable/disable a package for auditing
*  verified one audit applies to all <commit, packages> to the packages the
auditor owns
*  verified that re-parsing a commit won't have effect if there exists a
 relationship for <commit, package> already

Reviewers: epriestley, nh

Reviewed By: epriestley

CC: aran, benmathews, btrahan, mpodobnik, prithvi, TomL, epriestley

Differential Revision: 1242
2011-12-20 13:36:53 -08:00
epriestley
b5c9b9d059 Use remote credentials for 'git fetch' and 'hg pull' commands
Summary: These are "local" commands, but need remote credentials. If the daemon
runs as a user who does not have credentials, the initial clone will work but
subsequent updates will fail.

Test Plan:
  - Nuked a local copy of a Git repo.
  - Ran "phd debug fetch <phid>" as root (or any other user with no natural SSH
keys). Verified initial clone worked (since it passes credentials to the command
correctly).
  - Killed daemon, re-ran, verified "fetch" failed (no credentials passed).
  - Applied this patch.
  - Re-ran "phd debug fetch <phid>", verified it passed credentials and
succeeded.
  - Did all these steps for a Mercurial repo.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran, btrahan

Maniphest Tasks: T686

Differential Revision: 1236
2011-12-20 08:28:38 -08:00
jungejason
c13b7da290 Add Related Commits for Owners
Summary:
For each commit, find the affected packages, and provide a way to
search by package.

Test Plan:
create commits that touch and don't touch two packages, and verify
that they display correctly in all the UI pages.

Reviewers: epriestley, blair, nh, tuomaspelkonen

Reviewed By: epriestley

CC: benmathews, aran, epriestley, btrahan, jungejason, mpodobnik, prithvi

Maniphest Tasks: T83

Differential Revision: 1208
2011-12-14 22:48:57 -08:00
Marek Sapota
4ef18d35ac Fix error introduced with D1195.
Summary:
Not my best commit=/

Reviewers: epriestley

Test plan:
After the change commit parser started working again.

Differential Revision: 1206
2011-12-13 20:37:18 -08:00
epriestley
51b8168253 Fix fatal in commit message parser
Summary:
See D1195, which fataled this daemon.

https://secure.phabricator.com/daemon/log/2966/

Test Plan: Applied this patch to secure.phabricator.com, restarted daemon, it
picked up D1203.

Reviewers: btrahan, jungejason, mareksapota

Reviewed By: btrahan

CC: aran, btrahan, mareksapota

Differential Revision: 1204
2011-12-13 18:56:26 -08:00
Marek Sapota
9292cfd6a3 Recognise better who committed a revision.
Test Plan: Commit as not the author and see what shows up.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley, mareksapota, zeeg

Differential Revision: 1195
2011-12-13 14:00:13 -08:00
Bob Trahan
588b959c03 phabricator_format_timestamp => phabricator_datetime
Summary: make the change, kill the function.   be sure to get a good $user or
$viewer variable

Test Plan:
for each controller or view, look at it in the ui.   change timezone, refresh ui
and note change.   i did not test the OAuthSettingsPanelController; not sure how
to get to that badboy and i got a bit lazy

Maniphest Tasks: T222

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, btrahan, epriestley

Maniphest Tasks: T222

Differential Revision: 1166
2011-12-04 08:51:34 -08:00
John Stockdale
501c90bb30 Use Git's encoding flag instead of MBString
Summary:
0d5b0f21ad added string conversion but MBString always needs an argument for endcoding.

It looks like we can get away with doing this in git instead, with the --encoding='UTF-8' flag. Then we should be safe to remove the test for output type, and stay UTF-8 safe.

Test Plan:
Run updaters with change. Verify commits are updated.

Reviewers: epriestley

CC:

Differential Revision: 1108
2011-11-12 01:01:41 +00:00
David Reuss
0d5b0f21ad Convert author/message encoding if not UTF-8
Test Plan:
used the reparse.php script for reparsing commit messages and saw the
correct author name (and mapped correctly as a phabricator user) in diffusion.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: 1059
2011-10-28 08:06:03 -07:00
David Reuss
4e900c096f Convert "falsey" binary hunks if we have a repository encoding
Summary:
This adds an encoding detail to the repository, so we can attempt to
convert hunks previously detected as binary.

We also add the encoding information to the arcanist projectinfo
API so we can pull the information if we have it when uploading changes
via arc.

Test Plan:
Changed encoding through the edit UI, and saw "This is binary file", and
changed it back and saw the correct output from the diff.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: 1009
2011-10-28 08:04:57 -07:00
Edward Speyer
6737ae4828 Run a commit discovery daemon only once
Summary:
Discover commits then return; useful when initializing new repositories
in unit tests.

By which I mean "when initializing a new repository in my unit test that
I'm working on".

Test Plan: Using this in a PhabricatorTestCase.

Reviewers: epriestley, aran

Reviewed By: epriestley

CC: aran, edward, epriestley

Differential Revision: 948
2011-10-25 14:25:40 -07:00
epriestley
4156cf6bd9 Add an optional configuration option to set 'Precedence: bulk' headers on
transactional mail

Summary: See T571. SES refuses to deliver mail with this header and there are
various reports of other issues on the internet so I'm defaulting it to off.

Test Plan: Set config to true, tried to send mail, SES rejected it because of
"Precedence: bulk" header.

Reviewers: bmaurer, ola, jungejason, nh, aran

Reviewed By: aran

CC: aran, epriestley, bmaurer

Differential Revision: 1032
2011-10-23 14:25:13 -07:00
Nicholas Harper
4f365e1527 Clarify instructions for repository remote uri
Summary:
Clarified the instructions when editing a repository for the remote uri to
mention that a local path for the remote uri must be specified as
file:///local/path/to/repo instead of /local/path/to/repo. (The latter used to
work, but stopped functioning for new repositories as of D888.)

Test Plan:
loaded /repository/edit/NN/tracking (where NN is a repo number), and saw the
updated instruction message.

Reviewers: jungejason, epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: 1005
2011-10-12 12:25:49 -07:00
epriestley
254f606e89 Tie all the pieces for symbol cross-references together
Summary:
This makes symbol cross-references work in Differential. You need to do a little
legwork but I'll document that once the change has baked for a little while.

Basically:

  - Projects are annotated with indexed languages, and "shared library" projects
(for example, symbols in Phabricator should be searched for in Arcanist and
libphutil).
  - When we render a changeset, we check if its language is an indexed one. If
it is, we invoke the decorator Javascript.
  - The Javascript takes you to a lookup page, which either gives you a list of
matching symbols (if several match) or redirects you instantly to the
definition.

Test Plan: Clicked class and function symbols in a diff, got jumped into
sensible sorts of places in Diffusion.

Reviewers: jungejason, nh, tuomaspelkonen, aran

Reviewed By: jungejason

CC: aran, jungejason

Differential Revision: 980
2011-10-09 17:58:17 -07:00
epriestley
a8519c6837 Unbreak slop in commit change parser. 2011-10-02 12:37:25 -07:00
epriestley
8e8d91a1ff Allow Diffusion to display the initial commit in Git repositories
Summary: See T507. Since you can't do "xxxxxxxx^" where "xxxxxxxx" is the first
commit in a repository, fall back to diffing against the empty tree if we fail
to diff against the parent commit.

Test Plan: Looked at the first commit in libphutil on my local.

Reviewers: edward, jungejason, nh, tuomaspelkonen, aran

Reviewed By: nh

CC: aran, edward, epriestley, nh

Differential Revision: 953
2011-09-30 11:56:19 -07:00
epriestley
07f4772d0b Make all parsers use credentials
Summary:
We need to issue all commands as $repository->junk() so we can pick up
credentials. Some of this stuff predates that change landing.

(I removed the "https" vs "svn+ssh" fallback code since it's specific to
Facebook, affected a tiny number of commits, is basically an SVN bug with UTF-8
handling and HTTP support, and doesn't make sense in the general case. The user
has the tools they need to force it via "reparse.php" if it's really an issue.)

Test Plan: Created new authenticated-remote mercurial and git repositories and
pulled/discovered them with credentials.

Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran

Reviewed By: Makinde

CC: aran, Makinde

Differential Revision: 970
2011-09-28 11:01:47 -07:00
epriestley
b1e1b1f9bd Basic support for Mercurial in Diffusion
Summary: Change import script plus almost all the view stuff. Still some rough
edges but this seems to mostly work. Blame is currently unsupported but I think
everything else works properly.

Test Plan:
Imported the hg repository itself. It doesn't immediately seem completely
broken. Here are some screens:

https://secure.phabricator.com/file/view/PHID-FILE-1438b71cc7c4a2eb4569/
https://secure.phabricator.com/file/view/PHID-FILE-3cec4f72f39e7de2d041/
https://secure.phabricator.com/file/view/PHID-FILE-2ea4883f160e8e5098f9/
https://secure.phabricator.com/file/view/PHID-FILE-35f751a36ebf65399ade/

All the parsers were able to churn through it without errors.

Ran the new "reparse.php" script in various one-commit and repository modes.

Browsed/imported some git repos for good measure.

NOTE: The hg repository is only 15,000 commits and around 1,000 files.
Performance is okay but hg doesn't provide performant, native APIs to get some
data efficiently so we have to do some dumb stuff. If some of these interfaces
are cripplingly slow or whatever, let me know and we can start bundling some
Mercurial extensions with Arcanist.

Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran

Reviewed By: Makinde

CC: aran, Makinde, epriestley

Differential Revision: 960
2011-09-27 19:28:57 -07:00
epriestley
cd71098110 Detect commits by hash relationships
Summary:
When we discover a new commit and it has a known local commit or tree hash, mark
it committed.

This supports Mercurial and Git-Immutable workflows, and improves
hybrid-Git-Mutable workflows and covers some cases where poeple just make
mistakes or whatever.

Test Plan: Parsed Mercurial, Git and SVN commits.

Reviewers: Makinde

Reviewed By: Makinde

CC: aran, Makinde

Differential Revision: 963
2011-09-27 11:04:56 -07:00
epriestley
3ce0c602ec Improve Diffusion parser linking of author names
Summary: See T502. Under some VCS setups, we get full email addresses instead of
usernames or real names. Try harder to find matches, by falling back to email
address parsing if we don't get hits on the straight-up token parsing.

Test Plan:
This is difficult to test because it depends on the account state and repository
state, and hard to pull out so it's more testable without better mocking
facilities. I just dumped this into the parser to verify the behavior:

  foreach (array(
    'epriestley',
    'Evan Priestley',
    'epriestley@epriestley.com',
    'derp <epriestley@epriestley.com>',
    '"Evan Priestley" <derpderpderp@derpderpderp.com>',
    'quackderp <derpderpderp@derpderpderp.com>',
    ) as $email) {
    echo "{$email} = ".$this->resolveUserPHID($email)."\n";
  }
  die();

  Running PhabricatorRepositoryGitCommitMessageParserWorker...
  epriestley = PHID-USER-79f25616ea2635089a31
  Evan Priestley = PHID-USER-79f25616ea2635089a31
  epriestley@epriestley.com = PHID-USER-1bec59b91be6223f07fd
  derp <epriestley@epriestley.com> = PHID-USER-1bec59b91be6223f07fd
  "Evan Priestley" <derpderpderp@derpderpderp.com> =
PHID-USER-79f25616ea2635089a31
  quackderp <derpderpderp@derpderpderp.com> =

This is expected (all variations of my identity parsed correctly, and the bogus
one failed). There are two different user PHIDs in the result set because I have
like 30 different similar accounts on my local, including one called "derp" and
another one with address "derp@derp.com", which prevented an earlier version of
this test case from working correctly.

Reviewers: zachallia, aran, Makinde, jungejason, nh, tuomaspelkonen

Reviewed By: jungejason

CC: aran, jungejason

Differential Revision: 968
2011-09-27 11:04:49 -07:00
epriestley
1c1f749eba Add an "arcanist.projectinfo" Conduit call
Summary:
We currently rely on "remote_hooks_enabled" in .arcconfig to determine whether
commands like "arc amend" and "arc merge" should imply "arc mark-committed".

However, this is a historical artifact that is now bad for a bunch of reasons:

  - The option name is confusing, it really means 'repository is tracked'.
  - The option is hard to discover and generally sucks.
  - We can empirically determine the right answer since we now know if a project
is in a tracked repository.

Add a call which arcanist can make on these workflows to figure out if it is
interacting with a project in a tracked repository or not.

Also added an "isTracked()" convenience method to reduce the number of magic
strings all over the place.

Test Plan: Ran "arcanist.projectinfo" for nonexistent, untracked and tracked
projects.

Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran

Reviewed By: Makinde

CC: aran, epriestley, Makinde

Differential Revision: 945
2011-09-21 14:19:14 -07:00
epriestley
93b3bc8e89 Add a Mercurial message parser
Summary:
See D943, this is the second parse stage. This will mark Differential revisions
as "Committed" among other things.

Almost all the logic here is shared between VCSes so the implementation itself
is straightforward.

Test Plan: Parsed all messages for the official Mercurial repository.

Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran

Reviewed By: Makinde

CC: aran, Makinde

Differential Revision: 944
2011-09-16 11:09:39 -07:00
epriestley
e0b86cc81b Add a Mercurial commit discovery daemon
Summary:
Repository import has three major steps:

  - Commit discovery (serial)
  - Message parsing (parallel, mostly VCS independent)
  - Change parsing (parallel, highly VCS dependent)

This implements commit discovery for Mercurial, similar to git's parsing:

  - List the heads of all the branches.
  - If we haven't already discovered them, follow them back to their roots (or
the first commit we have discovered).
  - Import all the newly discovered commits, oldest first.

This is a little complicated but it ensures we discover commits in depth order,
so the discovery process is robust against interruption/failure. If we just
inserted commits as we went, we might read the tip, insert it, and then crash.
When we ran again, we'd think we had already discovered commits older than HEAD.

This also allows later stages to rely on being able to find Phabricator commit
IDs which correspond to parent commits.

NOTE: This importer is fairly slow because "hg" has a large startup time
(compare "hg --version" to "git --version" and "svn --version"; on my machine,
hg has 60ms of overhead for any command) and we need to run many commands (see
the whole "hg id" mess). You can expect something like 10,000 per hour, which
means you may need to run overnight to discover a large repository (IIRC, the
svn/git discovery processes are both about an order of magnitude faster). We
could improve this with batching, but I want to keep it as simple as possible
for now.

Test Plan: Discovered all the commits in the main Mercurial repository,
http://selenic.com/repo/hg.

Reviewers: Makinde, jungejason, nh, tuomaspelkonen, aran

Reviewed By: Makinde

CC: aran, Makinde

Differential Revision: 943
2011-09-16 11:08:52 -07:00
epriestley
43a3f4d234 Build an "affected path" index when attaching diffs to revisions
Summary: See T262. This creates the index on the Differential side which we need in order to execute this query efficiently on the Diffusion side.

Also renames "DiffusionGitPathIDQuery" to "DiffusionPathIDQuery", this query object has nothing to do with git.

Test Plan: Attached top-level and sub-level diffs to revisions and verified they populated the table with sensible data.

Reviewers: bmaurer, aravindn, fmoo, jungejason, nh, tuomaspelkonen, aran

CC:

Differential Revision: 931
2011-09-15 07:45:14 -07:00
epriestley
4da43b31a3 Add Mercurial repository configuration and local pull support
Summary: No actual parsing/import yet, but now you can define and pull Mercurial
repositories. I merged most of the local pull code so we can share it between
hg/git.

Test Plan:
  - Created a new Mercurial repository to track Codeigniter off Bitbucket
  - Edited / saved / etc.
  - Launched the mercurial pull daemon, it pulled the repo. Killed and
relaunched, it updated the repo.
  - Launched the git fetch deamon, it still works correctly.

Reviewers: Makinde, aran, jungejason, tuomaspelkonen

Reviewed By: Makinde

CC: aran, Makinde

Differential Revision: 793
2011-09-14 07:28:22 -07:00
epriestley
77ed7ade66 Add symbol import scripts for crossref features
Summary: Adds a script to import symbols from a ctags-like format, and another
to generate that format for PHP files.

Test Plan:
Ran it on Phabricator:

  mysql> select * from repository_symbol limit 200, 20;

+-------------------+---------------------------------------------------+------------+----------------+--------+------------+
  | arcanistProjectID | symbolName                                        |
symbolType | symbolLanguage | pathID | lineNumber |

+-------------------+---------------------------------------------------+------------+----------------+--------+------------+
  |                 1 | DifferentialDiffViewController                    |
class      | php            |     52 |         19 |
  |                 1 | DifferentialInlineCommentEditController           |
class      | php            |    308 |         19 |
  |                 1 | DifferentialInlineCommentPreviewController        |
class      | php            |  10543 |         19 |
  |                 1 | DifferentialRevisionEditController                |
class      | php            |  10544 |         19 |
  |                 1 | DifferentialRevisionListController                |
class      | php            |  10545 |         19 |
  |                 1 | DifferentialRevisionViewController                |
class      | php            |    142 |         19 |
  |                 1 | DifferentialSubscribeController                   |
class      | php            |  10546 |         19 |
  |                 1 | DifferentialRevisionListData                      |
class      | php            |     58 |         19 |
  |                 1 | DifferentialCommentEditor                         |
class      | php            |     39 |         19 |
  |                 1 | DifferentialRevisionEditor                        |
class      | php            |     42 |         24 |
  |                 1 | DifferentialFieldSpecificationIncompleteException |
class      | php            |  10547 |         19 |
  |                 1 | DifferentialFieldDataNotAvailableException        |
class      | php            |  10548 |         19 |
  |                 1 | DifferentialFieldParseException                   |
class      | php            |  10549 |         19 |
  |                 1 | DifferentialFieldValidationException              |
class      | php            |  10550 |         19 |
  |                 1 | DifferentialFieldSelector                         |
class      | php            |  10551 |         19 |
  |                 1 | DifferentialDefaultFieldSelector                  |
class      | php            |  10552 |         19 |
  |                 1 | DifferentialApplyPatchFieldSpecification          |
class      | php            |  10553 |         19 |
  |                 1 | DifferentialArcanistProjectFieldSpecification     |
class      | php            |  10554 |         19 |
  |                 1 | DifferentialAuthorFieldSpecification              |
class      | php            |  10555 |         19 |
  |                 1 | DifferentialFieldSpecification                    |
class      | php            |  10556 |         35 |

+-------------------+---------------------------------------------------+------------+----------------+--------+------------+
  20 rows in set (0.00 sec)

Reviewers: jungejason, nh, tuomaspelkonen, aran

Reviewed By: tuomaspelkonen

CC: aran, tuomaspelkonen

Differential Revision: 898
2011-09-13 08:49:44 -07:00
epriestley
cd05c960ff Add storage for repository symbol tracking
Summary: See T315 for an extensive description of this feature. Adds the
descibed storage table.

Test Plan: Used phpsh to read/write symbol objects.

Reviewers: jungejason, nh, tuomaspelkonen, aran

Reviewed By: tuomaspelkonen

CC: aran, epriestley, tuomaspelkonen

Differential Revision: 897
2011-09-13 08:49:44 -07:00
epriestley
8f772929ac Use a password input for HTTP Basic Auth in repositories
Summary: See D902. As @abdul notes, a password input is probably more
appropraite here.

Test Plan: Mashed stuff into it, got bullets instead of text.

Reviewers: abdul, jungejason, nh, tuomaspelkonen, aran

Reviewed By: nh

CC: aran, nh

Differential Revision: 913
2011-09-13 08:49:07 -07:00
Abdul Qabiz
6355b291ed - Added getRemoteCommandFuture(..) and getLocalCommand Future(..) methods to PhabricatorRepository
- Removed irrelevant csprintf(..)
  - Updated code to use $repository->getRemoteURI()
  - Updated code to use getRemoteCommandFuture(..) in Diffusion code
  - Updated code to use $repository->getRemoteURI()
2011-09-09 01:16:48 +05:30
epriestley
e3a9d73fe1 Add keyfile and HTTP Basic auth support to repositories
Summary: I still need to go through all the daemon and Diffusion code and change
the bare execx() calls to $repository->execxXXX() to actually make this work,
but we're getting close.

Test Plan: Configured repositories with various HTTP / SVN setups and ran the
test_connection.php script to verify keys were located and added and
username/password information was supplied.

Reviewers: jungejason, nh, tuomaspelkonen, aran

Reviewed By: nh

CC: aran, nh, jungejason

Differential Revision: 902
2011-09-07 13:22:08 -07:00
epriestley
1df7d4039e Store repository credentials with repositories
Summary:
Move toward storing credentials in configuration so it's easier to get the
daemons working. This should eventually solve all the key juggling junk you have
to do right now.

This only gets us part of the way to actually using these credentials in the
daemons since I have to go swap everything for $repository->execBlah().

I tried to write a web "Test Connection" button but it was too much of a mess to
get git to work since git doesn't give you access to its SSH command and SSH has
a bunch of interactive prompts which you can't really do anything about without
it or a bunch of ~/.ssh/config editing. This is what Git recommends:

https://git.wiki.kernel.org/index.php/GitFaq#How_do_I_specify_what_ssh_key_git_should_use.3F

..but it's not a great match for this use case.

Test Plan:
  - Only partial.
  - Ran "test_connection.php" on a Git repo with and without SSH, and with and
without valid credentials. This part works properly.
  - Ran "test_connection.php" on a public SVN repo, but I don't have private or
WEBDAV repos set up at the moment.
  - Mercurial doesn't work yet.
  - Daemons haven't been converted yet.

Reviewers: jungejason, tuomaspelkonen, aran

Reviewed By: jungejason

CC: aran, abdul, nmalcolm, epriestley, jungejason

Differential Revision: 888
2011-09-06 08:58:00 -07:00
epriestley
ae045a9cf2 When doing partial subdirectory parses in Subversion, stub out foreign commit
references

Summary:
See T325. We tentatively support doing partial subdirectory parses in
Phabricator for Subversion, so you can elect to import only "trunk/local/" or
similar. We do this by importing only some of the commits (those commits which
affected that directory).

In Subversion, you can also "svn cp
svn+ssh://example.com/svnroot/trunk/foreign/example.c@13 local.c". This means
that commits which reference "trunk/local/" may themselves reference foreign
commits.

Currently, we break in this case and can't find the commit reference. Instead,
generate a foreign commit stub so we can at least point at some reasonable
object.

Test Plan: Successfully imported trunk/a/ of the test repo in T325 without
errors. Verified commit 3 in that repo is imported as a foreign stub.

Reviewers: jungejason, nh, tuomaspelkonen, aran

Reviewed By: jungejason

CC: aran, jungejason, epriestley

Differential Revision: 892
2011-09-04 14:10:03 -07:00
epriestley
ed508247ba Fix a bug in the SVN parser which causes it to find commit refs in other SVN
repositories

Summary:
This query isn't scoped correctly to the repository ID, so we may identify
commits from other repositories.

This causes a somewhat subtle issue since we only use it to manage file
copies/moves, so you end up with a file "copied from" the same revision in
another repository. I think the UI probably even renders correctly.

Once I finish T325 and better understand what's going on here, I'll see how much
work is involved in writing an SQL patch to fix this.

Test Plan: Parsed the test repo from T325 with the expected error.

Reviewers: jungejason, nh, tuomaspelkonen, aran

Reviewed By: jungejason

CC: aran, jungejason

Differential Revision: 891
2011-09-04 14:10:02 -07:00
epriestley
6cae153569 Allow CommitTask daemon to recover from deleted repositories
Summary: If a user partially discovers a repository and then deletes it, the
timeline will have events from the old repository which this daemon won't be
able to parse.

Test Plan: @ajtrichards, can you apply this locally and restart your daemons
(##phd stop##, then relaunch them) and let me know if it fixes the issue?

Reviewers: ajtrichards, jungejason, tuomaspelkonen, aran

Reviewed By: ajtrichards

CC: aran, epriestley, ajtrichards

Differential Revision: 845
2011-08-22 15:41:27 -07:00
Svemir Brkic
e4093e8013 SVN error message may also be "File not found" 2011-08-22 17:34:37 -04:00
epriestley
6dc193d3d9 Fully update library map. 2011-08-18 09:52:36 -07:00
epriestley
a67ce60d67 Unguard two external POST endpoints from CSRF. 2011-08-18 07:25:45 -07:00
epriestley
90be65f6ec This write should also be unguarded, it is not vulnerable to CSRF. 2011-08-17 10:12:30 -07:00
epriestley
35d03d36c7 Improve display behavior of commit messages in Diffusion
Summary:
See T372. Always render commit messages on one display line, so the table
doesn't jump around as they AJAX in on browse views.

The goal here is to have the cell choose a size naturally and for its content to
render with "overflow: hidden" if the natural size isn't large enough to contain
the content. "white-space: pre" or "white-space: nowrap" would prevent wrapping
but potentially make the table exceed the display width when a better behavior
is to hide some of the commit message.

Also use utf8-aware shortening, now that we have a function for it.

Casting a wide net in case anyone has a better way to do the CSS here. It's kind
of nasty that we have to use so many DOM nodes.

Test Plan:
  - Resized window while viewing browse and history views in Safari, Chrome and
Firefox. Table exhibited described behavior.
  - Verified summaries render sensibly and are properly truncated to 100
characters.

Reviewed By: aran
Reviewers: aran, jungejason, tuomaspelkonen, tomo, mroch, cpojer
CC: aran, epriestley
Differential Revision: 750
2011-07-31 12:05:06 -07:00
epriestley
da8beefa6c Use AphrontFormLayoutView, not AphrontFormView, for GitHub 'Repository'
interface

Summary:
See T58. When I originally wrote this interface you couldn't make a form that
looked like a form but wasn't really a form, so I made it a real form. That
meant you could hit "return" in the text input and submit the form, which didn't
do anything but is vaguely silly.

After D422 there's a layout-only form element, so switch to that.

Test Plan: Loaded the tab, hit return in the textarea, it didn't do a form
submit.
Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, jungejason, epriestley
Differential Revision: 719
2011-07-24 18:10:40 -07:00
epriestley
ed5c46681d Allow SVN repositories to import subdirectories instead of the entire repository
Summary:
See T325. While this is a touch hacky it ends up being fairly clean, and we can
now do initial imports much more quickly and this actually cleaned up some of
the code. I also made the repository edit interface a little less foreboding.

@tuomaspelkonen, did you get anywhere with that bug you were chasing down a
couple days ago? We can hold this if it throws a wrench into stuff you're
working on.

Test Plan:
  - Imported a subdirectory of a midsized SVN project (jQuery UI).
  - Commit discovery for ~3500/4500 commits took just a few seconds.
  - Commit discovery correctly ignored commits which didn't affect this
directory.
  - Commit discovery correctly stopped at commit 13.
  - Browse interface shows an incomplete listing, but that's fine, and
everything is otherwise functionally correct. We can add a note or something
later ("this is a view of commits affecting a subdirectory, some paths aren't
available"), but this behavior probably won't be too startling to users.
  - Edited Git and SVN repositories to test form logic.

Reviewed By: jungejason
Reviewers: tuomaspelkonen, jungejason, aran, Girish
Commenters: tuomaspelkonen
CC: jcleveley, aran, jungejason, tuomaspelkonen
Differential Revision: 696
2011-07-20 10:56:02 -07:00
epriestley
11d8f1af18 Improve search result listing
Summary:
Make it prettier, paginate, add user pictures, show document types, clean some
stuff up a little. Plenty of room for improvement but this should make it a lot
more useful.

Test Plan:
Here's what the new one looks like:
https://secure.phabricator.com/file/view/PHID-FILE-edce2b83c2e3a121c2b7/

Reviewed By: jungejason
Reviewers: tomo, jungejason, aran, tuomaspelkonen, mroch
Commenters: tomo
CC: aran, tomo, jungejason, epriestley
Differential Revision: 545
2011-06-29 13:25:38 -07:00
Jason Ge
d282549166 Add commits to the Phabricator search index
Summary:
create the indexer for commit

Test Plan:
run the reindex_one_commit.php against one existing commit
and it is returned in search result on the webpage; run parse_one_commit
against another commit; modified reindex_everything.php to let it only
parse one commit after loading all commits.

Reviewed By: epriestley
Reviewers: epriestley, aran
CC: aran, jungejason, epriestley, debow
Differential Revision: 490
2011-06-25 13:15:54 -07:00
epriestley
c67a749036 Verify that SVN repository URI ends in a "/"
Summary:
Everything breaks if this isn't true, and it's easy to get subtly wrong right
now. There are other more magical ways we could do this (automatically add a "/"
in this form or at runtime) but I think making it explicit is the easiest and
most robust approach. See T67.

Test Plan:
Tried to save a URI without a trailing slash.

Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, jungejason
Differential Revision: 305
2011-05-18 10:37:53 -07:00
epriestley
f72c1acc63 Lockdown tool directory editing to administrators
Summary:
Someone has "defaced" secure.phabricator.com with a helpful suggestion that I
actually do this; fair enough. :P

Test Plan:
Logged in as myself, unable to edit directory information. Logged out, logged in
as admin, was able to edit directory information.

I need to fix some more CSS stuff since some of these tabs render out hideous in
the admin background, but I can followup with that.

Reviewed By: tuomaspelkonen
Reviewers: aran, jungejason, tuomaspelkonen
Commenters: aran
CC: aran, tuomaspelkonen, epriestley
Differential Revision: 296
2011-05-17 13:17:13 -07:00
epriestley
19b23e2dd0 Fix repository deletion
Summary:
I never actually wrote this controller.

Test Plan:
Deleted a repository via web UI.

Reviewed By: tuomaspelkonen
Reviewers: tuomaspelkonen, jungejason, aran
Commenters: aran
CC: aran, tuomaspelkonen, epriestley
Differential Revision: 231
2011-05-05 13:01:45 -07:00
jungejason
c728e4f7da Open database connection with 'w' instead of 'r' for writing
Summary:
there are several places we open an 'r' connection but use it
for writing. Fix them.

Test Plan:
ran parse_one_commit.php against one revision which executes
the code with problem. It used to throw exception. Now it works fine.

Reviewed By: Girish
Reviewers: tuomaspelkonen, Girish
Commenters: aran
CC: aran, Girish
Differential Revision: 213
2011-05-02 13:31:12 -07:00
epriestley
6bec3d2e4f Simplify and demuddle MetaMTA send pathways
Summary:
I pretty shortsightedly made sending a side effect of save() in the case that a
server is configured for immediate sending. Move this out, make it explicit, and
get rid of all the tangles surrounding it.

The web tool now ignores the server setting and only repsects the checkbox,
which makes far more sense.

Test Plan:
Sent mails from Maniphest, Differential, and the web console. Also ran all the
unit tests. Verified headers from Maniphest.

Reviewed By: rm
Reviewers: aran, rm
CC: tuomaspelkonen, rm, jungejason, aran
Differential Revision: 200
2011-05-02 03:07:30 -07:00
epriestley
2f96095014 Use %s and %b in GitCommitMessageParserWorker, not %B
Summary:
See T129, some older git doesn't have %B and we can reasonably fake it with %s
and %b.

Test Plan:
Reparsed all of the Phabricator repository with this worker, commit messages
look fine.

Reviewed By: rm
Reviewers: rm, aran, jungejason, tuomaspelkonen
CC: aran, rm
Differential Revision: 209
2011-05-02 01:26:18 -07:00
epriestley
bc1be958bd Allow GitCommitChangeParser to parse 'T' (type change) messages in
git history.
2011-04-14 14:16:53 -07:00
epriestley
fa38b70ba6 Fix message IDs and Herald URIs. 2011-04-10 08:46:39 -07:00
epriestley
3dcf902a39 In Herald commit notification emails, put a space between the
(author, reviewer) and Differential revision title.
2011-04-10 08:02:14 -07:00
epriestley
22297b71a0 Close the loop on Diffusion commits posting back to Differential. 2011-04-07 21:59:42 -07:00
epriestley
ed1eead587 Revision has getTitle() now, not getName(). 2011-04-07 21:22:41 -07:00
epriestley
361ec78b03 Add missing includes from XHPAST parse bug. 2011-04-06 23:14:58 -07:00
epriestley
9630123d6a Use pull-frequency to determine pull frequency, not some sort of random
algorithm that backs off to 15m which is crazy.
2011-04-06 19:36:48 -07:00
epriestley
40083bfa0e Allow Herald to be disabled while importing a repository so we don't send a
billion zillion emails.
2011-04-06 16:28:20 -07:00
epriestley
11ea93260a Sync up UUIDs and create project configs. 2011-04-05 21:55:04 -07:00
epriestley
bb79418615 Don't send sandbox URIs in Herald messages. 2011-04-04 14:23:03 -07:00
epriestley
cc66c4890d Commit + Herald integration. 2011-04-04 14:13:14 -07:00
epriestley
736f629a07 Lint stragglers. 2011-04-03 23:38:02 -07:00
epriestley
2834e9f6ce HeraldCommitAdapter and some related junk. 2011-04-03 23:23:36 -07:00
epriestley
5038ab850c Some owners read workflows. 2011-04-03 19:20:47 -07:00
epriestley
c2f10f19b1 Fix some daemon ish. 2011-04-02 15:02:06 -07:00
epriestley
1f88e08761 Move 'phd parse-commit' to a dedicated script; allow message parsing tasks to
be executed in isolation, provide a script to requeue all message reparses,
stop parse-commit from inserting side-effect tasks.
2011-04-02 13:23:22 -07:00
epriestley
82fffe466a Add basic detail-parser functionality. 2011-04-01 17:11:55 -07:00
epriestley
34e8d902c7 Restore "Shortcuts" feature to Diffusion. 2011-03-31 00:33:44 -07:00
epriestley
e25c58ed9c Restore some commit metadata to browse views. 2011-03-30 22:41:31 -07:00
epriestley
e6cf7a9cb0 More Diffusion junk. 2011-03-30 22:08:41 -07:00
epriestley
63a3edb29f Make SVN commit parsers actually parse commits again. 2011-03-27 00:39:29 -07:00
epriestley
fa8f168bd0 Allow commits to be marked as 'bad' so they won't be parsed. Useful if you work
at a company where someone deleted the entire repository *cough cough*.
2011-03-26 23:52:09 -07:00
epriestley
1a11297dd6 Allow parsing of rare extra-broken non-UTF8 messages. 2011-03-26 23:20:16 -07:00
epriestley
084c79d85a Herald skeleton. 2011-03-22 13:22:55 -07:00
epriestley
2eb30aed9e Improve search/phid flexibility. 2011-03-21 23:37:05 -07:00
epriestley
afe0079819 Improve awkward Diffusion query plans. 2011-03-20 17:46:02 -07:00
epriestley
01a20c0480 Fix various parsing bugs in Differential. 2011-03-19 14:42:17 -07:00
epriestley
0d292abbf0 Prevent daemons from spinning when they hit duplicate key exceptions. 2011-03-15 14:36:01 -07:00
epriestley
14a172d707 Make GitDiscovery terminate, maybe? 2011-03-15 14:25:47 -07:00
epriestley
bb144542d3 Diffusion/phd/console improvements. 2011-03-15 13:38:14 -07:00
epriestley
b5db12b6b2 Browse / git improvements. 2011-03-13 22:03:30 -07:00
epriestley
4893146815 Improve parser scalability, fix a bug or two, provide 'phd', the Phabricator
Daemon interface.
2011-03-13 14:27:03 -07:00
epriestley
383b3d740c Improve some diffusioney things.
Summary:

Test Plan:

Reviewers:

CC:
2011-03-12 22:51:40 -08:00
epriestley
485b5e5ded Make the Diffusion UI vaguely usable in some cases. 2011-03-12 16:17:34 -08:00
epriestley
48b5d6d025 Diffusion change parser for SVN.
Summary: there is a small possibility this works correctly

Test Plan:

Reviewers:

CC:
2011-03-11 14:13:23 -08:00
epriestley
452f4000ff Merge branch 'master' of github.com:facebook/phabricator 2011-03-11 09:36:04 -08:00
epriestley
8bcbeface1 Increase parseriness of some parsery things.
Summary: None of these are parsnips.

Test Plan:

Reviewers:

CC:
2011-03-11 09:34:22 -08:00
jungejason
6949807c51 Fix Repository tool's missing app link
Summary:
the Repository tool is missing the standard application link
at the top. The reason is that PhabricatorRepositoryListController and
several other classes are inheriting from a wrong class.

Test Plan:
play with the application to make sure it is not breaking
other stuff.

Reviewed By: epriestley
Reviewers: epriestley
CC: jungejason, epriestley
Differential Revision: 70
2011-03-10 17:27:48 -08:00
epriestley
71605e1c2d Minor improvements to PhabricatorRepositoryGitCommitDiscoveryDaemon.php 2011-03-07 22:16:53 -08:00
epriestley
57495c4287 Rough cut of repository tracking
Summary: Basic scaffolding for repository tracking, plus daemon infrastructure
(Timelines, Cursors) and some fixes (memory usage, mysql_connect() junk).

Test Plan: parsed Javelin git commit history via daemon

Reviewers:

CC:
2011-03-06 22:29:22 -08:00
epriestley
eccc76dae6 Fix some issues caught by HipHop, and work around some issues
caused by HipHop.
2011-02-26 21:01:42 -08:00
epriestley
580116b1c1 Herp derp.
Summary:

Test Plan:

Reviewers:

CC:
2011-02-12 18:49:28 -08:00
epriestley
f55a30a288 Rough cut of Repositories and GitHub post-receive hooks.
Summary:

Test Plan:

Reviewers:

CC:
2011-02-12 18:26:15 -08:00