1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-11 17:32:41 +01:00
Commit graph

69 commits

Author SHA1 Message Date
epriestley
938b5c69c6 Use PhutilProxyException to disambiguate repository pull errors
Summary: Ref T2569. When repository pulls fail, they don't necessarily list identifying information about which repository failed. Use a proxy exception to list that information.

Test Plan: {F51267}

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2569

Differential Revision: https://secure.phabricator.com/D6548
2013-07-23 16:58:20 -07:00
Lauri-Henrik Jalonen
93e37e9060 Phabricator event timeline removed
Summary: Removed related files and references

Test Plan: Crossed my fingers

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin, AnhNhan

Maniphest Tasks: T2003

Differential Revision: https://secure.phabricator.com/D5744

Conflicts:
	src/__phutil_library_map__.php
	src/infrastructure/storage/patch/PhabricatorBuiltinPatchList.php
2013-07-09 18:07:42 -07:00
epriestley
efbd3ecc48 Fix the type of some values passed to MySQL
Summary: Ref T3377. MySQL ignores indexes if we hand it mismatched datatypes. This seems colossally dumb, but give it what it expects.

Test Plan: wat

Reviewers: wez, btrahan

Reviewed By: wez

CC: aran

Maniphest Tasks: T3377

Differential Revision: https://secure.phabricator.com/D6201
2013-06-13 18:01:40 -07:00
epriestley
e7fde9a77c Make repository discovery partially testable
Summary:
Ref T2784. Begins pulling discovery into Engines and covering it with tests. In particular:

  - Discovery is currently a one-shot process where we find all the new commits and write them to the database in one go. Split it apart so we find and return the new commits first, then write them to the database separately. This makes things simpler and more testable.
  - This diff only brings SVN into an engine (and only the "find the commits" part), since it's simpler than Git or Mercurial.
  - Creates a base Engine class and moves common functionality there.
  - Restores the `--verbose` flag to `repository pull`.

Test Plan: Added unit tests. Ran `bin/repository discover`. Ran `bin/phd debug pulllocal`.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2784

Differential Revision: https://secure.phabricator.com/D5906
2013-05-12 19:08:37 -07:00
epriestley
105dd1899c Make repository pulls testable
Summary:
Ref T2784. This moves us toward being able to test the background and Conduit pipelines for repositories. In particular:

  - Separate the logic for pulling repositories (`git pull`, `hg pull`) out of `PhabricatorRepositoryPullLocalDaemon` and put it in `PhabricatorRepositoryPullEngine`. This allows repositories to be pulled directly without invoking the daemons.
  - Add tests for the engine, including a future-looking base test case.
  - Add basic `PhutilDirectoryFixture`-based repositories.

Next steps:

  # Do the same for repo discovery.
  # Then we can start writing tests against specific Conduit methods.

Test Plan: Ran unit tests. Ran `bin/repository pull` on SVN, Hg and Git repositories. Ran `bin/phd debug pulllocal`.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, nh

Maniphest Tasks: T2784

Differential Revision: https://secure.phabricator.com/D5904
2013-05-12 19:05:52 -07:00
Bob Trahan
7573ad9a70 DiffusionBranchQuery => Conduit
Summary: ref T2784. This one had a few fun spots where I had to move data around. Also, is there some common object (or should I add it?) that can do this toDictionary newFromConduit stuff? Also, this assumes D5803 is largely correct at the time of this diff.

Test Plan: browsed mercurial and git repository page. saw the branches i expected.

Reviewers: epriestley

Reviewed By: epriestley

CC: chad, aran, Korvin

Maniphest Tasks: T2784

Differential Revision: https://secure.phabricator.com/D5810
2013-05-01 14:56:36 -07:00
Nick Harper
0ba7d34f97 Allow phabricator to manage repositories
Summary:
This adds a configuration option for repositories to be marked as managed
by phabricator, which is then used by the pullLocal daemon to try to recover
from some errors it runs into.

Test Plan: Set a bogus uri for a repo, saw that the pullLocal daemon fixed it.

Reviewers: epriestley, vrana

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3680
2012-12-12 14:43:09 -08:00
vrana
ef85f49adc Delete license headers from files
Summary:
This commit doesn't change license of any file. It just makes the license implicit (inherited from LICENSE file in the root directory).

We are removing the headers for these reasons:

- It wastes space in editors, less code is visible in editor upon opening a file.
- It brings noise to diff of the first change of any file every year.
- It confuses Git file copy detection when creating small files.
- We don't have an explicit license header in other files (JS, CSS, images, documentation).
- Using license header in every file is not obligatory: http://www.apache.org/dev/apply-license.html#new.

This change is approved by Alma Chao (Lead Open Source and IP Counsel at Facebook).

Test Plan: Verified that the license survived only in LICENSE file and that it didn't modify externals.

Reviewers: epriestley, davidrecordon

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2035

Differential Revision: https://secure.phabricator.com/D3886
2012-11-05 11:16:51 -08:00
epriestley
5903ed650c Move completed tasks to an "archive" table and delete them in the GC
Summary:
Currently, when taskmasters complete a task it is immediately deleted. This prevents us from doing some general things, like:

  - Supporting the idea of permanent failure (e.g., after N failures just stop trying).
  - Showing the user how fast taskmasters are completing tasks.
  - Showing the user how long tasks took to complete.

Having better visibility into this is important to Drydock, which builds on the task system. Also, generally buff debug output for task execution.

Test Plan: Ran `bin/phd debug taskmaster`. Ran `bin/phd debug garbage`. Queued some tasks via various systems.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3852
2012-10-31 15:22:16 -07:00
epriestley
839f3df9c2 Stop trasforming "scp-style" URIs into normal URIs in Git
Summary:
See D3033, T1529. We currently transform "scp-style" `user@host:path` URIs into normal `ssh://user@host/path` URIs. This is undesirable for two reasons:

  - The paths aren't always equivalent. They are for GitHub, which is why I missed this originally, but in the general case the ":path" is resolved relatively and the "/path" is resolved absolutely. So this transformation can break things.
  - It confuses users, who do not think of "git@host:path" URIs as SSH URIs even though the SSH protocol is implied.

So stop using them, and just use the "git@host:path" URIs instead. This is a bit messy since we have some validation built up on top of URIs. Hopefully we can get rid of more of this in the future as we simplify repository management.

Test Plan: Unit tests cover this stuff pretty well. Made a new git repository with a "git@host:path" style URI and did pull/discover on it, verified the right URI was used.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1529

Differential Revision: https://secure.phabricator.com/D3036
2012-07-23 12:05:03 -07:00
epriestley
ddf67fce58 Add an example event listener, improve documentation, and add a commit discovery event
Summary: Improve documentation around Phabricator events.

Test Plan: Generated and read documentation. Ran test script.

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran

Maniphest Tasks: T1092

Differential Revision: https://secure.phabricator.com/D2917
2012-07-03 16:46:27 -07:00
epriestley
dcf3a31944 Minor, unlock after discovery exception. 2012-07-03 07:46:38 -07:00
epriestley
1c1fdf0448 Minor, move the "repair" instructions to the right place. 2012-07-02 07:22:56 -07:00
Nick Harper
d70f8dad3f Change lock in PullLocalDaemon to only lock on discovery
Summary:
The locks held by read-only pullLocal daemons were causing our discovery instance
to not get the lock and fail at discovery. We don't need to hold the lock while
pulling (only while discovering), so this moves the lock to the appropriate place.

Test Plan: tested in production

Reviewers: jungejason, epriestley, vrana

Reviewed By: jungejason

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D2890
2012-06-29 16:19:32 -07:00
epriestley
46008ce3b3 Record initial branch when recording commits
Summary:
We have a race condition right now, where we may insert a commit without `seenOnBranches`. This means shouldAutocloseCommit will return false (since it's not on any autoclose branches) if the message parser runs fast enough, causing it to associate the commit but not close the revision. This happened for D2851.

Also prompt the user to repair broken repositories.

Test Plan: Ran discovery / repair. Ran discovery on new commits. Verified 'seenOnBranches' value. Deleted some data, verified "repair" error. Repaired repository.

Reviewers: jungejason, nh, vrana, btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D2858
2012-06-28 14:00:00 -07:00
epriestley
1bd0a1f0d9 Add a global lock for the RepositoryPullDaemon
Summary: Allow multiple daemons to run without contention.

Test Plan: Ran multiple daemons simultaneously in "debug" mode, observed them acquiring (and sometimes failing to acquire) locks.

Reviewers: btrahan, jungejason, nh

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1400

Differential Revision: https://secure.phabricator.com/D2877
2012-06-28 09:54:55 -07:00
epriestley
7b50b2fbdc Use subprocess output streaming to improve performance of Git commit discovery
Summary:
Improve performance of large discovery tasks in Git by using subprocess streaming, like we do for Mercurial.

Basically, we save the cost of running many `git log` commands by running one big `git log` command but only parsing as much of it as we need to. This is pretty complicated, but we more or less need it for mercurial (which has ~100ms of 'hg' overhead instead of ~5ms of 'git' overhead) so we're already committed to most of the complexity costs. The git implementation is much simpler than the hg implementation because we don't need to handle all the weird parent rules (git gives us to them easily).

Test Plan:
Before, `discover --repair` on Phabricator took 35s:

  real	0m35.324s
  user	0m13.364s
  sys	0m21.088s

Now 7s:

  real	0m7.236s
  user	0m2.436s
  sys	0m3.444s

Note that most of the time is spent inserting rows after discover, the actual speedup of the git discovery part is much larger (subjectively, it runs in less than a second now, from ~28 seconds before).

Also ran discover/pull on single new commits in normal cases to verify that nothing broke in the common case.

Reviewers: jungejason, nh, vrana

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1401

Differential Revision: https://secure.phabricator.com/D2851
2012-06-25 15:21:48 -07:00
epriestley
ca31e3e84b Add a --repair flag to "bin/repository discover"
Summary:
If a repository is missing commits because they mysteriously vanished, there's no reasonable way to get them back right now. Provide a way to ignore the state in the database and rediscover the entire repository unconditionally.

We don't queue any reparses or anything, but when I move reparse into this script we can hook things up or something. This generally shouldn't be too important anyway.

Test Plan: Ran `repository discover --repair` on Phabricator.

Reviewers: jungejason, nh, vrana

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D2850
2012-06-25 12:35:47 -07:00
epriestley
a705f336a3 Add vebose logging to PhutilRepositoryPullDaemon
Summary: Add verbose logging. This logging is activated by setting "phd.verbose" in the config, running "phd debug", or explicitly in scripts/repository/pull.php and scripst/repository/discover.php

Test Plan:
  >>> orbital ~/devtools/phabricator $ ./scripts/repository/discover.php GTEST
  Discovering 'GTEST'...
  <VERB> PhabricatorRepositoryPullLocalDaemon Discovering commits in repository 'GTEST'...
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch '()_+abcd$100', at a37bc285a12efa7224fe19f3df54cd90fa2b897a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch '+abcd$100', at a37bc285a12efa7224fe19f3df54cd90fa2b897a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch '_+abcd$100', at a37bc285a12efa7224fe19f3df54cd90fa2b897a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch 'abcd$100', at a37bc285a12efa7224fe19f3df54cd90fa2b897a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch 'arcpatch', at 774c7737b2d560a291697126bf4513204ccf661a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch 'arcpatch-1', at dc97539bee07293f95990d71f4638335a2531d69.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch 'arcpatch-2', at 1acfaec313c46dd3caa90448800181fb91b0270f.

Reviewers: jungejason

Reviewed By: jungejason

CC: aran

Differential Revision: https://secure.phabricator.com/D2843
2012-06-24 15:06:40 -07:00
Jason Ge
de46a2550c Fix auto-close in a repo with missing commits in Phabricator db
Summary:
After adding more error logging: https://secure.phabricator.com/differential/diff/5295/, discover.php prints the output: https://secure.phabricator.com/P427

The problem is that there are gaps in the commits and revisions in Phabricator's DB for the repo. For some git commits, both the phabricator commits and the differential revision are missing (because of a mysterious DB data loss. nharper should be able to recover it); for some other git commits, the phabricator commits are missing (probably also because of the daba loss mentioned above). This can also happen if the git commits were too big to be parsed by phabricator.

Because of the gap, the pull daemon detects them as new autoclose commits on the branch
  (https://secure.phabricator.com/diffusion/P/browse/master/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php;382bafa271ec4962$267)
  (https://secure.phabricator.com/diffusion/P/browse/master/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php;382bafa271ec4962$592)

Then tries to update it
  (https://secure.phabricator.com/diffusion/P/browse/master/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php;382bafa271ec4962$616)
and will fail because the commit doesn't exist in the DB.
  (https://secure.phabricator.com/diffusion/P/browse/master/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php;382bafa271ec4962$341)
After the exception is thrown, all the commits just detected are not marked as 'seenOnBranches', so they won't be marked as closed by the CommitMessageParserWorker.

This also explains the weird behavior where the parsing of the commits is always paused for 15 minutes and then gets started again. The parsing pauses because the process listed at https://secure.phabricator.com/P427 takes quite some time to finish before it can record tha tasks. The parsing happens all the time because the process will be executed again and again.

A better fix might be to put all the missing commits in the repository_badcommit table and check against it.

Test Plan: will run 'discover.php FBCODE' to test it.

Reviewers: epriestley, vrana

Reviewed By: vrana

CC: nh, vrana, aran, Korvin

Differential Revision: https://secure.phabricator.com/D2846
2012-06-23 09:47:20 -07:00
Jason Ge
a75da4efac Revert "run record commit if its the first time seen"
Summary: This reverts commit 22e606d49e.

Test Plan: none

Reviewers: epriestley, nh

Reviewed By: nh

CC: nh, vrana, aran, Korvin

Differential Revision: https://secure.phabricator.com/D2833
2012-06-21 20:26:07 -07:00
Jason Ge
22e606d49e run record commit if its the first time seen
Test Plan: test live.

Reviewers: nh

Reviewed By: nh

CC: epriestley, aran

Differential Revision: https://secure.phabricator.com/D2832
2012-06-21 19:47:25 -07:00
Jason Ge
06c976b738 check commit before calling its method
Summary: it's calling pull daemon's failure. This is actually Nick's fix.

Test Plan: Nick already manually ran it on daemon machine.

Reviewers: vrana, nh

Reviewed By: vrana

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D2828
2012-06-21 18:36:31 -07:00
epriestley
e48830eecb Don't rediscover an entire branch when a new commit appears at HEAD
Summary:
The autoclose logic is currently doing a little too much work. We want to parse each commit at most twice:

  # When it first appears in the repository.
  # When it first appears on an autoclose branch.

These two events might not be distinct (i.e., it might first appear on an autoclose branch).

Currently, to discover commits initially appearing on autoclose branches, we check each branch, determine if it's an autoclose branch or not, and determine if the HEAD is already a known commit on an autoclose branch. This is correct so far, and allows us to ignore branches which either haven't changed or have commits at HEAD which we've already examined.

However, if an autoclose branch has a new commit, we start working backward through it. Prior to this patch, we only stop when we hit commits that we've already discovered lie on this branch. If the branch is new, none of the commits will be discovered on it (they're discovered in general, and likely discovered on other autoclose branches, but not discovered on this branch), so we'll parse all the way back to the root.

Instead, we want to stop when we hit commits that we've already discovered on //any// autoclose branch.

So do that.

Test Plan: Pushed a new branch, then pushed a new commit at HEAD. Ran discovery, verified we rediscovered only 1 commit, not every commit in history.

Reviewers: vrana, jungejason, aurelijus

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1389

Differential Revision: https://secure.phabricator.com/D2798
2012-06-19 13:47:01 -07:00
Ben Rogers
444c634b6c Allows branches to appear in diffusion with out triggering closing things from the commit message
Summary: Implemented it how it was suggested in ticket comments

Test Plan: create a revision in a branch, push that branch up, verify it's visible in diffusion and also that revision is not closed, then merge and push to master, verify that revision closed

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin, 20after4

Maniphest Tasks: T1210

Differential Revision: https://secure.phabricator.com/D2706
2012-06-15 14:10:15 -07:00
epriestley
375c921bb0 Minor, fix issue with defaults: wildcard and repeat default to array and may not have a different default. 2012-06-12 07:05:57 -07:00
vrana
2e484e257d Fix lint errors found by Nemo
Summary:
See also:

- https://github.com/tpyo/amazon-s3-php-class/pull/33
- https://github.com/stripe/stripe-php/pull/13

Test Plan: Ran a script analyzing sources by HPHP.

Reviewers: btrahan, jungejason, epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D2713
2012-06-11 19:09:42 -07:00
vrana
6cc196a2e5 Move files in Phabricator one level up
Summary:
- `kill_init.php` said "Moving 1000 files" - I hope that this is not some limit in `FileFinder`.
- [src/infrastructure/celerity] `git mv utils.php map.php; git mv api/utils.php api.php`
- Comment `phutil_libraries` in `.arcconfig` and run `arc liberate`.

NOTE: `arc diff` timed out so I'm pushing it without review.

Test Plan:
/D1234
Browsed around, especially in `applications/repository/worker/commitchangeparser` and `applications/` in general.

Auditors: epriestley

Maniphest Tasks: T1103
2012-06-01 12:32:44 -07:00
epriestley
09c8af4de0 Upgrade phabricator to libphutil v2
Summary: Mechanical changes from D2588. No "Class.php" moves yet.

Test Plan: See D2588.

Reviewers: vrana, btrahan, jungejason

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1103

Differential Revision: https://secure.phabricator.com/D2591
2012-05-30 14:26:29 -07:00
epriestley
820a6d407a Improve performance of repository discovery under Mercurial
Summary: Prelminary, I want to test this a bit more when I'm better rested. Provide a "bulk" import mode for Mercurial so we can do initial discovery more quickly.

Test Plan: Imported the Mercurial repository in 2m45s, blocked on MySQL I/O rather than Mercurial I/O.

Reviewers: csilvers, btrahan

Reviewed By: csilvers

CC: aran

Differential Revision: https://secure.phabricator.com/D2457
2012-05-11 18:29:14 -07:00
epriestley
7e0c06716f Minor, fix obvious typo that I missed in testing somehow.
Auditors: csilvers
2012-05-09 12:43:57 -07:00
epriestley
b800df8c1b Simplify daemon management: "phd start"
Summary:
  - Merge CommitTask daemon into PullLocal daemon. This is another artifact of past instability (and order-dependent parsers). We still publish to the timeline, although this was the last consumer. Long term we'll probably delete timeline and move to webhooks, since everyone who has asked about this stuff has been eager to trade away the durability and ordering of the timeline for the ease of use of webhooks. There's also no reason to timeline this anymore since parsing is no longer order-dependent.
  - Add `phd start` to start all the daemons you need. Add `phd restart` to restart all the daemons you need. So cool~
  - Simplify and improve phd and Diffusion daemon documentation.

Test Plan:
  - Ran `phd start`.
  - Ran `phd restart`.
  - Generated/read documentation.
  - Imported some stuff, got clean parses.

Reviewers: btrahan, csilvers

Reviewed By: csilvers

CC: aran, jungejason, nh

Differential Revision: https://secure.phabricator.com/D2433
2012-05-09 10:29:37 -07:00
epriestley
d2b01aead0 Use one daemon to discover commits in all repositories, not one per repository
Summary:
See D2418. This merges the commit discovery daemon into the same single daemon, and applies all the same rules to it.

There are relatively few implementation changes, but a few things did change:

  - I simplified/improved Mercurial importing, by finding full branch tip hashes with "--debug branches" and using "parents --template {node}" so we don't need to do separate "--debug id" calls.
  - Added a new "--not" flag to exclude repositories, since I switched to real arg parsing anyway.
  - I removed a web UI notification that you need to restart the daemons, this is no longer true.
  - I added a web UI notification that no pull daemon is running on the machine.

NOTE: @makinde, this doesn't change anything from your perspective, but it something breaks this is the likely cause.

This implicitly resolves T792, because discovery no longer runs before pulling.

Test Plan:

  - Swapped databases to a fresh install.
  - Ran "pulllocal" in debug mode. Verified it correctly does nothing (fixed a minor issue with min() on empty array).
  - Added an SVN repository. Verified it cloned and discovered correctly.
  - Added a Mercurial repository. Verified it cloned and discovered correctly.
  - Added a Git repository. Verified it cloned and discovered correctly.
  - Ran with arguments to verify behaviors: "--not MTEST --not STEST", "P --no-discovery", "P".

Reviewers: btrahan, csilvers, Makinde

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T792

Differential Revision: https://secure.phabricator.com/D2430
2012-05-08 12:53:41 -07:00
epriestley
1c62a35710 Run one daemon to pull all working copies, not one daemon per working copy
Summary:
Allow the pull daemon to take a list of repositories. By default, pull all repositories.

Make some effort to respect pull frequencies, although we'll necessarily suffer a bit if running with only one process.

NOTE: We still launch one discovery daemon per working copy, so this only cuts the daemon count in half.

Test Plan:
  - Ran `phd debug pulllocal`, verified behavior.
  - Ran `pull.php P MTEST SVNTEST --trace`, verified it pulled the repos and ran the right commands.
  - Ran `phd repository-launch-master`, verified the right daemons launched, checked daemon console.
  - Ran `phd repository-launch-readonly`, verified the right daemon launched, checked daemon console.

Reviewers: btrahan, csilvers, davidreuss

Reviewed By: csilvers

CC: aran

Differential Revision: https://secure.phabricator.com/D2418
2012-05-07 15:01:10 -07:00
vrana
38ffe45f8e Use committer date instead of author date for Git epoch
Summary:
This is somewhat controversial but push date is usually more useful than commit date (which can be for example a month before other people can see the commit).
We can also store both dates.

Test Plan:
  git log --pretty="%ct %at"

Reviewers: epriestley

Reviewed By: epriestley

CC: nh, aran, Koolvin

Differential Revision: https://secure.phabricator.com/D2319
2012-04-26 16:25:56 -07:00
epriestley
a5903d2a53 Use head_key() and last_key() to explicitly communicate intent
Summary:
PHP arrays have an internal "current position" marker. (I think because foreach() wasn't introduced until PHP 4 and there was no way to get rid of it by then?)

A few functions affect the position of the marker, like reset(), end(), each(), next(), and prev(). A few functions read the position of the marker, like each(), next(), prev(), current() and key().

For the most part, no one uses any of this because foreach() is vastly easier and more natural. However, we sometimes want to select the first or last key from an array. Since key() returns the key //at the current position//, and you can't guarantee that no one will introduce some next() calls somewhere, the right way to do this is reset() + key(). This is cumbesome, so we introduced head_key() and last_key() (like head() and last()) in D2161.

Switch all the reset()/end() + key() (or omitted reset() since I was feeling like taking risks + key()) calls to head_key() or last_key().

Test Plan: Verified most of these by visiting the affected pages.

Reviewers: btrahan, vrana, jungejason, Koolvin

Reviewed By: jungejason

CC: aran

Differential Revision: https://secure.phabricator.com/D2169
2012-04-09 11:08:59 -07:00
epriestley
ec736f9c50 Handle "hg pull" return code change between 2.1 and 2.1.1 more gracefully
Summary:
See <https://github.com/facebook/phabricator/issues/102>. Between Feb 1 and Mar 1, the hg released changed the exit code behavior of "hg pull". This broke us mildly (and a bunch of other applications more severely, which is why it was reverted).

Detect the common case of this (english) and don't fail.

Test Plan: @killermonk, can you try applying this? I'll try to do an upgrade to 2.1 and see if I can also do a proper test.

Reviewers: Makinde, btrahan, killermonk

Reviewed By: btrahan

CC: killermonk, aran, epriestley

Differential Revision: https://secure.phabricator.com/D1948
2012-03-19 19:19:48 -07:00
epriestley
d0af617818 Add "final" to (almost) everything else
Summary: Last of the big final patches. Left a few debatable classes (12 out of about 400) that I'll deal with individually eventually.

Test Plan: Ran testEverythingImplemented.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T795

Differential Revision: https://secure.phabricator.com/D1881
2012-03-13 16:21:04 -07:00
epriestley
def19bb8de Add additional protections against local repository misconfigurations
Summary: We've hit a couple of these in the wild, raise better error messages when the local repo is toast / broken / nonsense.

Test Plan: Broke my local repo in all of the different ways we test for, verified I got an error message in each case.

Reviewers: btrahan, abirchall

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T964, T924

Differential Revision: https://secure.phabricator.com/D1855
2012-03-12 10:34:37 -07:00
epriestley
89dac1cf19 When updating git repositories, use --prune to prune old branches
Summary: We'll keep deleted branches around right now because Git's behavior is not to remove them without --prune.

Test Plan: Ran "git fetch --all --prune" to make sure it at least ostensibly works.

Reviewers: btrahan, 20after4

Reviewed By: 20after4

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1833
2012-03-08 13:24:25 -08:00
Edward Speyer
17d801a50e GitFetch daemon: more verbose
Summary: A smidgen more messaging about what's going on.

Test Plan:
Ran it, saw this:

  2012-03-07 5:37:30 PM [STDO] >>> [0] <exec> $ /data/phabricator/bin/list_db_services --tier_name 'cdb.phabricator'
  <<< [0] <exec> 47,101 us
  ...
  >>> [9] <exec> $ mkdir -p '/var/repo'
  <<< [9] <exec> 41,374 us
  Creating new directory /var/repo/fbcode for repo FBCode
  >>> [10] <exec> $ git clone --origin origin 'ssh://fbcode.git.vip.facebook.com/data/gitrepos/fbcode.git' '/var/repo/fbcode'

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1819
2012-03-07 19:24:06 -08:00
epriestley
b10fa8023f Support Git implicit file:// URIs
Summary: @s reported an issue with implicit file:// URIs in Git, see P270.
Recognize and handle URIs in this format. For URIs we don't understand, raise an
exception.

Test Plan:
  - Added failing tests.
  - Fixed code.
  - Tests pass.

Reviewers: btrahan, jungejason, s

Reviewed By: s

CC: aran, epriestley, s

Differential Revision: https://secure.phabricator.com/D1362
2012-01-11 09:00:18 -08:00
epriestley
99231ebba4 Allow Git repositories to track only some branches
Summary:
Some installs use Git as the backbone of a CI framework or use a Git remote to
share patches. The tracker scripts currently recognize associated revisions as
"Committed" when they appear in any branch, even if that branch is
"alincoln-personal-development_test_hack" or whatever.

To address the broadest need here, allow Git repositories to be configured to
track only certain branches instead of all branches.

This doesn't allow you to import a branch into Diffusion but ignore it in
Differential. Supporting that is somewhat technically complicated because the
parser currently goes like this:

  - Look at HEAD of all branches.
  - For any commits we haven't seen before, follow them back to something we
have seen (or the root).
  - "Discover" everything new.

Since this doesn't track <branch, commit> pairs, we currently don't have enough
information to tell when a commit appears in a branch for the first time, so we
don't have anywhere we can put a test for whether that branch is tracked and do
the Differential hook only if it is.

However, I think this cruder patch satisfies most of the need and is simple and
obvious in its implementation.

See also D1263.

Test Plan:
  - Updated a Git repository with various filters: "", "master, remote", "derp",
"  ,,, master   ,,,,,"
  - Edited SVN and Mercurial repositories to verify they didn't get caught in
the crossfire.
  - Ran daemon in debug mode on libphutil with filter "derp", got exception
about no tracked branches. Ran with filter "master", got tracking. Ran with no
filter, got tracking.
  - Looked at Diffusion with "derp" and "master", saw no branches and "master"
respectively.
  - Added unit tests to cover filtering logic.

Reviewers: btrahan, jungejason, nh, fratrik

Reviewed By: fratrik

CC: aran, fratrik, epriestley

Maniphest Tasks: T270

Differential Revision: https://secure.phabricator.com/D1290
2012-01-04 10:20:34 -08:00
epriestley
efb0fa739f Make tracked git repositories use an implicit 'origin' remote
Summary:
See T624. I originally wrote this to require an explicit remote, but this
creates an ugly "origin:" in all the URIs and makes T270 more difficult.

Treat all branch names as implying 'origin/'.

Test Plan:
  - Pulled and imported a fresh copy of libphutil without issues.
  - Browsed various git repositories.
  - Browsed Javelin's various branches.
  - Ran upgrade script, got a bunch of clean 'origin/master' -> 'master'
conversions.
  - Tried to specify an explicit remote in a default branch name.
  - Unit tests.

Reviewers: nh, jungejason, btrahan

Reviewed By: btrahan

CC: aran, btrahan

Maniphest Tasks: T624

Differential Revision: https://secure.phabricator.com/D1269
2011-12-29 08:35:32 -08:00
epriestley
abd8efc358 Further relax same origin checks
Summary: Allow paths to match even if they differ by trailing slashes and
".git".

Test Plan: Ran unit tests.

Reviewers: jungejason, btrahan

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T710

Differential Revision: https://secure.phabricator.com/D1286
2011-12-26 12:01:59 -08:00
epriestley
e786c44b6f Relax the same-origin check in the Git commit discovery daemon
Summary:
Git accepts either "git@x:/path" or "ssh://git@x.com/path" URIs to mean the
exact same thing, which is causing some false positives and confusion,
particularly because we sometimes mutate URIs.

Since this is just a sanity check, we don't really care about the username,
domain or credentials -- matching the paths is good enough. We're just trying to
make it hard to shoot yourself in the foot by copy-pasting the same local path
into two repositories and forgetting to change one, like I did. :P

Relax the check to only verify the paths are the same.

Test Plan:
  - Ran unit tests, which should fully cover things.
  - Ran commit discovery daemon in debug mode on incorrectly and correctly
configured repositories.

Reviewers: ajtrichards, jungejason, btrahan

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T710

Differential Revision: https://secure.phabricator.com/D1279
2011-12-24 08:54:44 -08:00
epriestley
3b65864ee1 Fix two minor bugs with recent patches:
- Use the computed remote URI (which may have an explicit 'ssh://' under Git in some cases).
  - Use '$id' correctly rather than casting the URI to an int in the message parser.
2011-12-22 06:57:04 -08:00
epriestley
92f9741779 Add a safety check to commit discovery daemon
Summary:
Although I couldn't repro the issue in T692, I did manage to point the "Diviner"
repository at the "Phabricator" working copy and screw some stuff up on
secure.phabricator.com.

Before discovering commits in a repository, ensure the 'origin' remote points at
the configured URI. This prevents issues where the working copy gets configured
to point at an existing (but incorrect) checkout.

Test Plan:
  - Ran gitcommitdiscovery daemon normally under "phd debug", saw it execute the
"remote show -n" command and then start working.
  - Intentionally botched the config, got an exception:

  (Exception) Working copy '/INSECURE/repos/phabricator' has origin URL
  'ssh://git@github.com/facebook/phabricator.git', but the configured URL
  'git://github.com/facebook/diviner.git' is expected. Refusing to proceed.

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T692

Differential Revision: 1253
2011-12-20 19:52:08 -08:00
epriestley
b5c9b9d059 Use remote credentials for 'git fetch' and 'hg pull' commands
Summary: These are "local" commands, but need remote credentials. If the daemon
runs as a user who does not have credentials, the initial clone will work but
subsequent updates will fail.

Test Plan:
  - Nuked a local copy of a Git repo.
  - Ran "phd debug fetch <phid>" as root (or any other user with no natural SSH
keys). Verified initial clone worked (since it passes credentials to the command
correctly).
  - Killed daemon, re-ran, verified "fetch" failed (no credentials passed).
  - Applied this patch.
  - Re-ran "phd debug fetch <phid>", verified it passed credentials and
succeeded.
  - Did all these steps for a Mercurial repo.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran, btrahan

Maniphest Tasks: T686

Differential Revision: 1236
2011-12-20 08:28:38 -08:00
Edward Speyer
6737ae4828 Run a commit discovery daemon only once
Summary:
Discover commits then return; useful when initializing new repositories
in unit tests.

By which I mean "when initializing a new repository in my unit test that
I'm working on".

Test Plan: Using this in a PhabricatorTestCase.

Reviewers: epriestley, aran

Reviewed By: epriestley

CC: aran, edward, epriestley

Differential Revision: 948
2011-10-25 14:25:40 -07:00