1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-23 15:22:41 +01:00
Commit graph

1340 commits

Author SHA1 Message Date
James Rhodes
d8d1173f52 Implement support for leasing from Drydock hosts in Harbormaster
Summary:
This adds LeaseHostBuildStepImplementation for getting leases on hosts in Drydock via Harbormaster.  It stores the resulting lease in an artifact.

There is also a few bug fixes as well.

Test Plan: Created a build plan with a "Lease Host" build step.  Ran the build plan and saw the build pass and the artifact in the database.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T1049, T4111

Differential Revision: https://secure.phabricator.com/D7706
2013-12-05 12:46:23 +11:00
James Rhodes
53250d84df Introduce HarbormasterBuildTarget to snapshot build steps through a build
Summary: This implements build targets as outlined in D7582.  Build targets represent an instance of a build step particular to the build.  Logs and artifacts have been adjusted to attach to build targets instead of build / build step pairs.

Test Plan: Ran builds and clicked around the interface.  Everything seemed to work.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T4111, T1049

Differential Revision: https://secure.phabricator.com/D7703
2013-12-05 12:01:12 +11:00
epriestley
d2e9aee16d Reject dangerous changes in Git repositories by default
Summary: Ref T4189. This adds a per-repository "dangerous changes" flag, which defaults to off. This flag must be enabled to do non-appending branch mutation (delete branches / rewrite history).

Test Plan:
With flag on and off, performed various safe and dangerous pushes.

  >>> orbital ~/repos/POEMS $ git push origin :blarp
  remote: +---------------------------------------------------------------+
  remote: |      * * * PUSH REJECTED BY EVIL DRAGON BUREAUCRATS * * *     |
  remote: +---------------------------------------------------------------+
  remote:             \
  remote:              \                    ^    /^
  remote:               \                  / \  // \
  remote:                \   |\___/|      /   \//  .\
  remote:                 \  /V  V  \__  /    //  | \ \           *----*
  remote:                   /     /  \/_/    //   |  \  \          \   |
  remote:                   @___@`    \/_   //    |   \   \         \/\ \
  remote:                  0/0/|       \/_ //     |    \    \         \  \
  remote:              0/0/0/0/|        \///      |     \     \       |  |
  remote:           0/0/0/0/0/_|_ /   (  //       |      \     _\     |  /
  remote:        0/0/0/0/0/0/`/,_ _ _/  ) ; -.    |    _ _\.-~       /   /
  remote:                    ,-}        _      *-.|.-~-.           .~    ~
  remote:   \     \__/        `/\      /                 ~-. _ .-~      /
  remote:    \____(Oo)           *.   }            {                   /
  remote:    (    (--)          .----~-.\        \-`                 .~
  remote:    //__\\  \ DENIED!  ///.----..<        \             _ -~
  remote:   //    \\               ///-._ _ _ _ _ _ _{^ - - - - ~
  remote:
  remote:
  remote: DANGEROUS CHANGE: The change you're attempting to push deletes the branch 'blarp'.
  remote: Dangerous change protection is enabled for this repository.
  remote: Edit the repository configuration before making dangerous changes.
  remote:
  To ssh://dweller@localhost/diffusion/POEMS/
   ! [remote rejected] blarp (pre-receive hook declined)
  error: failed to push some refs to 'ssh://dweller@localhost/diffusion/POEMS/'

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, chad, richardvanvelzen

Maniphest Tasks: T4189

Differential Revision: https://secure.phabricator.com/D7689
2013-12-03 10:28:39 -08:00
James Rhodes
ba16df0fed Restructure Drydock so that blueprints are instances in the DB
Summary:
//(this diff used to be about applying policies to blueprints)//

This restructures Drydock so that blueprints are instances in the DB, with an associated implementation class.  Thus resources now have a `blueprintPHID` instead of `blueprintClass` and DrydockBlueprint becomes a DAO.  The old DrydockBlueprint is renamed to DrydockBlueprintImplementation, and the DrydockBlueprint DAO has a `blueprintClass` column on it.

This now just implements CAN_VIEW and CAN_EDIT policies for blueprints, although they are probably not enforced in all of the places they could be.

Test Plan: Used the `create-resource` and `lease` commands.  Closed resources and leases in the UI.  Clicked around the new and old lists to make sure everything is still working.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T4111, T2015

Differential Revision: https://secure.phabricator.com/D7638
2013-12-03 11:09:07 +11:00
epriestley
618b5cbbc4 Install pre-commit hooks in Git repositories
Summary:
Ref T4189. T4189 describes most of the intent here:

  - When updating hosted repositories, sync a pre-commit hook into them instead of doing a `git fetch`.
  - The hook calls into Phabricator. The acting Phabricator user is sent via PHABRICATOR_USER in the environment. The active repository is sent via CLI.
  - The hook doesn't do anything useful yet; it just veifies basic parameters, does a little parsing, and exits 0 to allow the commit.

Test Plan:
  - Performed Git pushes and pulls over SSH and HTTP.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4189

Differential Revision: https://secure.phabricator.com/D7682
2013-12-02 15:45:36 -08:00
James Rhodes
d571507651 Update a few things in Drydock for policies
Summary: DrydockResource has been updated to be policy-aware (although there are no policy columns).

Test Plan: Clicked around in Drydock, viewed resources and leases, everything still seemed to work.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T3605, T4111

Differential Revision: https://secure.phabricator.com/D7595
2013-11-28 08:32:51 +11:00
epriestley
8b19465a9d Render linked JIRA issues with a Doorkeeper tag
Summary: Fixes T3687. Instead of rendering "JIRA Issues" in Differential using plain links, render them using Doorkeeper tags so they get the nice "enhance with object name" effect.

Test Plan: {F84886}

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T3687

Differential Revision: https://secure.phabricator.com/D7648
2013-11-25 14:54:35 -08:00
epriestley
4b91c4f7ae Add UI for defining repository mirrors
Summary:
Ref T4038. This adds everything except the actual pushing part for mirrors.

This isn't the most beautiful or sophisticated UI, but I want get the authoritative repositories self-hosted and get users beta-ing hosting as soon as possible. We can do transactions, etc., later on.

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4038

Differential Revision: https://secure.phabricator.com/D7632
2013-11-22 15:23:50 -08:00
epriestley
61b26255bb Add "PassphraseKey" classes for code which needs to actually use credentials
Summary: Ref T4122. These classes provide typed, checked access to credentials, so you can say "give me this password, and throw if anything is funky".

Test Plan: Used in next revision.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4122

Differential Revision: https://secure.phabricator.com/D7625
2013-11-22 15:23:10 -08:00
James Rhodes
7c3cb5948c Drydock blueprint for preallocated remote hosts
Summary:
This adds a Drydock blueprint for preallocated, remote hosts.  This will be used by the Harbormaster interface to allow users to specify remote hosts that builds can be run on.

This adds a `canAllocateResource` method to Drydock blueprints; it is used to detect whether a blueprint can allocate a resource for the given type and attributes.

Test Plan:
Ran:

```
bin/drydock lease --type host --attributes remote=true,preallocated=true,host=192.168.56.101,port=22,user=james,keyfile=,path=C:\\Build\\,platform=windows
```

and saw the "C:\Build\<id>" folder appear on the remote Windows machine.  Viewed the lease and resource in Drydock as well.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran, jamesr

Maniphest Tasks: T4111

Differential Revision: https://secure.phabricator.com/D7593
2013-11-22 14:34:10 -08:00
epriestley
13275860b1 When stopping on redirect, show a full stack trace
Summary: Ref T4140. Provide more debugging information so we can figure out what's going on with redirect loops.

Test Plan: {F83868}

Reviewers: chad, btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4140

Differential Revision: https://secure.phabricator.com/D7620
2013-11-21 14:38:29 -08:00
epriestley
8f715d8edf Add a credential selection control to Passphrase
Summary: Ref T4122. Adds a control for choosing credentials.

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, lsave

Maniphest Tasks: T4122

Differential Revision: https://secure.phabricator.com/D7617
2013-11-21 12:35:36 -08:00
Bob Trahan
7b718bb033 Nuance - federate out the design of NuanceSource via NuanceSourceDefinition
Summary: ...and get the basic edit flow "working" for a new NuanceSourceDefinition - the Phabricator Form. ...and fix a dumb bug in the query class so when you redirect to the view page / try to edit an existing NuanceSource you don't fatal.

Test Plan: played around with the edit form and it worked!

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7585
2013-11-20 13:41:19 -08:00
epriestley
91d084624b Passphrase v0
Summary:
Ref T4122. Implements a credential management application for the uses described in T4122.

@chad, this needs an icon, HA HA HAHA HA BWW HA HA HA

bwahaha

Test Plan: See screenshots.

Reviewers: btrahan, chad

Reviewed By: btrahan

CC: chad, aran

Maniphest Tasks: T4122

Differential Revision: https://secure.phabricator.com/D7608
2013-11-20 09:13:35 -08:00
Aviv Eyal
dcf909ba56 Land to GitHub + support stuff
Summary:
A usable, Land to GitHub flow.

Still to do:
- Refactor all git/hg stratagies to a sane structure.
- Make the dialogs Workflow + explain why it's disabled.
- Show button and request Link Account if GH is enabled, but user is not linked.
- After refreshing token, user ends up in the settings stage.

Hacked something in LandController to be able to show an arbitrary dialog from a strategy.
It's not very nice, but I want to make some more refactoring to the controller/strategy/ies anyway.

Also made PhabricatorRepository::getRemoteURIObject() public, because it was very useful in getting
the domain and path for the repo.

Test Plan:
Went through these flows:
- load revision in hosted, github-backed, non-github backed repos to see button as needed.
- hit land with weak token - sent to refresh it with the extra scope.
- Land to repo I'm not allowed - got proper error message.
- Successfully landed; Failed to apply patch.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T182

Differential Revision: https://secure.phabricator.com/D7555
2013-11-13 17:25:24 -08:00
epriestley
f5ca647d2c Add bin/repository edit for CLI repository editing
Summary:
Ref T4039. This is mostly to deal with that, to prevent the security issues associated with mutable local paths. The next diff will lock them in the web UI.

I also added a confirmation prompt to `bin/repository delete`, which was a little scary without one.

See one comment inline about the `--as` flag. I don't love this, but when I started adding all the stuff we'd need to let this transaction show up as "Administrator" it quickly got pretty big.

Test Plan: Ran `bin/repository edit ...`, saw an edit with a transaction show up on the web UI.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4039

Differential Revision: https://secure.phabricator.com/D7579
2013-11-13 11:26:05 -08:00
epriestley
c8320923c4 Implement most of the administrative UI for approval queues
Summary:
Nothing fancy here, just:

  - UI to show users needing approval.
  - "Approve" and "Disable" actions.
  - Send "Approved" email on approve.
  - "Approve" edit + log operations.
  - "Wait for Approval" state for users who need approval.

There's still no natural way for users to end up not-approved -- you have to write directly to the database.

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7573
2013-11-13 11:24:18 -08:00
epriestley
85f505465e Support serving SVN repositories over SSH
Summary:
Ref T2230. The SVN protocol has a sensible protocol format with a good spec here:

http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_ra_svn/protocol

Particularly, compare this statement to the clown show that is the Mercurial wire protocol:

> It is possible to parse an item without knowing its type in advance.

WHAT A REASONABLE STATEMENT TO BE ABLE TO MAKE ABOUT A WIRE PROTOCOL

Although it makes substantially more sense than Mercurial, it's much heavier-weight than the Git or Mercurial protocols, since it isn't distributed.

It's also not possible to figure out if a request is a write request (or even which repository it is against) without proxying some of the protocol frames. Finally, several protocol commands embed repository URLs, and we need to reach into the protocol and translate them.

Test Plan: Ran various SVN commands over SSH (`svn log`, `svn up`, `svn commit`, etc).

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7556
2013-11-11 12:19:06 -08:00
epriestley
8840f60218 Enable Mercurial reads and writes over SSH
Summary:
Ref T2230. This is substantially more complicated than Git, but mostly because Mercurial's protocol is a like 50 ad-hoc extensions cobbled together. Because we must decode protocol frames in order to determine if a request is read or write, 90% of this is implementing a stream parser for the protocol.

Mercurial's own parser is simpler, but relies on blocking reads. Since we don't even have methods for blocking reads right now and keeping the whole thing non-blocking is conceptually better, I made the parser nonblocking. It ends up being a lot of stuff. I made an effort to cover it reasonably well with unit tests, and to make sure we fail closed (i.e., reject requests) if there are any parts of the protocol I got wrong.

A lot of the complexity is sharable with the HTTP stuff, so it ends up being not-so-bad, just very hard to verify by inspection as clearly correct.

Test Plan:
  - Ran `hg clone` over SSH.
  - Ran `hg fetch` over SSH.
  - Ran `hg push` over SSH, to a read-only repo (error) and a read-write repo (success).

Reviewers: btrahan, asherkin

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7553
2013-11-11 12:18:27 -08:00
epriestley
f2938bacd9 Generalize SSH passthru for repository hosting
Summary:
Ref T2230. In Git, we can determine if a command is read-only or read/write from the command itself, but this isn't the case in Mercurial or SVN.

For Mercurial and SVN, we need to proxy the protocol that's coming over the wire, look at each request from the client, and then check if it's a read or a write. To support this, provide a more flexible version of `passthruIO`.

The way this will work is:

  - The SSH IO channel is wrapped in a `ProtocolChannel` which can parse the the incoming stream into message objects.
  - The `willWriteCallback` will look at those messages and determine if they're reads or writes.
    - If they're writes, it will check for write permission.
    - If we're good to go, the message object is converted back into a byte stream and handed to the underlying command.

Test Plan: Executed `git clone`, `git clone --depth 3`, `git push` (against no-write repo, got error), `git push` (against valid repo).

Reviewers: btrahan

Reviewed By: btrahan

CC: hach-que, asherkin, aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7551
2013-11-11 12:12:21 -08:00
James Rhodes
79ef667dfd Render build status on revisions and commits
Summary:
This uses an event listener to render the status of builds on their buildables.  The revision and commit view now renders out the status of each of the builds.

Currently the revision controller has the results for the latest diff rendered out.  We might want to show the status of previous diffs in the future, but for now I think the latest diff should do fine.

There's also a number of bug fixes in this diff, including a particularly nasty one where builds would have a build plan PHID generated for them, which resulted in handle lookups always returning invalid objects.

Test Plan: Ran builds against diffs and commits, saw them appear on the revision and commit view controllers.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7544
2013-11-09 15:04:00 -08:00
epriestley
56c65e33ef Add a "Make an HTTP Request" build step
Summary:
Ref T1049. This is very minimal, but does what it says.

I merged the variable replacement code so Remote + HTTP can share more stuff.

Test Plan:
Ran "HTTP" and "Remote" build plans.

{F79886}

{F79887}

Reviewers: hach-que, btrahan

Reviewed By: hach-que

CC: zeeg, aran

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7541
2013-11-09 07:16:12 -08:00
epriestley
f357e159b1 Liberate. 2013-11-08 18:19:48 -08:00
James Rhodes
0ac1be7094 Implemented support for build logs
Summary:
Depends on D7519.

This implements support for build logs in Harbormaster.  This includes support for appending to a log from the "Run Remote Command" build step.

It also adds the ability to cancel builds.

Currently the build view page doesn't update the logs live; I'm sure this can be achieved with Javelin, but I don't have enough experience with Javelin to actually make it poll from updates to content in the background.

{F79151}

{F79153}

{F79150}

{F79152}

Test Plan:
Tested this by setting up SSH on a Windows machine and using a Remote Command configured with:

```
C:\Windows\system32\cmd.exe /C cd C:\Build && mkdir Build_${timestamp} && cd Build_${timestamp} && git clone --recursive https://github.com/hach-que/Tychaia.git && cd Tychaia && Protobuild.exe && C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe Tychaia.Windows.sln
```

and observed the output of the build stream from the Windows machine into Phabricator.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7521
2013-11-08 18:15:07 -08:00
Bob Trahan
3de4e91dac Naunce - capalities for Source object
Summary: another little piece here that basically just adds some permissions to source editing. serving it up before I do anything too complicated to make sure it seems kosher. in terms of what comes next this form needs to be dynamic based on source type so there'll be some fun there. That said, I plan to implement a more simple "phabricator form" only version to start here and flesh out a few other things like queues with that.

Test Plan: set permission to no one for source edit and got a nice error page.

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7535
2013-11-08 12:45:14 -08:00
Asher Baker
d700e7f22d Add support for landing to hosted Mercurial repos.
Summary: I've kept this as close as possible to the Git version for ease of review and later refactoring of them both together. At minimum, the functions to get the working dir should probably be cleaned up one day.

Test Plan: Landed a revision.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7534
2013-11-08 11:39:32 -08:00
epriestley
6324669748 Allow Phabricator to serve Mercurial repositories over HTTP
Summary: Ref T2230. This is easily the worst thing I've had to write in a while. I'll leave some notes inline.

Test Plan: Ran `hg clone http://...` on a hosted repo. Ran `hg push` on the same. Changed sync'd both ways.

Reviewers: asherkin, btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7520
2013-11-06 18:00:42 -08:00
epriestley
44a40eaf57 Split Diffusion VSC serve code into its own controller
Summary: This is starting to get a bit sizable and it turns out Mercurial is sort of a beast, so split the VCS serve stuff into a separate controller.

Test Plan: Pushed and pulled an authenticated Git repository.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, hach-que

Differential Revision: https://secure.phabricator.com/D7494
2013-11-06 17:55:46 -08:00
Bob Trahan
66ae64f7bc Nuance - get some scaffolding up there
Summary:
I updated the wiki too - https://secure.phabricator.com/w/projects/pebkac/ - with what I am thinking right now. Rough plan here is

 - next diff:
  - implement editors and transactions
  - implement "web type" for contact source
   - /pebkac/item/new/ will be the entry point for this
  - implement "actions" on a contact
  - probably some "polish" on the scaffolding laid out here; like "create" permissions maybs
 - diffs after that:
  - implement "twitter" type for source
  - implement email reply handler stuff for item and source

Probs a great time to blast huge holes in all this stuff. :D

Test Plan: these pages load and arc lint doesn't complain

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran, chad

Differential Revision: https://secure.phabricator.com/D7465
2013-11-06 17:00:09 -08:00
epriestley
4f0f95f7b5 Assign PHIDs to all diffs
Summary:
Ref T1049. Ref T2222. `DifferentialDiff` does not currently have a PHID, but we need it for Harbormaster and ApplicationTransactions. See some discussion in D7501.

(I split the SQL into two sections so we can't fail in the middle. At some point, I'd like to do a pass on the migration stuff and get this happening automatically, and also simplify the PatchList.)

Test Plan:
  - Ran `bin/storage upgrade`.
  - Checked for valid PHIDs in the database.
  - Used `phid.query` to look up a diff by PHID.
  - Created a new diff and verified it got a PHID.

Reviewers: btrahan, hach-que

Reviewed By: btrahan

CC: aran, vrana

Maniphest Tasks: T2222, T1049

Differential Revision: https://secure.phabricator.com/D7513
2013-11-06 13:59:06 -08:00
epriestley
bd29784a32 Add an administrative bin/repository importing command to list importing commits
Summary: Ref T4068. Adds a command to list all commits in an "importing" status. This will allow users to use `reparse.php` to diagnose and repair issues.

Test Plan:
  - Ran `bin/repository importing P`, etc.
  - Used `reparse.php` to reparse some commit stages and saw status update correctly.
  - Ran on a repo with no importing commits.
  - Ran with `... --simple | xargs`, which saves us having to put an `awk` or something in there for users.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4068

Differential Revision: https://secure.phabricator.com/D7515
2013-11-06 11:26:41 -08:00
epriestley
e3a5ab1f8c Add an administrative bin/repository mark-imported command
Summary:
Ref T4068. In some cases like that one, I anticipate a repository not fully importing when a handful of random commits are broken. In the long run we should just deal with that properly, but in the meantime provide an administrative escape hatch so you can mark the repository as imported and get it running normally.

The major reason to do this is that Herald, Feed, Harbormaster, etc., won't activate until a repository is "imported".

Test Plan:
  - Tried to mark an imported repository as imported, got an "already imported" message.
  - Same for not-imported.
  - Marked a repository not-imported.
  - Marked a repository imported.
  - Marked a repository not-imported, then waited for the daemons to mark it imported again automatically.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, kbrownlees

Maniphest Tasks: T4068

Differential Revision: https://secure.phabricator.com/D7514
2013-11-06 11:26:24 -08:00
James Rhodes
e4569e7e7e Implement interface for adding, editing and deleting build steps on plans.
Summary: This implements an interface for adding new build steps, editing existing build steps and deleting build steps from build plans.  It uses the settings definitions on the build implementation to work out what fields should be displayed on the edit page.

Test Plan:
See screenshots:

{F78529}

{F78532}

{F78528}

{F78531}

{F78527}

{F78530}

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7500
2013-11-05 14:08:44 -08:00
James Rhodes
c514d34b94 Add build step implementation infrastructure and sleep build step.
Summary:
Depends on D7498.

This implements support for a "build step implementation".  Build steps have an associated class name (which makes the class in PHP) and a details field, which is serialized JSON (same as PhabricatorRepository).

This also implements a SleepBuildStepImplementation which just pauses the build for a specified period of seconds.

Test Plan:
Inserted a build step with `insert into harbormaster_buildstep (phid, buildPlanPHID, className, details, dateCreated, dateModified) values ('', 'PHID-HMCP-zkh5w6czfbfpk2gxwdeo', 'SleepBuildStepImplementation', '{"seconds":5}', NOW(), NOW());` (adjusting the build plan PHID as appropriate).

Started the daemon and applied the build plan to a buildable, and saw the daemon take a 5 second delay after creating `SleepBuildStepImplementation`.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran, chad

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7499
2013-11-05 13:34:44 -08:00
Aviv Eyal
5c0edc9351 Land Revision button for hosted git repos
Summary:
ref T182.

Simple approach of clone, patch, push. While waiting for drydock, implement a hackish mutex
setup for the workspace, which should work ok as long as there's only one committer who is
carefull about theses things.

Less obvious note: This is taking the both author and commiter's 'primary email' for the commit -
which might rub some people wrong.

Test Plan:
With a hosted repo, created some diffs and landed them.
Also clicked button for some error cases, got the right error message.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: hach-que, Korvin, epriestley, aran

Maniphest Tasks: T182

Differential Revision: https://secure.phabricator.com/D7486
2013-11-05 13:00:13 -08:00
James Rhodes
ca5400d14b Implement basic Harbormaster daemon and start builds.
Summary: This implements a basic Harbormaster daemon that takes pending builds and builds them (currently just sleeps 15 seconds before moving to passed state).  It also implements an interface to apply a build plan to a buildable, so that users can kick off builds for a buildable.

Test Plan: Ran `bin/phd debug PhabricatorHarbormasterBuildDaemon` and used the interface to start some builds by applying a build plan.  Observed them move from 'pending' to 'building' to 'passed'.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7498
2013-11-05 12:48:36 -08:00
epriestley
4f20530856 Merge "expandshortcommitquery" and "stablecommitnamequery" into "resolverefs"
Summary:
Ref T1493. Diffusion has some garbagey behavior for things we can't resolve. Common cases are:

  - Looking at a branch that doesn't exist.
  - Looking at a repository with no branches.
  - Looking at a commit that doesn't exist.
  - Looking at an empty repository.

In these cases, we generally fatal unhelpfully. I want to untangle this mess.

This doesn't help much, but does clean things up a bit. We currently have two separate query paths, "stablecommitname" and "expandshortcommit". These are pretty much doing the same thing -- taking some ref like "master" or "default" or a tag name or part of a commit name, and turning it into a full commit name. Merge them into a single "resolverefs" method.

This simplifies the code a fair bit, and gives us better error messages. They still aren't great, but they're like this now:

  Ref "7498aec194ecf2d333e0e2baddd9d5cdf922d7f1" is ambiguous or does not exist.

...instead of just:

  ERR-INVALID-COMMIT

Test Plan: Looked at Git, Mercurial and Subversion repositories that were empty and non-empty. Looked at branches/heads. Tried to look at invalid commits. Looked at tags. All of this still works, and some behaviors are a bit better than they used to be.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1493

Differential Revision: https://secure.phabricator.com/D7484
2013-11-04 14:13:07 -08:00
epriestley
b90e51ab0e Clean up hg --debug branches calls
Summary: Ref T1493. Consolidate these a bit; they might need some more magic once we do `--noupdate` checkouts. Mostly just trying to clean up and centralize this code a bit.

Test Plan: Viewed and `bin/repository discover`'d Mercurial repos with and without any branches.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1493

Differential Revision: https://secure.phabricator.com/D7480
2013-11-04 12:15:32 -08:00
James Rhodes
24579e2956 Add phame.queryblog and phame.querypost Conduit calls.
Summary: This implements Conduit calls for querying Phame blogs and Phame posts.

Test Plan: Made some calls and they seem to generally work.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T3695

Differential Revision: https://secure.phabricator.com/D7478
2013-11-02 16:31:30 -07:00
epriestley
cd674931fc When creating a repository in Diffusion, prompt for "Create" or "Import" first
Summary:
Ref T2230. This will need some more refinement, but basically it adds a "Create" vs "Import" step before we go through the paged workflow.

  - If you choose "Create", we skip the remote URI / auth stuff, and then set the "hosted" flag.
  - If you choose "Import", we do what we do now.

Test Plan: Created and imported repos.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7475
2013-11-01 17:39:35 -07:00
epriestley
0278b15ceb Implementation of VCS passwords against user.
Summary: This allows users to set their HTTP access passwords via Diffusion interface.

Test Plan: Clicked the "Set HTTP Access Password" link, set a password and saw it appear in the DB.

Reviewers: #blessed_reviewers, hach-que, btrahan

Reviewed By: hach-que

CC: Korvin, epriestley, aran, jamesr

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7462
2013-11-01 08:34:11 -07:00
epriestley
40b0818207 Show additional status information during repository import
Summary:
Ref T2350. Fixes T2231.

  - Adds log flags around discovery.
  - Adds message flags for "needs update". This is basically an out-of-band hint to the daemons that a repository should be pulled sooner than normal. We set the flag when users push a revision, and expose a Conduit method that `arc land` will be able to use.

Test Plan: See screenshots.

Reviewers: btrahan, chad

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2350, T2231

Differential Revision: https://secure.phabricator.com/D7467
2013-10-31 15:46:57 -07:00
epriestley
3a39b01233 Add "RepositoryStatusMessage" and detailed information about initilization
Summary:
`RepositoryStatusMessage` is basically a key/value table associated with a repository that I'm using to let the daemons store the most recent event of a given type, so we can easily show it on the status dashboard. I think this will be a lot easier for users to figure out than digging through logfiles.

I'm also going to write the "this needs a pull" status here eventually, for reducing the time lapse between pushes and discovery.

  - Add storage for these messages.
  - Have the pull engine populate the INIT phase. I'll do the FETCH phase next.
  - Update the status readout to show all the various states.

Test Plan: See screenshots.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7461
2013-10-30 16:04:19 -07:00
epriestley
f08908ff35 Raise a setup warning for missing or invalid local repository directory
Summary: I'm planning to add more detailed info to Diffusion itself, but catch the big issue here.

Test Plan: Hit config issue locally, then resolved it.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7439
2013-10-30 13:07:09 -07:00
epriestley
3bf372c60c Consolidate querying of things which we can use git for-each-ref for
Summary: Ref T2230. This cleans up D7442, by using `git for-each-ref` everywhere we can, in a basically reasonable way.

Test Plan:
In bare and non-bare repositories:

  - Ran discovery with `bin/repository discover`;
  - listed branches on `/diffusion/X/`;
  - listed tags on `/diffusion/X/`;
  - listed tags, branches and refs on `/diffusion/rXnnnn`.

Reviewers: btrahan, avivey

Reviewed By: avivey

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7447
2013-10-30 13:06:09 -07:00
epriestley
c749fcc192 Reject SCP-style URIs with explicit protocols in Diffusion
Summary:
Fixes T3619. These URIs are valid:

  git@domain.com:/path        (Git SCP-style implicit SSH)
  ssh://git@domain.com/path   (Explicit SSH)

This URI, arrived at by adding "ssh://" to the front of an SCP-style URI, is not:

  ssh://git@domain.com:/path

Detect URIs in this form and reject them. See T3619.

Test Plan:
{F75486}

Also set some valid URIs.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T3619

Differential Revision: https://secure.phabricator.com/D7431
2013-10-29 15:32:51 -07:00
epriestley
c6665b1907 Serve git writes over SSH
Summary: Looks like this is pretty straightforward; same as the reads except mark it as needing PUSH.

Test Plan: Ran `git push`, pushed over SSH to a hosted repo.

Reviewers: btrahan

Reviewed By: btrahan

CC: hach-que, aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7425
2013-10-29 15:32:41 -07:00
epriestley
7d9dfb561d Serve Git reads over HTTP
Summary: Mostly ripped from D7391. No writes yet.

Test Plan: Ran `git clone` against a local over HTTP, got a clone.

Reviewers: btrahan, hach-que

Reviewed By: hach-que

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7423
2013-10-29 15:32:40 -07:00
epriestley
bb4904553f Route some VCS connections over SSH
Summary:
  - Add web UI for configuring SSH hosting.
  - Route git reads (`git-upload-pack` over SSH).

Test Plan:
  >>> orbital ~ $ git clone ssh://127.0.0.1/
  Cloning into '127.0.0.1'...
  Exception: Unrecognized repository path "/". Expected a path like "/diffusion/X/".
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
  >>> orbital ~ $ git clone ssh://127.0.0.1/diffusion/X/
  Cloning into 'X'...
  Exception: No repository "X" exists!
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
  >>> orbital ~ $ git clone ssh://127.0.0.1/diffusion/MT/
  Cloning into 'MT'...
  Exception: This repository is not available over SSH.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
  >>> orbital ~ $ git clone ssh://127.0.0.1/diffusion/P/
  Cloning into 'P'...
  Exception: TODO: Implement serve over SSH.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.

Reviewers: btrahan

Reviewed By: btrahan

CC: hach-que, aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7421
2013-10-29 15:32:40 -07:00
epriestley
c7f23f522a Accept and route VCS HTTP requests
Summary:
Mostly ripped from D7391, with some changes:

  - Serve repositories at `/diffusion/X/`, with no special `/git/` or `/serve/` URI component.
    - This requires a little bit of magic, but I got the magic working for Git, Mercurial and SVN, and it seems reasonable.
    - I think having one URI for everything will make it easier for users to understand.
    - One downside is that git will clone into `X` by default, but I think that's not a big deal, and we can work around that in the future easily enough.
  - Accept HTTP requests for Git, SVN and Mercurial repositories.
  - Auth logic is a little different in order to be more consistent with how other things work.
  - Instead of AphrontBasicAuthResponse, added "VCSResponse". Mercurial can print strings we send it on the CLI if we're careful, so support that. I did a fair amount of digging and didn't have any luck with git or svn.
  - Commands we don't know about are assumed to require "Push" capability by default.

No actual VCS data going over the wire yet.

Test Plan:
Ran a bunch of stuff like this:

  $ hg clone http://local.aphront.com:8080/diffusion/P/
  abort: HTTP Error 403: This repository is not available over HTTP.

...and got pretty reasonable-seeming errors in all cases. All this can do is produce errors for now.

Reviewers: hach-que, btrahan

Reviewed By: hach-que

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7417
2013-10-29 15:32:40 -07:00