1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-18 12:52:42 +01:00
Commit graph

468 commits

Author SHA1 Message Date
epriestley
d804598f17 Add some of a billing daemon skeleton
Summary:
Ref T6881. This adds the worker, and a script to make it easier to test. It doesn't actually invoice anything.

I'm intentionally allowing the script to double-bill since it makes testing way easier (by letting you bill the same period over and over again), and provides a tool for recovery if billing screws up.

(This diff isn't very interesting, just trying to avoid a 5K-line diff at the end.)

Test Plan: Used `bin/phortune invoice ...` to get the worker to print out some date ranges which it would theoretically invoice.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6881

Differential Revision: https://secure.phabricator.com/D11577
2015-01-30 11:29:05 -08:00
epriestley
8798083ad9 Proxy VCS SSH requests
Summary: Fixes T7034. Like HTTP, proxy requests to the correct host if a repository has an Almanac service host.

Test Plan: Ran VCS requests through the proxy.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7034

Differential Revision: https://secure.phabricator.com/D11543
2015-01-28 14:41:24 -08:00
epriestley
9b359affe7 Prepare SSH connections for proxying
Summary:
Ref T7034.

In a cluster environment, when a user connects with a VCS request over SSH (like `git pull`), the receiving server may need to proxy it to a server which can actually satisfy the request.

In order to proxy the request, we need to know which repository the user is interested in accessing.

Split the SSH workflow into two steps:

  # First, identify the repository.
  # Then, execute the operation.

In the future, this will allow us to put a possible "proxy the whole thing somewhere else" step in the middle, mirroring the behavior of Conduit.

This is trivially easy in `git` and `hg`. Both identify the repository on the commmand line.

This is fiendishly complex in `svn`, for the same reasons that hosting SVN was hard in the first place. Specifically:

  - The client doesn't tell us what it's after.
  - To get it to tell us, we have to send it a server capabilities string //first//.
  - We can't just start an `svnserve` process and read the repository out after a little while, because we may need to proxy the request once we figure out the repository.
  - We can't consume the client protocol frame that tells us what the client wants, because when we start the real server request it won't know what the client is after if it never receives that frame.
  - On the other hand, we must consume the second copy of the server protocol frame that would be sent to the client, or they'll get two "HELLO" messages and not know what to do.

The approach here is straightforward, but the implementation is not trivial. Roughly:

  - Start `svnserve`, read the "hello" frame from it.
  - Kill `svnserve`.
  - Send the "hello" to the client.
  - Wait for the client to send us "I want repository X".
  - Save the message it sent us in the "peekBuffer".
  - Return "this is a request for repository X", so we can proxy it.

Then, to continue the request:

  - Start the real `svnserve`.
  - Read the "hello" frame from it and throw it away.
  - Write the data in the "peekBuffer" to it, as though we'd just received it from the client.
  - State of the world is normal again, so we can continue.

Also fixed some other issues:

  - SVN could choke if `repository.default-local-path` contained extra slashes.
  - PHP might emit some complaints when executing the commit hook; silence those.

Test Plan: Pushed and pulled repositories in SVN, Mercurial and Git.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7034

Differential Revision: https://secure.phabricator.com/D11541
2015-01-28 10:18:07 -08:00
epriestley
834079f766 Pass cluster.instance to ssh-exec if it is defined
Summary: Ref T7034. This is a second special case, like commit hooks, where we need some help from Phabricator to make instance identity knowable.

Test Plan: Connected to an instance and ran SSH commands.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7034

Differential Revision: https://secure.phabricator.com/D11539
2015-01-28 10:17:40 -08:00
epriestley
d8550c114d Promote instance identity to the upstream and pass it to commit hooks
Summary:
Fixes T7019. In a cluster environment, pushes currently fail because the commit hook can't identify the instance.

For web processes, the hostname identifies the instance -- but we don't have a hostname in the hook.

For CLI processes, the environment identifies the instance -- but we don't have an environment in the hook under SVN.

Promote the instance identifier into the upstream and pack/unpack it explicitly for hooks. This is probably not useful for anyone but us, but the amount of special-purpose code we're introducing is very small.

I poked at trying to do this in a more general way, but:

  - We MUST know this BEFORE we run code, so the normal subclassing stuff is useless.
  - I couldn't come up with any other parameter which might ever be useful to pass in.

Test Plan: Used `git push` to push code through proxied HTTP, got a clean push.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7019

Differential Revision: https://secure.phabricator.com/D11495
2015-01-27 14:51:48 -08:00
Chad Little
32b85c065e Delete app icons
Summary: Unused at this point

Test Plan: Grep

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: hach-que, Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11506
2015-01-27 12:36:39 -08:00
Chad Little
6018ef91b8 Remove 1x AppIcons, use FontIcons instead
Summary: Removes the 1x application icons, and uses the fonticons instead. Feed was only known location.

Test Plan:
feed, dashboards, grep for use

{F275636}

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: hach-que, Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11496
2015-01-25 14:14:41 -08:00
epriestley
934df0e735 Add bin/trigger, for testing event triggers
Summary:
Ref T6881. This makes it easier to fire a trigger and make sure it works properly. You can use the `--now` flag to travel through time, and test scheduling conditions with `--last` and `--next`. It will tell you when the trigger would reschedule.

Better than waiting 24 hours to see if things work.

Test Plan: Fired some backups, got useful output which made me think my code probably works correctly.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6881

Differential Revision: https://secure.phabricator.com/D11438
2015-01-20 11:31:32 -08:00
Bob Trahan
648fa2e1bc Repositories - Move scripts/repository/reparse.php to bin/repository reparse
Summary:
Fixes T5966. Accomplishes a few things

 - see title
 - adds a force-autoclose flag and the plumbing for it
 - removes references to some HarborMaster thing that used to key off commits and seems long dead, but forgotten :/

Test Plan:
ran a few commands. These first three had great success:

`./repository reparse --all FIRSTREPO --message --change  --herald --owners`
`./repository reparse --all FIRSTREPO --message --change  --herald --owners --min-date yesterday`
`./repository reparse --all FIRSTREPO --message --change  --herald --owners --min-date yesterday --force-autoclose`

...and these next two showed me some errors as expected:

`./repository reparse --all FIRSTREPO --message --change  --herald --owners --min-date garbagedata`
`./repository reparse --all GARBAGEREPO --message --change  --herald --owners`

Also, made a diff in a repository with autoclose disabled and commited the diff. Later, reparse the diff with force-autoclose. Verified the diff closed and that the reason "why" had the proper message text.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: joshuaspence, epriestley, Korvin

Maniphest Tasks: T5966

Differential Revision: https://secure.phabricator.com/D10492
2015-01-06 11:42:15 -08:00
Joshua Spence
e12f0bb53a Correctly pass worker priority from reparse script
Summary: D11044 broke the `./scripts/repository/reparse.php` script.

Test Plan: `./scripts/repository/reparse.php --message --all X`

Reviewers: btrahan, #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11211
2015-01-05 08:34:05 +11:00
Joshua Spence
39ca2fdf64 Use new FutureIterator instead of Futures
Summary: Ref T6829. Deprecate the `Futures()` function.

Test Plan: N/A

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6829

Differential Revision: https://secure.phabricator.com/D11077
2014-12-30 23:13:38 +11:00
Chad Little
b925f42c55 Remove docs sprite, replace with FontAwesome
Summary: Removes the docs sprite in Conpherence with FontAwesome, adds additional icons. Unsure what happens if someone customized this config option.

Test Plan: Added images and files to a Conpherence, saw new icons.

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11028
2014-12-20 20:59:24 -08:00
Chad Little
1e0f1f4936 Remove payments sprite
Summary: Removes unused payments sprite and code, also some unused conpherence generated images. We use images in login (and could use FontAwesome, maybe).

Test Plan: grep codebase, pull up uiexamples icons page

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11025
2014-12-20 09:44:53 -08:00
Chad Little
f07f7bfaa4 Remove unused Conpherence sprites
Summary: These were refactored out a while ago

Test Plan: Grep codebase, use Conpherence on desktop, mobile.

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11023
2014-12-20 07:44:50 -08:00
Joshua Spence
54f8aa8cd9 Minor linter fixes
Summary: Self explanatory

Test Plan: `arc lint`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D10946
2014-12-09 18:37:32 +11:00
epriestley
7e1c312183 Add bin/worker flood, for flooding the task queue with work
Summary: Ref T6615. Ref T3554. We need better tooling around the queue eventually, so start here.

Test Plan: Added 100K+ tasks locally with `bin/worker flood`. Executed some of them with `bin/phd debug taskmaster` (we already have a TestWorker, used in unit tests).

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T3554, T6615

Differential Revision: https://secure.phabricator.com/D10894
2014-11-24 11:10:15 -08:00
epriestley
bf17b12daf Standardize SSH key storage
Summary:
Ref T5833. This fixes a few weird things with this table:

  - A bunch of columns were nullable for no reason.
  - We stored an MD5 hash of the key (unusual) but never used it and callers were responsible for manually populating it.
  - We didn't perform known-key-text lookups by using an index.

Test Plan:
  - Ran migrations.
  - Faked duplicate keys, saw them clean up correctly.
  - Added new keys.
  - Generated new keys.
  - Used `bin/auth-ssh` and `bin/auth-ssh-key`.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10805
2014-11-07 15:34:44 -08:00
epriestley
6f0d3b0796 Add a query/policy layer on top of SSH keys for Almanac
Summary:
Ref T5833. Currently, SSH keys are associated only with users, and are a bit un-modern. I want to let Almanac Devices have SSH keys so devices in a cluster can identify to one another.

For example, with hosted installs, initialization will go something like this:

  - A request comes in for `company.phacility.com`.
  - A SiteSource (from D10787) makes a Conduit call to Almanac on the master install to check if `company` is a valid install and pull config if it is.
  - This call can be signed with an SSH key which identifies a trusted Almanac Device.

In the cluster case, a web host can make an authenticated call to a repository host with similar key signing.

To move toward this, put a proper Query class on top of SSH key access (this diff). In following diffs, I'll:

  - Rename `userPHID` to `objectPHID`.
  - Move this to the `auth` database.
  - Provide UI for device/key association.

An alternative approach would be to build some kind of special token layer in Conduit, but I think that would be a lot harder to manage in the hosting case. This gives us a more direct attack on trusting requests from machines and recognizing machines as first (well, sort of second-class) actors without needing things like fake user accounts.

Test Plan:
  - Added and removed SSH keys.
  - Added and removed SSH keys from a bot account.
  - Tried to edit an unonwned SSH key (denied).
  - Ran `bin/ssh-auth`, got sensible output.
  - Ran `bin/ssh-auth-key`, got sensible output.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5833

Differential Revision: https://secure.phabricator.com/D10790
2014-11-06 12:37:02 -08:00
epriestley
917da08417 Fix various MySQL version issues with new charset stuff
Summary:
Ref T1191. Notable stuff:

  - Adds `--disable-utf8mb4` to `bin/storage` to make it easier to test what things will (approximately) do on old MySQL. This isn't 100% perfect but should catch all the major stuff. It basically makes us pretend the server is an old server.
  - Require utf8mb4 to dump a quickstart.
  - Fix some issues with quickstart generation, notably special casing the FULLTEXT handling.
  - Add an `--unsafe` flag to `bin/storage adjust` to let it truncate data to fix schemata.
  - Fix some old patches which don't work if the default table charset is utf8mb4.

Test Plan:
  - Dumped a quickstart.
  - Loaded the quickstart with utf8mb4.
  - Loaded the quickstart with `--disable-utf8mb4` (verified that we get binary columns, etc).
  - Adjusted schema with `--disable-utf8mb4` (got a long adjustment with binary columns, some truncation stuff with weird edge case test data).
  - Adjusted schema with `--disable-utf8mb4 --unsafe` (got truncations and clean adjust).
  - Adjusted schema back without `--disable-utf8mb4` (got a long adjustment with utf8mb4 columns, some invalid data on truncated utf8).
  - Adjusted schema without `--disable-utf8mb4`, but with `--unsafe` (got truncations on the invalid data).

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10757
2014-10-29 15:49:29 -07:00
Joshua Spence
3cf9a5820f Minor formatting changes
Summary: Apply some autofix linter rules.

Test Plan: `arc lint` and `arc unit`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D10585
2014-10-08 08:39:49 +11:00
James Rhodes
8fbebce501 Implement storage of a host ID and a public key for authorizing Conduit between servers
Summary:
Ref T4209.  This creates storage for public keys against authorized hosts, such that servers can be authorized to make Conduit calls as the omnipotent user.

Servers are registered into this system by running the following command once:

```
bin/almanac register
```

NOTE: This doesn't implement authorization between servers, just the storage of public keys.

Placing this against Almanac seemed like the most sensible place, since I'm imagining in future that the `register` command will accept more information (like the hostname of the server so it can be found in the service directory).

Test Plan: Ran `bin/almanac register` and saw the host (and public key information) appear in the database.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4209

Differential Revision: https://secure.phabricator.com/D10400
2014-10-03 22:52:41 +10:00
Joshua Spence
6f806bd12b Use PhutilConsoleProgressBar for the repository/reparse.php script
Summary: The output from this script is too verbose... all that I care about is the overall progress.

Test Plan:
Ran the script on a large repository.

```lang=bash
./scripts/repository/reparse.php --all XYZ --message
NOTE: This script will queue tasks to reparse the data. Once the tasks have been queued, you need to run Taskmaster daemons to execute them.

QUEUEING TASKS (92,969 Commits):
[                                                                  ]   0.1%
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D10441
2014-09-09 00:04:25 +10:00
epriestley
6be8d65763 Convert two missed phutil_utf8_shorten() callsites
Summary: Fixes T6006. These didn't get caught by D10392.

Test Plan: Forced migration to re-run; ran SSH commands against Phabricator.

Auditors: btrahan
2014-08-30 07:20:35 -07:00
epriestley
eaacb4a511 Replace ActionHeader minicons with Font icons
Summary:
Ref T5885. See D10276.

Currently, ActionHeaders can only have minicons, and we don't use them anywhere and they probably don't make much sense in the product anymore.

Instead, allow them to have font icons. Remove minicons, which have no callsites and probably won't in the future.

Test Plan:
{F190925}

  - Grepped for `minicons`.
  - Grepped for `setHeaderIcon()`.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T5885

Differential Revision: https://secure.phabricator.com/D10277
2014-08-15 11:08:28 -07:00
epriestley
237b1d769b Improve bin/storage upgrade behavior when run out-of-order
Summary:
Fixes T5770. This error occurs if you run `bin/storage upgrade` before you set up MySQL credentials.

This isn't what the setup guide says to do, but it's an easy mistake to make and should be a permitted install path since there's no reason you can't do things in this order.

Specifically, we use a mixture of "standard" (configured) and "administrative" (`--user` and `--password`) credentials, and if the standard ones are bogus bad things happen. We use the standard credentials to make some initialization order stuff easier, and because there's no `--host` flag and adding one would be silly, and because we only need administrative credentials to issue ALTER / CREATE statements.

Test Plan: Ran with bad standard credentials; ran with bad administrative credentials. Ran with good credentials.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5770

Differential Revision: https://secure.phabricator.com/D10199
2014-08-08 15:44:40 -07:00
Mukunda Modell
12aaa942ac Add a CanCDN flag to uploaded files
Summary:
CanCDN flag indicates that a file can be served + cached
via anonymous content distribution networks.

Once D10054 lands, any files that lack the CanCDN flag
will require a one-time-use token and headers will
prohibit cache to protect sensitive files from
unauthorized access.

This diff separates the CanCDN changes from the code that
enforces these restrictions in D10054 so that the changes
can be tested and refined independently.

Test Plan: Work in progress

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: rush898, qgil, epriestley, aklapper, Korvin

Maniphest Tasks: T5685

Differential Revision: https://secure.phabricator.com/D10166
2014-08-07 18:56:20 -07:00
Joshua Spence
9a679bf374 Allow worker tasks to have priorities
Summary: Fixes T5336. Currently, `PhabricatorWorkerLeaseQuery` is basically FIFO. It makes more sense for the queue to be a priority-queue, and to assign higher priorities to alerts (email and SMS).

Test Plan: Created dummy tasks in the queue (with different priorities). Verified that the priority field was set correctly in the DB and that the priority was shown on the `/daemon/` page. Started a `PhabricatorTaskmasterDaemon` and verified that the higher priority tasks were executed before lower priority tasks.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T5336

Differential Revision: https://secure.phabricator.com/D9871
2014-07-12 03:02:06 +10:00
Joshua Spence
97bb13d1d7 Discover workflows automatically.
Summary: Most scripts detect the relevant workflows automatically. Some scripts, however, use a hardcoded list of workflows.

Test Plan: Ran `./bin/aphlict`, `./bin/cache` and `./bin/phd`.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9564
2014-06-16 09:00:25 +10:00
Joshua Spence
d0128afa29 Applied various linter fixes.
Summary: Applied some more linter fixes that I previously missed because my global `arc` install was out-of-date.

Test Plan: Will run `arc unit` on another host.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9443
2014-06-09 16:04:12 -07:00
Joshua Spence
0a62f13464 Change double quotes to single quotes.
Summary: Ran `arc lint --apply-patches --everything` over rP, mainly to change double quotes to single quotes where appropriate. These changes also validate that the `ArcanistXHPASTLinter::LINT_DOUBLE_QUOTE` rule is working as expected.

Test Plan: Eyeballed it.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D9431
2014-06-09 11:36:50 -07:00
epriestley
fcc2a24d63 Update Aphlict documentation for new server management commands
Summary:
The docs are now a little out of date.

Also //possibly// we should call this `bin/notifications` or something, maybe?

Test Plan: read

Reviewers: joshuaspence

Reviewed By: joshuaspence

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D9398
2014-06-05 12:40:53 -07:00
Chad Little
dbebfea9cc FontAwesome RemarkupBar
Summary:
Replaces the icons with fonts from FontAwesome. Up in the air about the meme icon. Thoughts?

Also removed the second fullscreen/normal state. Seems obvious what it does, but assume someone complained previously?

Test Plan:
Tested all the icon states and made sure they still worked. Test fullscreen and help.

{F163485}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9385
2014-06-05 10:37:21 -07:00
Chad Little
14dab3f55c Convert ButtonBar to FontAwesome
Summary: Removes the sprites and images, uses fontawesome in examples and calendar

Test Plan:
UIExamples, Calendar

{F163375}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9371
2014-06-04 12:53:32 -07:00
epriestley
5b1262c98b Add a bin/hunks script to manage migrations of hunk data
Summary:
Ref T4045. Ref T5179. While we'll eventually need to force a migration, we can let installs (particularly large installs) do an online migration for now. This moves hunks to the new storage format one at a time.

(Note that nothing writes to the new store yet, so this is the only way to populate it.)

WARNING: Installs, don't run this yet! It won't compress the data. Wait until it can also do compression.

Test Plan: Added a `break;` after migrating one row and moved a few rows over. Spot checked them in the database and viewed the affected diffs.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4045, T5179

Differential Revision: https://secure.phabricator.com/D9291
2014-06-03 18:01:23 -07:00
epriestley
e3d7c16d8c Use "https://" instead of "git://" for anoymous GitHub requests in documentation
Summary: These both work, but "git://" uses a nonstandard and possibly firewalled port, is less familiar to users, and is not promoted in the GitHub UI.

Test Plan: `grep`, reviewed diff.

Reviewers: chad, avive, avivey

Reviewed By: avivey

Subscribers: epriestley, avivey

Differential Revision: https://secure.phabricator.com/D9360
2014-06-03 09:40:16 -07:00
epriestley
63ed126b2a Point github.com/facebook URIs at github.com/phacility insead
Summary: Point everything at the new canonical URI.

Test Plan: `grep`

Reviewers: chad

Reviewed By: chad

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D9328
2014-05-29 08:33:25 -07:00
Chad Little
ac05fe9c3b Replace action sprite with FontAwesome
Summary: Replaces the action icons in action headers with FontAwesome

Test Plan:
- grep SPRITE_ACTIONS
- grep sprite-actions
- Replace on UIExamples
- Replace on Workboards
- Replace on Dashboards
- Replace on FeedStories

{F157840}
{F157841}

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9230
2014-05-21 10:18:43 -07:00
Chad Little
3a81f8c68d Convert rest of SPRITE_STATUS to FontAwesome
Summary:
Updates policy, headers, typeaheads to FA over policy icons

Need advice - can't seem to place where icons come from on Typeahead? Wrong icons and wrong colors.... it is late

Test Plan:
- grepped for SPRITE_STATUS
- grepped for sprite-status
- grepped for setStatus for headers
- grepped individual icons names

Browsed numerous places, checked new dropdowns, see pudgy people.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4739

Differential Revision: https://secure.phabricator.com/D9179
2014-05-18 16:10:54 -07:00
Chad Little
b2f3001ec4 Replace Sprite-Icons with FontAwesome
Summary: The removes the sprite sheet 'icons' and replaces it with FontAwesome fonts.

Test Plan:
- Grep for SPRITE_ICONS and replace
- Grep for sprite-icons and replace
- Grep for PhabricatorActionList and choose all new icons
- Grep for Crumbs and fix icons
- Test/Replace PHUIList Icon support
- Test/Replace ObjectList Icon support (foot, epoch, etc)
- Browse as many pages as I could get to
- Remove sprite-icons and move remarkup to own sheet
- Review this diff in Differential

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D9052
2014-05-12 10:08:32 -07:00
Bob Trahan
e96c363eef Add SMS support
Summary:
Provides a working SMS implementation with support for Twilio.

This version doesn't really retry if we get any gruff at all. Future versions should retry.

Test Plan: used bin/sms to send messages and look at them.

Reviewers: chad, epriestley

Reviewed By: epriestley

Subscribers: aurelijus, epriestley, Korvin

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D8930
2014-05-09 12:47:21 -07:00
epriestley
889440ead0 Allow structured destruction of Differential Revisions
Summary:
Ref T4749. Ref T3265. Ref T4909.

  - Remove old "destroy revision" script.
  - Move to structured `bin/remove` destruction.
  - Fix some edge issues.
  - Add transaction destruction support.

Test Plan:
  - Destroyed a bunch of revisions.
  - Saw diffs, changesets, hunks, transactions, edges, and inlines also get wiped out.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4749, T4909, T3265

Differential Revision: https://secure.phabricator.com/D8943
2014-05-01 18:25:30 -07:00
epriestley
827fbb3782 Remove various old things in scripts/
Summary: These have all been obsolete for a reasonable amount of time, or are no longer relevant.

Test Plan: shrug~

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D8941
2014-05-01 18:23:48 -07:00
epriestley
2022a70e16 Implement bin/remove, for structured destruction of objects
Summary:
Ref T4749. Ref T3265. Ref T4909. Several goals here:

  - Move user destruction to the CLI to limit the power of rogue admins.
  - Start consolidating all "destroy named object" scripts into a single UI, to make it easier to know how to destroy things.
  - Structure object destruction so we can do a better and more automatic job of cleaning up transactions, edges, search indexes, etc.
  - Log when we destroy objects so there's a record if data goes missing.

Test Plan: Used `bin/remove destroy` to destroy several users.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T3265, T4749, T4909

Differential Revision: https://secure.phabricator.com/D8940
2014-05-01 18:23:31 -07:00
Felipe Bessa Coelho
f27f7dce52 Install PHP mbstring extension on RHEL & friends
Summary:
The mbstring extension for PHP is not a dependency to any of the already
listed packages on RHEL-like systems, and is needed by Phabricator
(showing a "install mbstring" message as the first thing if it is not
installed)

RHEL seems to have some extra steps to allow php-mbstring to be installed, though:

http://snippets.roozbehk.com/post/35750940300/php-mbstring-missing-on-red-hat-enterprise-linux-6

PS: disabled lint for this change because of the already >80 chars long "yum install" string

Test Plan:
* Created a new container with docker using both centos:6.4 and fedora:20 images
* Ran install script
* Started httpd and mysqld services
* Browsed to server's address
* Got error message
* Installed php-mbstring & restarted httpd
* Works

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D8772
2014-04-14 14:59:27 -07:00
epriestley
0fd12f9cad Remove nonfunctional/obsolete 'reconcile.php'
Summary: Ref T4780. This no longer works and barely ever worked. T4237 is now the relevant task.

Test Plan: Grepped for `reconcile.php`

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4780

Differential Revision: https://secure.phabricator.com/D8739
2014-04-09 16:12:29 -07:00
epriestley
af0b749369 Fix many lies in the "User Roles" document
Summary: Fixes T3047. Update this document and remove some lies ("menu bar is read in admin interfaces"!!!!).

Test Plan:
  - Read text.
  - Searched for "System Agent" in the UI and replaced it with "bot" or "bot/script" or similar.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T3047

Differential Revision: https://secure.phabricator.com/D8675
2014-04-02 12:06:56 -07:00
Joshua Spence
6270114767 Various linter fixes.
Summary:
- Removed trailing newlines.
- Added newline at EOF.
- Removed leading newlines.
- Trimmed trailing whitespace.
- Spelling fix.
- Added newline at EOF

Test Plan: N/A

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: hach-que, chad, Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D8344
2014-02-26 12:44:58 -08:00
Aviv Eyal
361e49dd67 Symbol import: preemptively error on non-utf8 symbols
Summary:
Put a test somewhere we can --ignore it.

Also fix path tests.

Test Plan: insert good/bad values at various positions.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D8353
2014-02-26 12:41:55 -08:00
Chad Little
eec593e313 Add ButtonBar icons
Summary: Adds an icon set for icons on button bars.

Test Plan: photoshop

Reviewers: epriestley, btrahan

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T4394

Differential Revision: https://secure.phabricator.com/D8170
2014-02-09 10:22:12 -08:00
epriestley
0726411cb4 Write a very basic string extractor
Summary: Ref T1139. This has some issues and glitches, but is a reasonable initial attempt that gets some of the big pieces in. We have about 5,200 strings in Phabricator.

Test Plan: {F108261}

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, chad

Maniphest Tasks: T1139

Differential Revision: https://secure.phabricator.com/D8138
2014-02-05 11:02:41 -08:00
Ricky Elrod
6b4998bf4b Increment copyright year and add 2017 dates to calendar import script
Summary: D4335 except a year later.

Test Plan: Nope.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D8106
2014-01-30 09:05:25 -08:00
Eric Stern
14f070a0af Skip anon functions in symbol generation script
Summary:
Filters closures out of symbol generator script, per @epriestley's
comment in T4334

Test Plan:
Before:
  eric@eric-dev ~/phabricator/scripts/symbols: echo 'closure.php' | ./generate_php_symbols.php
    function php  /closure.php
   d function php 10 /closure.php
    function php  /closure.php
   a class php 3 /closure.php
  a b method php 4 /closure.php

After:
  eric@eric-dev ~/phabricator/scripts/symbols: echo 'closure.php' | ./generate_php_symbols.php
   d function php 10 /closure.php
   a class php 3 /closure.php
  a b method php 4 /closure.php

eric@eric-dev ~/phabricator/scripts/symbols: cat closure.php
  <?php

  class a {
    function b() {
      $c = function() { return 1; };
      $c();
    }
  }

  function d() {
    return 2;
  }
  $e = function() {
    return 3;
  };

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: epriestley, Korvin, aran

Differential Revision: https://secure.phabricator.com/D8054
2014-01-23 17:01:11 -08:00
Andrej E Baranov
89bd84986a Fix typo
Summary: 'daeons' to 'daemons'

Test Plan: run `phd help`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: epriestley

CC: epriestley, aran

Differential Revision: https://secure.phabricator.com/D7973
2014-01-15 10:03:50 -08:00
John Mullanaphy
b960c8114b Changed \$callsign to \$argv[1] on commit_hook.php since is undefined causing an error when trying to report an error.
Summary:
Phabricator was going to give me an error message via commit_hook.php, unfortunately said error wasn't being set since
\$callsign was undefined. So, just changed \$callsign to \$argv[1] and now I get the appropriate commit.

Test Plan:
1. Add commit_hook.php to an SVN pre-commit.
2. Set the SVN to be hosted off of Phabricator.
3. Attempt to commit to commit to SVN repository.
Expected: Error message saying that the repository isn't hosted on Phabricator
Results: Error message saying undefined function.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7920
2014-01-09 10:36:01 -08:00
epriestley
972dfa7bfc Add 'hook.d/' directories to SVN and Git repositories for custom hooks
Summary:
Fixes T4189. Ref T4151. Allows repositories to have additional custom hooks for operations which can't be expressed with Herald (one such operation is lint).

This adds only local hook directories, since they're easier to use with existing hooks than global directories. I might add global directories eventually.

This doesn't support Mercurial since we have no demand for it and it's more complicated (we lose compatibility and power by just dropping a `hooks.d/` somewhere).

Test Plan:
  - Pulled hosted SVN and Git repos to verify the hook directories generate correctly.
  - Added a variety of hooks to the hook directories (echo + pass, fail).
  - Pushed commits and verified the hooks fired (output expected info, or failed).
  - Verified push log reflected the correct error code ("3", external) and detail ("nope.sh") when rejecting.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4151, T4189

Differential Revision: https://secure.phabricator.com/D7884
2014-01-03 12:26:10 -08:00
epriestley
2cfc3acf32 Allow Herald pre-commit rules to act on repository projects
Summary:
Fixes T4264. Adds:

  - New "Repository's projects" field to Herald pre-commit rules, so you can write global rules which act based on projects.
  - Allows pre-ref/pre-content rules to bind to projects, and fire for all repositories in that project, so users with limited power can write rules which apply to many repositories.
  - The pre-ref and pre-content classes were starting to share a fair amount of code, so I made them both extend an abstract base class.

Test Plan: Wrote new pre-ref and pre-content rules bound to projects, then pushed commits into repositories in those projects and not in those projects. The "repository projects" field populated, and the rules fired for repositories in the relevant projects.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4264

Differential Revision: https://secure.phabricator.com/D7883
2014-01-03 12:24:28 -08:00
lesha
8c114394e3 Enable Evil^H^H^H^HJSON for PHP on Ubuntu 13.10
Summary:
Good news. Starting with Ubuntu 13.10, Phabricator can legally be used by evil dictators, mad scientists, and toxic derivative creators.

The JSON implementation prohibiting evil (http://www.phoronix.com/scan.php?page=news_item&px=MTQ0MTY) was ripped out and replaced by the Evil-friendly PHP license: https://github.com/remicollet/pecl-json-c/blob/master/LICENSE

Test Plan: ran the shell script, Phabricator no longer fails with "Call to undefined function json_decode".

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7878
2014-01-02 11:59:12 -08:00
epriestley
2c35532256 Drive all Celerity operations from the new map
Summary:
Ref T4222.

  - Removes the old map and changes the CelerityResourceMap API to be entirely driven by the new map.
  - The new map is about 50% smaller and organized more sensibly.
  - This removes the `/pkg/` URI component. All resources are now required to have unique names, so we can tell if a resource is a package or not by looking at the name.
  - Removes some junky old APIs.
  - Cleans up some other APIs.
  - Added some feedback for `bin/celerity map`.
  - `CelerityResourceMap` is still a singleton which is inextricably bound to the Phabricator map; this will change in the future.

Test Plan:
  - Reloaded pages.
  - Verified packaging works by looking at generated includes.
  - Forced minification on and verified it worked.
  - Forced no-timestamps on and verified it worked.
  - Rebuilt map.
  - Ran old script and verified error message.
  - Checked logs.

Reviewers: btrahan, hach-que

Reviewed By: hach-que

CC: chad, aran

Maniphest Tasks: T4222

Differential Revision: https://secure.phabricator.com/D7872
2013-12-31 18:04:25 -08:00
epriestley
906ac21e54 Begin construction of bin/celerity map
Summary:
Ref T4222. Moves us toward a more modern Celerity CLI, and moves map discovery into the classtree. This is a little bit bulky (and means you can't ship completely standalone celerity maps) but has the advantage of being completely standard, and we could subclass maps into an auto-discovering map later if we have a need for it.

This doesn't affect the existing Celerity stuff. I'm going to build the new stuff in parallel, and then swap us over at the end.

Test Plan: Ran `bin/celerity map`, got reasonable-looking output.

Reviewers: btrahan, hach-que

Reviewed By: hach-que

CC: aran

Maniphest Tasks: T4222

Differential Revision: https://secure.phabricator.com/D7864
2013-12-31 18:02:41 -08:00
epriestley
95a806ada3 Remove celerity.resource-path config
Summary: Ref T4222. This was used by Facebook while developing Releeph, but should no longer be necessary since Releeph is in the upstream. I can't get an answer out of Facebook about whether they still use it or not (see T4227), so nuke it. We're going to replace it with a more general mechanism (see T4222).

Test Plan: Regenerated celerity map. Browsed some pages, still got resources.

Reviewers: btrahan, hach-que

Reviewed By: hach-que

CC: aran

Maniphest Tasks: T4222

Differential Revision: https://secure.phabricator.com/D7863
2013-12-31 18:02:35 -08:00
epriestley
db3228844a Note where to stop/start php-fpm in upgrade example script
Summary: This isn't as explicit as it could be.

Test Plan: Reading.

Reviewers: poop

Reviewed By: poop

CC: aran

Differential Revision: https://secure.phabricator.com/D7861
2013-12-30 16:47:05 -08:00
epriestley
26582eb82d Provide a more helpful message if a user connects via raw SSH
Summary: We currently print a fairly vague, technical message which is ambiguous about indicating success or an error if you aren't familiar with SSH.

Test Plan:
  $ ssh -T dweller@localhost
  phabricator-ssh-exec: Welcome to Phabricator.

  You are logged in as epriestley.

  You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH.

  Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH.

  Supported commands are: conduit, git-receive-pack, git-upload-pack, hg, svnserve.

Reviewers: btrahan, dctrwatson

Reviewed By: dctrwatson

CC: aran

Differential Revision: https://secure.phabricator.com/D7854
2013-12-30 13:21:56 -08:00
epriestley
ce78bf1de4 Make all bin/* scripts locate their workflows dynamically
Summary:
Ref T2015. Not directly related to Drydock, but I bumped into this. All these scripts currently enumerate their workflows explicitly.

Instead, use `PhutilSymbolLoader` to automatically discover workflows. This reduces code duplication and errors (see all the bad `extends` this diff fixes) and lets third parties add new workflows (not clearly valuable?).

Test Plan: Ran `bin/x help` for each modified script.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D7840
2013-12-27 13:15:48 -08:00
epriestley
1650874004 Modernize Drydock CLI management of task execution
Summary:
Ref T2015. Currently, Drydock has a `wait-for-lease` workflow which is invoked in the background by the `lease` workflow.

The goal of this mechanism is to allow `bin/drydock lease` to print out logs as the lease is acquired. However, this predates the `runAllTasksInProcess` flags, and they provide a simpler and more robust way (potentially with `--trace` and `PhutilConsole`) to do synchronous execution and debug logging.

Simplify this whole mechanism: just run everything in-process in `bin/drydock lease`, and do logging via `--trace`. We could thread a `PhutilConsole` through things too, but this seems good enough for now.

Also various cleanup/etc.

Test Plan: Ran `bin/drydock lease`. Ran `bin/harbormaster build X --plan Y`, for `Y` being a Drydock-dependent build plan.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D7835
2013-12-27 13:15:12 -08:00
epriestley
aad6b57c36 Add bin/harbormaster to make builds easier to debug
Summary:
Ref T1049. Adds `bin/harbormaster` and `bin/harbormaster build` for applying plans from the console. Since this gets `--trace`, it's much easier to debug what's going on.

This doesn't work properly with some of the Drydock steps yet, I need to look at those. I think `setRunAllTasksInProcess` probably obsoletes some of the mechanisms. It might also not work with "Wait for Builds" but I didn't check.

Test Plan: Used `bin/harbormaster` to run a bunch of builds. Ran builds from web UI.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7825
2013-12-26 10:40:52 -08:00
epriestley
d667b12206 Provide a standalone query for resolution of commit author/committer into Phabricator users
Summary:
Ref T4195. To implement the "Author" and "Committer" rules, I need to resolve author/committer strings into Phabricator users.

The code to do this is currently buried in the daemons. Extract it into a standalone query.

I also added `bin/repository lookup-users <commit>` to test this query, both to improve confidence I'm getting this right and to provide a diagnostic command for users, since there's occasionally some confusion over how author/committer strings resolve into valid users.

Test Plan:
I tested this using `bin/repository lookup-users` and `reparse.php --message` on Git, Mercurial and SVN commits. Here's the `lookup-users` output:

  >>> orbital ~/devtools/phabricator $ ./bin/repository lookup-users rINIS3
  Examining commit rINIS3...
  Raw author string: epriestley
  Phabricator user: epriestley (Evan Priestley   )
  Raw committer string: null
  >>> orbital ~/devtools/phabricator $ ./bin/repository lookup-users rPOEMS165b6c54f487c8
  Examining commit rPOEMS165b6c54f487...
  Raw author string: epriestley <git@epriestley.com>
  Phabricator user: epriestley (Evan Priestley   )
  Raw committer string: epriestley <git@epriestley.com>
  Phabricator user: epriestley (Evan Priestley   )
  >>> orbital ~/devtools/phabricator $ ./bin/repository lookup-users rINIH6d24c1aee7741e
  Examining commit rINIH6d24c1aee774...
  Raw author string: epriestley <hg@yghe.net>
  Phabricator user: epriestley (Evan Priestley   )
  Raw committer string: null
  >>> orbital ~/devtools/phabricator $

The `reparse.php` output was similar, and all VCSes resolved authors correctly.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1731, T4195

Differential Revision: https://secure.phabricator.com/D7801
2013-12-19 11:05:17 -08:00
epriestley
74251b3636 Support bookmark hook operations in Mercurial
Summary: Ref T4195. Turns bookmark mutations in Mercurial into log objects.

Test Plan:
Pushed a pile of bookmarks and got logs:

{F89313}

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4195

Differential Revision: https://secure.phabricator.com/D7764
2013-12-17 08:34:30 -08:00
epriestley
7f45824984 Fix two issues with creating Conpherence threads via mail on some configurations
Summary:
Ref T4107. Two issues:

  - With strict MySQL settings, we try to insert `null` into the non-nullable `messageCount` field. Add an `initializeNew...` method.
  - If we don't create a new conpherence (for example, because the message body is empty), we fatal on `getPHID()` right now.

Also, make this stuff a little easier to test.

Test Plan: Used `mail_handler.php` to receive empty conpherence mail, and new-thread conpherence mail.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4107

Differential Revision: https://secure.phabricator.com/D7760
2013-12-12 10:59:28 -08:00
epriestley
d846f6508b Fix some repository URI handling issues in Git and Mercurial
Summary:
See <https://github.com/facebook/phabricator/issues/467>. @dctrwatson also ran into an issue where we were trying to `setPass()` a GitURI.

  - For Git and Mercurial, properly generate credential URIs where relevant.
  - Don't try to `setPass()` on Git-style URIs.

This isn't perfect but should clean things up a bit.

Test Plan: Added unit tests. Lots of `grep`.

Reviewers: btrahan

Reviewed By: btrahan

CC: dctrwatson, aran

Differential Revision: https://secure.phabricator.com/D7759
2013-12-12 09:45:27 -08:00
Chad Little
3ad4be4d93 Multicolor headers
Summary: This adds a handful of 'Main Header' colors to change the look of Phabricator very slightly. I know I would probably set my dev header to a different color.

Test Plan: Tested each css class and color, can add more in the future.

Reviewers: epriestley, btrahan

Reviewed By: btrahan

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7731
2013-12-06 12:08:11 -08:00
epriestley
a74bfe5167 Minor, sort out a change request which was lost in the melee. 2013-12-05 17:37:04 -08:00
epriestley
5ca84589bd Add an SSH access log
Summary: Ref T4107. Ref T4189. This implements an SSH access log, similar to the HTTP access log.

Test Plan:
  [Thu, 05 Dec 2013 13:45:41 -0800]	77841	orbital	::1	dweller	epriestley	epriestley	git-receive-pack	/diffusion/POEMS/	0	324765	402	232
  [Thu, 05 Dec 2013 13:45:48 -0800]	77860	orbital	::1	dweller	epriestley	epriestley	git-receive-pack	/diffusion/POEMS/	0	325634	402	232

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4107, T4189

Differential Revision: https://secure.phabricator.com/D7719
2013-12-05 17:00:48 -08:00
epriestley
e28b848ab2 Store pusher remote address and push protocol in PushLog
Summary: Ref T4195. Stores remote address and protocol in the logs, where possible.

Test Plan: Pushed some stuff, looked at the log, saw data.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4195

Differential Revision: https://secure.phabricator.com/D7711
2013-12-05 11:59:22 -08:00
epriestley
d2e9aee16d Reject dangerous changes in Git repositories by default
Summary: Ref T4189. This adds a per-repository "dangerous changes" flag, which defaults to off. This flag must be enabled to do non-appending branch mutation (delete branches / rewrite history).

Test Plan:
With flag on and off, performed various safe and dangerous pushes.

  >>> orbital ~/repos/POEMS $ git push origin :blarp
  remote: +---------------------------------------------------------------+
  remote: |      * * * PUSH REJECTED BY EVIL DRAGON BUREAUCRATS * * *     |
  remote: +---------------------------------------------------------------+
  remote:             \
  remote:              \                    ^    /^
  remote:               \                  / \  // \
  remote:                \   |\___/|      /   \//  .\
  remote:                 \  /V  V  \__  /    //  | \ \           *----*
  remote:                   /     /  \/_/    //   |  \  \          \   |
  remote:                   @___@`    \/_   //    |   \   \         \/\ \
  remote:                  0/0/|       \/_ //     |    \    \         \  \
  remote:              0/0/0/0/|        \///      |     \     \       |  |
  remote:           0/0/0/0/0/_|_ /   (  //       |      \     _\     |  /
  remote:        0/0/0/0/0/0/`/,_ _ _/  ) ; -.    |    _ _\.-~       /   /
  remote:                    ,-}        _      *-.|.-~-.           .~    ~
  remote:   \     \__/        `/\      /                 ~-. _ .-~      /
  remote:    \____(Oo)           *.   }            {                   /
  remote:    (    (--)          .----~-.\        \-`                 .~
  remote:    //__\\  \ DENIED!  ///.----..<        \             _ -~
  remote:   //    \\               ///-._ _ _ _ _ _ _{^ - - - - ~
  remote:
  remote:
  remote: DANGEROUS CHANGE: The change you're attempting to push deletes the branch 'blarp'.
  remote: Dangerous change protection is enabled for this repository.
  remote: Edit the repository configuration before making dangerous changes.
  remote:
  To ssh://dweller@localhost/diffusion/POEMS/
   ! [remote rejected] blarp (pre-receive hook declined)
  error: failed to push some refs to 'ssh://dweller@localhost/diffusion/POEMS/'

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, chad, richardvanvelzen

Maniphest Tasks: T4189

Differential Revision: https://secure.phabricator.com/D7689
2013-12-03 10:28:39 -08:00
epriestley
f93c6985ad Support Mercurial pretxnchangegroup hooks
Summary: Ref T4189. Fixes T2066. Mercurial has a //lot// of hooks so I'm not 100% sure this is all we need to install (we may need separate hooks for tags/bookmarks) but it should cover most of what we're after at least.

Test Plan:
  - `bin/repository pull`'d a Mercurial repo and got a hook install.
  - Pushed to a Mercurial repository over SSH and HTTP, with good/bad hooks. Saw hooks fire.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2066, T4189

Differential Revision: https://secure.phabricator.com/D7685
2013-12-02 15:46:03 -08:00
epriestley
017d6ccd07 Support SVN pre-commit hoooks
Summary:
Ref T4189. This adds SVN support, which was a little more messy than I though. Principally, we can not use `PHABRICATOR_USER` for Subversion, because it strips away the entire environment for "security reasons".

Instead, use `--tunnel-user` plus `svnlook author` to figure out the author.

Also fix "ssh://" clone URIs, which needs to be "svn+ssh://".

Test Plan:
  - Made SVN commits through the hook.
  - Made Git commits, too, to make sure I didn't break anything.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4189

Differential Revision: https://secure.phabricator.com/D7683
2013-12-02 15:45:55 -08:00
epriestley
618b5cbbc4 Install pre-commit hooks in Git repositories
Summary:
Ref T4189. T4189 describes most of the intent here:

  - When updating hosted repositories, sync a pre-commit hook into them instead of doing a `git fetch`.
  - The hook calls into Phabricator. The acting Phabricator user is sent via PHABRICATOR_USER in the environment. The active repository is sent via CLI.
  - The hook doesn't do anything useful yet; it just veifies basic parameters, does a little parsing, and exits 0 to allow the commit.

Test Plan:
  - Performed Git pushes and pulls over SSH and HTTP.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4189

Differential Revision: https://secure.phabricator.com/D7682
2013-12-02 15:45:36 -08:00
epriestley
d59722321f Handle "-p port" flag in ssh-connect
Summary:
The documentation is explicit that Git does not pass this flag:

  The $GIT_SSH command will be given exactly two arguments: the username@host (or just host) from the URL and the shell command to execute on that remote system.

This isn't true; it does. Accommodate it.

I'll see if I can fix this in the upstream, too.

Test Plan: Ran various `ssh-connect` commands with -p, etc.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7680
2013-12-02 11:25:46 -08:00
epriestley
e19ad8ab8a Warn users about using bin/accountadmin for first time setup
Summary: The "easy setup" is sort of an exaggeration since it basically just amounts to getting you logged in correctly, but it will probably be more true in the future.

Test Plan: Ran `bin/accountadmin` with zero and more than zero accounts.

Reviewers: asherkin, btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7678
2013-12-02 11:25:42 -08:00
epriestley
1b026fa629 Unconditionally approve bin/accountadmin accounts
Summary: Some oldschool users create accounts via bin/accountadmin, which
messes up the first-time setup process. Auto-approve these accounts, as this
better aligns with user expectation.

Auditors: btrahan
2013-12-02 06:51:42 -08:00
Eric Stern
45c3a605ee Correctly handle case where no ssh keys are found
Summary: If no ssh keys are in phabricator, bin/ssh-auth errors with undefined `$lines`. This fixes that case and explicitly tells the user no rows were found.

Test Plan: Ran bin/ssh-auth before and after change with no ssh keys in the system. Error goes away after change.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: epriestley

CC: epriestley, aran, Korvin

Differential Revision: https://secure.phabricator.com/D7675
2013-11-30 18:56:23 -08:00
epriestley
58530669a2 Make ssh-connect more robust against known_host issues
I think the daemon is choking on this, prevent it from failing.

Auditors: btrahan
2013-11-22 16:24:24 -08:00
epriestley
6e41016077 Document and remove some scary warnings from repository hosting
Summary: Fixes T2230. This isn't a total walk in the park to configure, but should work for early adopters now.

Test Plan: Read documentation, browsed UI.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7634
2013-11-22 15:24:27 -08:00
epriestley
d09dd23bd7 Actually push to mirrors
Summary: Fixes T4038. Push repositories to mirrors.

Test Plan: Created a functional mirror of a local: https://github.com/epriestley/poems

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4038

Differential Revision: https://secure.phabricator.com/D7633
2013-11-22 15:24:09 -08:00
epriestley
51fb1ca16d Migrate repositories to use Passphrase for credential management
Summary: Fixes T4122. Ref T2230. Instead of storing credentials on each repository, store them in Passphrase. This allows easy creation/management of many repositories which share credentials.

Test Plan:
  - Upgraded repositories.
  - Created and edited repositories.
  - Pulled HTTP and SSH repositories.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2230, T4122

Differential Revision: https://secure.phabricator.com/D7629
2013-11-22 15:23:33 -08:00
James Rhodes
7c3cb5948c Drydock blueprint for preallocated remote hosts
Summary:
This adds a Drydock blueprint for preallocated, remote hosts.  This will be used by the Harbormaster interface to allow users to specify remote hosts that builds can be run on.

This adds a `canAllocateResource` method to Drydock blueprints; it is used to detect whether a blueprint can allocate a resource for the given type and attributes.

Test Plan:
Ran:

```
bin/drydock lease --type host --attributes remote=true,preallocated=true,host=192.168.56.101,port=22,user=james,keyfile=,path=C:\\Build\\,platform=windows
```

and saw the "C:\Build\<id>" folder appear on the remote Windows machine.  Viewed the lease and resource in Drydock as well.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran, jamesr

Maniphest Tasks: T4111

Differential Revision: https://secure.phabricator.com/D7593
2013-11-22 14:34:10 -08:00
Nikolajus Krauklis
67b8f9d35d Update manage_cache.php
tagline copied from mail

Reviewed by: epriestley

See: <https://github.com/facebook/phabricator/pull/446>
2013-11-21 12:36:00 -08:00
epriestley
ff8b48979e Simplify Repository remote and local command construction
Summary:
This cleans up some garbage:

  - We were specifying environmental variables with `X=y git ...`, but now have `setEnv()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setEnv()`.
  - We were specifying the working directory with `(cd %s && git ...)`, but now have `setCWD()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setCWD()`.
  - We were specifying the Git credentials with `ssh-agent -c (ssh-add ... && git ...)`. We can do this more cleanly with `GIT_SSH`. Use `GIT_SSH`.
  - Since we have to write a script for `GIT_SSH` anyway, use the same script for Subversion and Mercurial.

This fixes two specific issues:

  - Previously, we were not able to set `-o StrictHostKeyChecking=no` on Git commands, so the first time you cloned a git repo the daemons would generally prompt you to add `github.com` or whatever to `known_hosts`. Since this was non-interactive, things would mysteriously hang, in effect. With `GIT_SSH`, we can specify the flag, reducing the number of ways things can go wrong.
  - This adds `LANG=C`, which probably (?) forces the language to English for all commands. Apparently you need to install special language packs or something, so I don't know that this actually works, but at least two users with non-English languages have claimed it does (see <https://github.com/facebook/arcanist/pull/114> for a similar issue in Arcanist).

At some point in the future I might want to combine the Arcanist code for command execution with the Phabricator code for command execution (they share some stuff like LANG and HGPLAIN). However, credential management is kind of messy, so I'm adopting a "wait and see" approach for now. I expect to split this at least somewhat in the future, for Drydock/Automerge if nothing else.

Also I'm not sure if we use the passthru stuff at all anymore, I may just be able to delete that. I'll check in a future diff.

Test Plan: Browsed and pulled Git, Subversion and Mercurial repositories.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7600
2013-11-20 10:41:35 -08:00
William R. Otte
74d81f9be0 Bugfix for D7596
Summary: Fixed a small bug that caused the catch-all commit to purge previously added symbols in that session.

Test Plan: Re-ran the script, observed corrected behavior.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T4117

Differential Revision: https://secure.phabricator.com/D7597
2013-11-17 20:19:49 -08:00
William R. Otte
7d43e59110 Modified script to commit smaller batches of symbols to the database.
Summary:
Modified the import script so it will only try to load a configurable
number of symbols at a time to avoid exhausting memory for large project
imports.

I haven't written a line of PHP in more than a decade, so please forgive
any stylistic or technical errors.

Test Plan: Ran the script on symbol table generated from linux kernel.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T4117

Differential Revision: https://secure.phabricator.com/D7596
2013-11-17 16:33:33 -08:00
epriestley
f5ca647d2c Add bin/repository edit for CLI repository editing
Summary:
Ref T4039. This is mostly to deal with that, to prevent the security issues associated with mutable local paths. The next diff will lock them in the web UI.

I also added a confirmation prompt to `bin/repository delete`, which was a little scary without one.

See one comment inline about the `--as` flag. I don't love this, but when I started adding all the stuff we'd need to let this transaction show up as "Administrator" it quickly got pretty big.

Test Plan: Ran `bin/repository edit ...`, saw an edit with a transaction show up on the web UI.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4039

Differential Revision: https://secure.phabricator.com/D7579
2013-11-13 11:26:05 -08:00
epriestley
c0e1a63a63 Implement an approval queue
Summary:
  - Add an option for the queue.
  - By default, enable it.
  - Dump new users into the queue.
  - Send admins an email to approve them.

Test Plan:
  - Registered new accounts with queue on and off.
  - As an admin, approved accounts and disabled the queue from email.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7576
2013-11-13 11:24:56 -08:00
epriestley
7f11e8d740 Improve handling of email verification and "activated" accounts
Summary:
Small step forward which improves existing stuff or lays groudwork for future stuff:

  - Currently, to check for email verification, we have to single-query the email address on every page. Instead, denoramlize it into the user object.
    - Migrate all the existing users.
    - When the user verifies an email, mark them as `isEmailVerified` if the email is their primary email.
    - Just make the checks look at the `isEmailVerified` field.
  - Add a new check, `isUserActivated()`, to cover email-verified plus disabled. Currently, a non-verified-but-not-disabled user could theoretically use Conduit over SSH, if anyone deployed it. Tighten that up.
  - Add an `isApproved` flag, which is always true for now. In a future diff, I want to add a default-on admin approval queue for new accounts, to prevent configuration mistakes. The way it will work is:
    - When the queue is enabled, registering users are created with `isApproved = false`.
    - Admins are sent an email, "[Phabricator] New User Approval (alincoln)", telling them that a new user is waiting for approval.
    - They go to the web UI and approve the user.
    - Manually-created accounts are auto-approved.
    - The email will have instructions for disabling the queue.

I think this queue will be helpful for new installs and give them peace of mind, and when you go to disable it we have a better opportunity to warn you about exactly what that means.

Generally, I want to improve the default safety of registration, since if you just blindly coast through the path of least resistance right now your install ends up pretty open, and realistically few installs are on VPNs.

Test Plan:
  - Ran migration, verified `isEmailVerified` populated correctly.
  - Created a new user, checked DB for verified (not verified).
  - Verified, checked DB (now verified).
  - Used Conduit, People, Diffusion.

Reviewers: btrahan

Reviewed By: btrahan

CC: chad, aran

Differential Revision: https://secure.phabricator.com/D7572
2013-11-12 14:37:04 -08:00
epriestley
85f505465e Support serving SVN repositories over SSH
Summary:
Ref T2230. The SVN protocol has a sensible protocol format with a good spec here:

http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_ra_svn/protocol

Particularly, compare this statement to the clown show that is the Mercurial wire protocol:

> It is possible to parse an item without knowing its type in advance.

WHAT A REASONABLE STATEMENT TO BE ABLE TO MAKE ABOUT A WIRE PROTOCOL

Although it makes substantially more sense than Mercurial, it's much heavier-weight than the Git or Mercurial protocols, since it isn't distributed.

It's also not possible to figure out if a request is a write request (or even which repository it is against) without proxying some of the protocol frames. Finally, several protocol commands embed repository URLs, and we need to reach into the protocol and translate them.

Test Plan: Ran various SVN commands over SSH (`svn log`, `svn up`, `svn commit`, etc).

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7556
2013-11-11 12:19:06 -08:00
epriestley
8840f60218 Enable Mercurial reads and writes over SSH
Summary:
Ref T2230. This is substantially more complicated than Git, but mostly because Mercurial's protocol is a like 50 ad-hoc extensions cobbled together. Because we must decode protocol frames in order to determine if a request is read or write, 90% of this is implementing a stream parser for the protocol.

Mercurial's own parser is simpler, but relies on blocking reads. Since we don't even have methods for blocking reads right now and keeping the whole thing non-blocking is conceptually better, I made the parser nonblocking. It ends up being a lot of stuff. I made an effort to cover it reasonably well with unit tests, and to make sure we fail closed (i.e., reject requests) if there are any parts of the protocol I got wrong.

A lot of the complexity is sharable with the HTTP stuff, so it ends up being not-so-bad, just very hard to verify by inspection as clearly correct.

Test Plan:
  - Ran `hg clone` over SSH.
  - Ran `hg fetch` over SSH.
  - Ran `hg push` over SSH, to a read-only repo (error) and a read-write repo (success).

Reviewers: btrahan, asherkin

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7553
2013-11-11 12:18:27 -08:00
James Rhodes
448d8684e8 Quick hack to make symbol lookup work for C#.
Summary: This is kind of a quick hack to make symbol lookup work for C#.  ctags calls C# 'csharp', while pygments recognises it as 'cs' (or at least, I have to put 'cs' in the Arcanist indexed languages for the clickables to appear, while it's 'csharp' in the symbol database).

Test Plan: Tested this in my live install and it makes symbol lookup work.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7497
2013-11-09 15:08:50 -08:00
epriestley
bd29784a32 Add an administrative bin/repository importing command to list importing commits
Summary: Ref T4068. Adds a command to list all commits in an "importing" status. This will allow users to use `reparse.php` to diagnose and repair issues.

Test Plan:
  - Ran `bin/repository importing P`, etc.
  - Used `reparse.php` to reparse some commit stages and saw status update correctly.
  - Ran on a repo with no importing commits.
  - Ran with `... --simple | xargs`, which saves us having to put an `awk` or something in there for users.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4068

Differential Revision: https://secure.phabricator.com/D7515
2013-11-06 11:26:41 -08:00
epriestley
e3a5ab1f8c Add an administrative bin/repository mark-imported command
Summary:
Ref T4068. In some cases like that one, I anticipate a repository not fully importing when a handful of random commits are broken. In the long run we should just deal with that properly, but in the meantime provide an administrative escape hatch so you can mark the repository as imported and get it running normally.

The major reason to do this is that Herald, Feed, Harbormaster, etc., won't activate until a repository is "imported".

Test Plan:
  - Tried to mark an imported repository as imported, got an "already imported" message.
  - Same for not-imported.
  - Marked a repository not-imported.
  - Marked a repository imported.
  - Marked a repository not-imported, then waited for the daemons to mark it imported again automatically.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, kbrownlees

Maniphest Tasks: T4068

Differential Revision: https://secure.phabricator.com/D7514
2013-11-06 11:26:24 -08:00
epriestley
c6665b1907 Serve git writes over SSH
Summary: Looks like this is pretty straightforward; same as the reads except mark it as needing PUSH.

Test Plan: Ran `git push`, pushed over SSH to a hosted repo.

Reviewers: btrahan

Reviewed By: btrahan

CC: hach-que, aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7425
2013-10-29 15:32:41 -07:00
epriestley
bb4904553f Route some VCS connections over SSH
Summary:
  - Add web UI for configuring SSH hosting.
  - Route git reads (`git-upload-pack` over SSH).

Test Plan:
  >>> orbital ~ $ git clone ssh://127.0.0.1/
  Cloning into '127.0.0.1'...
  Exception: Unrecognized repository path "/". Expected a path like "/diffusion/X/".
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
  >>> orbital ~ $ git clone ssh://127.0.0.1/diffusion/X/
  Cloning into 'X'...
  Exception: No repository "X" exists!
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
  >>> orbital ~ $ git clone ssh://127.0.0.1/diffusion/MT/
  Cloning into 'MT'...
  Exception: This repository is not available over SSH.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
  >>> orbital ~ $ git clone ssh://127.0.0.1/diffusion/P/
  Cloning into 'P'...
  Exception: TODO: Implement serve over SSH.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.

Reviewers: btrahan

Reviewed By: btrahan

CC: hach-que, aran

Maniphest Tasks: T2230

Differential Revision: https://secure.phabricator.com/D7421
2013-10-29 15:32:40 -07:00