1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-27 01:02:42 +01:00
phorge-phorge/scripts/ssh/ssh-connect.php

160 lines
4 KiB
PHP
Raw Normal View History

Simplify Repository remote and local command construction Summary: This cleans up some garbage: - We were specifying environmental variables with `X=y git ...`, but now have `setEnv()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setEnv()`. - We were specifying the working directory with `(cd %s && git ...)`, but now have `setCWD()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setCWD()`. - We were specifying the Git credentials with `ssh-agent -c (ssh-add ... && git ...)`. We can do this more cleanly with `GIT_SSH`. Use `GIT_SSH`. - Since we have to write a script for `GIT_SSH` anyway, use the same script for Subversion and Mercurial. This fixes two specific issues: - Previously, we were not able to set `-o StrictHostKeyChecking=no` on Git commands, so the first time you cloned a git repo the daemons would generally prompt you to add `github.com` or whatever to `known_hosts`. Since this was non-interactive, things would mysteriously hang, in effect. With `GIT_SSH`, we can specify the flag, reducing the number of ways things can go wrong. - This adds `LANG=C`, which probably (?) forces the language to English for all commands. Apparently you need to install special language packs or something, so I don't know that this actually works, but at least two users with non-English languages have claimed it does (see <https://github.com/facebook/arcanist/pull/114> for a similar issue in Arcanist). At some point in the future I might want to combine the Arcanist code for command execution with the Phabricator code for command execution (they share some stuff like LANG and HGPLAIN). However, credential management is kind of messy, so I'm adopting a "wait and see" approach for now. I expect to split this at least somewhat in the future, for Drydock/Automerge if nothing else. Also I'm not sure if we use the passthru stuff at all anymore, I may just be able to delete that. I'll check in a future diff. Test Plan: Browsed and pulled Git, Subversion and Mercurial repositories. Reviewers: btrahan Reviewed By: btrahan CC: aran Maniphest Tasks: T2230 Differential Revision: https://secure.phabricator.com/D7600
2013-11-20 19:41:35 +01:00
#!/usr/bin/env php
<?php
// This is a wrapper script for Git, Mercurial, and Subversion. It primarily
// serves to inject "-o StrictHostKeyChecking=no" into the SSH arguments.
Install a SIGTERM handler in ssh-connect Summary: Ref T10547. This has been around for a while but I was never able to reproduce it. I caught a repro case in the cluster recently and I think this is the right fix. We tell Subversion to run `ssh-connect` instead of `ssh` so we can provide options and credentials, by using `SVN_SSH` in the environment. Subversion will sometimes kill the SSH tunnel subprocess aggressively with SIGTERM -- as of writing, you can search for `SIGTERM` in `make_tunnel()` here: http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_ra_svn/client.c By default, when a PHP process gets SIGTERM it just exits immediately, without running destructors or shutdown functions. Since destructors/shutdown functions don't run, `TempFile` doesn't get a chance to remove the file. I don't have a clear picture of //when// Subversion sends SIGTERM to the child process. I can't really get this to trigger locally via `svn`, although I was able to get it to trigger explicitly. So I'm only about 95% sure this fixes it, but it seems likely. Test Plan: Locally, I couldn't get this to reproduce "normally" even knowing the cause (maybe Subversion doesn't do the SIGTERM stuff on OSX?) but I was able to get it to reproduce reliabily by adding `posix_kill(getmypid(), SIGTERM);` to the body of the script. With that added, running the script with `PHABRICATOR_CREDENTIAL=PHID-CDTL-...` in the environment reliably left straggler temporary files. Adding `declare()` and a signal handler fixed this: the script now runs the `TempFile` destructor and longer leaves the stragglers around. Reviewers: chad Reviewed By: chad Maniphest Tasks: T10547 Differential Revision: https://secure.phabricator.com/D16102
2016-06-13 18:15:20 +02:00
// In some cases, Subversion sends us SIGTERM. If we don't catch the signal and
// react to it, we won't run object destructors by default and thus won't clean
// up temporary files. Declare ticks so we can install a signal handler.
if (function_exists('pcntl_async_signals')) {
pcntl_async_signals(true);
} else {
declare(ticks = 1);
}
Install a SIGTERM handler in ssh-connect Summary: Ref T10547. This has been around for a while but I was never able to reproduce it. I caught a repro case in the cluster recently and I think this is the right fix. We tell Subversion to run `ssh-connect` instead of `ssh` so we can provide options and credentials, by using `SVN_SSH` in the environment. Subversion will sometimes kill the SSH tunnel subprocess aggressively with SIGTERM -- as of writing, you can search for `SIGTERM` in `make_tunnel()` here: http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_ra_svn/client.c By default, when a PHP process gets SIGTERM it just exits immediately, without running destructors or shutdown functions. Since destructors/shutdown functions don't run, `TempFile` doesn't get a chance to remove the file. I don't have a clear picture of //when// Subversion sends SIGTERM to the child process. I can't really get this to trigger locally via `svn`, although I was able to get it to trigger explicitly. So I'm only about 95% sure this fixes it, but it seems likely. Test Plan: Locally, I couldn't get this to reproduce "normally" even knowing the cause (maybe Subversion doesn't do the SIGTERM stuff on OSX?) but I was able to get it to reproduce reliabily by adding `posix_kill(getmypid(), SIGTERM);` to the body of the script. With that added, running the script with `PHABRICATOR_CREDENTIAL=PHID-CDTL-...` in the environment reliably left straggler temporary files. Adding `declare()` and a signal handler fixed this: the script now runs the `TempFile` destructor and longer leaves the stragglers around. Reviewers: chad Reviewed By: chad Maniphest Tasks: T10547 Differential Revision: https://secure.phabricator.com/D16102
2016-06-13 18:15:20 +02:00
Simplify Repository remote and local command construction Summary: This cleans up some garbage: - We were specifying environmental variables with `X=y git ...`, but now have `setEnv()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setEnv()`. - We were specifying the working directory with `(cd %s && git ...)`, but now have `setCWD()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setCWD()`. - We were specifying the Git credentials with `ssh-agent -c (ssh-add ... && git ...)`. We can do this more cleanly with `GIT_SSH`. Use `GIT_SSH`. - Since we have to write a script for `GIT_SSH` anyway, use the same script for Subversion and Mercurial. This fixes two specific issues: - Previously, we were not able to set `-o StrictHostKeyChecking=no` on Git commands, so the first time you cloned a git repo the daemons would generally prompt you to add `github.com` or whatever to `known_hosts`. Since this was non-interactive, things would mysteriously hang, in effect. With `GIT_SSH`, we can specify the flag, reducing the number of ways things can go wrong. - This adds `LANG=C`, which probably (?) forces the language to English for all commands. Apparently you need to install special language packs or something, so I don't know that this actually works, but at least two users with non-English languages have claimed it does (see <https://github.com/facebook/arcanist/pull/114> for a similar issue in Arcanist). At some point in the future I might want to combine the Arcanist code for command execution with the Phabricator code for command execution (they share some stuff like LANG and HGPLAIN). However, credential management is kind of messy, so I'm adopting a "wait and see" approach for now. I expect to split this at least somewhat in the future, for Drydock/Automerge if nothing else. Also I'm not sure if we use the passthru stuff at all anymore, I may just be able to delete that. I'll check in a future diff. Test Plan: Browsed and pulled Git, Subversion and Mercurial repositories. Reviewers: btrahan Reviewed By: btrahan CC: aran Maniphest Tasks: T2230 Differential Revision: https://secure.phabricator.com/D7600
2013-11-20 19:41:35 +01:00
$root = dirname(dirname(dirname(__FILE__)));
require_once $root.'/scripts/__init_script__.php';
// Contrary to the documentation, Git may pass a "-p" flag. If it does, respect
// it and move it before the "--" argument.
$args = new PhutilArgumentParser($argv);
$args->parsePartial(
array(
array(
'name' => 'port',
'short' => 'p',
'param' => pht('port'),
'help' => pht('Port number to connect to.'),
),
Add parsing for ssh options (-o) which are passed when using GIT v2 wire protocol by git command (SSH transport) Summary: Makes `ssh-connect` compatible with Git v2 wire protocol over SSH More details about git V2 wire: https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html `git` command (2.18+) passes extra options (`-o "SendEnv GIT_PROTOCOL"`) to underlying `ssh` command to enable v2 wire protocol (environment variable enabling new protocol). Phabricator `ssh-connect` command doesn't understand `-o` options and interprets it as host parts hence when you enable git v2 all clones/ls-remotes crash with: ``` #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:525] #1 PhabricatorRepository::execxRemoteCommand(string, PhutilOpaqueEnvelope) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:400] #2 PhabricatorRepositoryPullEngine::loadGitRemoteRefs(PhabricatorRepository) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:343] #3 PhabricatorRepositoryPullEngine::executeGitUpdate() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:126] #4 PhabricatorRepositoryPullEngine::pullRepositoryWithLock() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:40] #5 PhabricatorRepositoryPullEngine::pullRepository() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:59] #6 PhabricatorRepositoryManagementUpdateWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441] #7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333] #8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22] COMMAND git ls-remote '********' STDOUT (empty) STDERR ssh: Could not resolve hostname -o: Name or service not known fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. at [<phutil>/src/future/exec/ExecFuture.php:369] ``` Test Plan: How to reproduce: 1. add repository to Phabricator which is accessed via `ssh` 2. Use git 2.18+ 3. Enable wire protocol in `/etc/gitconfig`: ``` [protocol] version = 2 ``` 4. Try refreshing repository: `phabricator/bin/repository update somecallsing` 5. Repository update fails with `ssh: Could not resolve hostname -o: Name or service not known` after this changes - updates will succeed Reviewers: epriestley, Pawka, #blessed_reviewers Reviewed By: epriestley, #blessed_reviewers Subscribers: Korvin Differential Revision: https://secure.phabricator.com/D19542
2018-08-01 13:42:52 +02:00
array(
'name' => 'options',
'short' => 'o',
'param' => pht('options'),
'repeat' => true,
'help' => pht('SSH options.'),
),
));
Add parsing for ssh options (-o) which are passed when using GIT v2 wire protocol by git command (SSH transport) Summary: Makes `ssh-connect` compatible with Git v2 wire protocol over SSH More details about git V2 wire: https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html `git` command (2.18+) passes extra options (`-o "SendEnv GIT_PROTOCOL"`) to underlying `ssh` command to enable v2 wire protocol (environment variable enabling new protocol). Phabricator `ssh-connect` command doesn't understand `-o` options and interprets it as host parts hence when you enable git v2 all clones/ls-remotes crash with: ``` #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:525] #1 PhabricatorRepository::execxRemoteCommand(string, PhutilOpaqueEnvelope) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:400] #2 PhabricatorRepositoryPullEngine::loadGitRemoteRefs(PhabricatorRepository) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:343] #3 PhabricatorRepositoryPullEngine::executeGitUpdate() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:126] #4 PhabricatorRepositoryPullEngine::pullRepositoryWithLock() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:40] #5 PhabricatorRepositoryPullEngine::pullRepository() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:59] #6 PhabricatorRepositoryManagementUpdateWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441] #7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333] #8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22] COMMAND git ls-remote '********' STDOUT (empty) STDERR ssh: Could not resolve hostname -o: Name or service not known fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. at [<phutil>/src/future/exec/ExecFuture.php:369] ``` Test Plan: How to reproduce: 1. add repository to Phabricator which is accessed via `ssh` 2. Use git 2.18+ 3. Enable wire protocol in `/etc/gitconfig`: ``` [protocol] version = 2 ``` 4. Try refreshing repository: `phabricator/bin/repository update somecallsing` 5. Repository update fails with `ssh: Could not resolve hostname -o: Name or service not known` after this changes - updates will succeed Reviewers: epriestley, Pawka, #blessed_reviewers Reviewed By: epriestley, #blessed_reviewers Subscribers: Korvin Differential Revision: https://secure.phabricator.com/D19542
2018-08-01 13:42:52 +02:00
$unconsumed_argv = $args->getUnconsumedArgumentVector();
Install a SIGTERM handler in ssh-connect Summary: Ref T10547. This has been around for a while but I was never able to reproduce it. I caught a repro case in the cluster recently and I think this is the right fix. We tell Subversion to run `ssh-connect` instead of `ssh` so we can provide options and credentials, by using `SVN_SSH` in the environment. Subversion will sometimes kill the SSH tunnel subprocess aggressively with SIGTERM -- as of writing, you can search for `SIGTERM` in `make_tunnel()` here: http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_ra_svn/client.c By default, when a PHP process gets SIGTERM it just exits immediately, without running destructors or shutdown functions. Since destructors/shutdown functions don't run, `TempFile` doesn't get a chance to remove the file. I don't have a clear picture of //when// Subversion sends SIGTERM to the child process. I can't really get this to trigger locally via `svn`, although I was able to get it to trigger explicitly. So I'm only about 95% sure this fixes it, but it seems likely. Test Plan: Locally, I couldn't get this to reproduce "normally" even knowing the cause (maybe Subversion doesn't do the SIGTERM stuff on OSX?) but I was able to get it to reproduce reliabily by adding `posix_kill(getmypid(), SIGTERM);` to the body of the script. With that added, running the script with `PHABRICATOR_CREDENTIAL=PHID-CDTL-...` in the environment reliably left straggler temporary files. Adding `declare()` and a signal handler fixed this: the script now runs the `TempFile` destructor and longer leaves the stragglers around. Reviewers: chad Reviewed By: chad Maniphest Tasks: T10547 Differential Revision: https://secure.phabricator.com/D16102
2016-06-13 18:15:20 +02:00
if (function_exists('pcntl_signal')) {
pcntl_signal(SIGTERM, 'ssh_connect_signal');
}
function ssh_connect_signal($signo) {
// This is just letting destructors fire. In particular, we want to clean
// up any temporary files we wrote. See T10547.
exit(128 + $signo);
}
Simplify Repository remote and local command construction Summary: This cleans up some garbage: - We were specifying environmental variables with `X=y git ...`, but now have `setEnv()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setEnv()`. - We were specifying the working directory with `(cd %s && git ...)`, but now have `setCWD()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setCWD()`. - We were specifying the Git credentials with `ssh-agent -c (ssh-add ... && git ...)`. We can do this more cleanly with `GIT_SSH`. Use `GIT_SSH`. - Since we have to write a script for `GIT_SSH` anyway, use the same script for Subversion and Mercurial. This fixes two specific issues: - Previously, we were not able to set `-o StrictHostKeyChecking=no` on Git commands, so the first time you cloned a git repo the daemons would generally prompt you to add `github.com` or whatever to `known_hosts`. Since this was non-interactive, things would mysteriously hang, in effect. With `GIT_SSH`, we can specify the flag, reducing the number of ways things can go wrong. - This adds `LANG=C`, which probably (?) forces the language to English for all commands. Apparently you need to install special language packs or something, so I don't know that this actually works, but at least two users with non-English languages have claimed it does (see <https://github.com/facebook/arcanist/pull/114> for a similar issue in Arcanist). At some point in the future I might want to combine the Arcanist code for command execution with the Phabricator code for command execution (they share some stuff like LANG and HGPLAIN). However, credential management is kind of messy, so I'm adopting a "wait and see" approach for now. I expect to split this at least somewhat in the future, for Drydock/Automerge if nothing else. Also I'm not sure if we use the passthru stuff at all anymore, I may just be able to delete that. I'll check in a future diff. Test Plan: Browsed and pulled Git, Subversion and Mercurial repositories. Reviewers: btrahan Reviewed By: btrahan CC: aran Maniphest Tasks: T2230 Differential Revision: https://secure.phabricator.com/D7600
2013-11-20 19:41:35 +01:00
$pattern = array();
$arguments = array();
$pattern[] = 'ssh';
$pattern[] = '-o';
$pattern[] = 'StrictHostKeyChecking=no';
// This prevents "known host" failures, and covers for issues where HOME is set
// to something unusual.
$pattern[] = '-o';
$pattern[] = 'UserKnownHostsFile=/dev/null';
Allow cluster devices to SSH to one another without acting as a user Summary: Ref T4292. When you run `git fetch` and connect to, say, `repo001.west.company.com`, we'll look at the current version of the repository in other nodes in the cluster. If `repo002.east.company.com` has a newer version of the repository, we'll fetch that version first, then respond to your request. To do this, we need to run `git fetch repo002.east.company.com ...` and have that connect to the other host and be able to fetch data. This change allows us to run `PHABRICATOR_AS_DEVICE=1 git fetch ...` to use device credentials to do this fetch. (Device credentials are already supported and used, they just always connect as a user right now, but these fetches should be doable without having a user. We will have a valid user when you run `git fetch` yourself, but we won't have one if the daemons notice that a repository is out of date and want to update it, so the update code should not depend on having a user.) Test Plan: ``` $ PHABRICATOR_AS_DEVICE=1 ./bin/ssh-connect local.phacility.com Warning: Permanently added 'local.phacility.com' (RSA) to the list of known hosts. PTY allocation request failed on channel 0 phabricator-ssh-exec: Welcome to Phabricator. You are logged in as device/daemon.phacility.net. You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH. Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH. Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve. Connection to local.phacility.com closed. ``` Reviewers: chad Reviewed By: chad Maniphest Tasks: T4292 Differential Revision: https://secure.phabricator.com/D15755
2016-04-19 15:19:39 +02:00
$as_device = getenv('PHABRICATOR_AS_DEVICE');
$credential_phid = getenv('PHABRICATOR_CREDENTIAL');
Allow cluster devices to SSH to one another without acting as a user Summary: Ref T4292. When you run `git fetch` and connect to, say, `repo001.west.company.com`, we'll look at the current version of the repository in other nodes in the cluster. If `repo002.east.company.com` has a newer version of the repository, we'll fetch that version first, then respond to your request. To do this, we need to run `git fetch repo002.east.company.com ...` and have that connect to the other host and be able to fetch data. This change allows us to run `PHABRICATOR_AS_DEVICE=1 git fetch ...` to use device credentials to do this fetch. (Device credentials are already supported and used, they just always connect as a user right now, but these fetches should be doable without having a user. We will have a valid user when you run `git fetch` yourself, but we won't have one if the daemons notice that a repository is out of date and want to update it, so the update code should not depend on having a user.) Test Plan: ``` $ PHABRICATOR_AS_DEVICE=1 ./bin/ssh-connect local.phacility.com Warning: Permanently added 'local.phacility.com' (RSA) to the list of known hosts. PTY allocation request failed on channel 0 phabricator-ssh-exec: Welcome to Phabricator. You are logged in as device/daemon.phacility.net. You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH. Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH. Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve. Connection to local.phacility.com closed. ``` Reviewers: chad Reviewed By: chad Maniphest Tasks: T4292 Differential Revision: https://secure.phabricator.com/D15755
2016-04-19 15:19:39 +02:00
if ($as_device) {
$device = AlmanacKeys::getLiveDevice();
if (!$device) {
throw new Exception(
pht(
'Attempting to create an SSH connection that authenticates with '.
'the current device, but this host is not configured as a cluster '.
'device.'));
}
if ($credential_phid) {
throw new Exception(
pht(
'Attempting to proxy an SSH connection that authenticates with '.
'both the current device and a specific credential. These options '.
'are mutually exclusive.'));
}
}
if ($credential_phid) {
$viewer = PhabricatorUser::getOmnipotentUser();
$key = PassphraseSSHKey::loadFromPHID($credential_phid, $viewer);
Simplify Repository remote and local command construction Summary: This cleans up some garbage: - We were specifying environmental variables with `X=y git ...`, but now have `setEnv()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setEnv()`. - We were specifying the working directory with `(cd %s && git ...)`, but now have `setCWD()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setCWD()`. - We were specifying the Git credentials with `ssh-agent -c (ssh-add ... && git ...)`. We can do this more cleanly with `GIT_SSH`. Use `GIT_SSH`. - Since we have to write a script for `GIT_SSH` anyway, use the same script for Subversion and Mercurial. This fixes two specific issues: - Previously, we were not able to set `-o StrictHostKeyChecking=no` on Git commands, so the first time you cloned a git repo the daemons would generally prompt you to add `github.com` or whatever to `known_hosts`. Since this was non-interactive, things would mysteriously hang, in effect. With `GIT_SSH`, we can specify the flag, reducing the number of ways things can go wrong. - This adds `LANG=C`, which probably (?) forces the language to English for all commands. Apparently you need to install special language packs or something, so I don't know that this actually works, but at least two users with non-English languages have claimed it does (see <https://github.com/facebook/arcanist/pull/114> for a similar issue in Arcanist). At some point in the future I might want to combine the Arcanist code for command execution with the Phabricator code for command execution (they share some stuff like LANG and HGPLAIN). However, credential management is kind of messy, so I'm adopting a "wait and see" approach for now. I expect to split this at least somewhat in the future, for Drydock/Automerge if nothing else. Also I'm not sure if we use the passthru stuff at all anymore, I may just be able to delete that. I'll check in a future diff. Test Plan: Browsed and pulled Git, Subversion and Mercurial repositories. Reviewers: btrahan Reviewed By: btrahan CC: aran Maniphest Tasks: T2230 Differential Revision: https://secure.phabricator.com/D7600
2013-11-20 19:41:35 +01:00
$pattern[] = '-l %P';
$arguments[] = $key->getUsernameEnvelope();
$pattern[] = '-i %P';
$arguments[] = $key->getKeyfileEnvelope();
Simplify Repository remote and local command construction Summary: This cleans up some garbage: - We were specifying environmental variables with `X=y git ...`, but now have `setEnv()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setEnv()`. - We were specifying the working directory with `(cd %s && git ...)`, but now have `setCWD()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setCWD()`. - We were specifying the Git credentials with `ssh-agent -c (ssh-add ... && git ...)`. We can do this more cleanly with `GIT_SSH`. Use `GIT_SSH`. - Since we have to write a script for `GIT_SSH` anyway, use the same script for Subversion and Mercurial. This fixes two specific issues: - Previously, we were not able to set `-o StrictHostKeyChecking=no` on Git commands, so the first time you cloned a git repo the daemons would generally prompt you to add `github.com` or whatever to `known_hosts`. Since this was non-interactive, things would mysteriously hang, in effect. With `GIT_SSH`, we can specify the flag, reducing the number of ways things can go wrong. - This adds `LANG=C`, which probably (?) forces the language to English for all commands. Apparently you need to install special language packs or something, so I don't know that this actually works, but at least two users with non-English languages have claimed it does (see <https://github.com/facebook/arcanist/pull/114> for a similar issue in Arcanist). At some point in the future I might want to combine the Arcanist code for command execution with the Phabricator code for command execution (they share some stuff like LANG and HGPLAIN). However, credential management is kind of messy, so I'm adopting a "wait and see" approach for now. I expect to split this at least somewhat in the future, for Drydock/Automerge if nothing else. Also I'm not sure if we use the passthru stuff at all anymore, I may just be able to delete that. I'll check in a future diff. Test Plan: Browsed and pulled Git, Subversion and Mercurial repositories. Reviewers: btrahan Reviewed By: btrahan CC: aran Maniphest Tasks: T2230 Differential Revision: https://secure.phabricator.com/D7600
2013-11-20 19:41:35 +01:00
}
Allow cluster devices to SSH to one another without acting as a user Summary: Ref T4292. When you run `git fetch` and connect to, say, `repo001.west.company.com`, we'll look at the current version of the repository in other nodes in the cluster. If `repo002.east.company.com` has a newer version of the repository, we'll fetch that version first, then respond to your request. To do this, we need to run `git fetch repo002.east.company.com ...` and have that connect to the other host and be able to fetch data. This change allows us to run `PHABRICATOR_AS_DEVICE=1 git fetch ...` to use device credentials to do this fetch. (Device credentials are already supported and used, they just always connect as a user right now, but these fetches should be doable without having a user. We will have a valid user when you run `git fetch` yourself, but we won't have one if the daemons notice that a repository is out of date and want to update it, so the update code should not depend on having a user.) Test Plan: ``` $ PHABRICATOR_AS_DEVICE=1 ./bin/ssh-connect local.phacility.com Warning: Permanently added 'local.phacility.com' (RSA) to the list of known hosts. PTY allocation request failed on channel 0 phabricator-ssh-exec: Welcome to Phabricator. You are logged in as device/daemon.phacility.net. You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH. Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH. Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve. Connection to local.phacility.com closed. ``` Reviewers: chad Reviewed By: chad Maniphest Tasks: T4292 Differential Revision: https://secure.phabricator.com/D15755
2016-04-19 15:19:39 +02:00
if ($as_device) {
$pattern[] = '-l %R';
$arguments[] = AlmanacKeys::getClusterSSHUser();
$pattern[] = '-i %R';
$arguments[] = AlmanacKeys::getKeyPath('device.key');
}
// Subversion passes us a host in the form "domain.com:port", which is not
// valid for normal SSH but which we can parse into a valid "-p" flag.
$passthru_args = $unconsumed_argv;
$host = array_shift($passthru_args);
$parts = explode(':', $host, 2);
$host = $parts[0];
$port = $args->getArg('port');
if (!$port) {
if (count($parts) == 2) {
$port = $parts[1];
}
}
if ($port) {
$pattern[] = '-p %d';
$arguments[] = $port;
}
Add parsing for ssh options (-o) which are passed when using GIT v2 wire protocol by git command (SSH transport) Summary: Makes `ssh-connect` compatible with Git v2 wire protocol over SSH More details about git V2 wire: https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html `git` command (2.18+) passes extra options (`-o "SendEnv GIT_PROTOCOL"`) to underlying `ssh` command to enable v2 wire protocol (environment variable enabling new protocol). Phabricator `ssh-connect` command doesn't understand `-o` options and interprets it as host parts hence when you enable git v2 all clones/ls-remotes crash with: ``` #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:525] #1 PhabricatorRepository::execxRemoteCommand(string, PhutilOpaqueEnvelope) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:400] #2 PhabricatorRepositoryPullEngine::loadGitRemoteRefs(PhabricatorRepository) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:343] #3 PhabricatorRepositoryPullEngine::executeGitUpdate() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:126] #4 PhabricatorRepositoryPullEngine::pullRepositoryWithLock() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:40] #5 PhabricatorRepositoryPullEngine::pullRepository() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:59] #6 PhabricatorRepositoryManagementUpdateWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441] #7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333] #8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22] COMMAND git ls-remote '********' STDOUT (empty) STDERR ssh: Could not resolve hostname -o: Name or service not known fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. at [<phutil>/src/future/exec/ExecFuture.php:369] ``` Test Plan: How to reproduce: 1. add repository to Phabricator which is accessed via `ssh` 2. Use git 2.18+ 3. Enable wire protocol in `/etc/gitconfig`: ``` [protocol] version = 2 ``` 4. Try refreshing repository: `phabricator/bin/repository update somecallsing` 5. Repository update fails with `ssh: Could not resolve hostname -o: Name or service not known` after this changes - updates will succeed Reviewers: epriestley, Pawka, #blessed_reviewers Reviewed By: epriestley, #blessed_reviewers Subscribers: Korvin Differential Revision: https://secure.phabricator.com/D19542
2018-08-01 13:42:52 +02:00
$options = $args->getArg('options');
$allowed_ssh_options = array('SendEnv=GIT_PROTOCOL');
if (!empty($options)) {
foreach ($options as $option) {
if (array_search($option, $allowed_ssh_options) !== false) {
$pattern[] = '-o %s';
$arguments[] = $option;
} else {
throw new Exception(
pht(
'Disallowed ssh option "%s" given with "-o". '.
'Allowed options are: %s.',
$option,
implode(', ', $allowed_ssh_options)));
}
}
}
Simplify Repository remote and local command construction Summary: This cleans up some garbage: - We were specifying environmental variables with `X=y git ...`, but now have `setEnv()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setEnv()`. - We were specifying the working directory with `(cd %s && git ...)`, but now have `setCWD()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setCWD()`. - We were specifying the Git credentials with `ssh-agent -c (ssh-add ... && git ...)`. We can do this more cleanly with `GIT_SSH`. Use `GIT_SSH`. - Since we have to write a script for `GIT_SSH` anyway, use the same script for Subversion and Mercurial. This fixes two specific issues: - Previously, we were not able to set `-o StrictHostKeyChecking=no` on Git commands, so the first time you cloned a git repo the daemons would generally prompt you to add `github.com` or whatever to `known_hosts`. Since this was non-interactive, things would mysteriously hang, in effect. With `GIT_SSH`, we can specify the flag, reducing the number of ways things can go wrong. - This adds `LANG=C`, which probably (?) forces the language to English for all commands. Apparently you need to install special language packs or something, so I don't know that this actually works, but at least two users with non-English languages have claimed it does (see <https://github.com/facebook/arcanist/pull/114> for a similar issue in Arcanist). At some point in the future I might want to combine the Arcanist code for command execution with the Phabricator code for command execution (they share some stuff like LANG and HGPLAIN). However, credential management is kind of messy, so I'm adopting a "wait and see" approach for now. I expect to split this at least somewhat in the future, for Drydock/Automerge if nothing else. Also I'm not sure if we use the passthru stuff at all anymore, I may just be able to delete that. I'll check in a future diff. Test Plan: Browsed and pulled Git, Subversion and Mercurial repositories. Reviewers: btrahan Reviewed By: btrahan CC: aran Maniphest Tasks: T2230 Differential Revision: https://secure.phabricator.com/D7600
2013-11-20 19:41:35 +01:00
$pattern[] = '--';
$pattern[] = '%s';
$arguments[] = $host;
Simplify Repository remote and local command construction Summary: This cleans up some garbage: - We were specifying environmental variables with `X=y git ...`, but now have `setEnv()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setEnv()`. - We were specifying the working directory with `(cd %s && git ...)`, but now have `setCWD()` on both `ExecFuture` and `PhutilExecPassthru`. Use `setCWD()`. - We were specifying the Git credentials with `ssh-agent -c (ssh-add ... && git ...)`. We can do this more cleanly with `GIT_SSH`. Use `GIT_SSH`. - Since we have to write a script for `GIT_SSH` anyway, use the same script for Subversion and Mercurial. This fixes two specific issues: - Previously, we were not able to set `-o StrictHostKeyChecking=no` on Git commands, so the first time you cloned a git repo the daemons would generally prompt you to add `github.com` or whatever to `known_hosts`. Since this was non-interactive, things would mysteriously hang, in effect. With `GIT_SSH`, we can specify the flag, reducing the number of ways things can go wrong. - This adds `LANG=C`, which probably (?) forces the language to English for all commands. Apparently you need to install special language packs or something, so I don't know that this actually works, but at least two users with non-English languages have claimed it does (see <https://github.com/facebook/arcanist/pull/114> for a similar issue in Arcanist). At some point in the future I might want to combine the Arcanist code for command execution with the Phabricator code for command execution (they share some stuff like LANG and HGPLAIN). However, credential management is kind of messy, so I'm adopting a "wait and see" approach for now. I expect to split this at least somewhat in the future, for Drydock/Automerge if nothing else. Also I'm not sure if we use the passthru stuff at all anymore, I may just be able to delete that. I'll check in a future diff. Test Plan: Browsed and pulled Git, Subversion and Mercurial repositories. Reviewers: btrahan Reviewed By: btrahan CC: aran Maniphest Tasks: T2230 Differential Revision: https://secure.phabricator.com/D7600
2013-11-20 19:41:35 +01:00
foreach ($passthru_args as $passthru_arg) {
$pattern[] = '%s';
$arguments[] = $passthru_arg;
}
$pattern = implode(' ', $pattern);
array_unshift($arguments, $pattern);
$err = newv('PhutilExecPassthru', $arguments)
->execute();
exit($err);