2016-04-19 00:22:52 +02:00
|
|
|
<?php
|
|
|
|
|
|
|
|
abstract class DiffusionCommandEngine extends Phobject {
|
|
|
|
|
|
|
|
private $repository;
|
|
|
|
private $protocol;
|
|
|
|
private $credentialPHID;
|
|
|
|
private $argv;
|
|
|
|
private $passthru;
|
Allow cluster devices to SSH to one another without acting as a user
Summary:
Ref T4292. When you run `git fetch` and connect to, say, `repo001.west.company.com`, we'll look at the current version of the repository in other nodes in the cluster.
If `repo002.east.company.com` has a newer version of the repository, we'll fetch that version first, then respond to your request.
To do this, we need to run `git fetch repo002.east.company.com ...` and have that connect to the other host and be able to fetch data.
This change allows us to run `PHABRICATOR_AS_DEVICE=1 git fetch ...` to use device credentials to do this fetch. (Device credentials are already supported and used, they just always connect as a user right now, but these fetches should be doable without having a user. We will have a valid user when you run `git fetch` yourself, but we won't have one if the daemons notice that a repository is out of date and want to update it, so the update code should not depend on having a user.)
Test Plan:
```
$ PHABRICATOR_AS_DEVICE=1 ./bin/ssh-connect local.phacility.com
Warning: Permanently added 'local.phacility.com' (RSA) to the list of known hosts.
PTY allocation request failed on channel 0
phabricator-ssh-exec: Welcome to Phabricator.
You are logged in as device/daemon.phacility.net.
You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH.
Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH.
Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve.
Connection to local.phacility.com closed.
```
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15755
2016-04-19 15:19:39 +02:00
|
|
|
private $connectAsDevice;
|
Synchronize (hosted, clustered, Git) repositories over Conduit + HTTP
Summary:
Ref T4292. We currently synchronize hosted, clustered, Git repositories when we receive an SSH pull or push.
Additionally:
- Synchronize before HTTP reads and writes.
- Synchronize reads before Conduit requests.
We could relax Conduit eventually and allow Diffusion to say "it's OK to give me stale data".
We could also redirect some set of these actions to just go to the up-to-date host instead of connecting to a random host and synchronizing it. However, this potentially won't work as well at scale: if you have a larger number of servers, it sends all of the traffic to the leader immediately following a write. That can cause "thundering herd" issues, and isn't efficient if replicas are in different geographical regions and the write just went to the east coast but most clients are on the west coast. In large-scale cases, it's better to go to the local replica, wait for an update, then serve traffic from it -- particularly given that writes are relatively rare. But we can finesse this later once things are solid.
Test Plan:
- Pushed and pulled a Git repository over HTTP.
- Browsed a Git repository from the web UI.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15758
2016-04-19 17:03:12 +02:00
|
|
|
private $sudoAsDaemon;
|
2016-04-19 00:22:52 +02:00
|
|
|
|
|
|
|
public static function newCommandEngine(PhabricatorRepository $repository) {
|
|
|
|
$engines = self::newCommandEngines();
|
|
|
|
|
|
|
|
foreach ($engines as $engine) {
|
|
|
|
if ($engine->canBuildForRepository($repository)) {
|
|
|
|
return id(clone $engine)
|
|
|
|
->setRepository($repository);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
throw new Exception(
|
|
|
|
pht(
|
|
|
|
'No registered command engine can build commands for this '.
|
|
|
|
'repository ("%s").',
|
|
|
|
$repository->getDisplayName()));
|
|
|
|
}
|
|
|
|
|
|
|
|
private static function newCommandEngines() {
|
|
|
|
return id(new PhutilClassMapQuery())
|
|
|
|
->setAncestorClass(__CLASS__)
|
|
|
|
->execute();
|
|
|
|
}
|
|
|
|
|
|
|
|
abstract protected function canBuildForRepository(
|
|
|
|
PhabricatorRepository $repository);
|
|
|
|
|
|
|
|
abstract protected function newFormattedCommand($pattern, array $argv);
|
|
|
|
abstract protected function newCustomEnvironment();
|
|
|
|
|
|
|
|
public function setRepository(PhabricatorRepository $repository) {
|
|
|
|
$this->repository = $repository;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function getRepository() {
|
|
|
|
return $this->repository;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function setProtocol($protocol) {
|
|
|
|
$this->protocol = $protocol;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function getProtocol() {
|
|
|
|
return $this->protocol;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function setCredentialPHID($credential_phid) {
|
|
|
|
$this->credentialPHID = $credential_phid;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function getCredentialPHID() {
|
|
|
|
return $this->credentialPHID;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function setArgv(array $argv) {
|
|
|
|
$this->argv = $argv;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function getArgv() {
|
|
|
|
return $this->argv;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function setPassthru($passthru) {
|
|
|
|
$this->passthru = $passthru;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function getPassthru() {
|
|
|
|
return $this->passthru;
|
|
|
|
}
|
|
|
|
|
Allow cluster devices to SSH to one another without acting as a user
Summary:
Ref T4292. When you run `git fetch` and connect to, say, `repo001.west.company.com`, we'll look at the current version of the repository in other nodes in the cluster.
If `repo002.east.company.com` has a newer version of the repository, we'll fetch that version first, then respond to your request.
To do this, we need to run `git fetch repo002.east.company.com ...` and have that connect to the other host and be able to fetch data.
This change allows us to run `PHABRICATOR_AS_DEVICE=1 git fetch ...` to use device credentials to do this fetch. (Device credentials are already supported and used, they just always connect as a user right now, but these fetches should be doable without having a user. We will have a valid user when you run `git fetch` yourself, but we won't have one if the daemons notice that a repository is out of date and want to update it, so the update code should not depend on having a user.)
Test Plan:
```
$ PHABRICATOR_AS_DEVICE=1 ./bin/ssh-connect local.phacility.com
Warning: Permanently added 'local.phacility.com' (RSA) to the list of known hosts.
PTY allocation request failed on channel 0
phabricator-ssh-exec: Welcome to Phabricator.
You are logged in as device/daemon.phacility.net.
You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH.
Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH.
Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve.
Connection to local.phacility.com closed.
```
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15755
2016-04-19 15:19:39 +02:00
|
|
|
public function setConnectAsDevice($connect_as_device) {
|
|
|
|
$this->connectAsDevice = $connect_as_device;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function getConnectAsDevice() {
|
|
|
|
return $this->connectAsDevice;
|
|
|
|
}
|
|
|
|
|
Synchronize (hosted, clustered, Git) repositories over Conduit + HTTP
Summary:
Ref T4292. We currently synchronize hosted, clustered, Git repositories when we receive an SSH pull or push.
Additionally:
- Synchronize before HTTP reads and writes.
- Synchronize reads before Conduit requests.
We could relax Conduit eventually and allow Diffusion to say "it's OK to give me stale data".
We could also redirect some set of these actions to just go to the up-to-date host instead of connecting to a random host and synchronizing it. However, this potentially won't work as well at scale: if you have a larger number of servers, it sends all of the traffic to the leader immediately following a write. That can cause "thundering herd" issues, and isn't efficient if replicas are in different geographical regions and the write just went to the east coast but most clients are on the west coast. In large-scale cases, it's better to go to the local replica, wait for an update, then serve traffic from it -- particularly given that writes are relatively rare. But we can finesse this later once things are solid.
Test Plan:
- Pushed and pulled a Git repository over HTTP.
- Browsed a Git repository from the web UI.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15758
2016-04-19 17:03:12 +02:00
|
|
|
public function setSudoAsDaemon($sudo_as_daemon) {
|
|
|
|
$this->sudoAsDaemon = $sudo_as_daemon;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function getSudoAsDaemon() {
|
|
|
|
return $this->sudoAsDaemon;
|
|
|
|
}
|
|
|
|
|
2016-04-19 00:22:52 +02:00
|
|
|
public function newFuture() {
|
|
|
|
$argv = $this->newCommandArgv();
|
|
|
|
$env = $this->newCommandEnvironment();
|
|
|
|
|
Synchronize (hosted, clustered, Git) repositories over Conduit + HTTP
Summary:
Ref T4292. We currently synchronize hosted, clustered, Git repositories when we receive an SSH pull or push.
Additionally:
- Synchronize before HTTP reads and writes.
- Synchronize reads before Conduit requests.
We could relax Conduit eventually and allow Diffusion to say "it's OK to give me stale data".
We could also redirect some set of these actions to just go to the up-to-date host instead of connecting to a random host and synchronizing it. However, this potentially won't work as well at scale: if you have a larger number of servers, it sends all of the traffic to the leader immediately following a write. That can cause "thundering herd" issues, and isn't efficient if replicas are in different geographical regions and the write just went to the east coast but most clients are on the west coast. In large-scale cases, it's better to go to the local replica, wait for an update, then serve traffic from it -- particularly given that writes are relatively rare. But we can finesse this later once things are solid.
Test Plan:
- Pushed and pulled a Git repository over HTTP.
- Browsed a Git repository from the web UI.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15758
2016-04-19 17:03:12 +02:00
|
|
|
if ($this->getSudoAsDaemon()) {
|
|
|
|
$command = call_user_func_array('csprintf', $argv);
|
|
|
|
$command = PhabricatorDaemon::sudoCommandAsDaemonUser($command);
|
|
|
|
$argv = array('%C', $command);
|
|
|
|
}
|
|
|
|
|
2016-04-19 00:22:52 +02:00
|
|
|
if ($this->getPassthru()) {
|
|
|
|
$future = newv('PhutilExecPassthru', $argv);
|
|
|
|
} else {
|
|
|
|
$future = newv('ExecFuture', $argv);
|
|
|
|
}
|
|
|
|
|
|
|
|
$future->setEnv($env);
|
|
|
|
|
|
|
|
return $future;
|
|
|
|
}
|
|
|
|
|
|
|
|
private function newCommandArgv() {
|
|
|
|
$argv = $this->argv;
|
|
|
|
$pattern = $argv[0];
|
|
|
|
$argv = array_slice($argv, 1);
|
|
|
|
|
|
|
|
list($pattern, $argv) = $this->newFormattedCommand($pattern, $argv);
|
|
|
|
|
|
|
|
return array_merge(array($pattern), $argv);
|
|
|
|
}
|
|
|
|
|
|
|
|
private function newCommandEnvironment() {
|
|
|
|
$env = $this->newCommonEnvironment() + $this->newCustomEnvironment();
|
|
|
|
foreach ($env as $key => $value) {
|
|
|
|
if ($value === null) {
|
|
|
|
unset($env[$key]);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return $env;
|
|
|
|
}
|
|
|
|
|
|
|
|
private function newCommonEnvironment() {
|
Allow cluster devices to SSH to one another without acting as a user
Summary:
Ref T4292. When you run `git fetch` and connect to, say, `repo001.west.company.com`, we'll look at the current version of the repository in other nodes in the cluster.
If `repo002.east.company.com` has a newer version of the repository, we'll fetch that version first, then respond to your request.
To do this, we need to run `git fetch repo002.east.company.com ...` and have that connect to the other host and be able to fetch data.
This change allows us to run `PHABRICATOR_AS_DEVICE=1 git fetch ...` to use device credentials to do this fetch. (Device credentials are already supported and used, they just always connect as a user right now, but these fetches should be doable without having a user. We will have a valid user when you run `git fetch` yourself, but we won't have one if the daemons notice that a repository is out of date and want to update it, so the update code should not depend on having a user.)
Test Plan:
```
$ PHABRICATOR_AS_DEVICE=1 ./bin/ssh-connect local.phacility.com
Warning: Permanently added 'local.phacility.com' (RSA) to the list of known hosts.
PTY allocation request failed on channel 0
phabricator-ssh-exec: Welcome to Phabricator.
You are logged in as device/daemon.phacility.net.
You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH.
Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH.
Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve.
Connection to local.phacility.com closed.
```
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15755
2016-04-19 15:19:39 +02:00
|
|
|
$repository = $this->getRepository();
|
|
|
|
|
2016-04-19 00:22:52 +02:00
|
|
|
$env = array();
|
|
|
|
// NOTE: Force the language to "en_US.UTF-8", which overrides locale
|
|
|
|
// settings. This makes stuff print in English instead of, e.g., French,
|
|
|
|
// so we can parse the output of some commands, error messages, etc.
|
|
|
|
$env['LANG'] = 'en_US.UTF-8';
|
|
|
|
|
|
|
|
// Propagate PHABRICATOR_ENV explicitly. For discussion, see T4155.
|
|
|
|
$env['PHABRICATOR_ENV'] = PhabricatorEnv::getSelectedEnvironmentName();
|
|
|
|
|
Allow cluster devices to SSH to one another without acting as a user
Summary:
Ref T4292. When you run `git fetch` and connect to, say, `repo001.west.company.com`, we'll look at the current version of the repository in other nodes in the cluster.
If `repo002.east.company.com` has a newer version of the repository, we'll fetch that version first, then respond to your request.
To do this, we need to run `git fetch repo002.east.company.com ...` and have that connect to the other host and be able to fetch data.
This change allows us to run `PHABRICATOR_AS_DEVICE=1 git fetch ...` to use device credentials to do this fetch. (Device credentials are already supported and used, they just always connect as a user right now, but these fetches should be doable without having a user. We will have a valid user when you run `git fetch` yourself, but we won't have one if the daemons notice that a repository is out of date and want to update it, so the update code should not depend on having a user.)
Test Plan:
```
$ PHABRICATOR_AS_DEVICE=1 ./bin/ssh-connect local.phacility.com
Warning: Permanently added 'local.phacility.com' (RSA) to the list of known hosts.
PTY allocation request failed on channel 0
phabricator-ssh-exec: Welcome to Phabricator.
You are logged in as device/daemon.phacility.net.
You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH.
Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH.
Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve.
Connection to local.phacility.com closed.
```
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15755
2016-04-19 15:19:39 +02:00
|
|
|
$as_device = $this->getConnectAsDevice();
|
|
|
|
$credential_phid = $this->getCredentialPHID();
|
|
|
|
|
|
|
|
if ($as_device) {
|
|
|
|
$device = AlmanacKeys::getLiveDevice();
|
|
|
|
if (!$device) {
|
|
|
|
throw new Exception(
|
|
|
|
pht(
|
|
|
|
'Attempting to build a reposiory command (for repository "%s") '.
|
|
|
|
'as device, but this host ("%s") is not configured as a cluster '.
|
|
|
|
'device.',
|
|
|
|
$repository->getDisplayName(),
|
|
|
|
php_uname('n')));
|
|
|
|
}
|
|
|
|
|
|
|
|
if ($credential_phid) {
|
|
|
|
throw new Exception(
|
|
|
|
pht(
|
|
|
|
'Attempting to build a repository command (for repository "%s"), '.
|
|
|
|
'but the CommandEngine is configured to connect as both the '.
|
|
|
|
'current cluster device ("%s") and with a specific credential '.
|
|
|
|
'("%s"). These options are mutually exclusive. Connections must '.
|
|
|
|
'authenticate as one or the other, not both.',
|
|
|
|
$repository->getDisplayName(),
|
|
|
|
$device->getName(),
|
|
|
|
$credential_phid));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2016-04-19 00:22:52 +02:00
|
|
|
if ($this->isAnySSHProtocol()) {
|
|
|
|
if ($credential_phid) {
|
|
|
|
$env['PHABRICATOR_CREDENTIAL'] = $credential_phid;
|
|
|
|
}
|
Allow cluster devices to SSH to one another without acting as a user
Summary:
Ref T4292. When you run `git fetch` and connect to, say, `repo001.west.company.com`, we'll look at the current version of the repository in other nodes in the cluster.
If `repo002.east.company.com` has a newer version of the repository, we'll fetch that version first, then respond to your request.
To do this, we need to run `git fetch repo002.east.company.com ...` and have that connect to the other host and be able to fetch data.
This change allows us to run `PHABRICATOR_AS_DEVICE=1 git fetch ...` to use device credentials to do this fetch. (Device credentials are already supported and used, they just always connect as a user right now, but these fetches should be doable without having a user. We will have a valid user when you run `git fetch` yourself, but we won't have one if the daemons notice that a repository is out of date and want to update it, so the update code should not depend on having a user.)
Test Plan:
```
$ PHABRICATOR_AS_DEVICE=1 ./bin/ssh-connect local.phacility.com
Warning: Permanently added 'local.phacility.com' (RSA) to the list of known hosts.
PTY allocation request failed on channel 0
phabricator-ssh-exec: Welcome to Phabricator.
You are logged in as device/daemon.phacility.net.
You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH.
Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH.
Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve.
Connection to local.phacility.com closed.
```
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15755
2016-04-19 15:19:39 +02:00
|
|
|
if ($as_device) {
|
|
|
|
$env['PHABRICATOR_AS_DEVICE'] = 1;
|
|
|
|
}
|
2016-04-19 00:22:52 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
return $env;
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function isSSHProtocol() {
|
|
|
|
return ($this->getProtocol() == 'ssh');
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function isSVNProtocol() {
|
|
|
|
return ($this->getProtocol() == 'svn');
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function isSVNSSHProtocol() {
|
|
|
|
return ($this->getProtocol() == 'svn+ssh');
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function isHTTPProtocol() {
|
|
|
|
return ($this->getProtocol() == 'http');
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function isHTTPSProtocol() {
|
|
|
|
return ($this->getProtocol() == 'https');
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function isAnyHTTPProtocol() {
|
|
|
|
return ($this->isHTTPProtocol() || $this->isHTTPSProtocol());
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function isAnySSHProtocol() {
|
|
|
|
return ($this->isSSHProtocol() || $this->isSVNSSHProtocol());
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function getSSHWrapper() {
|
|
|
|
$root = dirname(phutil_get_library_root('phabricator'));
|
|
|
|
return $root.'/bin/ssh-connect';
|
|
|
|
}
|
|
|
|
|
|
|
|
}
|