1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-30 18:52:42 +01:00
phorge-phorge/src/applications/harbormaster/query/HarbormasterBuildMessageQuery.php

99 lines
2.3 KiB
PHP
Raw Normal View History

Allow external systems to send messages to build targets Summary: Ref T1049. Allows external systems to send a message to a build target. The primary intended use case is: - You make an HTTP request to Jenkins. - The build goes into a "waiting" state. - Later, Jenkins calls `harbormaster.sendmessage` to report that the target passed or failed. - The build continues as appropriate. This is deceptively complicated because: - There are a lot of race concerns. We might get a message back from an external system before it even responds to the request we made. We want to make sure we process these messages no matter when we receive them. - These messages need to be sent to a build target (vs a build or buildable) because we'll get into trouble with parallelization later on otherwise (Jenkins is told to do 3 builds; we can't tell which ones failed or what overall state is unless the message are sent to targets). - I initially thought about implementing this as a separate "Wait for a response from an external system" build step. This gets a lot more complicated for users once we do parallelization, though. Particularly, in the case where you've told Jenkins to do 3 builds, the three "wait" steps need to know which target they're waiting for (and jenkins needs to know some unique identifier for each target). So this pretty much boils down to a more complicated, more error-prone version of using target PHIDs. This makes the already-muddy Build UI a bit worse, but it needs a general clarity pass anyway (it's showing way too much uninteresting data, and should show a better summary of results instead). Test Plan: - This doesn't really do anything interesting yet. - Used Conduit to send messages to build plans. - Viewed the messages on the build screen. Reviewers: btrahan Reviewed By: btrahan Subscribers: epriestley Maniphest Tasks: T1049 Differential Revision: https://secure.phabricator.com/D8604
2014-03-26 00:11:28 +01:00
<?php
final class HarbormasterBuildMessageQuery
extends PhabricatorCursorPagedPolicyAwareQuery {
private $ids;
private $buildTargetPHIDs;
private $consumed;
public function withIDs(array $ids) {
$this->ids = $ids;
return $this;
}
public function withBuildTargetPHIDs(array $phids) {
$this->buildTargetPHIDs = $phids;
return $this;
}
public function withConsumed($consumed) {
$this->consumed = $consumed;
return $this;
}
protected function loadPage() {
$table = new HarbormasterBuildMessage();
$conn_r = $table->establishConnection('r');
$data = queryfx_all(
$conn_r,
'SELECT * FROM %T %Q %Q %Q',
$table->getTableName(),
$this->buildWhereClause($conn_r),
$this->buildOrderClause($conn_r),
$this->buildLimitClause($conn_r));
return $table->loadAllFromArray($data);
}
protected function willFilterPage(array $page) {
$build_target_phids = array_filter(mpull($page, 'getBuildTargetPHID'));
if ($build_target_phids) {
$build_targets = id(new PhabricatorObjectQuery())
->setViewer($this->getViewer())
->withPHIDs($build_target_phids)
->setParentQuery($this)
->execute();
$build_targets = mpull($build_targets, null, 'getPHID');
} else {
$build_targets = array();
}
foreach ($page as $key => $message) {
$build_target_phid = $message->getBuildTargetPHID();
if (empty($build_targets[$build_target_phid])) {
unset($page[$key]);
continue;
}
$message->attachBuildTarget($build_targets[$build_target_phid]);
}
return $page;
}
protected function buildWhereClause(AphrontDatabaseConnection $conn_r) {
Allow external systems to send messages to build targets Summary: Ref T1049. Allows external systems to send a message to a build target. The primary intended use case is: - You make an HTTP request to Jenkins. - The build goes into a "waiting" state. - Later, Jenkins calls `harbormaster.sendmessage` to report that the target passed or failed. - The build continues as appropriate. This is deceptively complicated because: - There are a lot of race concerns. We might get a message back from an external system before it even responds to the request we made. We want to make sure we process these messages no matter when we receive them. - These messages need to be sent to a build target (vs a build or buildable) because we'll get into trouble with parallelization later on otherwise (Jenkins is told to do 3 builds; we can't tell which ones failed or what overall state is unless the message are sent to targets). - I initially thought about implementing this as a separate "Wait for a response from an external system" build step. This gets a lot more complicated for users once we do parallelization, though. Particularly, in the case where you've told Jenkins to do 3 builds, the three "wait" steps need to know which target they're waiting for (and jenkins needs to know some unique identifier for each target). So this pretty much boils down to a more complicated, more error-prone version of using target PHIDs. This makes the already-muddy Build UI a bit worse, but it needs a general clarity pass anyway (it's showing way too much uninteresting data, and should show a better summary of results instead). Test Plan: - This doesn't really do anything interesting yet. - Used Conduit to send messages to build plans. - Viewed the messages on the build screen. Reviewers: btrahan Reviewed By: btrahan Subscribers: epriestley Maniphest Tasks: T1049 Differential Revision: https://secure.phabricator.com/D8604
2014-03-26 00:11:28 +01:00
$where = array();
if ($this->ids) {
$where[] = qsprintf(
$conn_r,
'id IN (%Ld)',
$this->ids);
}
if ($this->buildTargetPHIDs) {
$where[] = qsprintf(
$conn_r,
'buildTargetPHID IN (%Ls)',
$this->buildTargetPHIDs);
}
if ($this->consumed !== null) {
$where[] = qsprintf(
$conn_r,
'isConsumed = %d',
Allow Harbormaster build targets to wait for messages Summary: This hooks up all the pieces of the build pipeline so `harbormaster.sendmessage` actually works. Particularly: - Candidate build steps (i.e., those which interact with external systems) can now "Wait for Message". This pauses them indefinitely when they complete, until something calls `harbormaster.sendmessage`. - After processing a target, we check if we should move it to PASSED or WAITING. - Before updating a build, we move WAITING targets with pending messages to either PASSED or FAILED. - I added an explicit "Building" state, which doesn't affect workflows but communicates more information to human users. A big part of this is avoiding races. I believe we get the correct behavior no matter which order events occur in: - We update builds after targets complete and after we receive messages, so we're guaranteed to update once both these conditions are true. This means messages can't be lost (even if they arrive before a build completes). - The minor changes to the build engine logic mean that firing additional build updates is always safe, no matter what the current state of the build is. - The build itself is protected by a lock in the build engine. - The target is not covered by an explicit lock, but for all states only the engine (waiting) //or// the worker (all other states) can interact with it. All of the interactions also move the target state forward to the same destination and have no other side effects. - Messages are only consumed inside the engine lock, so they don't need an explicit lock. Test Plan: - Made an HTTP request wait after completion, then ran a pile of builds through it using `bin/harbormaster build` and the web UI. - Passed and failed message-awaiting builds with `harbormaster.sendmessage`. Reviewers: btrahan Reviewed By: btrahan Subscribers: epriestley, zeeg Differential Revision: https://secure.phabricator.com/D8788
2014-04-16 22:01:46 +02:00
(int)$this->consumed);
Allow external systems to send messages to build targets Summary: Ref T1049. Allows external systems to send a message to a build target. The primary intended use case is: - You make an HTTP request to Jenkins. - The build goes into a "waiting" state. - Later, Jenkins calls `harbormaster.sendmessage` to report that the target passed or failed. - The build continues as appropriate. This is deceptively complicated because: - There are a lot of race concerns. We might get a message back from an external system before it even responds to the request we made. We want to make sure we process these messages no matter when we receive them. - These messages need to be sent to a build target (vs a build or buildable) because we'll get into trouble with parallelization later on otherwise (Jenkins is told to do 3 builds; we can't tell which ones failed or what overall state is unless the message are sent to targets). - I initially thought about implementing this as a separate "Wait for a response from an external system" build step. This gets a lot more complicated for users once we do parallelization, though. Particularly, in the case where you've told Jenkins to do 3 builds, the three "wait" steps need to know which target they're waiting for (and jenkins needs to know some unique identifier for each target). So this pretty much boils down to a more complicated, more error-prone version of using target PHIDs. This makes the already-muddy Build UI a bit worse, but it needs a general clarity pass anyway (it's showing way too much uninteresting data, and should show a better summary of results instead). Test Plan: - This doesn't really do anything interesting yet. - Used Conduit to send messages to build plans. - Viewed the messages on the build screen. Reviewers: btrahan Reviewed By: btrahan Subscribers: epriestley Maniphest Tasks: T1049 Differential Revision: https://secure.phabricator.com/D8604
2014-03-26 00:11:28 +01:00
}
$where[] = $this->buildPagingClause($conn_r);
return $this->formatWhereClause($where);
}
public function getQueryApplicationClass() {
return 'PhabricatorHarbormasterApplication';
Allow external systems to send messages to build targets Summary: Ref T1049. Allows external systems to send a message to a build target. The primary intended use case is: - You make an HTTP request to Jenkins. - The build goes into a "waiting" state. - Later, Jenkins calls `harbormaster.sendmessage` to report that the target passed or failed. - The build continues as appropriate. This is deceptively complicated because: - There are a lot of race concerns. We might get a message back from an external system before it even responds to the request we made. We want to make sure we process these messages no matter when we receive them. - These messages need to be sent to a build target (vs a build or buildable) because we'll get into trouble with parallelization later on otherwise (Jenkins is told to do 3 builds; we can't tell which ones failed or what overall state is unless the message are sent to targets). - I initially thought about implementing this as a separate "Wait for a response from an external system" build step. This gets a lot more complicated for users once we do parallelization, though. Particularly, in the case where you've told Jenkins to do 3 builds, the three "wait" steps need to know which target they're waiting for (and jenkins needs to know some unique identifier for each target). So this pretty much boils down to a more complicated, more error-prone version of using target PHIDs. This makes the already-muddy Build UI a bit worse, but it needs a general clarity pass anyway (it's showing way too much uninteresting data, and should show a better summary of results instead). Test Plan: - This doesn't really do anything interesting yet. - Used Conduit to send messages to build plans. - Viewed the messages on the build screen. Reviewers: btrahan Reviewed By: btrahan Subscribers: epriestley Maniphest Tasks: T1049 Differential Revision: https://secure.phabricator.com/D8604
2014-03-26 00:11:28 +01:00
}
}