1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-28 17:52:43 +01:00
phorge-phorge/src/applications/harbormaster
epriestley 4d5e8a149a Split Harbormaster workers apart so build steps can run in parallel
Summary:
Ref T1049. Currently, the Harbormaster worker looks like this:

  foreach (step) {
    run_step(step);
  }

This means steps can't ever be run in parallel. Instead, split it into two workers. The "Build" worker starts things off, and basically does:

  update_build();

(We could theoretically do this in the original process because it should never take very long, but since there's a lock and it's a little bit complex it seemed cleaner to separate it.)

The "Target" worker runs an individual target (like a command, or an HTTP request, or whatever), then updates the build:

  run_one_step(step);
  update_build();

The new `update_build()` mechanism in `HarbormasterBuildEngine` does this, roughly:

  figure_out_overall_status_of_all_steps();
  if (build is done) { done(); }
  if (build is fail) { fail(); }
  foreach (step that is ready to run) {
    queue_target_worker_for_step(step);
  }

So, overall:

  - The part of the code that updates Builds is completely separated from the part of the code that updates Targets.
  - Targets can run in parallel.

Test Plan:
  - Ran a bunch of builds via `bin/harbormaster build`.
  - Ran a bunch of builds via web UI.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7890
2014-01-06 12:32:10 -08:00
..
application Formalize "manual" buildables in Harbormaster 2013-12-26 10:40:43 -08:00
capability Harbormaster v(-2) 2013-10-22 15:01:06 -07:00
config Migrate "Run Command" to use Drydock hosts 2013-12-05 14:06:22 +11:00
controller Add bin/harbormaster to make builds easier to debug 2013-12-26 10:40:52 -08:00
editor Allow Harbormaster build plans to be disabled 2013-12-26 10:40:22 -08:00
engine Split Harbormaster workers apart so build steps can run in parallel 2014-01-06 12:32:10 -08:00
event Use correct method name in Harbormaster UI event listener 2013-12-26 11:37:25 -08:00
interface Add bin/harbormaster to make builds easier to debug 2013-12-26 10:40:52 -08:00
management Extend all "ManagementWorkflow" classes from a base class 2013-12-27 13:15:40 -08:00
phid Render build status on revisions and commits 2013-11-09 15:04:00 -08:00
query Formalize "manual" buildables in Harbormaster 2013-12-26 10:40:43 -08:00
remarkup Harbormaster v(-2) 2013-10-22 15:01:06 -07:00
step Make Drydock more broadly aware of policies 2013-12-27 13:15:19 -08:00
storage Make Drydock more broadly aware of policies 2013-12-27 13:15:19 -08:00
view Replace some hsprintf() by phutil_tag() 2013-11-11 09:23:23 -08:00
worker Split Harbormaster workers apart so build steps can run in parallel 2014-01-06 12:32:10 -08:00