llvm-premerge-checks/docs/development.md

• Overview
• Phabricator integration
• Buildkite pipelines
• Life of a pre-merge check
• Cluster parts
  • Ingress and public addresses
  • Linux agents
  • Windows agents
• Enabled projects and project detection
• Agent machines
• Compilation caching
• Buildkite monitoring

Overview

• Buildkite orchestrates each build.

• multiple Linux and windows agents connected to Buildkite. Agents are run at Google Cloud Platform.

• small proxy service that takes build requests from reviews.llvm.org and converts them into Buildkite build request. Buildkite job sends build results directly to Phabricator.

• every review creates a new branch in fork of llvm-project.

Phabricator integration

• Harbormaster build plan the Phabricator side these things were configured

• Herald rule for everyone and for beta testers. Note that right now there is no difference between beta and "normal" builds.

• the merge_guards_bot user account for writing comments.

Buildkite pipelines

Buildkite allows dynamically define pipelines as the output of a command. That gives us the flexibility to generate pipeline code using a script from a specific branch of pre-merge checks.

For example, "pre-merge" pipeline has a single fixed step, that checks out this repo and runs a python script to generate further steps:

export SRC=${BUILDKITE_BUILD_PATH}/llvm-premerge-checks
rm -rf ${SRC}
git clone --depth 1 --branch ${scripts_branch:-master} https://github.com/google/llvm-premerge-checks.git ${SRC}
export SCRIPT_DIR=${SRC}/scripts
${SCRIPT_DIR}/buildkite/build_branch_pipeline.py | tee /dev/tty | buildkite-agent pipeline upload

One typically edits corresponding script, not a pipeline definition in the Buildkite interface. How to test changes.

Life of a pre-merge check

When new diff arrives for review it triggers a Herald rule ("everyone" or "beta testers").

That in sends an HTTP POST request to phab-proxy that submits a new buildkite job diff-checks. All parameters from the original request are put in the build's environment with ph_ prefix (to avoid shadowing any Buildkite environment variable). "scripts_branch" parameter defines which branch of llvm-premerge-checks to use.

diff-checks pipeline (create_branch_pipeline.py) downloads a patch (or series of patches) and applies it to a fork of the llvm-project repository. Then it pushes a new state as a new branch (e.g. "phab-diff-288211") and triggers "premerge-checks" on it (all "ph_" env variables are passed to it). This new branch can now be used to reproduce the build or by another tooling. Periodical cleanup-branches pipeline deletes branches older than 30 days.

premerge-checks pipeline (build_branch_pipeline.py) builds and tests changes on Linux and Windows agents. Then it uploads a combined result to Phabricator.

Cluster parts

Ingress and public addresses

https://build.llvm-merge-guard.org/ URL points to phabricator proxy application.

We use NGINX ingress for Kubernetes and Let's Encrypt certificate manager. Follow up to date docs to install reverse proxy and certificate manager.

lets-encrypt configuration.

Access to the service is restricted with basic HTTP auth. It's configured with k8s secret 'http-auth' in 'buildkite' namespace (see how to update auth)

llvm-merge-guard.org domain is managed by Google Domains.

Linux agents

• docker image buildkite-premerge-debian.

• Kubernetes manifests.

Windows agents

• docker image agent-windows-buildkite.

• VMs are manually managed and updated, use RDP to access.

• there is an 'windows development' VM to do Windows-related development.

Enabled projects and project detection

To reduce build times and mask unrelated problems, we're only building and testing the projects that were modified by a patch. choose_projects.py uses manually maintained config file to define inter-project dependencies and exclude projects:

1. Get prefix (e.g. "llvm", "clang") of all paths modified by a patch.

2. Add all dependant projects.

3. Add all projects that this extended list depends on, completing the dependency subtree.

4. Remove all disabled projects.

Agent machines

All build machines are running from Docker containers so that they can be debugged, updated, and scaled easily:

• Linux. We use Kubernetes deployment to manage these agents.

• Windows based on Windows vs2019. At the moment they are run as multiple individual VM instances.

See playbooks how to manage and set up machines.

Compilation caching

Each build is performed on a clean copy of the git repository. To speed up the builds ccache is used on Linux and sccache on Windows.

Buildkite monitoring

VM instance buildkite-monitoring exposes Buildkite metrics to GCP. To set up a new instance:

1. Create as small Linux VM with full access to Stackdriver Monitoring API.

2. Follow instructions to install monitoring agent and enable statsd plugin.

3. Download recent release of buildkite-agent-metrics.

4. Run in SSH session:

chmod +x buildkite-agent-metrics-linux-amd64
nohup ./buildkite-agent-metrics-linux-amd64 -token XXXX -interval 30s -backend statsd &

Metrics are exported as "custom/statsd/gauge".

TODO: update "Testing scripts locally" playbook on how to run Linux build locally with Docker. TODO: migrate 'builkite-monitoring' to k8s deployment.