llvm-premerge-checks/docs/monitoring_main_branch.md

Monitoring the LLVM main branch

This document proposes a design to improve the stability of the LLVM main branch for use in pre-merge testing.

Background

The stability of the pre-merge testing largely depends on the stability of the LLVM main branch: Whenever something is broken on the main branch, also pre-merge testing will fail: The patches from Phabricator are applied to some revision on the main branch before they can be built. So the fewer revisions of the main branch being broken, the more stable pre-merge testing will be.

High-level Design

We propose to run a Buildbot worker on the main branch with the same Docker image we're using for pre-merge testing. That worker shall check all commits to main for build and test failures with regards to the configuration we're using for pre-merge testing. Whenever these builds fails, Buildbot notifies the commiters and gives them the opportunity to fix or revert their patch.

This is much faster than a having a human investigate the issue and notify the committers. By having faster feedback the main branch is broken for fewer revisions and this the probability of a false-positive pre-merge test is lower.

Machine setup

We would deploy another container as part of the existing Kubernetes cluster for pre-merge testing. The image would be based on buildkite-premerge-debian , and we would just add the things needed for the Buildbot agent by resuing the setup of an existing worker (e.g. clangd-ubuntu-clang ).

Build configuration

The Buildbot worker should run as much of the pre-merge testing scripts as possible to get the results as close as we can to that of what we would see happen in pre-merge testing.

Currently we're running these steps on Buildkite to check the main branch:

export SRC=${BUILDKITE_BUILD_PATH}/llvm-premerge-checks
rm -rf ${SRC}
git clone --depth 1 https://github.com/google/llvm-premerge-checks.git "${SRC}"
cd "${SRC}"
git fetch origin "${ph_scripts_refspec:-master}":x
git checkout x
cd "$BUILDKITE_BUILD_CHECKOUT_PATH"
pip install -q -r ${SRC}/scripts/requirements.txt
${SRC}/scripts/pipeline_main.py | tee /dev/tty | buildkite-agent pipeline upload

We should run similar steps on Buildbot by creating a new BuilderFactory to the existing list and then configuring it as usual.

Open questions

1. Do we actually need to create a BuilderFactory or can we also just run a shell script?
2. How can we quickly test and iterate on the Python scripting? How can we run our own Buildbot master for testing?
3. The current solution by only wrapping the existing pre-merge testing scripts will not result in nice build steps as the other builders are producing as buildkite does not have that concept. Is that an issue?
4. We do need to check out the pre-merge testing config and scripts form another git repo. Is that an issue?