revi-archive/llvm-premerge-checks
1
0
Fork 0
Code Activity

update docs

Browse source 
- move "user_doc" to "README" as it's publicly facing page visible when one opens repo
- move developer docs to "development"
- update docs to reflect current buildkite setup (not completely)
- grammar
This commit is contained in:
Mikhail Goncharov 2020-08-27 11:21:35 +02:00
parent 46c9fefb77
commit 056e5cdbaa
6 changed files with 181 additions and 180 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

71
README.md
View file

@ -1,39 +1,66 @@
# Overview # Overview
This repository contains the configuration files for the pre-merge checks for the LLVM project. This github project contains the documentation and the server configuration cluster of build machines that are used to check all incoming commits to the LLVM project. The *pre-merge checks* for the [LLVM project](http://llvm.org/) are a [continuous integration (CI)](https://en.wikipedia.org/wiki/Continuous_integration) workflow. The workflow checks the patches the developers upload to the [LLVM Phabricator](https://reviews.llvm.org) instance. *Phabricator* is the code review tool in the LLVM project. The workflow checks the patches before a user merges them the master branch - thus the term *pre-merge testing*. When a user uploads a patch to the LLVM Phabricator, Phabricator triggers the checks and then displays the results.
# User documentation The CI system checks the patches **before** a user merges them to the master branch. This way bugs in a patch are contained during the code review stage and do not pollute the master branch. The more bugs the CI system can catch during the code review phase, the more stable and bug-free the master branch will become.
If you are interested in a high-level overview and how to best use the tool see [docs/user_doc.md](docs/user_doc.md)
# Cluster overview This repository contains the configurations and script to run pre-merge checks for the LLVM project.
The cluster consists of these services: ## Feedback
* Jenkins build server: http://jenkins.llvm-merge-guard.org
* a set of Jenkins agents running the builds
* an nginx server with the build results/logs http://results.llvm-merge-guard.org
![deployment diagram](http://www.plantuml.com/plantuml/proxy?src=https://raw.githubusercontent.com/google/llvm-premerge-checks/master/docs/deployment.plantuml) If you notice issues or have an idea on how to improve pre-merge checks, please create a [new issue](https://github.com/google/llvm-premerge-checks/issues/new) or give a
:heart: to an existing one.
# Jenkins-Phabricator integration ## Sign up for beta-test
The Jenkins-Phabricator is based on the instructions provided with the [Phabricator-Jenkins Plugin](https://github.com/uber/phabricator-jenkins-plugin). To get the latest features and help us developing the project, sign up for the pre-merge beta testing by adding yourself to the ["pre-merge beta testing" project](https://reviews.llvm.org/project/members/78/) on Phabricator.
On the Phabricator side these things were configured: ## Opt-out
* the Harbormaster [build plan](https://reviews.llvm.org/harbormaster/plan/3/)
* the Herald [rule](https://reviews.llvm.org/H511)
* the [merge_guards_bot user](https://reviews.llvm.org/p/merge_guards_bot/) writing the comments
On the Jenkins side: In case you want to opt-out entirely of pre-merge testing, add yourself to the [OPT OUT project](https://reviews.llvm.org/project/view/83/).
* in the Jenkins configuration page as explained in the instrucitons
* in the build [job](http://jenkins.llvm-merge-guard.org/job/Phabricator/)
* The Phabricator pluging is *not* used, as it's not flexible enough. Rather Phabricator just triggers the build via an HTTP request. The `arc patch` operations by scripts. The build feedback is also uploaded by scripts via the [harbormaster.sendmessage](https://secure.phabricator.com/conduit/method/harbormaster.sendmessage/) and [differential.revision.edit](https://secure.phabricator.com/conduit/method/differential.revision.edit/) APIs.
There is no backup of the credentials. If you need to change it, generate a new one and update it in Jenkins and Phabricator. If you decide to opt-out, please let us know why, so we might be able to improve in the future.
# Requirements
The builds are only triggered if the Revision in Phabricator is created/updated via `arc diff`.
If you update a Revision via the Web UI it will [not trigger](https://secure.phabricator.com/Q447) a build.
To get a patch on Phabricator tested the build server must be able to apply the patch to the checked out git repository.
If you want to get your patch tested, please make sure that either:
* You set a git hash as `sourceControlBaseRevision` in Phabricator which is available on the Github repository,
* **or** you define the dependencies of your patch in Phabricator,
* **or** your patch can be applied to the master branch.
Only then can the build server apply the patch locally and run the builds and tests.
# Accessing results on Phabricator
TODO: move closer to top and explain what different links mean currently. Update screenshots.
Phabricator will automatically trigger a build for every new patch you upload or modify.
Phabricator shows the build results at the top of the entry:
![build status](docs/images/diff_detail.png)
CI will compile and run tests, run clang-format and [clang-tidy](docs/clang_tidy.md) on lines changed.
If a unit test failed, this is shown below the build status. You can also expand the unit test to see the details:
![unit test results](docs/images/unit_tests.png).
# Contributing
We're happy to get help on improving the infrastructure and workflows!
Please check [contibuting](docs/contributing.md) first.
[Development](docs/development.md) gives an overview how different parts interact together.
[Playbooks](docs/playbooks.md) shows concrete examples how to, for example, build and run agents locally.
If you have any questions please contact [Christian Kühnel](mailto:kuhnel@google.com) or [Mikhail Goncharov](mailto:goncahrov@google.com).
# Additional Information # Additional Information
* [Playbooks](docs/playbooks.md) for installing/upgrading agents and testing changes. * [Playbooks](docs/playbooks.md) for installing/upgrading agents and testing changes.
* [User documentation](docs/user_doc.md)
* [Log of the service operations](https://github.com/google/llvm-premerge-checks/wiki/LLVM-pre-merge-tests-operations-blog) * [Log of the service operations](https://github.com/google/llvm-premerge-checks/wiki/LLVM-pre-merge-tests-operations-blog)
# License # License
This project is licensed unter the "Apache 2.0 with LLVM Exception" license. See [LICENSE](LICENSE) for details. This project is licensed under the "Apache 2.0 with LLVM Exception" license. See [LICENSE](LICENSE) for details.

2
docs/clang_tidy.md
View file

@ -26,4 +26,4 @@ If your are confident that some files are in good shape already, please
---- ----
[about pre-merge checks](docs/user_doc.md) [about pre-merge checks](../README.md)

59
docs/deployment.plantuml
View file

@ -1,53 +1,24 @@
@startuml @startuml
component phabricator
node Phabricator component buildkite
node github { node github {
component LLVM_monorepo component "llvm-project"
component "llvm-project fork"
component "llvm-premerge-checks"
} }
cloud "Google Cloud" { node "GCP" {
component "buildkite agents"
node "premerge-test Kubernetes cluster" { component "phab-proxy"
node "reverse proxy" {
component nginx
}
node "nginx results" {
component "Linux results"
}
node "Jenkins" {
component "Jenkins master"
}
node "Linux agent" {
component "Swarm plugin (Linux)"
component "LLVM build tools (Linux)"
[Swarm plugin (Linux)] -- [Jenkins master]
[Swarm plugin (Linux)] --> [Linux results]
}
} }
node "Google Cloud Storage" { [phabricator] --> [phab-proxy]
component "Windows results" [phab-proxy] --> [buildkite]
} [buildkite] <-- [buildkite agents]
[buildkite agents] <-- [llvm-project]
node "Windows agent" { [buildkite agents] <--> [llvm-project fork]
component "Swarm plugin (Win)" [buildkite agents] --> [phabricator]
component "LLVM build tools (Win)" [buildkite agents] <-- [llvm-premerge-checks]
[Swarm plugin (Win)] -- [Jenkins master]
[Swarm plugin (Win)] --> [Windows results]
}
}
Phabricator --> nginx
nginx --> [Jenkins master]
[Jenkins master] --> LLVM_monorepo
[Swarm plugin (Linux)] --> LLVM_monorepo
[Swarm plugin (Win)] --> LLVM_monorepo
@enduml @enduml

100
docs/development.md Normal file
View file

@ -0,0 +1,100 @@
- [Overview](#overview)
- [Phabricator integration](#phabricator-integration)
- [Buildkite pipelines](#buildkite-pipelines)
- [Life of pre-merge check](#life-of-pre-merge-check)
- [Enabled projects and project detection](#enabled-projects-and-project-detection)
- [Agent machines](#agent-machines)
- [Compilation caching](#compilation-caching)
- [Buildkite monitoring](#buildkite-monitoring)
<small><i><a href='http://ecotrust-canada.github.io/markdown-toc/'>Table of contents generated with markdown-toc</a></i></small>
# Overview
* [Buildkite](https://buildkite.com/llvm-project) orchestrates each build.
* multiple Linux and windows agents connected to Buildkite. Agents are run at Google Cloud Platform.
* [small proxy service](/phabricator-proxy) that takes build requests from [reviews.llvm.org](http://reviews.llvm.org) and converts them into Buildkite build request. Buildkite job sends build results directly to Phabricator.
* every review creates a new branch in [fork of llvm-project](https://github.com/llvm-premerge-tests/llvm-project).
![deployment diagram](http://www.plantuml.com/plantuml/proxy?src=https://raw.githubusercontent.com/google/llvm-premerge-checks/master/docs/deployment.plantuml)
# Phabricator integration
* [Harbormaster build plan](https://reviews.llvm.org/harbormaster/plan/5On the Phabricator side these things were configured:/)
* Herald [rule for everyone](https://reviews.llvm.org/H576) and for [beta testers](https://reviews.llvm.org/H511).
Note that right now there is no difference between beta and "normal" builds.
* the [merge_guards_bot user](https://reviews.llvm.org/p/merge_guards_bot/) account for writing comments.
# Buildkite pipelines
Buildkite allows [dynamically define pipelines as the output of a command](https://buildkite.com/docs/pipelines/defining-steps#dynamic-pipelines).
That gives us the flexibility to generate pipeline code using a script from a specific branch of pre-merge checks.
For example, "pre-merge" pipeline has a single fixed step, that checks out this repo and runs a python script to
generate further steps:
```shell script
export SRC=${BUILDKITE_BUILD_PATH}/llvm-premerge-checks
rm -rf ${SRC}
git clone --depth 1 --branch ${scripts_branch:-master} https://github.com/google/llvm-premerge-checks.git ${SRC}
export SCRIPT_DIR=${SRC}/scripts
${SCRIPT_DIR}/buildkite/build_branch_pipeline.py | tee /dev/tty | buildkite-agent pipeline upload
```
One typically edits corresponding script, not a pipeline definition in the Buildkite interface. [How to test changes](playbooks.md#testing-changes-before-merging).
# Life of pre-merge check
When new diff arrives for review it triggers Herald rule ("everyone" or "beta testers").
That in sends an HTTP POST request to [**phab-proxy**](../phabricator-proxy) that submits a new buildkite job **diff-checks**.
All parameters from the original request are put in the build's environment with `ph_` prefix (to avoid shadowing any Buildkite environment variable).
"scripts_branch" parameter defines which branch of llvm-premerge-checks to use.
**diff-checks** pipeline ([create_branch_pipeline.py](../scripts/buildkite/create_branch_pipeline.py)) downloads a patch (or series of patches) and applies it to a fork of the llvm-project repository.
Then it pushes a new state as a new branch (e.g. "phab-diff-288211") and triggers "premerge-checks" on it (all "ph_" env variables are passed to it).
This new branch can now be used to reproduce the build or by another tooling.
Periodical **cleanup-branches** pipeline deletes branches older than 30 days.
**premerge-checks** pipeline ([build_branch_pipeline.py](../scripts/buildkite/build_branch_pipeline.py)) builds and tests
changes on Linux and Windows agents. Then it uploads a combined result to Phabricator.
# Enabled projects and project detection
To reduce build times and mask unrelated problems, we're only building and testing the projects that were modified by a patch.
[choose_projects.py](../scripts/choose_projects.py) uses manually maintained [config file](../scripts/llvm-dependencies.yaml) to define inter-project dependencies and exclude projects:
1. Get prefix (e.g. "llvm", "clang") of all paths modified by a patch.
1. Add all dependant projects.
1. Add all projects that this extended list depends on, completing the dependency subtree.
1. Remove all disabled projects.
# Agent machines
All build machines are running from Docker containers so that they can be debugged, updated, and scaled easily:
* [Linux](../containers/buildkite-premerge-debian/Dockerfile).
We use [Kubernetes deployemnt](../kubernetes/buildkite) to manage these agents.
* [Windows](../containers/agent-windows-buildkite/Dockerfile) based on [Windows vs2019](../containers/agent-windows-vs2019).
At the moment they are run as multiple individual VM instances.
See [playbooks](playbooks.md) how to manage and set up machines.
# Compilation caching
Each build is performed on a clean copy of the git repository. To speed up the builds [ccache](https://ccache.dev/) is used on Linux and [sccache](https://github.com/mozilla/sccache) on Windows.
# Buildkite monitoring
VM instance `buildkite-monitoring` exposes Buildkite metrics to GCP.
To set up a new instance:
1. Create as small Linux VM with full access to *Stackdriver Monitoring API*.
2. Follow instructions to [install monitoring agent](https://cloud.google.com/monitoring/agent/install-agent) and [enable statsd plugin](https://cloud.google.com/monitoring/agent/plugins/statsd).
3. Download recent release of [buildkite-agent-metrics](https://github.com/buildkite/buildkite-agent-metrics/releases).
4. Run in SSH session:
```bash
chmod +x buildkite-agent-metrics-linux-amd64
nohup ./buildkite-agent-metrics-linux-amd64 -token XXXX -interval 30s -backend statsd &
```
.
TODO: update "Testing scripts locally" playbook on how to run Linux build locally with Docker.

31
docs/playbooks.md
View file

@ -1,5 +1,20 @@
# Playbooks - [Playbooks](#playbooks)
* [deployment to a clean infrastructure](#deployment-to-a-clean-infrastructure)
* [creating basic authentication for reverse proxy](#creating-basic-authentication-for-reverse-proxy)
* [Creating docker containers on Windows](#creating-docker-containers-on-windows)
* [Spawning a new windows agent](#spawning-a-new-windows-agent)
+ [Buildkite](#buildkite)
+ [Jenkins](#jenkins)
* [Testing scripts locally](#testing-scripts-locally)
* [Custom environment variables](#custom-environment-variables)
* [Testing changes before merging](#testing-changes-before-merging)
- [Phabricator integration](#phabricator-integration)
* [Herald](#herald)
* [Harbormaster](#harbormaster)
<small><i><a href='http://ecotrust-canada.github.io/markdown-toc/'>Table of contents generated with markdown-toc</a></i></small>
# Playbooks
## deployment to a clean infrastructure ## deployment to a clean infrastructure
@ -150,20 +165,6 @@ schtasks.exe /create /tn "Start Buildkite agent" /ru SYSTEM /SC ONSTART /DELAY 0
C:\llvm-premerge-checks\scripts\windows_agent_start_buildkite.ps1 [-testing] [-version latest] C:\llvm-premerge-checks\scripts\windows_agent_start_buildkite.ps1 [-testing] [-version latest]
``` ```
## Buildkite monitoring
VM instance `buildkite-monitoring` exposes Buildkite metrics to GCP.
To setup a new instance:
1. Create as small linux VM with full access to *Stackdriver Monitoring API*.
2. Follow instructions to [install moninorign agent](https://cloud.google.com/monitoring/agent/install-agent) and [enable statsd plugin](https://cloud.google.com/monitoring/agent/plugins/statsd).
3. Download recent release of [buildkite-agent-metrics](https://github.com/buildkite/buildkite-agent-metrics/releases).
4. Run in SSH session:
```bash
chmod +x buildkite-agent-metrics-linux-amd64
nohup ./buildkite-agent-metrics-linux-amd64 -token XXXX -interval 30s -backend statsd &
```
.
Metrics are exported as "custom/statsd/gauge". Metrics are exported as "custom/statsd/gauge".
## Testing scripts locally ## Testing scripts locally

98
docs/user_doc.md
View file

@ -1,98 +0,0 @@
# Pre-merge checks
The *pre-merge checks* for the [LLVM project](http://llvm.org/) are a [continuous integration (CI)](https://en.wikipedia.org/wiki/Continuous_integration) workflow. The workflow checks the patches the developers upload to the [LLVM Phabricator](https://reviews.llvm.org) instance. *Phabricator* is the code review tool in the LLVM project. The workflow checks the patches before a user merges them the master branch - thus the term *pre-merge testing*. When a user uploads a patch to the LLVM Phabricator, Phabricator triggers the checks and then displays the results.
The CI system checks the patches **before** a user merges them to the master branch. This way bugs in a patch are contained during the code review stage and do not pollute the master branch. The more bugs the CI system can catch during the code review phase, the more stable and bug-free the master branch will become.
## Stages
The *checks* comprise of separate stages:
* Apply patch
1. Checkout of the LLVM git repository
1. Apply the patch -- `arc patch`
1. Create a new git branch and store it in https://github.com/llvm-premerge-tests/llvm-project/branches
1. Upload build results to Phabricator
* Linux
1. Checkout of the branch (from apply patch)
1. Guess which projects were modified, run Cmake for those.
1. Build the binaries -- `ninja all`
1. Run the test suite -- `ninja check-all`
1. Run clang-format and clang-tidy on the diff.
1. Upload build results to Phabricator
* Windows (beta testing only)
1. Checkout of the branch (from apply patch)
1. Guess which projects were modified, run Cmake for those.
1. Build the binaries -- `ninja all`
1. Run the test suite -- `ninja check-all`
1. Upload build results to Phabricator
The checks are executed on one Linux platform (Debian Testing on amd64 with the clang-8 tool chain) at the moment. Builds and Test for Windows (Windows 10, amd64, Visual Studio 2019). The plan is to add more platforms, in the future.
## Enabled projects and project detection
To reduce build times and mask unrelated problems, we're only building and testing the projects that were modified by a patch. The logic for that looks like this:
1. Get prefix (e.g. llvm, clang) of all paths modified by the patch.
1. Identify the projects that depend on these, based a manually maintained [config file](https://github.com/google/llvm-premerge-checks/blob/master/scripts/llvm-dependencies.yaml).
1. Add all projects that this extended list depends on to be built, based on the same [config file](https://github.com/google/llvm-premerge-checks/blob/master/scripts/llvm-dependencies.yaml)
1. Remove all `excludedProjects` projects, based on the same [config file](https://github.com/google/llvm-premerge-checks/blob/master/scripts/llvm-dependencies.yaml). These projects were blacklisted as they fail building and/or testing on the current machines.
1. Then use the list of projects as arguments in `cmake -D LLVM_ENABLE_PROJECTS=<project list>`.
## Machine configuration
All build machines are running from Docker containers so that they can be debugged, updated and scaled easily:
* [Linux image](https://github.com/google/llvm-premerge-checks/blob/master/containers/agent-debian-testing-ssd/Dockerfile)
* [Windows base image](https://github.com/google/llvm-premerge-checks/blob/master/containers/agent-windows-vs2019/Dockerfile) and [Windows Jenkins config](https://github.com/google/llvm-premerge-checks/blob/master/containers/agent-windows-jenkins/Dockerfile) on top of base image
## Clean builds and caching
Each build is performed on a clean copy of the git repository. To speed up the builds [ccache](https://ccache.dev/) is used on Linux and [sccache](https://github.com/mozilla/sccache) on Windows.
## Feedback
If you find any problems please raise an [issue on github](https://github.com/google/llvm-premerge-checks/issues).
## Opt out
In case you want to opt out entirely of pre-merge testing, add yourself to the [OPT OUT project](https://reviews.llvm.org/project/view/83/).
If you decide to opt out, please let us know why, so we might be able to improve in the future.
## Sign up for beta test
To get the latest features, sign up for the pre-merge beta testing by adding yourself to the ["pre-merge beta testing" project](https://reviews.llvm.org/project/members/78/) on Phabricator.
# Requirements
The builds are only triggered if the Revision in Phabricator is created/updated via `arc diff`. If you update a Revision via the Web UI it will [not trigger](https://secure.phabricator.com/Q447) a build.
To get a patch on Phabricator tested the build server must be able to apply the patch to the checked out git repository. If you want to get your patch tested, please make sure that that either:
* You set a git hash as `sourceControlBaseRevision` in Phabricator which is available on the Github repository,
* **or** you define the dependencies of your patch in Phabricator,
* **or** your patch can be applied to the master branch.
Only then can the build server apply the patch locally and run the builds and tests.
# Integration in Phabricator
Once you're signed up, Phabricator will automatically trigger a build for every new patch you upload or every existing patch you modify. Phabricator shows the build results at the top of the entry:
![build status](images/diff_detail.png)
Bot will compile and run tests, run clang-format and [clang-tidy](docs/clang_tidy.md) on lines changed.
If a unit test failed, this is shown below the build status. You can also expand the unit test to see the details:
![unit test results](images/unit_tests.png)
The build logs are stored for 90 days and automatically deleted after that.
You can also trigger a build manually by using the "Run Plan Manually" link on the [Harbormaster page](https://reviews.llvm.org/harbormaster/plan/3/) and entering a revision ID in the pop-up window.
# Reporting issues
If you notice any bugs, please create a [new issue](https://github.com/google/llvm-premerge-checks/issues/new).
# Contributing
We're happy to get help on improving the infrastructure and the workflows. If you're interested please contact [Christian Kühnel](mailto:kuhnel@google.com).