revi-archive/llvm-premerge-checks
1
0
Fork 0
Code Activity

remove obsolete code

Browse source 
most of the scripts were used for Phabricator integration that is now disabled and current scripts are located in `llvm-project/.ci` . Also removed obsolete playbooks and scripts that were not used and likely no longer work. Updated wording and minor typos.

I have tried to look through potential references to the scripts and disabled some of buildkite pipelines that referred to them. That is not a guarantee that this will not break anything as I have not tried to rebuild containers and redeploy cluster.
This commit is contained in:
Mikhail Goncharov 2024-02-20 10:07:48 +01:00
parent 2d073b0307
commit 7f0564d978
114 changed files with 3 additions and 9405 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

87
README.md
View file

@ -1,6 +1,6 @@
This repo is holding VM configurations for machine cluster and scripts to run pre-merge tests triggered by http://reviews.llvm.org. This repo is holding VM configurations for machine cluster and scripts to run pre-merge tests triggered by http://reviews.llvm.org.
As LLVM project has moved to Pull Requests and Phabricator will no longer trigger builds, this repository will likely be gone. As LLVM project has moved to Pull Requests and Phabricator no longer triggers builds, this repository will likely be gone.
[Pull request migration schedule](https://discourse.llvm.org/t/pull-request-migration-schedule/71595). [Pull request migration schedule](https://discourse.llvm.org/t/pull-request-migration-schedule/71595).
@ -10,100 +10,17 @@ Presentation by Louis Dione on LLVM devmtg 2021 https://youtu.be/B7gB6van7Bw
[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/B7gB6van7Bw/0.jpg)](https://www.youtube.com/watch?v=B7gB6van7Bw) [![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/B7gB6van7Bw/0.jpg)](https://www.youtube.com/watch?v=B7gB6van7Bw)
The *pre-merge checks* for the [LLVM project](http://llvm.org/) are a
[continuous integration
(CI)](https://en.wikipedia.org/wiki/Continuous_integration) workflow. The
workflow checks the patches the developers upload to the [LLVM
Phabricator](https://reviews.llvm.org) instance.
*Phabricator* (https://reviews.llvm.org) is the code review tool in the LLVM
project.
The workflow checks the patches before a user merges them to the main branch -
thus the term *pre-merge testing**. When a user uploads a patch to the LLVM
Phabricator, Phabricator triggers the checks and then displays the results.
The CI system checks the patches **before** a user merges them to the main
branch. This way bugs in a patch are contained during the code review stage and
do not pollute the main branch. The more bugs the CI system can catch during
the code review phase, the more stable and bug-free the main branch will
become. <sup>[citation needed]()</sup>
This repository contains the configurations and script to run pre-merge checks
for the LLVM project.
## Feedback ## Feedback
If you notice issues or have an idea on how to improve pre-merge checks, please If you notice issues or have an idea on how to improve pre-merge checks, please
create a [new issue](https://github.com/google/llvm-premerge-checks/issues/new) create a [new issue](https://github.com/google/llvm-premerge-checks/issues/new)
or give a :heart: to an existing one. or give a :heart: to an existing one.
## Sign up for beta-test
To get the latest features and help us developing the project, sign up for the
pre-merge beta testing by adding yourself to the ["pre-merge beta testing"
project](https://reviews.llvm.org/project/members/78/) on Phabricator.
## Opt-out
In case you want to opt-out entirely of pre-merge testing, add yourself to the
[OPT OUT project](https://reviews.llvm.org/project/view/83/).
If you decide to opt-out, please let us know why, so we might be able to improve
in the future.
# Requirements
The builds are only triggered if the Revision in Phabricator is created/updated
via `arc diff`. If you update a Revision via the Web UI it will [not
trigger](https://secure.phabricator.com/Q447) a build.
To get a patch on Phabricator tested the build server must be able to apply the
patch to the checked out git repository. If you want to get your patch tested,
please make sure that either:
* You set a git hash as `sourceControlBaseRevision` in Phabricator which is
* available on the Github repository, **or** you define the dependencies of your
* patch in Phabricator, **or** your patch can be applied to the main branch.
Only then can the build server apply the patch locally and run the builds and
tests.
# Accessing results on Phabricator
Phabricator will automatically trigger a build for every new patch you upload or
modify. Phabricator shows the build results at the top of the entry: ![build
status](docs/images/diff_detail.png)
The CI will compile and run tests, run clang-format and
[clang-tidy](docs/clang_tidy.md) on lines changed.
If a unit test failed, this is shown below the build status. You can also expand
the unit test to see the details: ![unit test
results](docs/images/unit_tests.png).
# Contributing # Contributing
We're happy to get help on improving the infrastructure and workflows! We're happy to get help on improving the infrastructure and workflows!
Please check [contibuting](docs/contributing.md) first. Please check [contributing](docs/contributing.md).
[Development](docs/development.md) gives an overview how different parts
interact together.
[Playbooks](docs/playbooks.md) shows concrete examples how to, for example,
build and run agents locally.
If you have any questions please contact by [mail](mailto:goncahrov@google.com)
or find user "goncharov" on [LLVM Discord](https://discord.gg/xS7Z362).
# Additional Information
- [Playbooks](docs/playbooks.md) for installing/upgrading agents and testing
changes.
- [Log of the service
operations](https://github.com/google/llvm-premerge-checks/wiki/LLVM-pre-merge-tests-operations-blog)
# License # License

29
actions-runner-controller/README.md
View file

@ -1,29 +0,0 @@
# Installation
Install helm (along with other tools in `local_setup.sh`).
Once per cluster:
`helm install arc --namespace "arc-system" --create-namespace oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller`
## Add new set of runners
Create runner set first time:
- copy 'values.yml` and updata parameters: APP ID, target node set, repo etc
- run
```
helm install <runner-set> --namespace <k8s-namespace> --create-namespace \
--values=$(readlink -f <values.yml>)
--set-file=githubConfigSecret.github_app_private_key=$(readlink -f <pem>) \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
```
After that update values.yml to use `githubConfigSecret: arc-runner-set-gha-rs-github-secret`.
## Update
Example command for linux set:
`helm upgrade arc-google-linux --namespace arc-linux-prod -f $(readlink -f ~/src/merge-checks/actions-runner-controller/values-llvm.yaml) oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set`

194
actions-runner-controller/values-llvm.yaml
View file

@ -1,194 +0,0 @@
# See options doc in https://github.com/actions/actions-runner-controller/tree/master/charts/actions-runner-controller
## githubConfigUrl is the GitHub url for where you want to configure runners
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
githubConfigUrl: "https://github.com/llvm/llvm-project"
## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
githubConfigSecret:
### GitHub Apps Configuration
## NOTE: IDs MUST be strings, use quotes
github_app_id: "418336"
github_app_installation_id: "43821912"
## Pass --set-file=githubConfigSecret.github_app_private_key=<path to pem>
# First installation creates this secret.
# githubConfigSecret: arc-runner-set-gha-rs-github-secret
## proxy can be used to define proxy settings that will be used by the
## controller, the listener and the runner of this scale set.
#
# proxy:
# http:
# url: http://proxy.com:1234
# credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
# https:
# url: http://proxy.com:1234
# credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
# noProxy:
# - example.com
# - example.org
## maxRunners is the max number of runners the autoscaling runner set will scale up to.
maxRunners: 3
## minRunners is the min number of runners the autoscaling runner set will scale down to.
minRunners: 1
runnerGroup: "generic-google-cloud-2"
## name of the runner scale set to create. Defaults to the helm release name
# runnerScaleSetName: ""
## A self-signed CA certificate for communication with the GitHub server can be
## provided using a config map key selector. If `runnerMountPath` is set, for
## each runner pod ARC will:
## - create a `github-server-tls-cert` volume containing the certificate
## specified in `certificateFrom`
## - mount that volume on path `runnerMountPath`/{certificate name}
## - set NODE_EXTRA_CA_CERTS environment variable to that same path
## - set RUNNER_UPDATE_CA_CERTS environment variable to "1" (as of version
## 2.303.0 this will instruct the runner to reload certificates on the host)
##
## If any of the above had already been set by the user in the runner pod
## template, ARC will observe those and not overwrite them.
## Example configuration:
#
# githubServerTLS:
# certificateFrom:
# configMapKeyRef:
# name: config-map-name
# key: ca.crt
# runnerMountPath: /usr/local/share/ca-certificates/
## Container mode is an object that provides out-of-box configuration
## for dind and kubernetes mode. Template will be modified as documented under the
## template object.
##
## If any customization is required for dind or kubernetes mode, containerMode should remain
## empty, and configuration should be applied to the template.
# containerMode:
# type: "dind" ## type can be set to dind or kubernetes
# ## the following is required when containerMode.type=kubernetes
# kubernetesModeWorkVolumeClaim:
# accessModes: ["ReadWriteOnce"]
# # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
# storageClassName: "dynamic-blob-storage"
# resources:
# requests:
# storage: 1Gi
# kubernetesModeServiceAccount:
# annotations:
## template is the PodSpec for each listener Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
# listenerTemplate:
# spec:
# containers:
# # Use this section to append additional configuration to the listener container.
# # If you change the name of the container, the configuration will not be applied to the listener,
# # and it will be treated as a side-car container.
# - name: listener
# securityContext:
# runAsUser: 1000
# # Use this section to add the configuration of a side-car container.
# # Comment it out or remove it if you don't need it.
# # Spec for this container will be applied as is without any modifications.
# - name: side-car
# image: example-sidecar
## template is the PodSpec for each runner Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
## template.spec will be modified if you change the container mode
## with containerMode.type=dind, we will populate the template.spec with following pod spec
## template:
## spec:
## initContainers:
## - name: init-dind-externals
## image: ghcr.io/actions/actions-runner:latest
## command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
## volumeMounts:
## - name: dind-externals
## mountPath: /home/runner/tmpDir
## containers:
## - name: runner
## image: ghcr.io/actions/actions-runner:latest
## command: ["/home/runner/run.sh"]
## env:
## - name: DOCKER_HOST
## value: unix:///run/docker/docker.sock
## volumeMounts:
## - name: work
## mountPath: /home/runner/_work
## - name: dind-sock
## mountPath: /run/docker
## readOnly: true
## - name: dind
## image: docker:dind
## args:
## - dockerd
## - --host=unix:///run/docker/docker.sock
## - --group=$(DOCKER_GROUP_GID)
## env:
## - name: DOCKER_GROUP_GID
## value: "123"
## securityContext:
## privileged: true
## volumeMounts:
## - name: work
## mountPath: /home/runner/_work
## - name: dind-sock
## mountPath: /run/docker
## - name: dind-externals
## mountPath: /home/runner/externals
## volumes:
## - name: work
## emptyDir: {}
## - name: dind-sock
## emptyDir: {}
## - name: dind-externals
## emptyDir: {}
######################################################################################################
## with containerMode.type=kubernetes, we will populate the template.spec with following pod spec
template:
spec:
containers:
- name: runner
image: us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux:latest
command: ["/bin/bash"]
args: ["-c", "/entrypoint.sh /home/runner/run.sh"]
env:
- name: ACTIONS_RUNNER_CONTAINER_HOOKS
value: /home/runner/k8s/index.js
- name: ACTIONS_RUNNER_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
value: "false"
- name: WORKDIR
value: "/home/runner/_work"
resources:
limits:
cpu: 31
memory: 80Gi
requests:
cpu: 31
memory: 80Gi
volumeMounts:
- name: work
mountPath: /home/runner/_work
volumes:
- name: work
emptyDir: {}
nodeSelector:
cloud.google.com/gke-nodepool: linux-agents-2
## Optional controller service account that needs to have required Role and RoleBinding
## to operate this gha-runner-scale-set installation.
## The helm chart will try to find the controller deployment and its service account at installation time.
## In case the helm chart can't find the right service account, you can explicitly pass in the following value
## to help it finish RoleBinding with the right service account.
## Note: if your controller is installed to only watch a single namespace, you have to pass these values explicitly.
# controllerServiceAccount:
# namespace: arc-system
# name: test-arc-gha-runner-scale-set-controller

194
actions-runner-controller/values-test.yaml
View file

@ -1,194 +0,0 @@
# See options doc in https://github.com/actions/actions-runner-controller/tree/master/charts/actions-runner-controller
## githubConfigUrl is the GitHub url for where you want to configure runners
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
githubConfigUrl: "https://github.com/metafloworg/llvm-project"
## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
#githubConfigSecret:
### GitHub Apps Configuration
## NOTE: IDs MUST be strings, use quotes
#github_app_id: "427039"
#github_app_installation_id: "43840704"
## Pass --set-file=githubConfigSecret.github_app_private_key=<path to pem>
# First installation creates this secret.
githubConfigSecret: arc-runner-set-gha-rs-github-secret
## proxy can be used to define proxy settings that will be used by the
## controller, the listener and the runner of this scale set.
#
# proxy:
# http:
# url: http://proxy.com:1234
# credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
# https:
# url: http://proxy.com:1234
# credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
# noProxy:
# - example.com
# - example.org
## maxRunners is the max number of runners the autoscaling runner set will scale up to.
maxRunners: 3
## minRunners is the min number of runners the autoscaling runner set will scale down to.
minRunners: 0
# runnerGroup: "default"
## name of the runner scale set to create. Defaults to the helm release name
# runnerScaleSetName: ""
## A self-signed CA certificate for communication with the GitHub server can be
## provided using a config map key selector. If `runnerMountPath` is set, for
## each runner pod ARC will:
## - create a `github-server-tls-cert` volume containing the certificate
## specified in `certificateFrom`
## - mount that volume on path `runnerMountPath`/{certificate name}
## - set NODE_EXTRA_CA_CERTS environment variable to that same path
## - set RUNNER_UPDATE_CA_CERTS environment variable to "1" (as of version
## 2.303.0 this will instruct the runner to reload certificates on the host)
##
## If any of the above had already been set by the user in the runner pod
## template, ARC will observe those and not overwrite them.
## Example configuration:
#
# githubServerTLS:
# certificateFrom:
# configMapKeyRef:
# name: config-map-name
# key: ca.crt
# runnerMountPath: /usr/local/share/ca-certificates/
## Container mode is an object that provides out-of-box configuration
## for dind and kubernetes mode. Template will be modified as documented under the
## template object.
##
## If any customization is required for dind or kubernetes mode, containerMode should remain
## empty, and configuration should be applied to the template.
# containerMode:
# type: "dind" ## type can be set to dind or kubernetes
# ## the following is required when containerMode.type=kubernetes
# kubernetesModeWorkVolumeClaim:
# accessModes: ["ReadWriteOnce"]
# # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
# storageClassName: "dynamic-blob-storage"
# resources:
# requests:
# storage: 1Gi
# kubernetesModeServiceAccount:
# annotations:
## template is the PodSpec for each listener Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
# listenerTemplate:
# spec:
# containers:
# # Use this section to append additional configuration to the listener container.
# # If you change the name of the container, the configuration will not be applied to the listener,
# # and it will be treated as a side-car container.
# - name: listener
# securityContext:
# runAsUser: 1000
# # Use this section to add the configuration of a side-car container.
# # Comment it out or remove it if you don't need it.
# # Spec for this container will be applied as is without any modifications.
# - name: side-car
# image: example-sidecar
## template is the PodSpec for each runner Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
## template.spec will be modified if you change the container mode
## with containerMode.type=dind, we will populate the template.spec with following pod spec
## template:
## spec:
## initContainers:
## - name: init-dind-externals
## image: ghcr.io/actions/actions-runner:latest
## command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
## volumeMounts:
## - name: dind-externals
## mountPath: /home/runner/tmpDir
## containers:
## - name: runner
## image: ghcr.io/actions/actions-runner:latest
## command: ["/home/runner/run.sh"]
## env:
## - name: DOCKER_HOST
## value: unix:///run/docker/docker.sock
## volumeMounts:
## - name: work
## mountPath: /home/runner/_work
## - name: dind-sock
## mountPath: /run/docker
## readOnly: true
## - name: dind
## image: docker:dind
## args:
## - dockerd
## - --host=unix:///run/docker/docker.sock
## - --group=$(DOCKER_GROUP_GID)
## env:
## - name: DOCKER_GROUP_GID
## value: "123"
## securityContext:
## privileged: true
## volumeMounts:
## - name: work
## mountPath: /home/runner/_work
## - name: dind-sock
## mountPath: /run/docker
## - name: dind-externals
## mountPath: /home/runner/externals
## volumes:
## - name: work
## emptyDir: {}
## - name: dind-sock
## emptyDir: {}
## - name: dind-externals
## emptyDir: {}
######################################################################################################
## with containerMode.type=kubernetes, we will populate the template.spec with following pod spec
template:
spec:
containers:
- name: runner
image: us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux:latest
command: ["/bin/bash"]
args: ["-c", "/entrypoint.sh /home/runner/run.sh"]
env:
- name: ACTIONS_RUNNER_CONTAINER_HOOKS
value: /home/runner/k8s/index.js
- name: ACTIONS_RUNNER_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
value: "false"
- name: WORKDIR
value: "/home/runner/_work"
resources:
limits:
cpu: 31
memory: 80Gi
requests:
cpu: 31
memory: 80Gi
volumeMounts:
- name: work
mountPath: /home/runner/_work
volumes:
- name: work
emptyDir: {}
nodeSelector:
cloud.google.com/gke-nodepool: linux-agents-2
## Optional controller service account that needs to have required Role and RoleBinding
## to operate this gha-runner-scale-set installation.
## The helm chart will try to find the controller deployment and its service account at installation time.
## In case the helm chart can't find the right service account, you can explicitly pass in the following value
## to help it finish RoleBinding with the right service account.
## Note: if your controller is installed to only watch a single namespace, you have to pass these values explicitly.
# controllerServiceAccount:
# namespace: arc-system
# name: test-arc-gha-runner-scale-set-controller

3
containers/README.md
View file

@ -5,9 +5,6 @@ for the merge guards.
## Scripts ## Scripts
The scripts are written in bash (for Linux) and powershell (for Windows). The scripts are written in bash (for Linux) and powershell (for Windows).
### build_run.(sh|ps1)
Build the docker image and run it locally. This is useful for testing it.
### build_deploy.(sh|ps1) ### build_deploy.(sh|ps1)
Build the docker and deploy it to the GCP registry. This is useful for deploying Build the docker and deploy it to the GCP registry. This is useful for deploying
it in the Kubernetes cluster. it in the Kubernetes cluster.

1
containers/bazel/.dockerignore
View file

@ -1 +0,0 @@
cloudbuild.yaml

80
containers/bazel/Dockerfile
View file

@ -1,80 +0,0 @@
FROM debian:stable
RUN echo 'intall packages'; \
apt-get update; \
apt-get install -y --no-install-recommends \
locales openssh-client gnupg ca-certificates \
zip wget git \
gdb build-essential \
ninja-build \
libelf-dev libffi-dev gcc-multilib \
# for llvm-libc tests that build mpfr and gmp from source
autoconf automake libtool \
ccache \
python3 python3-psutil \
python3-pip python3-setuptools \
lsb-release software-properties-common \
swig python3-dev libedit-dev libncurses5-dev libxml2-dev liblzma-dev golang rsync jq \
# for llvm installation script
sudo;
# debian stable cmake is 3.18, we need to install a more recent version.
RUN wget --no-verbose -O /cmake.sh https://github.com/Kitware/CMake/releases/download/v3.23.3/cmake-3.23.3-linux-x86_64.sh; \
chmod +x /cmake.sh; \
mkdir -p /etc/cmake; \
/cmake.sh --prefix=/etc/cmake --skip-license; \
ln -s /etc/cmake/bin/cmake /usr/bin/cmake; \
cmake --version; \
rm /cmake.sh
# LLVM must be installed after prerequsite packages.
ENV LLVM_VERSION=16
RUN echo 'install llvm ${LLVM_VERSION}'; \
wget --no-verbose https://apt.llvm.org/llvm.sh; \
chmod +x llvm.sh; \
./llvm.sh ${LLVM_VERSION};\
apt-get update; \
apt install -y clang-${LLVM_VERSION} clang-format-${LLVM_VERSION} clang-tidy-${LLVM_VERSION} lld-${LLVM_VERSION}; \
ln -s /usr/bin/clang-${LLVM_VERSION} /usr/bin/clang;\
ln -s /usr/bin/clang++-${LLVM_VERSION} /usr/bin/clang++;\
ln -s /usr/bin/clang-tidy-${LLVM_VERSION} /usr/bin/clang-tidy;\
ln -s /usr/bin/clang-tidy-diff-${LLVM_VERSION}.py /usr/bin/clang-tidy-diff;\
ln -s /usr/bin/clang-format-${LLVM_VERSION} /usr/bin/clang-format;\
ln -s /usr/bin/clang-format-diff-${LLVM_VERSION} /usr/bin/clang-format-diff;\
ln -s /usr/bin/lld-${LLVM_VERSION} /usr/bin/lld;\
ln -s /usr/bin/lldb-${LLVM_VERSION} /usr/bin/lldb;\
ln -s /usr/bin/ld.lld-${LLVM_VERSION} /usr/bin/ld.lld
RUN echo 'configure locale'; \
sed --in-place '/en_US.UTF-8/s/^#//' /etc/locale.gen ;\
locale-gen ;\
echo 'make python 3 default'; \
rm -f /usr/bin/python && ln -s /usr/bin/python3 /usr/bin/python; \
pip3 install wheel
# Configure locale
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN wget --no-verbose -O /usr/bin/bazelisk https://github.com/bazelbuild/bazelisk/releases/download/v1.17.0/bazelisk-linux-amd64; \
chmod +x /usr/bin/bazelisk; \
bazelisk --version
RUN echo 'install buildkite' ;\
apt-get install -y apt-transport-https gnupg;\
sh -c 'echo deb https://apt.buildkite.com/buildkite-agent stable main > /etc/apt/sources.list.d/buildkite-agent.list' ;\
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 32A37959C2FA5C3C99EFBC32A79206696452D198 ;\
apt-get update ;\
apt-get install -y buildkite-agent tini gosu; \
apt-get clean;
COPY *.sh /usr/local/bin/
RUN chmod og+rx /usr/local/bin/*.sh
COPY --chown=buildkite-agent:buildkite-agent pre-checkout /etc/buildkite-agent/hooks
COPY --chown=buildkite-agent:buildkite-agent post-checkout /etc/buildkite-agent/hooks
# buildkite working directory
VOLUME /var/lib/buildkite-agent
ENTRYPOINT ["entrypoint.sh"]
CMD ["gosu", "buildkite-agent", "buildkite-agent", "start", "--no-color"]

39
containers/bazel/entrypoint.sh
View file

@ -1,39 +0,0 @@
#!/usr/bin/env bash
# Copyright 2021 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -eo pipefail
USER=buildkite-agent
P="${BUILDKITE_BUILD_PATH:-/var/lib/buildkite-agent}"
set -u
mkdir -p "$P"
chown -R ${USER}:${USER} "$P"
export CCACHE_DIR="${P}"/ccache
export CCACHE_MAXSIZE=20G
mkdir -p "${CCACHE_DIR}"
chown -R ${USER}:${USER} "${CCACHE_DIR}"
# /mnt/ssh should contain known_hosts, id_rsa and id_rsa.pub .
mkdir -p /var/lib/buildkite-agent/.ssh
if [ -d /mnt/ssh ]; then
cp /mnt/ssh/* /var/lib/buildkite-agent/.ssh || echo "no "
chmod 700 /var/lib/buildkite-agent/.ssh
chmod 600 /var/lib/buildkite-agent/.ssh/*
chown -R buildkite-agent:buildkite-agent /var/lib/buildkite-agent/.ssh/
else
echo "/mnt/ssh is not mounted"
fi
exec /usr/bin/tini -g -- $@

26
containers/bazel/post-checkout
View file

@ -1,26 +0,0 @@
# Copyright 2021 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# The `post-checkout` hook runs after checkout.
set -eu
# Run prune as git will start to complain "warning: There are too many unreachable loose objects; run 'git prune' to remove them.".
# That happens as we routinely drop old branches.
git prune
if [ -f ".git/gc.log" ]; then
echo ".git/gc.log exist"
cat ./.git/gc.log
rm -f ./.git/gc.log
fi

23
containers/bazel/pre-checkout
View file

@ -1,23 +0,0 @@
#!/bin/bash
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# The `pre-checkout` hook will run just before your pipelines source code is
# checked out from your SCM provider
set -e
# Convert https://github.com/llvm-premerge-tests/llvm-project.git -> llvm-project
# to use the same directory for fork and origin.
BUILDKITE_BUILD_CHECKOUT_PATH="${BUILDKITE_BUILD_PATH}/$(echo $BUILDKITE_REPO | sed -E "s#.*/([^/]*)#\1#" | sed "s/.git$//")"

46
containers/build_run.ps1
View file

@ -1,46 +0,0 @@
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# define command line arguments
param(
[Parameter(Mandatory=$true)][string]$IMAGE_NAME,
[Parameter(Mandatory=$false)][string]$CMD="",
[string]$token
)
# set script to stop on first error
$ErrorActionPreference = "Stop"
# some docs recommend setting 2GB memory limit
docker build `
--memory 2GB `
-t $IMAGE_NAME `
--build-arg token=$token `
"$PSScriptRoot\$IMAGE_NAME"
If ($LastExitCode -ne 0) {
exit
}
$DIGEST=$(docker image inspect --format "{{range .RepoDigests}}{{.}}{{end}}" $IMAGE_NAME) -replace ".*@sha256:(.{6})(.*)$","`$1"
# mount a persistent workspace for experiments
docker run -it `
-v C:/ws:C:/ws `
-v C:/credentials:C:/credentials `
-e BUILDKITE_BUILD_PATH=C:\ws `
-e IMAGE_DIGEST=${DIGEST} `
-e PARENT_HOSTNAME=$env:computername `
$IMAGE_NAME $CMD
If ($LastExitCode -ne 0) {
exit
}

26
containers/build_run.sh
View file

@ -1,26 +0,0 @@
#!/bin/bash
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Starts a new instances of a docker image. Example:
# sudo build_run.sh buildkite-premerge-debian /bin/bash
set -eux
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
IMAGE_NAME="${1%/}"
cd "${DIR}/${IMAGE_NAME}"
docker build -t ${IMAGE_NAME} .
docker run -i -t -v ~/.llvm-premerge-checks:/credentials -v ${DIR}/workspace:/workspace ${IMAGE_NAME} ${2}

2
containers/buildkite-windows/start_agent.ps1
View file

@ -1,5 +1,3 @@
# TODO: remove after migrating scripts to LLVM repo.
git clone https://github.com/google/llvm-premerge-checks.git c:\llvm-premerge-checks
# Checkout path is set to a fixed short value (e.g. c:\ws\src) to keep paths # Checkout path is set to a fixed short value (e.g. c:\ws\src) to keep paths
# short as many tools break on Windows with paths longer than 250. # short as many tools break on Windows with paths longer than 250.
if ($null -eq $env:BUILDKITE_BUILD_PATH) { $env:BUILDKITE_BUILD_PATH = 'c:\ws' } if ($null -eq $env:BUILDKITE_BUILD_PATH) { $env:BUILDKITE_BUILD_PATH = 'c:\ws' }

105
containers/github-linux/Dockerfile
View file

@ -1,105 +0,0 @@
FROM ubuntu:latest
RUN echo 'intall packages'; \
apt-get update; \
apt-get upgrade; \
apt-get install -y --no-install-recommends \
gosu \
locales openssh-client gnupg ca-certificates \
zip unzip wget curl git \
gdb build-essential \
ninja-build \
libelf-dev libffi-dev gcc-multilib libmpfr-dev libpfm4-dev \
# for llvm-libc tests that build mpfr and gmp from source
autoconf automake libtool \
python3 python3-psutil python3-pip python3-setuptools \
lsb-release software-properties-common \
swig python3-dev libedit-dev libncurses5-dev libxml2-dev liblzma-dev golang rsync jq \
# for llvm installation script
sudo \
# build scripts
nodejs \
# shell users
less vim
# debian stable cmake is 3.18, we need to install a more recent version.
RUN wget --no-verbose -O /cmake.sh https://github.com/Kitware/CMake/releases/download/v3.23.3/cmake-3.23.3-linux-x86_64.sh; \
chmod +x /cmake.sh; \
mkdir -p /etc/cmake; \
/cmake.sh --prefix=/etc/cmake --skip-license; \
ln -s /etc/cmake/bin/cmake /usr/bin/cmake; \
cmake --version; \
rm /cmake.sh
# LLVM must be installed after prerequsite packages.
ENV LLVM_VERSION=16
RUN echo 'install llvm ${LLVM_VERSION}' && \
wget --no-verbose https://apt.llvm.org/llvm.sh && \
chmod +x llvm.sh && \
./llvm.sh ${LLVM_VERSION} && \
apt-get update && \
apt-get install -y clang-${LLVM_VERSION} clang-format-${LLVM_VERSION} clang-tidy-${LLVM_VERSION} lld-${LLVM_VERSION} && \
ln -s /usr/bin/clang-${LLVM_VERSION} /usr/bin/clang && \
ln -s /usr/bin/clang++-${LLVM_VERSION} /usr/bin/clang++ && \
ln -s /usr/bin/clang-tidy-${LLVM_VERSION} /usr/bin/clang-tidy && \
ln -s /usr/bin/clang-tidy-diff-${LLVM_VERSION}.py /usr/bin/clang-tidy-diff && \
ln -s /usr/bin/clang-format-${LLVM_VERSION} /usr/bin/clang-format && \
ln -s /usr/bin/clang-format-diff-${LLVM_VERSION} /usr/bin/clang-format-diff && \
ln -s /usr/bin/lld-${LLVM_VERSION} /usr/bin/lld && \
ln -s /usr/bin/lldb-${LLVM_VERSION} /usr/bin/lldb && \
ln -s /usr/bin/ld.lld-${LLVM_VERSION} /usr/bin/ld.lld && \
clang --version
RUN echo 'configure locale' && \
sed --in-place '/en_US.UTF-8/s/^#//' /etc/locale.gen && \
locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN curl -o sccache-v0.5.4-x86_64-unknown-linux-musl.tar.gz -L https://github.com/mozilla/sccache/releases/download/v0.5.4/sccache-v0.5.4-x86_64-unknown-linux-musl.tar.gz \
&& echo "4bf3ce366aa02599019093584a5cbad4df783f8d6e3610548c2044daa595d40b sccache-v0.5.4-x86_64-unknown-linux-musl.tar.gz" | shasum -a 256 -c \
&& tar xzf ./sccache-v0.5.4-x86_64-unknown-linux-musl.tar.gz \
&& mv sccache-v0.5.4-x86_64-unknown-linux-musl/sccache /usr/bin \
&& chown root:root /usr/bin/sccache \
&& ls -la /usr/bin/sccache \
&& sccache --version
RUN groupadd -g 121 runner \
&& useradd -mr -d /home/runner -u 1001 -g 121 runner \
&& mkdir -p /home/runner/_work
WORKDIR /home/runner
# https://github.com/actions/runner/releases
ARG RUNNER_VERSION="2.311.0"
ARG RUNNER_ARCH="x64"
RUN curl -f -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz \
&& tar xzf ./runner.tar.gz \
&& rm runner.tar.gz
# https://github.com/actions/runner-container-hooks/releases
ARG RUNNER_CONTAINER_HOOKS_VERSION="0.4.0"
RUN curl -f -L -o runner-container-hooks.zip https://github.com/actions/runner-container-hooks/releases/download/v${RUNNER_CONTAINER_HOOKS_VERSION}/actions-runner-hooks-k8s-${RUNNER_CONTAINER_HOOKS_VERSION}.zip \
&& unzip ./runner-container-hooks.zip -d ./k8s \
&& rm runner-container-hooks.zip
RUN chown -R runner:runner /home/runner
# Copy entrypoint but script. There is no point of adding
# `ENTRYPOINT ["/entrypoint.sh"]` as k8s runs it with explicit command and
# entrypoint is ignored.
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
# USER runner
# We don't need entry point and token logic as it's all orchestrated by action runner controller.
# BUT we need to start server etc
# RUN chmod +x /entrypoint.sh /token.sh
# # try: USER runner instead of gosu
# ENTRYPOINT ["/entrypoint.sh"]
# CMD ["sleep", "infinity"]
# CMD ["./bin/Runner.Listener", "run", "--startuptype", "service"]

5
containers/github-linux/cloudbuild.yaml
View file

@ -1,5 +0,0 @@
steps:
- name: 'gcr.io/cloud-builders/docker'
args: [ 'build', '-t', 'us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux', '.' ]
images:
- 'us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux:latest'

33
containers/github-linux/entrypoint.sh
View file

@ -1,33 +0,0 @@
#!/usr/bin/env bash
# Copyright 2023 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -ueo pipefail
export PATH=${PATH}:/home/runner
USER=runner
WORKDIR=${WORKDIR:-/home/runner/_work}
export SCCACHE_DIR="${WORKDIR}/sccache"
mkdir -p "${SCCACHE_DIR}"
chown -R ${USER}:${USER} "${SCCACHE_DIR}"
chmod oug+rw "${SCCACHE_DIR}"
gosu runner bash -c 'SCCACHE_DIR="${SCCACHE_DIR}" SCCACHE_IDLE_TIMEOUT=0 SCCACHE_CACHE_SIZE=20G sccache --start-server'
sccache --show-stats
[[ ! -d "${WORKDIR}" ]] && mkdir -p "${WORKDIR}"
# exec /usr/bin/tini -g -- $@
gosu runner "$@"

50
containers/github-linux/token.sh
View file

@ -1,50 +0,0 @@
#!/bin/bash
# https://github.com/myoung34/docker-github-actions-runner/blob/master/token.sh
# Licensed under MIT
# https://github.com/myoung34/docker-github-actions-runner/blob/master/LICENSE
set -euo pipefail
_GITHUB_HOST=${GITHUB_HOST:="github.com"}
# If URL is not github.com then use the enterprise api endpoint
if [[ ${GITHUB_HOST} = "github.com" ]]; then
URI="https://api.${_GITHUB_HOST}"
else
URI="https://${_GITHUB_HOST}/api/v3"
fi
API_VERSION=v3
API_HEADER="Accept: application/vnd.github.${API_VERSION}+json"
AUTH_HEADER="Authorization: token ${ACCESS_TOKEN}"
CONTENT_LENGTH_HEADER="Content-Length: 0"
case ${RUNNER_SCOPE} in
org*)
_FULL_URL="${URI}/orgs/${ORG_NAME}/actions/runners/registration-token"
;;
ent*)
_FULL_URL="${URI}/enterprises/${ENTERPRISE_NAME}/actions/runners/registration-token"
;;
*)
_PROTO="https://"
# shellcheck disable=SC2116
_URL="$(echo "${REPO_URL/${_PROTO}/}")"
_PATH="$(echo "${_URL}" | grep / | cut -d/ -f2-)"
_ACCOUNT="$(echo "${_PATH}" | cut -d/ -f1)"
_REPO="$(echo "${_PATH}" | cut -d/ -f2)"
_FULL_URL="${URI}/repos/${_ACCOUNT}/${_REPO}/actions/runners/registration-token"
;;
esac
RUNNER_TOKEN="$(curl -XPOST -fsSL \
-H "${CONTENT_LENGTH_HEADER}" \
-H "${AUTH_HEADER}" \
-H "${API_HEADER}" \
"${_FULL_URL}" \
| jq -r '.token')"
echo "{\"token\": \"${RUNNER_TOKEN}\", \"full_url\": \"${_FULL_URL}\"}"

32
containers/stats/Dockerfile
View file

@ -1,32 +0,0 @@
FROM ubuntu:latest
RUN apt-get update ;\
apt-get upgrade -y;\
apt-get install -y --no-install-recommends \
locales openssh-client gnupg ca-certificates \
build-essential \
zip wget git less vim \
python3 python3-psutil python3-pip python3-setuptools pipenv \
# PostgreSQL
libpq-dev \
# debugging
less vim wget curl \
rsync jq tini gosu;
RUN echo 'configure locale' && \
sed --in-place '/en_US.UTF-8/s/^#//' /etc/locale.gen && \
locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
COPY *.sh /usr/local/bin/
# requirements.txt generated by running `pipenv lock -r > ../../containers/stats/requirements.txt` in `scripts/metrics`.
# COPY requirements.txt /tmp/
# RUN pip install -q -r /tmp/requirements.txt
RUN chmod og+rx /usr/local/bin/*.sh
RUN wget -q https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O /usr/local/bin/cloud_sql_proxy;\
chmod +x /usr/local/bin/cloud_sql_proxy
ENTRYPOINT ["entrypoint.sh"]
CMD ["/bin/bash"]

22
containers/stats/entrypoint.sh
View file

@ -1,22 +0,0 @@
#!/usr/bin/env bash
# Copyright 2021 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -euo pipefail
git clone --depth 1 https://github.com/google/llvm-premerge-checks.git ~/llvm-premerge-checks
cd ~/llvm-premerge-checks
git fetch origin "${SCRIPTS_REFSPEC:=main}":x
git checkout x
exec /usr/bin/tini -g -- $@

27
containers/stats/requirements.txt
View file

@ -1,27 +0,0 @@
backoff==1.10.0
certifi==2023.7.22
chardet==4.0.0
ftfy==6.0.1; python_version >= '3.6'
gitdb==4.0.7
gitpython==3.1.41
idna==2.10
lxml==4.9.1
mailchecker==4.0.7
pathspec==0.8.1
phabricator==0.8.1
phonenumbers==8.12.23
psycopg2-binary==2.8.6
pyaml==20.4.0
python-benedict==0.24.0
python-dateutil==2.8.1
python-fsutil==0.5.0
python-slugify==5.0.2
pyyaml==5.4.1
requests==2.31.0
six==1.16.0
smmap==4.0.0
text-unidecode==1.3
toml==0.10.2
urllib3==1.26.18
wcwidth==0.2.5
xmltodict==0.12.0

29
docs/clang_tidy.md
View file

@ -1,29 +0,0 @@
# clang-tidy checks
## Warning is not useful
If you found that a warning produced by clang-tidy is not useful:
- If clang-tidy must not run for some files at all (e.g. lit test), please
[add files to ignorelist](../scripts/clang-tidy.ignore).
- Consider fixing or [suppressing diagnostic](https://clang.llvm.org/extra/clang-tidy/#suppressing-undesired-diagnostics)
if there is a good reason.
- [File a bug](https://github.com/google/llvm-premerge-checks/issues/new?assignees=&labels=bug&template=bug_report.md&title=)
if build process should be improved.
- If you believe that you found a clang-tidy bug then please keep in mind that clang-tidy version run by bot
might not be most recent. Please reproduce your issue on current version before submitting a bug to clang-tidy.
## Review comments
Build bot leaves inline comments only for a small subset of files that are not blacklisted for analysis (see above) *and*
specifically whitelisted for comments.
That is done to avoid potential noise when a file already contains a number of warnings.
If your are confident that some files are in good shape already, please
[whitelist them](../scripts/clang-tidy-comments.ignore).
----
[about pre-merge checks](../README.md)

69
docs/decisions/1_migrate_from_jenkins_to_buildkite.md
View file

@ -1,69 +0,0 @@
## Goal
Migrate from the current build infrastructure on Jenkins. It works but:
- due to security concerns we cannot give users access to build interface and has to generate reports,copy artifacts etc. to mimic that.
- creating builds with new configurations, e.g. libc++ tests, requires a lot of cooperation between us and interested user.
- it's not possible for contributors to add new agents with different configurations. It also not clear how to make jenkins auto-scale number of agents.
- intial idea of "look we have a plugin to integrate Jenkins with Phabricator, that will make everything easy" does not hold: we have to interact with Conduit API and understand Phabricator quirks anyway to provide experience we want.
- it's hard to reproduce builds locally.
- relying only on Herald notifications does not seems to work ([#96](https://github.com/google/llvm-premerge-checks/issues/96))
**Non goal**
No changes to the current review workflow, e.g. run tests only if commit if is LGTM-ed [#60](https://github.com/google/llvm-premerge-checks/issues/60)).
## Alternatives considered
I have looked at [GitHub actions](https://help.github.com/en/actions) and [Buildkite](https://buildkite.com/docs/tutorials/getting-started) as alternatives to current setup.
While GitHub actions might be providing some interesting integration features with GitHub (maybe) I found it lacks some nice features that Buildkite have. Namely actions have:
- a limited support for shell and proposes javascript or docker containers as an "executor". That might not be as easy to debug or use to reproduce locally;
- a workflows that are more or less predefined, so to start specific steps for, e.g. 'clang-extra', we will likely need to start a job and detect within it that build is not needed. Buildkite give ability to dynamically generate steps as we go;
- very limited support for scaling / monitoring self-hosted agents ([issue](https://github.community/t5/GitHub-Actions/Self-hosted-runner-autoscaling/td-p/38416)) as well as [security issues](https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories).
## Proposed setup
Use buildkite pipelines and define them here, in llvm-premerge-checks repo. Diffs are going to be reflected in llvm-project fork.
Hopefully buildkite alone should be enought for the common workflow. If it's not enough and/or there are some issues with phabricator-buildkite itegration that require client update (Phabricator, like we had with [Jenkins CSRF](https://github.com/google/llvm-premerge-checks/issues/163) we can introduce a separate "conrol" service. This service can also have a DB to collect stats, maintain cohesive history of every revisions, manage tricky workflow cases, and manage scaling of buildkite agents. As much as I don't wan't to maintain a new custom service it seems to be a better idea on a long run to work around missing features or having an ability to maintain a complex state or history.
### Common workflow
In the most simple case build will be triggered directly by the Harbormaster "buildkite" rule. Build pipeline will receive PHID / DiffID and:
- using the ID, communicate with the Phabricator and decide what diffs have to be applied to represent the change
- report back to Phabricator a link to the build in buildkite
- create a new branch in llvm-project fork and apply changes as a separate commits
- run "build planning" script that will generate further steps based on e.g. author or the diff (if she participates in beta-testing), affected paths, trottling...; there might be multiple steps like this further down the line.
- complete every build and attach relevant artifacts to the build; add linter comments
- report final build result back.
Later this workflow can be split into two steps: one will setup github branch and create a PR. Another one will be triggered by GitHub and report PR build status to it. There are challenges on who and when is going to communicate back to Phabricator results of the build. Alternatively we can create a separate pipeline for the reviews on GitHub.
### Cleaning up branches / PR
Branches / PR are going to be kept for some time (for example month) after review is completed of diff seems to be abandoned.
That can intially be done by running a scheduled build on buildkite that will list branches and communicate with Phabricator to understand what to be deleted. Alternatively we can use "control" as can easily maintain state and, thus can do less work and even get notifications from Phabricator about revisions being closed.
### Missed diffs
We see that in some cases [#96](https://github.com/google/llvm-premerge-checks/issues/96) phabricator does not trigger a build. But we still want to run them. We can periodically (hourly) list recent reviews in phabricator and figure out if builds already being run for them. I see two possibilites here:
1. (preferred) find a way to trigger a build for such missed revision. It's definitely possible to do from the UI / command line but missing from the conduit API. This way "Common workflow" will kick in and we will get a build.
2. start a build "out of bound" and attach a comment to the review, describing that situation and providing a link to the build status. Update comment when build is done.
### Custom worflows / build agents
This setup allows community to relatively easily add own agents with relevant "purpose-tag". E.g. libc++ contributors can add a new condition into the "build planning" step that will add new steps (including ones that target their own agents) if relevant code was modified.
### Autoscaling
Buildkite provides a [number of ways](https://buildkite.com/docs/tutorials/parallel-builds#auto-scaling-your-build-agents) to collect agent stats and decide if we should up or downscale the number of agents. Technically that can be probably be done as a separate build pipeline though for security reasons and ease of implementation we might want to do that either in "control" service or in a completely separate one that is not accessible publicly.

24
docs/deployment.plantuml
View file

@ -1,24 +0,0 @@
@startuml
component phabricator
component buildkite
node github {
component "llvm-project"
component "llvm-project fork"
component "llvm-premerge-checks"
}
node "GCP" {
component "buildkite agents"
component "phab-proxy"
}
[phabricator] --> [phab-proxy]
[phab-proxy] --> [buildkite]
[buildkite] <-- [buildkite agents]
[buildkite agents] <-- [llvm-project]
[buildkite agents] <--> [llvm-project fork]
[buildkite agents] --> [phabricator]
[buildkite agents] <-- [llvm-premerge-checks]
@enduml

189
docs/development.md
View file

@ -1,189 +0,0 @@
- [Overview](#overview)
- [Buildkite agents](#buildkite-agents)
- [Build steps](#build-steps)
- [Phabricator integration](#phabricator-integration)
- [Life of a pre-merge check](#life-of-a-pre-merge-check)
- [Cluster parts](#cluster-parts)
- [Ingress and public addresses](#ingress-and-public-addresses)
- [Enabled projects and project detection](#enabled-projects-and-project-detection)
- [Agent machines](#agent-machines)
- [Compilation caching](#compilation-caching)
- [Buildkite monitoring](#buildkite-monitoring)
# Overview
- [Buildkite](https://buildkite.com/llvm-project) orchestrates each build.
- multiple Linux and windows agents connected to Buildkite. Agents are run at
Google Cloud Platform.
- [small proxy service](/phabricator-proxy) that takes build requests from
[reviews.llvm.org](http://reviews.llvm.org) and converts them into Buildkite
build request. Buildkite job sends build results directly to Phabricator.
- every review creates a new branch in [fork of
llvm-project](https://github.com/llvm-premerge-tests/llvm-project).
![deployment diagram](http://www.plantuml.com/plantuml/proxy?src=https://raw.githubusercontent.com/google/llvm-premerge-checks/main/docs/deployment.plantuml)
# Buildkite agents
Agents are deployed in two clusters `llvm-premerge-checks` and `windows-cluster`.
The latter is for windows machines (as it is controlled on cluster level if
machines can run windows containers).
Container configurations are in ./containers and deployment configurations are
in ./kubernetes. Most important ones are:
- Windows agents: container `containers/buildkite-windows`, deployment `kubernetes/buildkite/windows.yaml`. TODO: at the moment Docker image is created and uploaded
from a windows machine (e.g. win-dev). It would be great to setup a cloudbuild.
- Linux agents: run tests for linux x86 config, container `containers/buildkite-linux`, deployment `kubernetes/buildkite/linux.yaml`.
- Service agents: run low CPU build steps (e.g. generate pipeline steps) container `containers/buildkite-linux`, deployment `kubernetes/buildkite/service.yaml`.
All deployments have a copy `..-test` to be used as a test playground to check
container / deployment changes before modifying "prod" setup.
# Build steps
Buildkite allows [dynamically define pipelines as the output of a
command](https://buildkite.com/docs/pipelines/defining-steps#dynamic-pipelines).
And most of pipelines use this pattern of running a script and using the
resulting yaml. E.g. script to run pull-request checks is llvm-project [.ci/generate-buildkite-pipeline-premerge](https://github.com/llvm/llvm-project/blob/main/.ci/generate-buildkite-pipeline-premerge). Thus any changes to steps to run should
go into that script.
We have a legacy set of scripts in `/scripts` in this repo but discourage new
use and development of them - they are mostly kept to make Phabricator integration
to function.
# Phabricator integration
Note: this section is about integrating with Phabricator that is now discouraged,
some things might already be renamed or straight broken as we moving to Pull Requests.
- [Harbormaster build plan](https://reviews.llvm.org/harbormaster/plan/5) the
Phabricator side these things were configured
- Herald [rule for everyone](https://reviews.llvm.org/H576) and for [beta
testers](https://reviews.llvm.org/H511). Note that right now there is no
difference between beta and "normal" builds.
- the [merge_guards_bot user](https://reviews.llvm.org/p/merge_guards_bot/)
account for writing comments.
## Life of a pre-merge check
When new diff arrives for review it triggers a Herald rule ("everyone" or "beta
testers").
That in sends an HTTP POST request to [**phab-proxy**](../phabricator-proxy)
that submits a new buildkite job **diff-checks**. All parameters from the
original request are put in the build's environment with `ph_` prefix (to avoid
shadowing any Buildkite environment variable). "ph_scripts_refspec" parameter
defines refspec of llvm-premerge-checks to use ("main" by default).
**diff-checks** pipeline
([create_branch_pipeline.py](../scripts/create_branch_pipeline.py))
downloads a patch (or series of patches) and applies it to a fork of the
llvm-project repository. Then it pushes a new state as a new branch (e.g.
"phab-diff-288211") and triggers "premerge-checks" on it (all "ph_" env
variables are passed to it). This new branch can now be used to reproduce the
build or by another tooling. Periodical **cleanup-branches** pipeline deletes
branches older than 30 days.
**premerge-checks** pipeline
([build_branch_pipeline.py](../scripts/build_branch_pipeline.py))
builds and tests changes on Linux and Windows agents. Then it uploads a
combined result to Phabricator.
## Cluster parts
### Ingress and public addresses
We use NGINX ingress for Kubernetes. Right now it's only used to provide basic
HTTP authentication and forwards all requests from load balancer to
[phabricator proxy](../phabricator-proxy) application.
Follow up to date docs to install [reverse
proxy](https://kubernetes.github.io/ingress-nginx/deploy/#gce-gke).
[cert-manager] is installed with helm https://cert-manager.io/docs/installation/helm/
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.9.1 \
--set installCRDs=true
We also have [certificate manager](https://cert-manager.io/docs/) and
[lets-encrypt configuration](../kubernetes/cert-issuer.yaml) in place, but they are
not used at the moment and should be removed if we decide to live with static IP.
HTTP auth is configured with k8s secret 'http-auth' in 'buildkite' namespace
(see [how to update auth](playbooks.md#update-http-auth-credentials)).
## Enabled projects and project detection
To reduce build times and mask unrelated problems, we're only building and
testing the projects that were modified by a patch.
[choose_projects.py](../scripts/choose_projects.py) uses manually maintained
[config file](../scripts/llvm-dependencies.yaml) to define inter-project
dependencies and exclude projects:
1. Get prefix (e.g. "llvm", "clang") of all paths modified by a patch.
1. Add all dependant projects.
1. Add all projects that this extended list depends on, completing the
dependency subtree.
1. Remove all disabled projects.
## Agent machines
All build machines are running from Docker containers so that they can be
debugged, updated, and scaled easily:
- [Linux](../containers/buildkite-premerge-debian/Dockerfile). We use
[Kubernetes deployment](../kubernetes/buildkite) to manage these agents.
- [Windows](../containers/agent-windows-buildkite/Dockerfile). At the moment they are run as
multiple individual VM instances.
See [playbooks](playbooks.md) how to manage and set up machines.
## Compilation caching
Each build is performed on a clean copy of the git repository. To speed up the
builds [ccache](https://ccache.dev/) is used on Linux and
[sccache](https://github.com/mozilla/sccache) on Windows.
# Buildkite monitoring
FIXME: does not work as of 2023-09-11. Those metrics could allow
us to setup auto-scaling of machines to the current demend.
VM instance `buildkite-monitoring` exposes Buildkite metrics to GCP.
To set up a new instance:
1. Create as small Linux VM with full access to *Stackdriver Monitoring API*.
1. Follow instructions to [install monitoring
agent](https://cloud.google.com/monitoring/agent/install-agent) and [enable
statsd plugin](https://cloud.google.com/monitoring/agent/plugins/statsd).
1. Download recent release of
[buildkite-agent-metrics](https://github.com/buildkite/buildkite-agent-metrics/releases).
1. Run in SSH session:
```bash
chmod +x buildkite-agent-metrics-linux-amd64
nohup ./buildkite-agent-metrics-linux-amd64 -token XXXX -interval 30s -backend statsd &
```
Metrics are exported as "custom/statsd/gauge".
TODO: update "Testing scripts locally" playbook on how to run Linux build locally with Docker.
TODO: migrate 'builkite-monitoring' to k8s deployment.

BIN
docs/images/bot_comment.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 22 KiB

BIN
docs/images/diff_detail.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 15 KiB

BIN
docs/images/unit_tests.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 25 KiB

74
docs/monitoring_main_branch.md
View file

@ -1,74 +0,0 @@
# Monitoring the LLVM main branch
This document proposes a design to improve the stability of the LLVM main branch
for use in pre-merge testing.
## Background
The stability of the pre-merge testing largely depends on the stability of the
LLVM main branch: Whenever something is broken on the main branch, also
pre-merge testing will fail: The patches from Phabricator are applied to some
revision on the main branch before they can be built. So the fewer revisions of
the main branch being broken, the more stable pre-merge testing will be.
## High-level Design
We propose to run a Buildbot worker on the main branch with the same
Docker image we're using for pre-merge testing. That worker shall check all
commits to main for build and test failures with regards to the configuration
we're using for pre-merge testing. Whenever these builds fails, Buildbot
notifies the commiters and gives them the opportunity to fix or revert their
patch.
This is much faster than a having a human investigate the issue and notify the
committers. By having faster feedback the main branch is broken for fewer
revisions and this the probability of a false-positive pre-merge test is lower.
## Machine setup
We would deploy another container as part of the existing Kubernetes cluster for
pre-merge testing. The image would be based on
[buildkite-premerge-debian](https://github.com/google/llvm-premerge-checks/blob/main/containers/buildkite-premerge-debian/Dockerfile)
, and we would just add the things needed for the Buildbot agent by resuing the
setup of an existing worker (e.g.
[clangd-ubuntu-clang](https://github.com/llvm/llvm-zorg/blob/main/Buildbot/google/docker/Buildbot-clangd-ubuntu-clang/Dockerfile.)
).
## Build configuration
The Buildbot worker should run as much of the pre-merge testing scripts as
possible to get the results as close as we can to that of what we would see
happen in pre-merge testing.
Currently we're running these steps on Buildkite to check the main branch:
``` bash
export SRC=${BUILDKITE_BUILD_PATH}/llvm-premerge-checks
rm -rf ${SRC}
git clone --depth 1 https://github.com/google/llvm-premerge-checks.git "${SRC}"
cd "${SRC}"
git fetch origin "${ph_scripts_refspec:-master}":x
git checkout x
cd "$BUILDKITE_BUILD_CHECKOUT_PATH"
pip install -q -r ${SRC}/scripts/requirements.txt
${SRC}/scripts/pipeline_main.py | tee /dev/tty | buildkite-agent pipeline upload
```
We should run similar steps on Buildbot by creating a new `BuilderFactory` to
the [existing
list](https://github.com/llvm/llvm-zorg/tree/main/zorg/buildbot/builders) and
then configuring it [as
usual](https://github.com/llvm/llvm-zorg/blob/main/buildbot/osuosl/master/config/builders.py).
## Open questions
1. Do we actually need to create a `BuilderFactory` or can we also just run a
shell script?
2. How can we quickly test and iterate on the Python scripting? How can we run
our own Buildbot master for testing?
3. The current solution by only wrapping the existing pre-merge testing scripts
will not result in nice build steps as the other builders are producing as
buildkite does not have that concept. Is that an issue?
4. We do need to check out the pre-merge testing config and scripts form another
git repo. Is that an issue?

258
docs/playbooks.md
View file

@ -1,258 +0,0 @@
- [Playbooks](#playbooks)
* [Testing scripts locally](#testing-scripts-locally)
* [Testing changes before merging](#testing-changes-before-merging)
* [Deployment to a clean infrastructure](#deployment-to-a-clean-infrastructure)
* [Creating docker containers on Windows](#creating-docker-containers-on-windows)
* [Spawning a new windows agent](#spawning-a-new-windows-agent)
+ [Buildkite](#buildkite)
* [Custom environment variables](#custom-environment-variables)
* [Update HTTP auth credentials](#update-http-auth-credentials)
# Playbooks
## Development environment
You need will need recent python 3 installed, e.g. follow this
[installation guide](https://cloud.google.com/python/docs/setup?hl=en).
To install required packages run:
```shell script
pip install -r ./scripts/requirements.txt
```
optional:
```shell script
pip install jupyterlab pandas seaborn # for jupyter labs.
```
## Testing scripts locally
Build and run agent docker image `sudo ./containers/build_run.sh buildkite-premerge-debian /bin/bash`.
Set `CONDUIT_TOKEN` with your personal one from `https://reviews.llvm.org/settings/user/<USERNAME>/page/apitokens/`.
## Testing changes before merging
It's recommended to test even smallest changes before committing them to the `main` branch.
1. Create a pull request here.
1. Manually create a buildkite build in the pipeline you are updating and specify
environment variable `ph_scripts_refspec="pull/123/head"`. Replace `123`
with your PR number. If you don't have access to create buildkite builds,
please ask a reviewer to do that.
To test "premerge-tests" pipeline pick an existing build and copy "ph_"
parameters from it, omitting "ph_target_phid" to skip updating an existing
review.
See also [custom environment variables](#custom-environment-variables).
1. Wait for build to complete and maybe attach a link to it to your PR.
To test changes for the pipeline "setup" step please experiment on a copy first.
## Deployment to a clean infrastructure
General remarks:
* GCP does not route any traffic to your services unless the service is
"healthy". It might take a few minutes after startup before the services is
classified as healthy. Until then, you will only see some generic error
message.
These are the steps to set up the build server on a clean infrastructure:
1. Configure the tools on your local machine:
```bash
./local_setup.sh
```
If you not running docker under your user, you might need to
`sudo gcloud auth login --no-launch-browser && sudo gcloud auth configure-docker`
before running other commands under sudo.
1. Delete the old cluster, if it still exists:
```bash
cd kubernetes/cluster
./cluster_delete.sh
```
1. Create the cluster:
```bash
cd kubernetes/cluster
./cluster_create.sh
```
1. Push the docker images to gcr.io:
```bash
cd containers
#for each subfolder:
./build_deploy.sh <foldername>
```
1. Deploy the stack:
```bash
cd kubernetes
./deploy.sh
```
1. Configure it
## Running Docker containers
`./containers/build_run.sh` provides a good way to run a container once to check that it builds and contains good package versions. To work with a container and
persist it state do (taking 'base-debian' as example):
```
cd ./containers/base-debian
sudo docker build -t base-debian .
sudo docker run -it base-debian /bin/bash
```
to resume it from where you left
```
sudo docker start -a -i $(sudo docker ps -a -f ancestor=base-debian -q -l)
```
## Creating docker containers on Windows
If you want to build/update/test docker container for Windows, you need to do this on a Windows machine.
**Note**: There is an existing *windows-development* machine that you can resume and use for development. Please stop it after use.
To setup new machine in GCP:
1. Pick a GCP Windows image with Desktop Support.
* pick a "persistent SSD" as boot Disk. This is much faster.
* make sure that you give enough permissions in "Identity and API access" to be able to e.g. push new docker images to GCR.
1. Format the local SSD partition and use it as workspace.
1. install [Chocolately](https://chocolatey.org/docs/installation):
```powershell
iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
```
1. Install development tools: `choco install -y git googlechrome vscode`
1. (optionally) If you want to be able to push changes to github, you need to set up your github SSH keys and user name:
```powershell
ssh-keygen
git config --global user.name <your name>
git config --global user.email <your email>
```
1. Clone premerge checks sources:
```powershell
cd c:\
git clone https://github.com/google/llvm-premerge-checks
```
1. Install [Docker Enterprise](https://docs.docker.com/ee/docker-ee/windows/docker-ee/) and reboot:
```powershell
Install-Module DockerMsftProvider -Force
Install-Package Docker -ProviderName DockerMsftProvider -Force
Restart-Computer
```
1. Configure the Docker credentials for GCP:
```powershell
gcloud init # set options according to ./k8s_config here
gcloud components install docker-credential-gcr
docker-credential-gcr configure-docker
```
## Build / test docker for Windows.
1. To build and run a dockerfile:
```powershell
cd llvm-premerge-checks\containers
.\build_deploy.ps1 agent-windows-buildkite
1. To deploy container:
```powershell
cd llvm-premerge-checks\containers
.\build_deploy.ps1 agent-windows-buildkite
```
or
```powershell
cd llvm-premerge-checks\containers
.\build_run.ps1 agent-windows-buildkite cmd
```
Test this newly uploaded image:
```powershell
c:\llvm-premerge-check\scripts\windows_agent_start_buildkite.ps1
```
## Spawning a new windows agent
To spawn a new Windows agent:
1. Go to the [GCP page](https://pantheon.corp.google.com/compute/instances?project=llvm-premerge-checks&instancessize=50).
1. Add new windows machine wih OS "Windows Server" and version with "desktop experience" (so you can RDP) and boot disk size ~500 Gb. There is a "windows-agent-template" that might not be up to date.
1. Go to the [GCP page](https://pantheon.corp.google.com/compute/instances?project=llvm-premerge-checks&instancessize=50) again
1. Login to the new machine via RDP (you will need a RDP client, e.g. Chrome app).
1. (optional, quality of life) Add a powershell shortcut at desktop with "run as admin" flag. Create a folder with machine name (e.g "w16c2-2") somewhere and click "add new toolbar" on windows toolbar: this way it will be easier to identify which machine you are working with later.
1. Run these commands in the power shell under admin to bootstrap the Windows machine:
```powershell
Invoke-WebRequest -uri 'https://raw.githubusercontent.com/google/llvm-premerge-checks/main/scripts/windows_agent_bootstrap.ps1' -OutFile c:\windows_agent_bootstrap.ps1
c:/windows_agent_bootstrap.ps1
```
VM will be restarted after a prompt.
To start agent manually:
```powershell
C:\llvm-premerge-checks\scripts\windows_agent_start_buildkite.ps1 [-workdir D:\] [-testing] [-version latest]
```
## Custom environment variables
Buildkite pipelines have a number of custom environment variables one can set to change their behavior. That is useful to debug issues
or test changes. They are mostly used by pipleine generators, e.g. [pipeline_main](../scripts/pipeline_main.py),
please refer to the source code for the details. These variables have `ph_` prefix and can be set with URL parameters in Harbormaster build.
Most commonly used are:
- `ph_scripts_refspec`: ("main" by default): refspec branch of llvm-premerge-checks to use. This variable is also used in pipeline "bootstrap" in Buildkite interface. Use "branch-name" for branches and "pull/123/head" for Pull Requests.
- `ph_dry_run_report`: do not report any results back to Phabricator.
- `ph_no_cache`: (if set to any value) clear compilation cache before the build.
- `ph_projects`: which projects to use (semicolon separated), "detect" will look on diff to infer the projects, "default" selects all projects.
- `ph_notify_email`: comma-separated list of email addresses to be notified when build is complete.
- `ph_log_level` ("DEBUG", "INFO", "WARNING" (default) or "ERROR"): log level for build scripts.
- `ph_linux_agents`, `ph_windows_agents`: custom JSON constraints on agents. For example, you might put one machine to a custom queue if it's errornous and send jobs to it with `ph_windows_agents={"queue": "custom"}`.
- `ph_skip_linux`, `ph_skip_windows` (if set to any value): skip build on this OS.
- `ph_skip_generated`: don't run custom steps generated from within llvm-project.
While trying a new patch for premerge scripts it's typical to start a new build by copying "ph_"
env variables from one of the recent builds and appending
```shell
ph_dry_run_report=yes
ph_skip_windows=yes
ph_skip_generated=yes
ph_scripts_refspec="<branch name>"
ph_log_level=DEBUG
```
## Update HTTP auth credentials
To update e.g. buildkite http-auth:
```shell script
kubectl get secret http-auth -n buildkite -o yaml
# base64 decode it's data to 'auth'.
echo <data/auth from yaml> | base64 --decode > auth
# add / update passwords
htpasswd -b auth <user> <pass>
# update secret
kubectl delete secret http-auth -n buildkite
kubectl create secret generic http-auth -n buildkite --from-file=./auth
```
## Update github-ssh secret
This Kubernetes stores github ssh access files for llvm.premerge.tests@gmail.com user.
To access it run `kubectl get secret github-ssh -n buildkite -o yaml | tee github-ssh.yam`.
It has 3 entries: "id_rsa", "id_rsa.pub", "known_hosts".
If you need to updated it, edit resulting yaml above with updated base 64 values.
To updated "known_hosts" run `ssh-keyscan github.com` and store base64 encoded output in "known_hosts".
Then run `kubectl apply -f github-ssh.yaml`.
To create this secret from scratch run
`kubectl create secret generic github-ssh --namespace buildkite --from-file ./id_rsa --from-file ./id_rsa.pub --from-file ./known_hosts`
providing all files needed.
Note that containers have to be restarted as ssh settings are copied at startup
(entrypoint.sh).

57
kubernetes/cert-issuer.yaml
View file

@ -1,57 +0,0 @@
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# based on documentation on
# https://github.com/jetstack/cert-manager/blob/master/docs/tutorials/acme/quick-start/index.rst
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: goncharov@google.com
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- dns01:
cloudDNS:
# The ID of the GCP project
project: "llvm-premerge-checks"
# This is the secret used to access the service account
serviceAccountSecretRef:
name: clouddns-dns01-solver-svc-acct
key: key.json
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: goncharov@google.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- dns01:
cloudDNS:
project: "llvm-premerge-checks"
serviceAccountSecretRef:
name: clouddns-dns01-solver-svc-acct
key: key.json
# - http01:
# ingress:
# class: gce

48
kubernetes/cluster_create.sh
View file

@ -1,48 +0,0 @@
#!/bin/bash
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -eux
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
ROOT_DIR="$(dirname ${DIR})"
# get config options
source "${ROOT_DIR}/k8s_config"
# create the cluster
gcloud container clusters create $GCP_CLUSTER --zone $GCP_ZONE \
--machine-type=n1-standard-4 --num-nodes=1
# Linux agents node pool with local ssd.
# https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd
gcloud container node-pools create linux-agents --cluster $GCP_CLUSTER --zone $GCP_ZONE \
--machine-type=n1-standard-32 --num-nodes=2 --local-ssd-count=1
# created separate cluster for windows, as we need "ip-alias" enabled
# this can't be changed in a running cluster...
gcloud beta container clusters create $GCP_CLUSTER_WINDOWS \
--enable-ip-alias \
--num-nodes=1 \
--release-channel=rapid \
--enable-private-nodes
# Windows agents with local ssd
gcloud container node-pools create windows-pool --cluster $GCP_CLUSTER_WINDOWS \
--image-type=WINDOWS_SAC --no-enable-autoupgrade \
--machine-type=n1-standard-16 --local-ssd-count=1
# create static IP address
# IP can be created, but not used in Ingress. Not sure why
gcloud compute addresses create web-static-ip --zone=$GCP_ZONE

27
kubernetes/cluster_delete.sh
View file

@ -1,27 +0,0 @@
#!/bin/bash
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -eux
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
ROOT_DIR="$(dirname ${DIR})"
# get config options
source "${ROOT_DIR}/k8s_config"
#TODO: check gcloud auth login
# create the cluster
gcloud container clusters delete $GCP_CLUSTER

63
kubernetes/cron/buildbots.yaml
View file

@ -1,63 +0,0 @@
# Copyright 2021 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: buildbot-stats
namespace: buildkite
spec:
schedule: "40 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 24
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
containers:
- name: collect-buildkite-stats
image: gcr.io/llvm-premerge-checks/stats:latest
args: ["/root/llvm-premerge-checks/scripts/metrics/connect_db.sh python3 buildbot_monitoring.py"]
env:
- name: BUILDKITE_AGENT_TOKEN
valueFrom:
secretKeyRef:
name: buildkite-agent-token
key: token
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: CONDUIT_TOKEN
valueFrom:
secretKeyRef:
name: conduit-api-token
key: token
- name: BUILDKITE_API_TOKEN
valueFrom:
secretKeyRef:
name: buildkite-api-token
key: token
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-stats
key: password
- name: SCRIPTS_REFSPEC
value: "main"
restartPolicy: Never
nodeSelector:
cloud.google.com/gke-nodepool: service

63
kubernetes/cron/buildkite.yaml
View file

@ -1,63 +0,0 @@
# Copyright 2021 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: batch/v1
kind: CronJob
metadata:
name: buildkite-stats
namespace: buildkite
spec:
schedule: "0 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 24
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
containers:
- name: collect-buildkite-stats
image: gcr.io/llvm-premerge-checks/stats:latest
args: ["/root/llvm-premerge-checks/scripts/metrics/connect_db.sh download_buildkite_builds_pg.sh"]
env:
- name: BUILDKITE_AGENT_TOKEN
valueFrom:
secretKeyRef:
name: buildkite-agent-token
key: token
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: CONDUIT_TOKEN
valueFrom:
secretKeyRef:
name: conduit-api-token
key: token
- name: BUILDKITE_API_TOKEN
valueFrom:
secretKeyRef:
name: buildkite-api-token
key: token
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-stats
key: password
- name: SCRIPTS_REFSPEC
value: "main"
restartPolicy: Never
nodeSelector:
cloud.google.com/gke-nodepool: default-pool

63
kubernetes/cron/uptime.yaml
View file

@ -1,63 +0,0 @@
# Copyright 2021 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: uptime-stats
namespace: buildkite
spec:
schedule: "20 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 24
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
containers:
- name: collect-buildkite-stats
image: gcr.io/llvm-premerge-checks/stats:latest
args: ["/root/llvm-premerge-checks/scripts/metrics/connect_db.sh python3 server_monitoring.py"]
env:
- name: BUILDKITE_AGENT_TOKEN
valueFrom:
secretKeyRef:
name: buildkite-agent-token
key: token
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: CONDUIT_TOKEN
valueFrom:
secretKeyRef:
name: conduit-api-token
key: token
- name: BUILDKITE_API_TOKEN
valueFrom:
secretKeyRef:
name: buildkite-api-token
key: token
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-stats
key: password
- name: SCRIPTS_REFSPEC
value: "main"
restartPolicy: Never
nodeSelector:
cloud.google.com/gke-nodepool: service

72
kubernetes/github/linux-test.yaml
View file

@ -1,72 +0,0 @@
# Copyright 2023 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: apps/v1
kind: Deployment
metadata:
name: github-linux-test
spec:
replicas: 1
strategy:
rollingUpdate:
maxUnavailable: 1
maxSurge: 0
type: RollingUpdate
selector:
matchLabels:
app: github-linux-test
template:
metadata:
labels:
app: github-linux-test
spec:
containers:
- name: runner
image: us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux:latest
resources:
limits:
cpu: 31
memory: 80Gi
requests:
cpu: 31
memory: 80Gi
volumeMounts:
- name: workdir
mountPath: /work
env:
- name: WORKDIR
value: "/work"
- name: ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: github-register-agent-pat
key: token
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: RUNNER_NAME
value: "$(POD_NAME)"
- name: RUNNER_SCOPE
value: "org"
- name: ORG_NAME
value: "metafloworg"
- name: LABELS
value: "linux"
volumes:
- name: workdir
emptyDir: {}
nodeSelector:
cloud.google.com/gke-nodepool: linux-agents-2
terminationGracePeriodSeconds: 30

39
kubernetes/ingress.yaml
View file

@ -1,39 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-llvm-premerge
annotations:
# nginx.ingress.kubernetes.io/auth-type: basic
# nginx.ingress.kubernetes.io/auth-secret: http-auth
# nginx.ingress.kubernetes.io/auth-realm: "LLVM pre-merge checks"
# nginx.ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/ingress.class: "nginx"
# cert-manager
cert-manager.io/cluster-issuer: "letsencrypt-prod"
# kubernetes.io/ingress.global-static-ip-name: "llvm-premerge"
acme.cert-manager.io/http01-edit-in-place: "true"
# ^ cert-manager
spec:
ingressClassName: nginx
# cert-manager
tls:
- hosts:
- llvm-premerge.org
secretName: llvm-premerge-org-cert
# ^ cert-manager
defaultBackend:
service:
name: phabricator-proxy
port:
number: 8080
rules:
- host: llvm-premerge.org
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: phabricator-proxy
port:
number: 8080

57
kubernetes/phabricator-proxy/deployment.yaml
View file

@ -1,57 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: apps/v1
kind: Deployment
metadata:
name: phabricator-proxy
spec:
selector:
matchLabels:
app: phabricator-proxy
replicas: 1
template:
metadata:
labels:
app: phabricator-proxy
spec:
containers:
- name: phabricator-proxy
image: gcr.io/llvm-premerge-checks/phabricator-proxy:latest
ports:
- containerPort: 8080
protocol: TCP
env:
- name: BUILDKITE_API_TOKEN
valueFrom:
secretKeyRef:
name: buildkite-api-token
key: token
readinessProbe:
httpGet:
path: /
port: 8080
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 2
failureThreshold: 5
resources:
limits:
cpu: 500m
memory: 1500Mi
requests:
cpu: 500m
memory: 1500Mi
nodeSelector:
cloud.google.com/gke-nodepool: default-pool

22
kubernetes/phabricator-proxy/kustomization.yaml
View file

@ -1,22 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: buildkite
resources:
- Deployment.yaml
- Services.yaml
- Ingress.yaml

25
kubernetes/phabricator-proxy/service.yaml
View file

@ -1,25 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
kind: Service
apiVersion: v1
metadata:
name: phabricator-proxy
spec:
selector:
app: phabricator-proxy
ports:
- protocol: TCP
port: 8080
targetPort: 8080

33
kubernetes/windows_agent_create.sh
View file

@ -1,33 +0,0 @@
#!/bin/bash
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#-------------------------------------------------------------------------------
# create a new windows agent
set -eux
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
ROOT_DIR="$(dirname ${DIR})"
# get config options
source "${ROOT_DIR}/k8s_config"
NAME=$1
gcloud beta compute instances create "${NAME}" \
--project="${GCP_PROJECT}" \
--zone="${GCP_ZONE}" \
--machine-type=n2d-standard-16 \
--image=windows-server-2019-dc-for-containers-v20200714 \
--image-project=windows-cloud \
--boot-disk-size=200GB --boot-disk-type=pd-ssd

1
scripts/.gitignore vendored
View file

@ -1 +0,0 @@
benchmark/

0
scripts/__init__.py
View file

33
scripts/add_phabricator_artifact.py
View file

@ -1,33 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse
import logging
import os
from phabtalk.phabtalk import PhabTalk
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Runs premerge checks8')
parser.add_argument('--url', type=str)
parser.add_argument('--name', type=str)
parser.add_argument('--phid', type=str)
parser.add_argument('--log-level', type=str, default='WARNING')
args = parser.parse_args()
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
dry = os.getenv('ph_dry_run_report') is not None
PhabTalk(os.getenv('CONDUIT_TOKEN'), dry_run_updates=dry).maybe_add_url_artifact(args.phid, args.url, args.name)

38
scripts/apply_patch.sh
View file

@ -1,38 +0,0 @@
#!/usr/bin/env bash
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Scripts that use secrets has to be a separate files, not inlined in pipeline:
# https://buildkite.com/docs/pipelines/secrets#anti-pattern-referencing-secrets-in-your-pipeline-yaml
set -uo pipefail
scripts/patch_diff.py $ph_buildable_diff \
--path "${BUILDKITE_BUILD_PATH}"/llvm-project-fork \
--token $CONDUIT_TOKEN \
--url $PHABRICATOR_HOST \
--log-level $LOG_LEVEL \
--commit $BASE_COMMIT \
--phid ${ph_target_phid} \
--push-branch
EXIT_STATUS=$?
if [ $EXIT_STATUS -ne 0 ]; then
scripts/add_phabricator_artifact.py --phid="$ph_target_phid" --url="$BUILDKITE_BUILD_URL" --name="patch application failed"
scripts/set_build_status.py
echo failed
fi
exit $EXIT_STATUS

150
scripts/benchmark.py
View file

@ -1,150 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Run benchmark of the various steps in the pre-merge tests.
This can be used to tune the build times.
"""
import argparse
import csv
import datetime
import multiprocessing
import os
import platform
import psutil
import subprocess
import sys
from typing import Optional, Dict, List
class Cmd:
"""Command to be executed as part of the benchmark.
If name is not set, the results will not be logged to the result file.
"""
def __init__(self, cmd: str, title: str = None):
self.cmd = cmd # type: str
self.title = title # type: Optional[str]
self.execution_time = None # type: Optional[datetime.timedelta]
@property
def has_title(self) -> bool:
return self.title is not None
@property
def was_executed(self) -> bool:
return self.execution_time is not None
class Remove(Cmd):
"""Remove command, sensitive to OS."""
def __init__(self, path: str):
if platform.system() == 'Windows':
cmd = 'cmd /c rd /s/q {}'.format(path)
else:
cmd = 'rm -rf {}'.format(path)
super().__init__(cmd, "rm -rf {}".format(path))
# commands for the benchmark
COMMANDS = [
Cmd('git clone https://github.com/llvm/llvm-project .', 'git clone'),
Cmd('git checkout release/10.x', 'git checkout release'),
Cmd('git pull', 'git pull'),
Cmd('git checkout {commit}'),
# clean build, make sure ccache (or similar) is empty
Cmd('{pmt_root_path}/scripts/run_cmake.py', 'cmake 1st'),
Cmd('{pmt_root_path}/scripts/run_ninja.py all', 'ninja all 1st'),
Cmd('{pmt_root_path}/scripts/run_ninja.py check-all', 'ninja check-all 1st'),
Remove('build'),
# now rebuild from ccache (or similar)
Cmd('{pmt_root_path}/scripts/run_cmake.py', 'cmake 2nd'),
Cmd('{pmt_root_path}/scripts/run_ninja.py all', 'ninja all 2nd'),
Cmd('{pmt_root_path}/scripts/run_ninja.py check-all', 'ninja check-all 2nd'),
]
def run_benchmark(commit: str, name: str, result_file_path: str, workdir: str, pmt_root_path: str):
"""Tun the benchmark, write the results to a file."""
print('Usingn workdir {}'.format(workdir))
print('Using scripts from {}'.format(pmt_root_path))
cmd_parameters = {
'pmt_root_path': pmt_root_path,
'commit': commit,
}
if os.path.exists(workdir):
run_cmd(Remove(workdir), cmd_parameters, '.')
os.makedirs(workdir)
try:
for command in COMMANDS:
run_cmd(command, cmd_parameters, workdir)
finally:
write_results(COMMANDS, result_file_path, name)
def write_results(commands: List[Cmd], result_file_path: str, name: str):
fieldnames = ['name', 'cores', 'CPU', 'RAM', 'timestamp', 'OS']
fieldnames.extend(cmd.title for cmd in COMMANDS if cmd.has_title)
exists = os.path.exists(result_file_path)
with open(result_file_path, 'a') as csv_file:
writer = csv.DictWriter(csv_file, fieldnames=fieldnames, dialect=csv.excel)
if not exists:
writer.writeheader()
row = {
'name': name,
'cores': multiprocessing.cpu_count(),
'CPU': platform.processor(),
'RAM': psutil.virtual_memory().total,
'timestamp': datetime.datetime.now().timestamp(),
'OS': platform.platform()
}
for command in (cmd for cmd in commands if (cmd.has_title and cmd.was_executed)):
row[command.title] = command.execution_time.total_seconds()
writer.writerow(row)
print('Benchmark completed.')
def run_cmd(command: Cmd, cmd_parameters: Dict[str, str], workdir: str):
"""Run a single command."""
cmdline = command.cmd.format(**cmd_parameters)
print('Running: {}'.format(cmdline))
start_time = datetime.datetime.now()
proc = subprocess.Popen(cmdline, shell=True, cwd=workdir, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()
if proc.poll() != 0:
print(stdout)
print(stderr)
print('Benchmark failed.')
sys.exit(1)
end_time = datetime.datetime.now()
command.execution_time = end_time - start_time
print(' Execution time was: {}'.format(command.execution_time))
if __name__ == '__main__':
pmt_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
parser = argparse.ArgumentParser(
description='Benchmark for LLVM pre-merge tests.')
parser.add_argument('--commit', type=str, default='main', help="LLVM commit to run this benchmark on.")
parser.add_argument('--result-file', type=str, default='pmt-benchmark.csv',
help="path to CSV file where to store the benchmark results")
parser.add_argument('--workdir', type=str, default=os.path.join(os.getcwd(), 'benchmark'),
help='Folder to store the LLVM checkout.')
parser.add_argument('--name', type=str, default=None, help="name for the benchmark")
args = parser.parse_args()
run_benchmark(args.commit, args.name, args.result_file, args.workdir, pmt_root)

46
scripts/buildkite/sample_build.py
View file

@ -1,46 +0,0 @@
import os
import json
import argparse
import requests
import time
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Run sample build on buildkite.')
parser.add_argument('--dryrun', action='store_true')
parser.add_argument('--commit')
parser.add_argument('--pipeline', default='llvm-main')
args = parser.parse_args()
pipeline=args.pipeline
time.sleep(2)
d = json.dumps({
'branch': 'main',
'commit': args.commit,
'env': {
'ph_log_level': 'DEBUG',
#'ph_skip_linux': 'skip',
'ph_linux_agents': '{"queue": "linux-test-google"}',
#'ph_linux_agents': '{"queue": "linux-test"}',
# 'ph_linux_agents': '{"queue": "linux-clang15-test"}',
'ph_skip_windows': 'skip',
#'ph_windows_agents': f'{{"name": "win-dev", "queue": "windows-test"}}',
# 'ph_windows_agents': '{"queue": "windows-test"}',
# 'ph_scripts_refspec': 'windows-vscmd',
# 'ph_projects': 'all',
'ph_skip_generated': 'skip',
# 'ph_windows_agents': f'{{"name": "", "queue": "{queue}"}}',
}})
print(d)
if (args.dryrun):
exit(0)
token = os.getenv('BUILDKITE_API_TOKEN')
if token is None:
print("'BUILDKITE_API_TOKEN' environment variable is not set")
exit(1)
re = requests.post(f'https://api.buildkite.com/v2/organizations/llvm-project/pipelines/{pipeline}/builds',
data=d,
headers={'Authorization': f'Bearer {token}'})
print(re.status_code)
j = re.json()
print(j['web_url'])

109
scripts/buildkite_utils.py
View file

@ -1,109 +0,0 @@
import logging
import os
import re
import subprocess
import urllib.parse
from typing import Optional
from benedict import benedict
import backoff
import requests
context_style = {}
previous_context = 'default'
styles = ['default', 'info', 'success', 'warning', 'error']
def upload_file(base_dir: str, file: str):
"""
Uploads artifact to buildkite and returns URL to it
"""
r = subprocess.run(f'buildkite-agent artifact upload "{file}"', shell=True, capture_output=True, cwd=base_dir)
logging.debug(f'upload-artifact {r}')
match = re.search('Uploading artifact ([^ ]*) ', r.stderr.decode())
logging.debug(f'match {match}')
if match:
url = f'https://buildkite.com/organizations/llvm-project/pipelines/premerge-checks/builds/{os.getenv("BUILDKITE_BUILD_NUMBER")}/jobs/{os.getenv("BUILDKITE_JOB_ID")}/artifacts/{match.group(1)}'
logging.info(f'uploaded {file} to {url}')
return url
else:
logging.warning(f'could not find artifact {base_dir}/{file}')
return None
def set_metadata(key: str, value: str):
r = subprocess.run(f'buildkite-agent meta-data set "{key}" "{value}"', shell=True, capture_output=True)
if r.returncode != 0:
logging.warning(r)
def annotate(message: str, style: str = 'default', context: Optional[str] = None, append: bool = True):
"""
Adds an annotation for that currently running build.
Note that last `style` applied to the same `context` takes precedence.
"""
global previous_context, styles, context_style
if style not in styles:
style = 'default'
if context is None:
context = previous_context
previous_context = context
# Pick most severe style so far.
context_style.setdefault(context, 0)
context_style[context] = max(styles.index(style), context_style[context])
style = styles[context_style[context]]
if append:
message += '\n\n'
cmd = ['buildkite-agent', 'annotate', message, '--style', style, '--context', context]
if append:
cmd.append('--append')
r = subprocess.run(cmd, capture_output=True)
logging.debug(f'annotate call {r}')
if r.returncode != 0:
logging.warning(r)
def feedback_url():
title = f"buildkite build {os.getenv('BUILDKITE_PIPELINE_SLUG')} {os.getenv('BUILDKITE_BUILD_NUMBER')}"
return f'https://github.com/google/llvm-premerge-checks/issues/new?assignees=&labels=bug' \
f'&template=bug_report.md&title={urllib.parse.quote(title)}'
class BuildkiteApi:
def __init__(self, token: str, organization: str):
self.token = token
self.organization = organization
def get_build(self, pipeline: str, build_number: str):
# https://buildkite.com/docs/apis/rest-api/builds#get-a-build
return benedict(self.get(f'https://api.buildkite.com/v2/organizations/{self.organization}/pipelines/{pipeline}/builds/{build_number}').json())
def list_running_revision_builds(self, pipeline: str, rev: str):
return self.get(f'https://api.buildkite.com/v2/organizations/{self.organization}/pipelines/{pipeline}/builds?state[]=scheduled&state[]=running&meta_data[ph_buildable_revision]={rev}').json()
@backoff.on_exception(backoff.expo, Exception, max_tries=3, logger='', factor=3)
def get(self, url: str):
authorization = f'Bearer {self.token}'
response = requests.get(url, allow_redirects=True, headers={'Authorization': authorization})
if response.status_code != 200:
raise Exception(f'Buildkite responded with non-OK status: {response.status_code}')
return response
# cancel a build. 'build' is a json object returned by API.
def cancel_build(self, build):
build = benedict(build)
url = f'https://api.buildkite.com/v2/organizations/{self.organization}/pipelines/{build.get("pipeline.slug")}/builds/{build.get("number")}/cancel'
authorization = f'Bearer {self.token}'
response = requests.put(url, headers={'Authorization': authorization})
if response.status_code != 200:
raise Exception(f'Buildkite responded with non-OK status: {response.status_code}')
def format_url(url: str, name: Optional[str] = None):
if name is None:
name = url
return f"\033]1339;url='{url}';content='{name}'\a\n"
def strip_emojis(s: str) -> str:
return re.sub(r':[^:]+:', '', s).strip()

215
scripts/choose_projects.py
View file

@ -1,215 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Compute the LLVM_ENABLE_PROJECTS for cmake from diff.
This script will compute which projects are affected by the diff provided via STDIN.
It gets the modified files in the patch, assigns them to projects and based on a
project dependency graph it will get the transitively affected projects.
"""
import argparse
import logging
import os
import platform
import sys
from typing import Any, Dict, List, Set, TextIO, Tuple, Optional, Union
from unidiff import PatchSet # type: ignore
import yaml
class ChooseProjects:
# file where dependencies are defined
SCRIPT_DIR = os.path.dirname(__file__)
DEPENDENCIES_FILE = os.path.join(SCRIPT_DIR, 'llvm-dependencies.yaml')
def __init__(self, llvm_dir: Optional[str]):
self.llvm_dir = llvm_dir
self.defaultProjects: Dict[str, Dict[str, str]] = {}
# List of projects this project depends on, transitive closure.
# E.g. compiler-rt -> [llvm, clang].
self.dependencies: Dict[str,Set[str]] = {}
# List of projects that depends on this project. It's a full closure.
# E.g. llvm -> [clang, libcxx, ...]
self.usages: Dict[str, Set[str]] = dict()
self.all_projects: List[str] = ['all']
self.config: Dict[str, Any] = {}
self._load_config()
def _load_config(self):
logging.info('loading project config from {}'.format(self.DEPENDENCIES_FILE))
with open(self.DEPENDENCIES_FILE) as dependencies_file:
self.config = yaml.load(dependencies_file, Loader=yaml.SafeLoader)
for k, v in self.config['dependencies'].items():
self.dependencies[k] = set(v)
# Closure of dependencies.
while True:
updated = False
for s in self.dependencies.values():
n = len(s)
extend = set()
for d in s:
extend.update(self.dependencies.get(d, set()))
s.update(extend)
if len(s) > n:
updated = True
if not updated:
break
# Usages don't need to be closed as dependencies already are.
for project, deps in self.dependencies.items():
for d in deps:
self.usages.setdefault(d, set()).add(project)
logging.info(f'computed dependencies: {self.dependencies}')
logging.info(f'computed usages: {self.usages}')
self.all_projects = self.config['allprojects'].keys()
def get_excluded(self, os: str) -> Set[str]:
"""Returns transitive closure for excluded projects"""
return self.get_affected_projects(set(self.config['excludedProjects'].get(os, [])))
def get_check_targets(self, projects: Set[str]) -> Set[str]:
"""Return the `check-xxx` targets to pass to ninja for the given list of projects"""
if 'all' in projects:
return set(["check-all"])
targets = set()
all_projects = self.config['allprojects']
for project in projects:
targets.update(set(all_projects.get(project, [])))
return targets
@staticmethod
def _detect_os() -> str:
"""Detect the current operating system."""
if platform.system() == 'Windows':
return 'windows'
return 'linux'
def choose_projects(self, patch: str = None, os_name: Optional[str] = None) -> List[str]:
"""List all touched project with all projects that they depend on and also
all projects that depend on them"""
if self.llvm_dir is None:
raise ValueError('path to llvm folder must be set in ChooseProject.')
llvm_dir = os.path.abspath(os.path.expanduser(self.llvm_dir))
logging.info('Scanning LLVM in {}'.format(llvm_dir))
if not self.match_projects_dirs():
logging.warning(f'{llvm_dir} does not look like a llvm-project directory')
return self.get_all_enabled_projects(os_name)
changed_files = self.get_changed_files(patch)
changed_projects, unmapped_changes = self.get_changed_projects(changed_files)
if unmapped_changes:
logging.warning('There were changes that could not be mapped to a project.'
'Building all projects instead!')
return self.get_all_enabled_projects(os_name)
return self.extend_projects(changed_projects, os_name)
def extend_projects(self, projects: Set[str], os_name : Optional[str] = None) -> List[str]:
"""Given a set of projects returns a set of projects to be tested taking
in account exclusions from llvm-dependencies.yaml.
"""
logging.info(f'projects: {projects}')
if not os_name:
os_name = self._detect_os()
# Find all affected by current set.
affected_projects = self.get_affected_projects(projects)
logging.info(f'all affected projects(*) {affected_projects}')
# Exclude everything that is affected by excluded.
excluded_projects = self.get_excluded(os_name)
logging.info(f'all excluded projects(*) {excluded_projects}')
affected_projects = affected_projects - excluded_projects
logging.info(f'effective projects list {affected_projects}')
return sorted(affected_projects)
def run(self):
affected_projects = self.choose_projects()
print("Affected:", ';'.join(affected_projects))
print("Dependencies:", ';'.join(self.get_dependencies(affected_projects)))
print("Check targets:", ';'.join(self.get_check_targets(affected_projects)))
return 0
def match_projects_dirs(self) -> bool:
"""Make sure that all projects are folders in the LLVM dir.
"""
subdirs = os.listdir(self.llvm_dir)
for project in self.all_projects:
if project not in subdirs:
logging.error('Project not found in LLVM root folder: {}'.format(project))
return False
return True
@staticmethod
def get_changed_files(patch_str: Union[str, TextIO, None] = None) -> Set[str]:
"""get list of changed files from the patch or from STDIN.
e.g. ['compiler-rt/lib/tsan/CMakeLists.txt']"""
if patch_str is None:
patch_str = sys.stdin
patch = PatchSet(patch_str)
changed_files = set({f.path for f in patch.modified_files + patch.added_files + patch.removed_files})
logging.info('Files modified by this patch:\n ' + '\n '.join(sorted(changed_files)))
return changed_files
def get_changed_projects(self, changed_files: Set[str]) -> Tuple[Set[str], bool]:
"""Get list of projects affected by the change."""
logging.info("Get list of projects affected by the change.")
changed_projects = set()
unmapped_changes = False
for changed_file in changed_files:
project = changed_file.split('/', maxsplit=1)
# There is no utils project.
if project[0] == 'utils':
continue
if (project is None) or (project[0] not in self.all_projects):
unmapped_changes = True
logging.warning('Could not map file to project: {}'.format(changed_file))
else:
changed_projects.add(project[0])
logging.info('Projects directly modified by this patch:\n ' + '\n '.join(sorted(changed_projects)))
return changed_projects, unmapped_changes
def get_affected_projects(self, changed_projects: Set[str]) -> Set[str]:
"""Compute transitive closure of affected projects based on the
dependencies between the projects (including initially passed)."""
affected: Set[str] = set(changed_projects)
for p in changed_projects:
affected.update(self.usages.get(p, set()))
logging.info(f'added {affected - changed_projects} projects as they are affected')
return affected
def get_dependencies(self, projects: Set[str]) -> Set[str]:
"""Return transitive dependencies for a given projects (including the projects themself).
These are the required dependencies for given `projects` so that they can be built.
"""
affected: Set[str] = set(projects)
for p in projects:
affected.update(self.dependencies.get(p, set()))
return affected
def get_all_enabled_projects(self, os_name: Optional[str] = None) -> List[str]:
"""Get list of all not-excluded projects for current platform."""
return self.extend_projects(set(self.all_projects), os_name)
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description='Compute the projects affected by a change. A patch file is expected on stdin.')
parser.add_argument('--llvmdir', type=str, default='.')
parser.add_argument('--log-level', type=str, default='INFO')
args = parser.parse_args()
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
logging.info(f'checking changes in {args.llvmdir}')
chooser = ChooseProjects(args.llvmdir)
sys.exit(chooser.run())

2
scripts/clang-format.ignore
View file

@ -1,2 +0,0 @@
# Patterns for clang-format to ignore.
**/test

5
scripts/clang-tidy-comments.ignore
View file

@ -1,5 +0,0 @@
# Files that are allowed by clang-tidy.ignore but should not receive inline review comments.
# Right now it works in whitelist mode and only some files / directories are whitelisted.
# FIXME: is is used?
*
!clang-tools-extra/clangd/**

56
scripts/clang-tidy.ignore
View file

@ -1,56 +0,0 @@
# Files to be ignored by clang-tidy. Will not appear in short report and inline comments.
debuginfo-tests/
libcxxabi/test
libcxxabi/test/libcxxabi/test
libcxx/test
libcxx/utils/libcxx/test
libcxx/utils/google-benchmark/test
llvm/test
llvm/utils/gn/secondary/llvm/test
llvm/utils/gn/secondary/lld/test
llvm/utils/gn/secondary/clang-tools-extra/test
llvm/utils/gn/secondary/clang-tools-extra/clangd/test
llvm/utils/gn/secondary/compiler-rt/test
llvm/utils/gn/secondary/clang/test
llvm/utils/benchmark/test
polly/test
lldb/examples/test
lldb/test
lldb/utils/test
lldb/tools/intel-features/intel-mpx/test
lldb/packages/Python/lldbsuite/test
lldb/packages/Python/lldbsuite/test/tools/lldb-server/test
lldb/packages/Python/lldbsuite/test/commands/expression/test
lldb/packages/Python/lldbsuite/test/test_runner/test
lldb/third_party/Python/module/unittest2/unittest2/test
lld/test
clang-tools-extra/test
clang-tools-extra/clangd/clients/clangd-vscode/test
clang-tools-extra/clangd/test
pstl/test
libc/test
llgo/test
compiler-rt/test
compiler-rt/test/builtins/Unit/ppc/test
compiler-rt/test/builtins/Unit/test
compiler-rt/lib/sanitizer_common/sanitizer.*.inc
compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp
compiler-rt/lib/sanitizer_common/sanitizer_common_syscalls.inc
debuginfo-tests/dexter/dex/tools/test
debuginfo-tests/dexter/feature_tests/subtools/test
clang/lib/Headers/__clang_hip_*.h
clang/lib/Headers/wasm_simd128.h
clang/test
libclc
mlir/examples/standalone
mlir/include/mlir-c/Bindings
mlir/include/mlir/Bindings
mlir/lib/Bindings
mlir/test
mlir/tools/mlir-cuda-runner
mlir/tools/mlir-rocm-runner
openmp/libomptarget/test
openmp/libomptarget/deviceRTLs/nvptx/test
openmp/runtime/test
libunwind/test
libunwind/test/libunwind/test

111
scripts/clang_format_report.py
View file

@ -1,111 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse
import logging
import os
import subprocess
from typing import Tuple, Optional
import pathspec
import unidiff
from phabtalk.phabtalk import Report, Step
from buildkite_utils import annotate
def get_diff(base_commit) -> Tuple[bool, str]:
r = subprocess.run(f'python3 clang/tools/clang-format/git-clang-format {base_commit}', shell=True)
logging.debug(f'git-clang-format {r}')
if r.returncode != 0:
logging.error(f'git-clang-format returned an non-zero exit code {r.returncode}')
r = subprocess.run(f'git checkout -- .', stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
logging.debug(f'git reset {r}')
return False, ''
diff_run = subprocess.run(f'git diff -U0 --no-prefix --exit-code', capture_output=True, shell=True)
logging.debug(f'git diff {diff_run}')
r = subprocess.run(f'git checkout -- .', stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
logging.debug(f'git reset {r}')
return True, diff_run.stdout.decode()
def run(base_commit, ignore_config, step: Optional[Step], report: Optional[Report]):
"""Apply clang-format and return if no issues were found."""
if report is None:
report = Report() # For debugging.
if step is None:
step = Step() # For debugging.
step.reproduce_commands.append(f'git-clang-format {base_commit}')
r, patch = get_diff(base_commit)
if not r:
step.success = False
return
add_artifact = False
patches = unidiff.PatchSet(patch)
ignore_lines = []
if ignore_config is not None and os.path.exists(ignore_config):
ignore_lines = open(ignore_config, 'r').readlines()
logging.debug("ignore patterns: " + str(';'.join(ignore_lines)))
ignore = pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern, ignore_lines)
patched_file: unidiff.PatchedFile
success = True
for patched_file in patches:
add_artifact = True
if ignore.match_file(patched_file.source_file) or ignore.match_file(patched_file.target_file):
logging.info(f'patch of {patched_file.patch_info} is ignored')
continue
hunk: unidiff.Hunk
for hunk in patched_file:
lines = [str(x) for x in hunk]
success = False
m = 10 # max number of lines to report.
description = 'please reformat the code\n```\n'
n = len(lines)
cut = n > m + 1
if cut:
lines = lines[:m]
description += ''.join(lines) + '\n```'
if cut:
description += f'\n{n - m} diff lines are omitted. See full path.'
report.add_lint({
'name': 'clang-format',
'severity': 'autofix',
'code': 'clang-format',
'path': patched_file.source_file,
'line': hunk.source_start,
'char': 1,
'description': description,
})
if add_artifact:
patch_file = 'clang-format.patch'
with open(patch_file, 'w') as f:
f.write(patch)
report.add_artifact(os.getcwd(), patch_file, 'clang-format')
if not success:
step.success = False
annotate(f'clang-format: Please format your changes with clang-format by running `git-clang-format HEAD^`'
f'or applying the attached patch.', style='error')
logging.debug(f'report: {report}')
logging.debug(f'step: {step}')
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Runs clang-format against given diff with given commit. '
'Produces patch and attaches linter comments to a review.')
parser.add_argument('--base', default='HEAD~1')
parser.add_argument('--ignore-config', default=None, help='path to file with patters of files to ignore')
parser.add_argument('--log-level', type=str, default='INFO')
args = parser.parse_args()
logging.basicConfig(level=args.log_level)
run(args.base, args.ignore_config, None)

118
scripts/clang_tidy_report.py
View file

@ -1,118 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse
import logging
import os
import re
import subprocess
from typing import Optional
import pathspec
import ignore_diff
from buildkite_utils import annotate
from phabtalk.phabtalk import Report, Step
def run(base_commit, ignore_config, step: Optional[Step], report: Optional[Report]):
"""Apply clang-tidy and return if no issues were found."""
if report is None:
report = Report() # For debugging.
if step is None:
step = Step() # For debugging.
r = subprocess.run(f'git diff -U0 --no-prefix {base_commit}', shell=True, capture_output=True)
logging.debug(f'git diff {r}')
diff = r.stdout.decode("utf-8", "replace")
if ignore_config is not None and os.path.exists(ignore_config):
ignore = pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern,
open(ignore_config, 'r').readlines())
diff = ignore_diff.remove_ignored(diff.splitlines(keepends=True), open(ignore_config, 'r'))
logging.debug(f'filtered diff: {diff}')
else:
ignore = pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern, [])
p = subprocess.Popen(['clang-tidy-diff', '-p0', '-quiet'], stdout=subprocess.PIPE, stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
step.reproduce_commands.append(f'git diff -U0 --no-prefix {base_commit} | clang-tidy-diff -p0')
a = ''.join(diff)
logging.info(f'clang-tidy input: {a}')
out = p.communicate(input=a.encode())[0].decode()
logging.debug(f'clang-tidy-diff {p}: {out}')
# Typical finding looks like:
# [cwd/]clang/include/clang/AST/DeclCXX.h:3058:20: error: ... [clang-diagnostic-error]
pattern = '^([^:]*):(\\d+):(\\d+): (.*): (.*)'
add_artifact = False
logging.debug("cwd", os.getcwd())
errors_count = 0
warn_count = 0
inline_comments = 0
for line in out.splitlines(keepends=False):
line = line.strip()
line = line.replace(os.getcwd() + os.sep, '')
logging.debug(line)
if len(line) == 0 or line == 'No relevant changes found.':
continue
add_artifact = True
match = re.search(pattern, line)
if match:
file_name = match.group(1)
line_pos = match.group(2)
char_pos = match.group(3)
severity = match.group(4)
text = match.group(5)
text += '\n[[{} | not useful]] '.format(
'https://github.com/google/llvm-premerge-checks/blob/main/docs/clang_tidy.md#warning-is-not-useful')
if severity in ['warning', 'error']:
if severity == 'warning':
warn_count += 1
if severity == 'error':
errors_count += 1
if ignore.match_file(file_name):
print('{} is ignored by pattern and no comment will be added'.format(file_name))
else:
inline_comments += 1
report.add_lint({
'name': 'clang-tidy',
'severity': 'warning',
'code': 'clang-tidy',
'path': file_name,
'line': int(line_pos),
'char': int(char_pos),
'description': '{}: {}'.format(severity, text),
})
else:
logging.debug('does not match pattern')
if add_artifact:
p = 'clang-tidy.txt'
with open(p, 'w') as f:
f.write(out)
report.add_artifact(os.getcwd(), p, 'clang-tidy')
if errors_count + warn_count != 0:
step.success = False
url = "https://github.com/google/llvm-premerge-checks/blob/main/docs/clang_tidy.md#review-comments."
annotate(f'clang-tidy found {errors_count} errors and {warn_count} warnings. {inline_comments} of them were '
f'added as review comments [why?]({url})', style='error')
logging.debug(f'report: {report}')
logging.debug(f'step: {step}')
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Runs clang-format against given diff with given commit. '
'Produces patch and attaches linter comments to a review.')
parser.add_argument('--base', default='HEAD~1')
parser.add_argument('--ignore-config', default=None, help='path to file with patters of files to ignore')
parser.add_argument('--log-level', type=str, default='INFO')
args = parser.parse_args()
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
run(args.base, args.ignore_config, None, None)

85
scripts/cleanup_branches.py
View file

@ -1,85 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""The script will delete old git branches."""
import argparse
import datetime
import functools
import git
import operator
import os
import re
import sys
from typing import List
def delete_old_branches(repo_path: str, max_age: datetime.datetime, branch_patterns: List[re.Pattern],
*, dry_run: bool = True, remote_name: str = 'origin') -> bool:
"""Deletes 'old' branches from a git repo.
This script assumes that $repo_path contains a current checkout of the repository ot be cleaned up.
:retrun True IFF branches could be deleted successfully.
"""
repo = git.Repo(repo_path)
remote = repo.remote(name=remote_name)
repo.git.fetch('--prune')
refs = remote.refs
print('Found {} branches at {} in total.'.format(len(refs), remote_name))
del_count = 0
fail_count = 0
if dry_run:
print('DRY RUN. NO BRANCHES WILL BE DELETED', flush=True)
print('Deleting: \n', flush=True)
for reference in refs:
committed_date = datetime.datetime.fromtimestamp(reference.commit.committed_date)
if committed_date < max_age and _has_pattern_match(reference.name, branch_patterns):
print(reference.name, flush=True)
if not dry_run:
try:
remote.push(refspec=':{}'.format(reference.remote_head))
del_count += 1
except git.GitCommandError as err:
print('ERROR: Failed to delete "{}": {}'.format(reference.name, err), flush=True)
fail_count += 1
print('Deleted {} branches.'.format(del_count))
if fail_count > 0:
print('Failed to delete {} branches.'.format(fail_count))
return fail_count == 0
def _has_pattern_match(name: str, patterns) -> bool:
"""Check if name matches any of the patterns"""
return functools.reduce(
operator.or_,
map(lambda r: r.search(name) is not None, patterns),
False)
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Clean a git repository')
parser.add_argument('repo_path', type=str, nargs='?', default=os.getcwd())
parser.add_argument('--days', type=int, default=30)
parser.add_argument('--pattern', action='append', type=str)
parser.add_argument('--dryrun', action='store_true')
args = parser.parse_args()
max_age = datetime.datetime.now() - datetime.timedelta(days=args.days)
branch_pattern = [re.compile(r) for r in args.pattern]
success = delete_old_branches(args.repo_path, max_age, branch_pattern, dry_run=args.dryrun)
if not success:
sys.exit(1)

24
scripts/command_utils.py
View file

@ -1,24 +0,0 @@
#!/usr/bin/env python3
# Copyright 2023 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import sys
def get_env_or_die(name: str):
v = os.environ.get(name)
if not v:
sys.stderr.write(f"Error: '{name}' environment variable is not set.\n")
exit(2)
return v

50
scripts/common.ps1
View file

@ -1,50 +0,0 @@
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# common header file for all .ps1 scripts
# stop script on errors
Set-StrictMode -Version Latest
$PSDefaultParameterValues['*:ErrorAction']='Stop'
# Invokes a Cmd.exe shell script and updates the environment.
# from: https://stackoverflow.com/questions/41399692/running-a-build-script-after-calling-vcvarsall-bat-from-powershell
function Invoke-CmdScript {
param(
[String] $scriptName
)
$cmdLine = """$scriptName"" $args & set"
& $Env:SystemRoot\system32\cmd.exe /c $cmdLine |
select-string '^([^=]*)=(.*)$' | foreach-object {
$varName = $_.Matches[0].Groups[1].Value
$varValue = $_.Matches[0].Groups[2].Value
set-item Env:$varName $varValue
}
}
# call an executable and check the error code
# from: https://stackoverflow.com/questions/9948517/how-to-stop-a-powershell-script-on-the-first-error
function Invoke-Call {
param (
[scriptblock]$ScriptBlock,
[string]$ErrorAction = "Stop"
)
& @ScriptBlock *>&1 | ForEach-Object { "$_" }
if (($lastexitcode -ne 0) -and $ErrorAction -eq "Stop") {
Write-Error "Command $ScriptBlock exited with $lastexitcode."
exit $lastexitcode
}
}

13
scripts/etc/run_in_container.sh
View file

@ -1,13 +0,0 @@
# Sample script that you can run in docker container to build vllm
#!/usr/bin/env bash
su buildkite-agent
cd /var/lib/buildkite-agent
git clone https://github.com/llvm-premerge-tests/llvm-project.git llvm-project
cd llvm-project
rm -rf build
mkdir build
cd build
export CC="clang"
export CXX="clang++"
export LD="LLD"
cmake ../llvm -D LLVM_ENABLE_PROJECTS="clang;mlir;lldb;llvm" -G Ninja -D CMAKE_BUILD_TYPE=Release -D LLVM_ENABLE_ASSERTIONS=ON -D LLVM_BUILD_EXAMPLES=ON -D LLVM_LIT_ARGS="-v --xunit-xml-output test-results.xml" -D LLVM_ENABLE_LLD=ON -D CMAKE_CXX_FLAGS=-gmlt -DBOLT_CLANG_EXE=/usr/bin/clang

15
scripts/etc/run_sample_linux_build.sh
View file

@ -1,15 +0,0 @@
#!/usr/bin/env bash
[ ! -d llvm-project ] && git clone https://github.com/llvm/llvm-project.git llvm-project
cd llvm-project
git fetch --all
git checkout ${1:-20ba079dda7be1a72d64cebc9f55d909bf29f6c1}
rm -rf build
mkdir build
cd build
export CC="clang"
export CXX="clang++"
export LD="LLD"
cmake ../llvm -D LLVM_ENABLE_PROJECTS="bolt;compiler-rt;libc;pstl;polly;llvm;libclc;clang;mlir;clang-tools-extra;flang;lld" -G Ninja -D CMAKE_BUILD_TYPE=Release -D LLVM_ENABLE_ASSERTIONS=ON -D LLVM_BUILD_EXAMPLES=ON -D LLVM_LIT_ARGS="-v --xunit-xml-output test-results.xml --resultdb-output resultdb.json" -D LLVM_ENABLE_LLD=ON -D CMAKE_CXX_FLAGS=-gmlt -DBOLT_CLANG_EXE=/usr/bin/clang
# ^note that compiler cache arguments are omitted
ln -s $PWD/compile_commands.json ../compile_commands.json
ninja check-all

64
scripts/exec_utils.py
View file

@ -1,64 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import asyncio
import logging
import os
from asyncio.subprocess import PIPE
from typing import Callable, AnyStr
async def read_stream_and_display(stream, display):
while True:
line = await stream.readline()
if not line:
break
display(line) # assume it doesn't block
async def read_and_display(write_stdout, write_stderr, *cmd, **kwargs):
logging.debug(f'subprocess called with {cmd}; {kwargs}')
process = await asyncio.create_subprocess_shell(*cmd, stdout=PIPE, stderr=PIPE, **kwargs)
try:
await asyncio.gather(
read_stream_and_display(process.stdout, write_stdout),
read_stream_and_display(process.stderr, write_stderr))
except Exception:
process.kill()
raise
finally:
return await process.wait()
def tee(s: AnyStr, write1: Callable[[AnyStr], None], write2: Callable[[AnyStr], None]):
write1(s)
write2(s)
def if_not_matches(s: AnyStr, regexp, write: Callable[[AnyStr], None]):
x = s
if isinstance(s, (bytes, bytearray)):
x = s.decode()
if regexp.match(x) is None:
write(s)
def watch_shell(write_stdout, write_stderr, *cmd, **kwargs):
if os.name == 'nt':
loop = asyncio.ProactorEventLoop() # Windows
asyncio.set_event_loop(loop)
else:
loop = asyncio.get_event_loop()
rc = loop.run_until_complete(read_and_display(write_stdout, write_stderr, *cmd, **kwargs))
return rc

58
scripts/git_utils.py
View file

@ -1,58 +0,0 @@
# Copyright 2022 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import git
import os
import logging
"""URL of upstream LLVM repository."""
LLVM_GITHUB_URL = 'ssh://git@github.com/llvm/llvm-project'
FORK_REMOTE_URL = 'ssh://git@github.com/llvm-premerge-tests/llvm-project'
def initLlvmFork(path: str) -> git.Repo:
if not os.path.isdir(path):
logging.info(f'{path} does not exist, cloning repository...')
git.Repo.clone_from(FORK_REMOTE_URL, path)
repo = git.Repo(path)
# Remove index lock just in case.
lock_file = f"{repo.working_tree_dir}/.git/index.lock"
try:
os.remove(lock_file)
logging.info(f"removed {lock_file}")
except FileNotFoundError:
logging.info(f"{lock_file} does not exist")
repo.remote('origin').set_url(FORK_REMOTE_URL)
if 'upstream' not in repo.remotes:
repo.create_remote('upstream', url=LLVM_GITHUB_URL)
else:
repo.remote('upstream').set_url(LLVM_GITHUB_URL)
repo.git.fetch('--all')
repo.git.clean('-ffxdq')
repo.git.reset('--hard')
repo.heads.main.checkout()
repo.git.pull('origin', 'main')
return repo
def syncLlvmFork(repo: git.Repo):
repo.remote('origin').set_url(FORK_REMOTE_URL)
if 'upstream' not in repo.remotes:
repo.create_remote('upstream', url=LLVM_GITHUB_URL)
repo.remote('upstream').set_url(LLVM_GITHUB_URL)
syncRemotes(repo, 'upstream', 'origin')
pass
def syncRemotes(repo: git.Repo, fromRemote, toRemote):
"""sync one remote from another"""
repo.remotes[fromRemote].fetch()
repo.git.push(toRemote, '-f', f'refs/remotes/{fromRemote}/*:refs/heads/*')

109
scripts/git_utils_test.py
View file

@ -1,109 +0,0 @@
# Copyright 2022 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import git
import os
import scripts.git_utils as git_utils
def assertForkIsSynced(upstreamPath, forkPath):
upstream = git.Repo(path=upstreamPath)
fork = git.Repo(path=forkPath)
forkBranches = {}
for b in fork.branches:
forkBranches[b.name] = b
for b in upstream.branches:
assert b.name in forkBranches
assert b.commit.hexsha == forkBranches[b.name].commit.hexsha, f'branch {b.name} head'
def forkIsSynced(upstreamPath, forkPath) -> bool:
upstream = git.Repo(path=upstreamPath)
fork = git.Repo(path=forkPath)
forkBranches = {}
for b in fork.branches:
forkBranches[b.name] = b
for b in upstream.branches:
if b.name not in forkBranches:
return False
if b.commit.hexsha != forkBranches[b.name].commit.hexsha:
return False
return True
def add_simple_commit(remote, name: str):
with open(os.path.join(remote.working_tree_dir, name), 'wt') as f:
f.write('first line\n')
remote.index.add([os.path.join(remote.working_tree_dir, name)])
remote.index.commit(name)
def test_sync_branches(tmp_path):
upstreamRemote = os.path.join(tmp_path, 'upstreamBare')
forkRemote = os.path.join(tmp_path, 'forkBare')
git.Repo.init(path=upstreamRemote, bare=True)
git.Repo.init(path=forkRemote, bare=True)
upstreamPath = os.path.join(tmp_path, 'upstream')
forkPath = os.path.join(tmp_path, 'fork')
upstream = git.Repo.clone_from(url=upstreamRemote, to_path=upstreamPath)
add_simple_commit(upstream, '1')
upstream.git.push('origin', 'main')
fork = git.Repo.clone_from(url=forkRemote, to_path=forkPath)
fork.create_remote('upstream', url=upstreamRemote)
git_utils.syncRemotes(fork, 'upstream', 'origin')
fork.remotes.upstream.fetch()
fork.create_head('main', fork.remotes.upstream.refs.main)
fork.heads.main.checkout()
# Sync init commit.
git_utils.syncRemotes(fork, 'upstream', 'origin')
assertForkIsSynced(upstreamRemote, forkRemote)
# Add new change upstream.
add_simple_commit(upstream, '2')
upstream.git.push('--all')
git_utils.syncRemotes(fork, 'upstream', 'origin')
assertForkIsSynced(upstreamRemote, forkRemote)
# Add new branch.
upstream.create_head('branch1')
upstream.heads['branch1'].checkout()
add_simple_commit(upstream, '3')
upstream.git.push('--all')
git_utils.syncRemotes(fork, 'upstream', 'origin')
assertForkIsSynced(upstreamRemote, forkRemote)
# Add another branch commit.
add_simple_commit(upstream, '4')
upstream.git.push('--all')
git_utils.syncRemotes(fork, 'upstream', 'origin')
assertForkIsSynced(upstreamRemote, forkRemote)
# Discard changes in fork.
fork.remotes.origin.pull()
fork.heads.main.checkout()
add_simple_commit(fork, '5')
fork.remotes.origin.push()
upstream.remotes.origin.pull('main')
upstream.heads.main.checkout()
add_simple_commit(upstream, '6')
upstream.remotes.origin.push()
assert not forkIsSynced(upstreamRemote, forkRemote)
assert os.path.isfile(os.path.join(fork.working_tree_dir, '5'))
git_utils.syncRemotes(fork, 'upstream', 'origin')
assertForkIsSynced(upstreamRemote, forkRemote)
fork.git.pull('origin', 'main')
fork.heads.main.checkout()
assert not os.path.isfile(os.path.join(fork.working_tree_dir, '5'))
assert os.path.isfile(os.path.join(fork.working_tree_dir, '6'))

49
scripts/ignore_diff.py
View file

@ -1,49 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse
import logging
import re
import sys
import pathspec
def remove_ignored(diff_lines, ignore_patterns_lines):
logging.debug(f'ignore pattern {ignore_patterns_lines}')
ignore = pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern, ignore_patterns_lines)
good = True
result = []
for line in diff_lines:
match = re.search(r'^diff --git (.*) (.*)$', line)
if match:
good = not (ignore.match_file(match.group(1)) and ignore.match_file(match.group(2)))
if not good:
logging.debug(f'skip {line.rstrip()}')
continue
result.append(line)
return result
if __name__ == "__main__":
# Maybe FIXME: Replace this tool usage with flags for tidy/format, use paths relative to `__file__`
parser = argparse.ArgumentParser(description='Takes an output of git diff and removes files ignored by patten '
'specified by ignore file')
parser.add_argument('ignore_config', default=None,
help='path to file with patters of files to ignore')
parser.add_argument('--log-level', type=str, default='WARNING')
args = parser.parse_args()
logging.basicConfig(level=args.log_level)
filtered = remove_ignored([x for x in sys.stdin], open(args.ignore_config, 'r').readlines())
for x in filtered:
sys.stdout.write(x)

90
scripts/llvm-dependencies.yaml
View file

@ -1,90 +0,0 @@
# This mapping is only used to determine which projects need to be rebuild.
# E.g. all builds are still in-tree, so 'llvm' will always be included in the
# built projects.
dependencies:
llvm: []
bolt:
- llvm
- lld
clang:
- llvm
clang-tools-extra:
- clang
- llvm
# FIXME: "compiler-rt" depends on "llvm" only for configuration, right?
# it means we can miss breakages in configuration changes.
# Same for libcxx, libc and other projects that don't have 'llvm'
# as a dependency.
compiler-rt:
- clang
flang:
- llvm
- mlir
- clang
libc:
- clang
- clang-tools-extra
libcxx:
- clang
- lldb
libcxxabi:
- clang
- lldb
libclc: []
lld:
- llvm
lldb:
- clang
- llvm
mlir:
- llvm
openmp:
- clang
polly:
- llvm
pstl: []
cross-project-tests:
- clang
- lld
# List of all projects in the LLVM monorepository. This list is taken from
# llvm/CMakeLists.txt in "set(LLVM_ALL_PROJECTS ..."
# The value for all project is the list of ninja targets to run.
allprojects:
bolt: ["check-bolt"]
clang: ["check-clang"]
clang-tools-extra: ["check-clang-tools"]
compiler-rt: ["check-all"] # check-compiler-rt seems to exist only in standalone builds.
cross-project-tests: ["check-cross-project"]
flang: ["check-flang"]
libc: ["check-libc"]
libclc: ["check-all"] # There does not seem to be a more specific target.
libcxx: ["check-cxx"]
libcxxabi: ["check-cxxabi"]
lld: ["check-lld"]
lldb: ["check-all"] # FIXME: `check-lldb` may not include every lldb tests?
mlir: ["check-mlir"]
openmp: ["check-openmp"]
polly: ["check-polly"]
pstl: ["check-all"] # There does not seem to be a more specific target.
llvm: ["check-llvm"]
# projects excluded from automatic configuration as they could not be built
excludedProjects:
windows:
- bolt # tests are not supported yet
- lldb # failing tests
- libcxx # no windows support
- libcxxabi # no windows support
- libc # no windows support
- openmp # TODO: check: kuhnel has trouble with the Perl installation
- cross-project-tests # test failing
- check-cxxabi
- compiler-rt # tests taking too long
linux:
- libcxx # has custom pipeline
- libcxxabi # has custom pipeline
- cross-project-tests # tests failing
- lldb # tests failing
- openmp # https://github.com/google/llvm-premerge-checks/issues/410

1
scripts/metrics/.gitignore vendored
View file

@ -1 +0,0 @@
tmp

21
scripts/metrics/Pipfile
View file

@ -1,21 +0,0 @@
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
backoff = "*"
GitPython = "*"
lxml = "*"
pathspec = "*"
phabricator = "==0.8.1"
pyaml = "*"
requests = "*"
python-benedict = "*"
psycopg2-binary = "*"
chardet = "*"
[dev-packages]
[requires]
python_version = "3.9"

386
scripts/metrics/Pipfile.lock generated
View file

@ -1,386 +0,0 @@
{
"_meta": {
"hash": {
"sha256": "e3acba0f267d3eda606997edc26c93b7ce252026303108a9f1ae7d354c616aed"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.9"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.python.org/simple",
"verify_ssl": true
}
]
},
"default": {
"backoff": {
"hashes": [
"sha256:5e73e2cbe780e1915a204799dba0a01896f45f4385e636bcca7a0614d879d0cd",
"sha256:b8fba021fac74055ac05eb7c7bfce4723aedde6cd0a504e5326bcb0bdd6d19a4"
],
"index": "pypi",
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==1.10.0"
},
"certifi": {
"hashes": [
"sha256:539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082",
"sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9"
],
"markers": "python_version >= '3.6'",
"version": "==2023.7.22"
},
"chardet": {
"hashes": [
"sha256:0d6f53a15db4120f2b08c94f11e7d93d2c911ee118b6b30a04ec3ee8310179fa",
"sha256:f864054d66fd9118f2e67044ac8981a54775ec5b67aed0441892edb553d21da5"
],
"index": "pypi",
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==4.0.0"
},
"ftfy": {
"hashes": [
"sha256:0ffd33fce16b54cccaec78d6ec73d95ad370e5df5a25255c8966a6147bd667ca",
"sha256:bfc2019f84fcd851419152320a6375604a0f1459c281b5b199b2cd0d2e727f8f"
],
"markers": "python_version >= '3.6'",
"version": "==6.1.1"
},
"gitdb": {
"hashes": [
"sha256:6eb990b69df4e15bad899ea868dc46572c3f75339735663b81de79b06f17eb9a",
"sha256:c286cf298426064079ed96a9e4a9d39e7f3e9bf15ba60701e95f5492f28415c7"
],
"markers": "python_version >= '3.7'",
"version": "==4.0.10"
},
"gitpython": {
"hashes": [
"sha256:5f4c4187de49616d710a77e98ddf17b4782060a1788df441846bddefbb89ab33",
"sha256:f9b9ddc0761c125d5780eab2d64be4873fc6817c2899cbcb34b02344bdc7bc54"
],
"markers": "python_version >= '3.7'",
"version": "==3.1.37"
},
"idna": {
"hashes": [
"sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6",
"sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
"version": "==2.10"
},
"lxml": {
"hashes": [
"sha256:04da965dfebb5dac2619cb90fcf93efdb35b3c6994fea58a157a834f2f94b318",
"sha256:0538747a9d7827ce3e16a8fdd201a99e661c7dee3c96c885d8ecba3c35d1032c",
"sha256:0645e934e940107e2fdbe7c5b6fb8ec6232444260752598bc4d09511bd056c0b",
"sha256:079b68f197c796e42aa80b1f739f058dcee796dc725cc9a1be0cdb08fc45b000",
"sha256:0f3f0059891d3254c7b5fb935330d6db38d6519ecd238ca4fce93c234b4a0f73",
"sha256:10d2017f9150248563bb579cd0d07c61c58da85c922b780060dcc9a3aa9f432d",
"sha256:1355755b62c28950f9ce123c7a41460ed9743c699905cbe664a5bcc5c9c7c7fb",
"sha256:13c90064b224e10c14dcdf8086688d3f0e612db53766e7478d7754703295c7c8",
"sha256:1423631e3d51008871299525b541413c9b6c6423593e89f9c4cfbe8460afc0a2",
"sha256:1436cf0063bba7888e43f1ba8d58824f085410ea2025befe81150aceb123e345",
"sha256:1a7c59c6ffd6ef5db362b798f350e24ab2cfa5700d53ac6681918f314a4d3b94",
"sha256:1e1cf47774373777936c5aabad489fef7b1c087dcd1f426b621fda9dcc12994e",
"sha256:206a51077773c6c5d2ce1991327cda719063a47adc02bd703c56a662cdb6c58b",
"sha256:21fb3d24ab430fc538a96e9fbb9b150029914805d551deeac7d7822f64631dfc",
"sha256:27e590352c76156f50f538dbcebd1925317a0f70540f7dc8c97d2931c595783a",
"sha256:287605bede6bd36e930577c5925fcea17cb30453d96a7b4c63c14a257118dbb9",
"sha256:2aaf6a0a6465d39b5ca69688fce82d20088c1838534982996ec46633dc7ad6cc",
"sha256:32a73c53783becdb7eaf75a2a1525ea8e49379fb7248c3eeefb9412123536387",
"sha256:41fb58868b816c202e8881fd0f179a4644ce6e7cbbb248ef0283a34b73ec73bb",
"sha256:4780677767dd52b99f0af1f123bc2c22873d30b474aa0e2fc3fe5e02217687c7",
"sha256:4878e667ebabe9b65e785ac8da4d48886fe81193a84bbe49f12acff8f7a383a4",
"sha256:487c8e61d7acc50b8be82bda8c8d21d20e133c3cbf41bd8ad7eb1aaeb3f07c97",
"sha256:4beea0f31491bc086991b97517b9683e5cfb369205dac0148ef685ac12a20a67",
"sha256:4cfbe42c686f33944e12f45a27d25a492cc0e43e1dc1da5d6a87cbcaf2e95627",
"sha256:4d5bae0a37af799207140652a700f21a85946f107a199bcb06720b13a4f1f0b7",
"sha256:4e285b5f2bf321fc0857b491b5028c5f276ec0c873b985d58d7748ece1d770dd",
"sha256:57e4d637258703d14171b54203fd6822fda218c6c2658a7d30816b10995f29f3",
"sha256:5974895115737a74a00b321e339b9c3f45c20275d226398ae79ac008d908bff7",
"sha256:5ef87fca280fb15342726bd5f980f6faf8b84a5287fcc2d4962ea8af88b35130",
"sha256:603a464c2e67d8a546ddaa206d98e3246e5db05594b97db844c2f0a1af37cf5b",
"sha256:6653071f4f9bac46fbc30f3c7838b0e9063ee335908c5d61fb7a4a86c8fd2036",
"sha256:6ca2264f341dd81e41f3fffecec6e446aa2121e0b8d026fb5130e02de1402785",
"sha256:6d279033bf614953c3fc4a0aa9ac33a21e8044ca72d4fa8b9273fe75359d5cca",
"sha256:6d949f53ad4fc7cf02c44d6678e7ff05ec5f5552b235b9e136bd52e9bf730b91",
"sha256:6daa662aba22ef3258934105be2dd9afa5bb45748f4f702a3b39a5bf53a1f4dc",
"sha256:6eafc048ea3f1b3c136c71a86db393be36b5b3d9c87b1c25204e7d397cee9536",
"sha256:830c88747dce8a3e7525defa68afd742b4580df6aa2fdd6f0855481e3994d391",
"sha256:86e92728ef3fc842c50a5cb1d5ba2bc66db7da08a7af53fb3da79e202d1b2cd3",
"sha256:8caf4d16b31961e964c62194ea3e26a0e9561cdf72eecb1781458b67ec83423d",
"sha256:8d1a92d8e90b286d491e5626af53afef2ba04da33e82e30744795c71880eaa21",
"sha256:8f0a4d179c9a941eb80c3a63cdb495e539e064f8054230844dcf2fcb812b71d3",
"sha256:9232b09f5efee6a495a99ae6824881940d6447debe272ea400c02e3b68aad85d",
"sha256:927a9dd016d6033bc12e0bf5dee1dde140235fc8d0d51099353c76081c03dc29",
"sha256:93e414e3206779ef41e5ff2448067213febf260ba747fc65389a3ddaa3fb8715",
"sha256:98cafc618614d72b02185ac583c6f7796202062c41d2eeecdf07820bad3295ed",
"sha256:9c3a88d20e4fe4a2a4a84bf439a5ac9c9aba400b85244c63a1ab7088f85d9d25",
"sha256:9f36de4cd0c262dd9927886cc2305aa3f2210db437aa4fed3fb4940b8bf4592c",
"sha256:a60f90bba4c37962cbf210f0188ecca87daafdf60271f4c6948606e4dabf8785",
"sha256:a614e4afed58c14254e67862456d212c4dcceebab2eaa44d627c2ca04bf86837",
"sha256:ae06c1e4bc60ee076292e582a7512f304abdf6c70db59b56745cca1684f875a4",
"sha256:b122a188cd292c4d2fcd78d04f863b789ef43aa129b233d7c9004de08693728b",
"sha256:b570da8cd0012f4af9fa76a5635cd31f707473e65a5a335b186069d5c7121ff2",
"sha256:bcaa1c495ce623966d9fc8a187da80082334236a2a1c7e141763ffaf7a405067",
"sha256:bd34f6d1810d9354dc7e35158aa6cc33456be7706df4420819af6ed966e85448",
"sha256:be9eb06489bc975c38706902cbc6888f39e946b81383abc2838d186f0e8b6a9d",
"sha256:c4b2e0559b68455c085fb0f6178e9752c4be3bba104d6e881eb5573b399d1eb2",
"sha256:c62e8dd9754b7debda0c5ba59d34509c4688f853588d75b53c3791983faa96fc",
"sha256:c852b1530083a620cb0de5f3cd6826f19862bafeaf77586f1aef326e49d95f0c",
"sha256:d9fc0bf3ff86c17348dfc5d322f627d78273eba545db865c3cd14b3f19e57fa5",
"sha256:dad7b164905d3e534883281c050180afcf1e230c3d4a54e8038aa5cfcf312b84",
"sha256:e5f66bdf0976ec667fc4594d2812a00b07ed14d1b44259d19a41ae3fff99f2b8",
"sha256:e8f0c9d65da595cfe91713bc1222af9ecabd37971762cb830dea2fc3b3bb2acf",
"sha256:edffbe3c510d8f4bf8640e02ca019e48a9b72357318383ca60e3330c23aaffc7",
"sha256:eea5d6443b093e1545ad0210e6cf27f920482bfcf5c77cdc8596aec73523bb7e",
"sha256:ef72013e20dd5ba86a8ae1aed7f56f31d3374189aa8b433e7b12ad182c0d2dfb",
"sha256:f05251bbc2145349b8d0b77c0d4e5f3b228418807b1ee27cefb11f69ed3d233b",
"sha256:f1be258c4d3dc609e654a1dc59d37b17d7fef05df912c01fc2e15eb43a9735f3",
"sha256:f9ced82717c7ec65a67667bb05865ffe38af0e835cdd78728f1209c8fffe0cad",
"sha256:fe17d10b97fdf58155f858606bddb4e037b805a60ae023c009f760d8361a4eb8",
"sha256:fe749b052bb7233fe5d072fcb549221a8cb1a16725c47c37e42b0b9cb3ff2c3f"
],
"index": "pypi",
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==4.9.1"
},
"mailchecker": {
"hashes": [
"sha256:8b083907da42daa448d93eb19f0c978dde2e8565617a5e76271b35a5f7285ccd"
],
"version": "==5.0.9"
},
"pathspec": {
"hashes": [
"sha256:86379d6b86d75816baba717e64b1a3a3469deb93bb76d613c9ce79edc5cb68fd",
"sha256:aa0cb481c4041bf52ffa7b0d8fa6cd3e88a2ca4879c533c9153882ee2556790d"
],
"index": "pypi",
"version": "==0.8.1"
},
"phabricator": {
"hashes": [
"sha256:a276fb41eb91b550c46612aba874160807664ff7fe430adaa0ffa319249df981",
"sha256:fabf0f63f020256bb44bb9ea0eba82a1d9493b2c2c5cf7a37fcdab5d4f9404e2"
],
"index": "pypi",
"version": "==0.8.1"
},
"phonenumbers": {
"hashes": [
"sha256:34d6cb279dd4a64714e324c71350f96e5bda3237be28d11b4c555c44701544cd",
"sha256:869e44fcaaf276eca6b953a401e2b27d57461f3a18a66cf5f13377e7bb0e228c"
],
"version": "==8.13.23"
},
"psycopg2-binary": {
"hashes": [
"sha256:0deac2af1a587ae12836aa07970f5cb91964f05a7c6cdb69d8425ff4c15d4e2c",
"sha256:0e4dc3d5996760104746e6cfcdb519d9d2cd27c738296525d5867ea695774e67",
"sha256:11b9c0ebce097180129e422379b824ae21c8f2a6596b159c7659e2e5a00e1aa0",
"sha256:15978a1fbd225583dd8cdaf37e67ccc278b5abecb4caf6b2d6b8e2b948e953f6",
"sha256:1fabed9ea2acc4efe4671b92c669a213db744d2af8a9fc5d69a8e9bc14b7a9db",
"sha256:2dac98e85565d5688e8ab7bdea5446674a83a3945a8f416ad0110018d1501b94",
"sha256:42ec1035841b389e8cc3692277a0bd81cdfe0b65d575a2c8862cec7a80e62e52",
"sha256:6422f2ff0919fd720195f64ffd8f924c1395d30f9a495f31e2392c2efafb5056",
"sha256:6a32f3a4cb2f6e1a0b15215f448e8ce2da192fd4ff35084d80d5e39da683e79b",
"sha256:7312e931b90fe14f925729cde58022f5d034241918a5c4f9797cac62f6b3a9dd",
"sha256:7d92a09b788cbb1aec325af5fcba9fed7203897bbd9269d5691bb1e3bce29550",
"sha256:833709a5c66ca52f1d21d41865a637223b368c0ee76ea54ca5bad6f2526c7679",
"sha256:89705f45ce07b2dfa806ee84439ec67c5d9a0ef20154e0e475e2b2ed392a5b83",
"sha256:8cd0fb36c7412996859cb4606a35969dd01f4ea34d9812a141cd920c3b18be77",
"sha256:950bc22bb56ee6ff142a2cb9ee980b571dd0912b0334aa3fe0fe3788d860bea2",
"sha256:a0c50db33c32594305b0ef9abc0cb7db13de7621d2cadf8392a1d9b3c437ef77",
"sha256:a0eb43a07386c3f1f1ebb4dc7aafb13f67188eab896e7397aa1ee95a9c884eb2",
"sha256:aaa4213c862f0ef00022751161df35804127b78adf4a2755b9f991a507e425fd",
"sha256:ac0c682111fbf404525dfc0f18a8b5f11be52657d4f96e9fcb75daf4f3984859",
"sha256:ad20d2eb875aaa1ea6d0f2916949f5c08a19c74d05b16ce6ebf6d24f2c9f75d1",
"sha256:b4afc542c0ac0db720cf516dd20c0846f71c248d2b3d21013aa0d4ef9c71ca25",
"sha256:b8a3715b3c4e604bcc94c90a825cd7f5635417453b253499664f784fc4da0152",
"sha256:ba28584e6bca48c59eecbf7efb1576ca214b47f05194646b081717fa628dfddf",
"sha256:ba381aec3a5dc29634f20692349d73f2d21f17653bda1decf0b52b11d694541f",
"sha256:bd1be66dde2b82f80afb9459fc618216753f67109b859a361cf7def5c7968729",
"sha256:c2507d796fca339c8fb03216364cca68d87e037c1f774977c8fc377627d01c71",
"sha256:cec7e622ebc545dbb4564e483dd20e4e404da17ae07e06f3e780b2dacd5cee66",
"sha256:d14b140a4439d816e3b1229a4a525df917d6ea22a0771a2a78332273fd9528a4",
"sha256:d1b4ab59e02d9008efe10ceabd0b31e79519da6fb67f7d8e8977118832d0f449",
"sha256:d5227b229005a696cc67676e24c214740efd90b148de5733419ac9aaba3773da",
"sha256:e1f57aa70d3f7cc6947fd88636a481638263ba04a742b4a37dd25c373e41491a",
"sha256:e74a55f6bad0e7d3968399deb50f61f4db1926acf4a6d83beaaa7df986f48b1c",
"sha256:e82aba2188b9ba309fd8e271702bd0d0fc9148ae3150532bbb474f4590039ffb",
"sha256:ee69dad2c7155756ad114c02db06002f4cded41132cc51378e57aad79cc8e4f4",
"sha256:f5ab93a2cb2d8338b1674be43b442a7f544a0971da062a5da774ed40587f18f5"
],
"index": "pypi",
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
"version": "==2.8.6"
},
"pyaml": {
"hashes": [
"sha256:29a5c2a68660a799103d6949167bd6c7953d031449d08802386372de1db6ad71",
"sha256:67081749a82b72c45e5f7f812ee3a14a03b3f5c25ff36ec3b290514f8c4c4b99"
],
"index": "pypi",
"version": "==20.4.0"
},
"python-benedict": {
"hashes": [
"sha256:1aa65ea78bd0bfd269e9289edb05d6f4e82ce62669ad0604a27d80db00a1a575",
"sha256:af30f2a93aa15c0136c13bb528ded18b0e23171e8c567e968ee522a2b1cfa3b9"
],
"index": "pypi",
"version": "==0.24.0"
},
"python-dateutil": {
"hashes": [
"sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86",
"sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
"version": "==2.8.2"
},
"python-fsutil": {
"hashes": [
"sha256:5f02ed7764cac27633b4aaaa9cf56af2245a64a6cc048866b6f02792a8c5bf86",
"sha256:e0421440b5763d08129dcc469364105f3b434fc9736c3bc6ed181b1c623148ee"
],
"version": "==0.10.0"
},
"python-slugify": {
"hashes": [
"sha256:70ca6ea68fe63ecc8fa4fcf00ae651fc8a5d02d93dcd12ae6d4fc7ca46c4d395",
"sha256:ce0d46ddb668b3be82f4ed5e503dbc33dd815d83e2eb6824211310d3fb172a27"
],
"markers": "python_version >= '3.7'",
"version": "==8.0.1"
},
"pyyaml": {
"hashes": [
"sha256:04ac92ad1925b2cff1db0cfebffb6ffc43457495c9b3c39d3fcae417d7125dc5",
"sha256:062582fca9fabdd2c8b54a3ef1c978d786e0f6b3a1510e0ac93ef59e0ddae2bc",
"sha256:0d3304d8c0adc42be59c5f8a4d9e3d7379e6955ad754aa9d6ab7a398b59dd1df",
"sha256:1635fd110e8d85d55237ab316b5b011de701ea0f29d07611174a1b42f1444741",
"sha256:184c5108a2aca3c5b3d3bf9395d50893a7ab82a38004c8f61c258d4428e80206",
"sha256:18aeb1bf9a78867dc38b259769503436b7c72f7a1f1f4c93ff9a17de54319b27",
"sha256:1d4c7e777c441b20e32f52bd377e0c409713e8bb1386e1099c2415f26e479595",
"sha256:1e2722cc9fbb45d9b87631ac70924c11d3a401b2d7f410cc0e3bbf249f2dca62",
"sha256:1fe35611261b29bd1de0070f0b2f47cb6ff71fa6595c077e42bd0c419fa27b98",
"sha256:28c119d996beec18c05208a8bd78cbe4007878c6dd15091efb73a30e90539696",
"sha256:326c013efe8048858a6d312ddd31d56e468118ad4cdeda36c719bf5bb6192290",
"sha256:40df9b996c2b73138957fe23a16a4f0ba614f4c0efce1e9406a184b6d07fa3a9",
"sha256:42f8152b8dbc4fe7d96729ec2b99c7097d656dc1213a3229ca5383f973a5ed6d",
"sha256:49a183be227561de579b4a36efbb21b3eab9651dd81b1858589f796549873dd6",
"sha256:4fb147e7a67ef577a588a0e2c17b6db51dda102c71de36f8549b6816a96e1867",
"sha256:50550eb667afee136e9a77d6dc71ae76a44df8b3e51e41b77f6de2932bfe0f47",
"sha256:510c9deebc5c0225e8c96813043e62b680ba2f9c50a08d3724c7f28a747d1486",
"sha256:5773183b6446b2c99bb77e77595dd486303b4faab2b086e7b17bc6bef28865f6",
"sha256:596106435fa6ad000c2991a98fa58eeb8656ef2325d7e158344fb33864ed87e3",
"sha256:6965a7bc3cf88e5a1c3bd2e0b5c22f8d677dc88a455344035f03399034eb3007",
"sha256:69b023b2b4daa7548bcfbd4aa3da05b3a74b772db9e23b982788168117739938",
"sha256:6c22bec3fbe2524cde73d7ada88f6566758a8f7227bfbf93a408a9d86bcc12a0",
"sha256:704219a11b772aea0d8ecd7058d0082713c3562b4e271b849ad7dc4a5c90c13c",
"sha256:7e07cbde391ba96ab58e532ff4803f79c4129397514e1413a7dc761ccd755735",
"sha256:81e0b275a9ecc9c0c0c07b4b90ba548307583c125f54d5b6946cfee6360c733d",
"sha256:855fb52b0dc35af121542a76b9a84f8d1cd886ea97c84703eaa6d88e37a2ad28",
"sha256:8d4e9c88387b0f5c7d5f281e55304de64cf7f9c0021a3525bd3b1c542da3b0e4",
"sha256:9046c58c4395dff28dd494285c82ba00b546adfc7ef001486fbf0324bc174fba",
"sha256:9eb6caa9a297fc2c2fb8862bc5370d0303ddba53ba97e71f08023b6cd73d16a8",
"sha256:a0cd17c15d3bb3fa06978b4e8958dcdc6e0174ccea823003a106c7d4d7899ac5",
"sha256:afd7e57eddb1a54f0f1a974bc4391af8bcce0b444685d936840f125cf046d5bd",
"sha256:b1275ad35a5d18c62a7220633c913e1b42d44b46ee12554e5fd39c70a243d6a3",
"sha256:b786eecbdf8499b9ca1d697215862083bd6d2a99965554781d0d8d1ad31e13a0",
"sha256:ba336e390cd8e4d1739f42dfe9bb83a3cc2e80f567d8805e11b46f4a943f5515",
"sha256:baa90d3f661d43131ca170712d903e6295d1f7a0f595074f151c0aed377c9b9c",
"sha256:bc1bf2925a1ecd43da378f4db9e4f799775d6367bdb94671027b73b393a7c42c",
"sha256:bd4af7373a854424dabd882decdc5579653d7868b8fb26dc7d0e99f823aa5924",
"sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34",
"sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43",
"sha256:c8098ddcc2a85b61647b2590f825f3db38891662cfc2fc776415143f599bb859",
"sha256:d2b04aac4d386b172d5b9692e2d2da8de7bfb6c387fa4f801fbf6fb2e6ba4673",
"sha256:d483d2cdf104e7c9fa60c544d92981f12ad66a457afae824d146093b8c294c54",
"sha256:d858aa552c999bc8a8d57426ed01e40bef403cd8ccdd0fc5f6f04a00414cac2a",
"sha256:e7d73685e87afe9f3b36c799222440d6cf362062f78be1013661b00c5c6f678b",
"sha256:f003ed9ad21d6a4713f0a9b5a7a0a79e08dd0f221aff4525a2be4c346ee60aab",
"sha256:f22ac1c3cac4dbc50079e965eba2c1058622631e526bd9afd45fedd49ba781fa",
"sha256:faca3bdcf85b2fc05d06ff3fbc1f83e1391b3e724afa3feba7d13eeab355484c",
"sha256:fca0e3a251908a499833aa292323f32437106001d436eca0e6e7833256674585",
"sha256:fd1592b3fdf65fff2ad0004b5e363300ef59ced41c2e6b3a99d4089fa8c5435d",
"sha256:fd66fc5d0da6d9815ba2cebeb4205f95818ff4b79c3ebe268e75d961704af52f"
],
"markers": "python_version >= '3.6'",
"version": "==6.0.1"
},
"requests": {
"hashes": [
"sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804",
"sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e"
],
"index": "pypi",
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==2.25.1"
},
"six": {
"hashes": [
"sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
"sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
"version": "==1.16.0"
},
"smmap": {
"hashes": [
"sha256:dceeb6c0028fdb6734471eb07c0cd2aae706ccaecab45965ee83f11c8d3b1f62",
"sha256:e6d8668fa5f93e706934a62d7b4db19c8d9eb8cf2adbb75ef1b675aa332b69da"
],
"markers": "python_version >= '3.7'",
"version": "==5.0.1"
},
"text-unidecode": {
"hashes": [
"sha256:1311f10e8b895935241623731c2ba64f4c455287888b18189350b67134a822e8",
"sha256:bad6603bb14d279193107714b288be206cac565dfa49aa5b105294dd5c4aab93"
],
"version": "==1.3"
},
"toml": {
"hashes": [
"sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b",
"sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"
],
"markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
"version": "==0.10.2"
},
"urllib3": {
"hashes": [
"sha256:34b97092d7e0a3a8cf7cd10e386f401b3737364026c45e622aa02903dffe0f07",
"sha256:f8ecc1bba5667413457c529ab955bf8c67b45db799d159066261719e328580a0"
],
"index": "pypi",
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'",
"version": "==1.26.18"
},
"wcwidth": {
"hashes": [
"sha256:77f719e01648ed600dfa5402c347481c0992263b81a027344f3e1ba25493a704",
"sha256:8705c569999ffbb4f6a87c6d1b80f324bd6db952f5eb0b95bc07517f4c1813d4"
],
"version": "==0.2.8"
},
"xmltodict": {
"hashes": [
"sha256:341595a488e3e01a85a9d8911d8912fd922ede5fecc4dce437eb4b6c8d037e56",
"sha256:aa89e8fd76320154a40d19a0df04a4695fb9dc5ba977cbb68ab3e4eb225e7852"
],
"markers": "python_version >= '3.4'",
"version": "==0.13.0"
}
},
"develop": {}
}

83
scripts/metrics/README.md
View file

@ -1,83 +0,0 @@
# Metrics
To measure the impact and usefulness of the pre-merge checks, we want to collect
a set of metrics. This doc will summarize the metrics and tools. All of the data
shall be collected as time series, so that we can see changes over time.
* Impact - The metrics we ultimately want to improve
* Percentage of [build-bot build](http://lab.llvm.org:8011/) on main
failing. (Buildbot_percentage_failing)
* Time to fix a broken main build: Time between start of failing builds
until the build is fixed. (BuildBot_time_to_fix)
* Percentage of Revisions on Phabricator where a broken build was fixed
afterwards. This would indicate that a bug was found and fixed during
the code review phase. (Premerge_fixes)
* Number of reverts on main. This indicates that something was broken on
main that slipped through the pre-merge tests or was submitted without
any review. (Upstream_reverts)
* Users and behavior - Interesting to see and useful to adapt our approach.
* Percentage of commits to main that went through Phabricator.
* Number of participants in pre-merge tests.
* Percentage of Revisions with pre-merge tests executed
* Number of 30-day active committers on main and Phabricator.
* Builds - See how the infrastructure is doing.
* Time between upload of diff until build results available.
* Percentage of Revisions with successful/failed tests
* Number of pre-merge builds/day.
* Build queuing time.
* Individual times for `cmake`, `ninja all`, `ninja check-all` per
OS/architecture.
* Result storage size.
* Percentage of builds failing.
# Requirements
* Must:
* Do not collect/store personal data.
* Should:
* Minimize the amount of additional tools/scripts we need to maintain.
* Collect all metrics in a central location for easy evaluation (e.g.
database, CSV files).
* Nice to have:
* As the data is from an open source project and available anyway, give
public access to the metrics (numbers and charts).
* Send out alerts/notifications.
* Show live data in charts.
# Data sources
This section will explain where we can get the data from.
* build bot statistics
# Solution
We need to find solutions for these parts:
* Collect the data (regularly).
* Store the time series somewhere.
* Create & display charts.
Some ideas for this:
* bunch of scripts:
* Run a bunch of scripts manually to generate the metrics every now and
then. Phabricator already has a database and most entries there have
timestamps. So we could also reconstruct the history from that.
* TODO: Figure out if we can collect the most important metrics this way.
This requires that we can reconstruct historic values from the current
logs/git/database/... entries.
* Jenkins + CSV + Sheets:
* collect data with jenkins
* store numbers as CSV in this repo
* Charts are created manually on Google Sheets
* do it yourself:
* Collect data with Jenkins jobs
* Store the data on Prometheus
* Visualize with Grafana
* host all tools ourselves
* Stackdriver on GCP:
* TODO: figure out if we can get all the required data into Stackdriver
* Jupyter notebooks:
* TODO: figure out how that works

17
scripts/metrics/SQL_queries/buildbot_failing_builders.pgsql
View file

@ -1,17 +0,0 @@
/* list builders with success rate <70% over the last 7 days
these are probably worth investigating
*/
Select *,
format('https://lab.llvm.org/buildbot/#/builders/%s', builder_id) as link
FROM (
SELECT
builder_id,
count(*) as num_builds,
(100.0*count(CASE WHEN buildbot_overview.result='build successful' THEN 1 END)/count(*)) as success_prct
FROM buildbot_overview
WHERE completed_at > current_date - interval '7' day
GROUP BY builder_id
) as builder_success
WHERE success_prct < 70

9
scripts/metrics/SQL_queries/buildbot_per_commit.pgsql
View file

@ -1,9 +0,0 @@
/* list last 100 git commits with number of builds and success rate */
SELECT git_commits.hash as revision,
count(*) as num_builds,
(100.0*count(CASE WHEN buildbot_overview.result='build successful' THEN 1 END)/count(*)) as success_prct
FROM buildbot_overview, git_commits
WHERE buildbot_overview.revision = git_commits.hash
GROUP BY git_commits.hash
ORDER BY git_commits.commit_time DesC
LIMIT 100

7
scripts/metrics/SQL_queries/buildbot_success_daily.pgsql
View file

@ -1,7 +0,0 @@
/* list number of builds and percentage of successful builds per day */
SELECT to_date(cast(git_commits.commit_time as TEXT),'YYYY-MM-DD') as date,
count(*) as num_builds,
(100.0*count(CASE WHEN buildbot_overview.result='build successful' THEN 1 END)/count(*)) as success_prct
FROM buildbot_overview, git_commits
WHERE buildbot_overview.revision = git_commits.hash
GROUP BY date ORDER BY date ASC;

18
scripts/metrics/SQL_queries/buildbot_views.pgsql
View file

@ -1,18 +0,0 @@
/* Aggregated build information from multiple tables
also extracts additional columns from json data to make access easier. */
CREATE OR REPLACE VIEW buildbot_overview AS
SELECT
buildbot_buildsets.data -> 'sourcestamps' -> 0 ->> 'revision' AS revision,
buildbot_builds.build_id,
buildbot_builds.builder_id,
buildbot_builds.build_number,
buildbot_builds.build_data ->>'state_string' AS result,
format('https://lab.llvm.org/buildbot/#/builders/%s/builds/%s', buildbot_builds.builder_id, buildbot_builds.build_number) as link,
to_timestamp(CAST(build_data ->> 'complete_at' as int))::date as completed_at,
to_timestamp(CAST(build_data ->> 'started_at' as int))::date as started_at
FROM buildbot_buildsets, buildbot_buildrequests, buildbot_builds
WHERE buildbot_buildrequests.buildset_id = buildbot_buildsets.buildset_id AND
CAST(buildbot_builds.build_data ->> 'buildrequestid' AS int) = buildbot_buildrequests.buildrequest_id;

7
scripts/metrics/SQL_queries/server_statistics.pgsql
View file

@ -1,7 +0,0 @@
/* get server uptime statistics in percent */
SELECT DATE(timestamp) as date,
count(*) as measurements,
(100.0*count(CASE WHEN phabricator THEN 1 END)/count(*)) as phabricator_up_prcnt,
(100.0*count(CASE WHEN buildbot THEN 1 END)/count(*)) as buildbot_up_prcnt
FROM server_status GROUP BY date ORDER BY date ASC;

323
scripts/metrics/analyze_jobs.ipynb
View file

@ -1,323 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## License\n",
"\n",
"Copyright 2020 Google LLC\n",
"\n",
"Licensed under the the Apache License v2.0 with LLVM Exceptions (the \"License\");\n",
"you may not use this file except in compliance with the License.\n",
"You may obtain a copy of the License at\n",
"\n",
"https://llvm.org/LICENSE.txt\n",
"\n",
"Unless required by applicable law or agreed to in writing, software\n",
"distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"See the License for the specific language governing permissions and\n",
"limitations under the License."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"import os\n",
"import logging\n",
"import argparse\n",
"import json\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cache_file = \"cache.json\"\n",
"token = f'Bearer {os.getenv(\"BUILDKITE_API_TOKEN\")}'\n",
"builds = []"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"if os.path.exists(cache_file):\n",
" with open(cache_file) as f:\n",
" builds = json.load(f)\n",
" print(f'loaded {len(builds)} entries') "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"# load new jobs from Buildkite API\n",
"max_pages = 2000\n",
"if True:\n",
" existing = set()\n",
" for j in builds:\n",
" existing.add(j['id'])\n",
"\n",
" # uncomment to reset\n",
"# builds = []\n",
"# existing = set()\n",
" page = 1\n",
" stop = False\n",
" while page <= max_pages:\n",
" print('loading page', page)\n",
" re = requests.get('https://api.buildkite.com/v2/organizations/llvm-project/builds',\n",
" params={'page': page},\n",
" headers={'Authorization': token})\n",
" if re.status_code != 200:\n",
" print('response status', re.status_code, re)\n",
" break\n",
" x = re.json()\n",
" if x == []:\n",
" print('empty response')\n",
" break\n",
" for j in x:\n",
" if j['id'] in existing:\n",
"# print('found existing job', j['id'])\n",
" # load some more pages as some of them might be running before and wasn't added. \n",
" max_pages = min(page + 5, max_pages)\n",
" else:\n",
" # skip running jobs \n",
" if (j['state'] == 'running') or (j['state'] == 'scheduled'):\n",
" print(j['web_url'], 'is', j['state'], ', skipping')\n",
" continue\n",
" builds.append(j)\n",
" page += 1\n",
" print(len(builds), 'jobs in total') \n",
" with open(cache_file, 'w') as f:\n",
" json.dump(builds, f)\n",
" print(f'saved {len(builds)} entries')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"d = {\n",
" 'id': [],\n",
" 'number': [],\n",
" 'pipeline': [],\n",
"}\n",
"\n",
"jobs = {\n",
" 'pipeline': [],\n",
" 'name': [],\n",
" 'step_key': [],\n",
" 'state': [],\n",
" 'exit_status': [],\n",
" 'agent_id': [],\n",
" \n",
" 'agent_name': [],\n",
" 'runnable_at': [],\n",
" 'started_at': [],\n",
" 'wait_duration': [],\n",
" 'finished_at': [],\n",
" 'run_duration': [],\n",
"}\n",
"\n",
"sec = np.timedelta64(1, 's')\n",
"\n",
"for b in builds:\n",
" env = b['env']\n",
"# if 'ph_windows_agents' not in env:\n",
"# continue\n",
"# if 'scripts_branch' not in env:\n",
"# continue\n",
" d['id'].append(b['id'])\n",
" d['number'].append(b['number'])\n",
" d['pipeline'].append(b['pipeline']['slug'])\n",
" for x in b['jobs']:\n",
" if x['state'] in ['waiting_failed', 'canceled', 'skipped', 'broken']:\n",
" continue\n",
" try:\n",
" jobs['pipeline'].append(b['pipeline']['slug'])\n",
" jobs['name'].append(x['name'])\n",
" jobs['step_key'].append(x['step_key'] if 'step_key' in x else '')\n",
" jobs['state'].append(x['state'] )\n",
" jobs['exit_status'].append(x['exit_status'] if 'exit_status' in x else -1)\n",
" jobs['agent_id'].append(x['agent']['id'] if 'agent' in x else '')\n",
" jobs['agent_name'].append(x['agent']['name'] if 'agent' in x else '')\n",
" runnable = np.datetime64(x['runnable_at'].replace('Z', ''))\n",
" started = np.datetime64(x['started_at'].replace('Z', ''))\n",
" finished = np.datetime64(x['finished_at'].replace('Z', ''))\n",
" jobs['runnable_at'].append(runnable)\n",
" jobs['started_at'].append(started)\n",
" jobs['wait_duration'].append((started - runnable) / sec)\n",
" jobs['finished_at'].append(finished)\n",
" jobs['run_duration'].append((finished - started) / sec)\n",
" except Exception as e:\n",
" print(x)\n",
" raise e \n",
"jobs = pd.DataFrame(jobs)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"jobs.pipeline.unique()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"jobs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ds = jobs[jobs['pipeline'] == 'llvm-main-build'][jobs['step_key'] == 'windows'][jobs['state']=='passed'][~jobs['agent_name'].str.startswith('buildkite-')][jobs['started_at'] > np.datetime64('2020-01-22')]\n",
"ds = jobs[jobs['pipeline'] == 'llvm-main-build'][jobs['step_key'] == 'windows'][jobs['state']=='passed'][~jobs['agent_name'].str.startswith('buildkite-')][jobs['started_at'] > np.datetime64('2020-01-22')]\n",
"ds = ds.drop_duplicates()\n",
"# remove one slowest run (repo checkout)\n",
"# t = ds.loc[ds.groupby([\"agent_name\"])[\"run_duration\"].idxmax()]\n",
"# ds = pd.concat([ds, t]).drop_duplicates(keep=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"fig, ax = plt.subplots(figsize=(10,10)) # size of the plot (width, height)\n",
"\n",
"sns.swarmplot(\n",
" ax=ax,\n",
" x='run_duration',\n",
" y='agent_name',\n",
" split=True,\n",
" data=ds)\n",
"\n",
"plt.xlim(0, None)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"t = pd.pivot_table(ds, values=['run_duration'], index=['agent_name'],\n",
" aggfunc=[np.median, np.mean, np.std, np.count_nonzero])\n",
"t = t.reindex(t.sort_values(by=('median', 'run_duration'), ascending=False).index)\n",
"t"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"// TODO: try with w64 w/o AV (temporary disabled)\n",
"w1 - 16 - non ssd 100 + ssd\n",
"w1a - 16 - non ssd 100 + ssd - AV (temp)\n",
"w4a - 16 - non ssd 100 + ssd - AV\n",
"w16-1 - 16 - non sdd 200 + ssd\n",
"w32-1 - 32 - non ssd 100 + ssd\n",
"w32-2 - 32 - non ssd 100 + ssd - AV\n",
"w32-2d - 32 - non ssd 200 + ssd\n",
"w64 - 64 - ssd 200 + ssd \n",
"w64a - 64 - ssd 200 + ssd - AV (temp)\n",
"wh63-3 - 64 low ram - non ssd 100 + ssd - AV\n",
"\n",
"buildkite-windows-16n1-exp\n",
"gcloud beta compute instances create \"${NAME}\" \\\n",
" --project=\"${GCP_PROJECT}\" \\\n",
" --zone=\"${GCP_ZONE}\" \\\n",
" --machine-type=n1-standard-16 \\\n",
" --local-ssd=device-name=local-ssd-0 \\\n",
" --image=windows-server-2019-dc-for-containers-v20200714 \\\n",
" --image-project=windows-cloud \\\n",
" --boot-disk-size=200GB --boot-disk-type=pd-ssd\n",
"\n",
"buildkite-32-experimental-1\n",
"gcloud beta compute instances create \"${NAME}\" \\\n",
" --project=\"${GCP_PROJECT}\" \\\n",
" --zone=\"${GCP_ZONE}\" \\\n",
" --machine-type=n1-standard-32 \\\n",
" --local-ssd=device-name=local-ssd-0 \\\n",
" --image=windows-server-2019-dc-for-containers-v20200714 \\\n",
" --image-project=windows-cloud \\\n",
" --boot-disk-size=200GB --boot-disk-type=pd-ssd\n",
"\n",
" \n",
"buildkite-16c2-exp\n",
"gcloud beta compute instances create \"${NAME}\" \\\n",
" --project=\"${GCP_PROJECT}\" \\\n",
" --zone=\"${GCP_ZONE}\" \\\n",
" --machine-type=c2-standard-16 \\\n",
" --local-ssd=device-name=local-ssd-0 \\\n",
" --local-ssd=interface=SCSI \\\n",
" --image=windows-server-2019-dc-for-containers-v20200714 \\\n",
" --image-project=windows-cloud \\\n",
" --boot-disk-size=200GB --boot-disk-type=pd-ssd\n",
" \n",
"...a: as is D:/ is a workdir (ealy run with full cache build)\n",
"...b: C:/ws as a workdir, no cache\n",
"...c: c:/ws as workdir, no av\n",
"```\n",
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.5rc1"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

322
scripts/metrics/buildbot_monitoring.py
View file

@ -1,322 +0,0 @@
#!/usr/bin/env python3
import logging
import psycopg2
import os
import datetime
import requests
from typing import Optional, Dict, List
import json
PHABRICATOR_URL = "https://reviews.llvm.org/api/"
BUILDBOT_URL = "https://lab.llvm.org/buildbot/api/v2/"
# TODO(kuhnel): retry on connection issues, maybe resuse
# https://github.com/google/llvm-premerge-checks/blob/main/scripts/phabtalk/phabtalk.py#L44
# TODO(kuhnel): Import the step data so we can figure out in which step a build fails
# (e.g. compile vs. test)
def connect_to_db() -> psycopg2.extensions.connection:
"""Connect to the database."""
conn = psycopg2.connect(
f"host=127.0.0.1 sslmode=disable dbname=stats user=stats password={os.getenv('DB_PASSWORD')}")
return conn
def create_tables(conn: psycopg2.extensions.connection):
cur = conn.cursor()
cur.execute(
"""CREATE TABLE IF NOT EXISTS buildbot_workers (
timestamp timestamp NOT NULL,
worker_id integer NOT NULL,
data jsonb NOT NULL
);"""
)
cur.execute(
"""CREATE INDEX IF NOT EXISTS buildbot_worker_ids
ON buildbot_workers
(worker_id);"""
)
cur.execute(
"""CREATE INDEX IF NOT EXISTS buildbot_worker_timestamp
ON buildbot_workers
(timestamp);"""
)
# Note: step_data is not yet populated with data!
cur.execute(
"""CREATE TABLE IF NOT EXISTS buildbot_builds (
build_id integer PRIMARY KEY,
builder_id integer NOT NULL,
build_number integer NOT NULL,
build_data jsonb NOT NULL,
step_data jsonb
);"""
)
cur.execute(
"""CREATE TABLE IF NOT EXISTS buildbot_buildsets (
buildset_id integer PRIMARY KEY,
data jsonb NOT NULL
);"""
)
cur.execute(
"""CREATE TABLE IF NOT EXISTS buildbot_buildrequests (
buildrequest_id integer PRIMARY KEY,
buildset_id integer NOT NULL,
data jsonb NOT NULL
);"""
)
cur.execute(
"""CREATE TABLE IF NOT EXISTS buildbot_builders (
builder_id integer PRIMARY KEY,
timestamp timestamp NOT NULL,
name text NOT NULL,
data jsonb NOT NULL
);"""
)
conn.commit()
def get_worker_status(
worker_id: int, conn: psycopg2.extensions.connection
) -> Optional[Dict]:
"""Note: postgres returns a dict for a stored json object."""
cur = conn.cursor()
cur.execute(
"SELECT data FROM buildbot_workers WHERE worker_id = %s ORDER BY timestamp DESC;",
[worker_id],
)
row = cur.fetchone()
if row is None:
return None
return row[0]
def get_builder_status(
builder_id: int, conn: psycopg2.extensions.connection
) -> Optional[Dict]:
"""Note: postgres returns a dict for a stored json object."""
cur = conn.cursor()
cur.execute(
"""SELECT data FROM buildbot_builders WHERE builder_id = %s
ORDER BY timestamp DESC;""",
[builder_id],
)
row = cur.fetchone()
if row is None:
return None
return row[0]
def set_worker_status(
timestamp: datetime.datetime,
worker_id: int,
data: str,
conn: psycopg2.extensions.connection,
):
cur = conn.cursor()
cur.execute(
"""INSERT INTO buildbot_workers (timestamp, worker_id, data)
values (%s,%s,%s);""",
(timestamp, worker_id, data),
)
def update_workers(conn: psycopg2.extensions.connection):
logging.info("Updating worker status...")
response = requests.get(BUILDBOT_URL + "workers")
timestamp = datetime.datetime.now()
for worker in response.json()["workers"]:
worker_id = worker["workerid"]
data = json.dumps(worker)
# TODO: It would be faster if request all worker info and cache it
# locally
old_data = get_worker_status(worker_id, conn)
# only update worker information if it has changed as this data is quite
# static
if old_data is None or worker != old_data:
set_worker_status(timestamp, worker_id, data, conn)
conn.commit()
def update_builders(conn: psycopg2.extensions.connection):
"""get list of all builder ids."""
logging.info("Updating builder status...")
response = requests.get(BUILDBOT_URL + "builders")
timestamp = datetime.datetime.now()
for builder in response.json()["builders"]:
builder_id = builder["builderid"]
data = json.dumps(builder)
# TODO: It would be faster if request all builder info and cache it
# locally
old_data = get_builder_status(builder_id, conn)
# only update worker information if it has changed as this data is quite
# static
if old_data is None or builder != old_data:
set_worker_status(timestamp, builder_id, data, conn)
conn.commit()
def get_last_build(conn: psycopg2.extensions.connection) -> int:
"""Get the latest build number for a builder.
This is used to only get new builds."""
cur = conn.cursor()
cur.execute("SELECT MAX(build_id) FROM buildbot_builds")
row = cur.fetchone()
if row is None or row[0] is None:
return 0
return row[0]
def update_build_status(conn: psycopg2.extensions.connection):
start_id = get_last_build(conn)
logging.info("Updating build results, starting with {}...".format(start_id))
url = BUILDBOT_URL + "builds"
cur = conn.cursor()
for result_set in rest_request_iterator(url, "builds", "buildid", start_id=start_id):
args_str = b",".join(
cur.mogrify(
b" (%s,%s,%s,%s) ",
(
build["buildid"],
build["builderid"],
build["number"],
json.dumps(build, sort_keys=True),
),
)
for build in result_set
if build["complete"]
)
cur.execute(
b"INSERT INTO buildbot_builds (build_id, builder_id, build_number, build_data) values "
+ args_str
)
logging.info("last build id: {}".format(result_set[-1]["buildid"]))
conn.commit()
def rest_request_iterator(
url: str,
array_field_name: str,
id_field_name: str,
start_id: int = 0,
step: int = 1000,
):
"""Request paginated data from the buildbot master.
This returns a generator. Each call to it gives you shards of
<=limit results. This can be used to do a mass-SQL insert of data.
Limiting the range of the returned IDs causes Buildbot to sort the data.
This makes incremental imports much easier.
"""
while True:
count = 0
stop_id = start_id + step
response = requests.get(
url
+ "?{id_field_name}__gt={start_id}&{id_field_name}__le={stop_id}&".format(
**locals()
)
)
if response.status_code != 200:
raise Exception(
"Got status code {} on request to {}".format(response.status_code, url)
)
results = response.json()[array_field_name]
if len(results) == 0:
return
yield results
start_id = stop_id
def get_latest_buildset(conn: psycopg2.extensions.connection) -> int:
"""Get the maximumg buildset id.
This is useful for incremental updates."""
cur = conn.cursor()
cur.execute("SELECT MAX(buildset_id) from buildbot_buildsets;")
row = cur.fetchone()
if row[0] is None:
return 0
return row[0]
def update_buildsets(conn: psycopg2.extensions.connection):
start_id = get_latest_buildset(conn)
logging.info("Getting buildsets, starting with {}...".format(start_id))
url = BUILDBOT_URL + "buildsets"
cur = conn.cursor()
for result_set in rest_request_iterator(
url, "buildsets", "bsid", start_id=start_id
):
args_str = b",".join(
cur.mogrify(
b" (%s,%s) ",
(buildset["bsid"], json.dumps(buildset, sort_keys=True)),
)
for buildset in result_set
if buildset["complete"]
)
if len(args_str) == 0:
break
cur.execute(
b"INSERT INTO buildbot_buildsets (buildset_id, data) values " + args_str
)
logging.info("last id {}".format(result_set[-1]["bsid"]))
conn.commit()
def get_latest_buildrequest(conn: psycopg2.extensions.connection) -> int:
cur = conn.cursor()
cur.execute("SELECT MAX(buildrequest_id) from buildbot_buildrequests;")
row = cur.fetchone()
if row[0] is None:
return 0
return row[0]
def update_buildrequests(conn: psycopg2.extensions.connection):
start_id = get_latest_buildrequest(conn)
logging.info("Getting buildrequests, starting with {}...".format(start_id))
url = BUILDBOT_URL + "buildrequests"
cur = conn.cursor()
for result_set in rest_request_iterator(
url, "buildrequests", "buildrequestid", start_id=start_id
):
# cur.mogrify returns a byte string, so we need to join on a byte string
args_str = b",".join(
cur.mogrify(
" (%s,%s,%s) ",
(
buildrequest["buildrequestid"],
buildrequest["buildsetid"],
json.dumps(buildrequest),
),
)
for buildrequest in result_set
if buildrequest["complete"]
)
if len(args_str) == 0:
break
cur.execute(
b"INSERT INTO buildbot_buildrequests (buildrequest_id, buildset_id, data) values "
+ args_str
)
logging.info("{}".format(result_set[-1]["buildrequestid"]))
conn.commit()
if __name__ == "__main__":
logging.basicConfig(level='INFO', format='%(levelname)-7s %(message)s')
conn = connect_to_db()
create_tables(conn)
update_workers(conn)
update_builders(conn)
update_build_status(conn)
update_buildsets(conn)
update_buildrequests(conn)

141
scripts/metrics/buildbot_status_emails.py
View file

@ -1,141 +0,0 @@
#!/usr/bin/env python3
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import csv
import datetime
import gzip
import os
import mailbox
import requests
import re
from typing import List, Dict, Set
EMAIL_ARCHIVE_URL = 'http://lists.llvm.org/pipermail/llvm-dev/{year}-{month}.txt.gz'
TMP_DIR = os.path.join(os.path.dirname(__file__), 'tmp')
class LLVMBotArchiveScanner:
def __init__(self):
self._tmpdir = TMP_DIR
@staticmethod
def _generate_archive_url(month: datetime.date) -> str:
return EMAIL_ARCHIVE_URL.format(year=month.year, month=month.strftime('%B'))
def _download_archive(self, month: datetime.date):
os.makedirs(self._tmpdir, exist_ok=True)
filename = os.path.join(self._tmpdir, 'llvmdev-{year}-{month:02d}.txt'.format(year=month.year, month=month.month))
url = self._generate_archive_url(month)
# FIXME: decompress the files
self.download(url, filename)
def get_archives(self, start_month: datetime.date):
print('Downloading data...')
month = start_month
today = datetime.date.today()
while month < today:
self._download_archive(month)
if month.month < 12:
month = datetime.date(year=month.year, month=month.month+1, day=1)
else:
month = datetime.date(year=month.year+1, month=1, day=1)
def extract_emails(self) -> List[mailbox.Message]:
result = []
for archive_name in (d for d in os.listdir(self._tmpdir) if d.startswith('llvmdev-')):
print('Scanning {}'.format(archive_name))
mb = mailbox.mbox(os.path.join(self._tmpdir, archive_name), factory=mbox_reader)
for mail in mb.values():
subject = mail.get('subject')
if subject is None:
continue
if 'Buildbot numbers' in mail['subject']:
yield(mail)
yield
def get_attachments(self, email: mailbox.Message):
if email is None:
return
week_str = re.search(r'(\d+/\d+/\d+)', email['subject']).group(1)
week = datetime.datetime.strptime(week_str, '%m/%d/%Y').date()
attachment_url = re.search(r'Name: completed_failed_avr_time.csv[^<]*URL: <([^>]+)>', email.get_payload(), re.DOTALL).group(1)
filename = os.path.join(self._tmpdir, 'buildbot_stats_{}.csv'.format(week.isoformat()))
self.download(attachment_url, filename)
@staticmethod
def download(url, filename):
if os.path.exists(filename):
return
r = requests.get(url)
print('Getting {}'.format(filename))
with open(filename, 'wb') as f:
f.write(r.content)
def merge_results(self):
def _convert_int(s: str) -> int:
if len(s) == 0:
return 0
return int(s)
bot_stats = {} # type: Dict[str, Dict[datetime.date, float]]
weeks = set() # type: Set[datetime.date]
for csv_filename in (d for d in os.listdir(self._tmpdir) if d.startswith('buildbot_stats_')):
week_str = re.search(r'(\d+-\d+-\d+)', csv_filename).group(1)
week = datetime.datetime.fromisoformat(week_str).date()
weeks.add(week)
with open(os.path.join(self._tmpdir, csv_filename)) as csv_file:
reader = csv.DictReader(csv_file)
for row in reader:
name = row['name']
red_build = _convert_int(row['red_builds'])
all_builds = _convert_int(row['all_builds'])
percentage = 100.0 * red_build / all_builds
bot_stats.setdefault(name, {})
bot_stats[name][week] = percentage
with open(os.path.join(self._tmpdir, 'buildbot_weekly.csv'), 'w') as csv_file:
fieldnames = ['week']
filtered_bots = sorted(b for b in bot_stats.keys()) # if len(bot_stats[b]) == len(weeks)
fieldnames.extend(filtered_bots)
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
writer.writeheader()
for week in sorted(weeks):
row = {'week': week.isoformat()}
for bot in filtered_bots:
percentage = bot_stats[bot].get(week)
if percentage is None:
continue
row[bot] = percentage
writer.writerow(row)
def mbox_reader(stream):
"""Read a non-ascii message from mailbox.
Based on https://stackoverflow.com/questions/37890123/how-to-trap-an-exception-that-occurs-in-code-underlying-python-for-loop
"""
data = stream.read()
text = data.decode(encoding="utf-8")
return mailbox.mboxMessage(text)
if __name__ == '__main__':
scanner = LLVMBotArchiveScanner()
scanner.get_archives(datetime.date(year=2019, month=8, day=1))
for message in scanner.extract_emails():
scanner.get_attachments(message)
scanner.merge_results()

138
scripts/metrics/buildbots.py
View file

@ -1,138 +0,0 @@
#!/usr/bin/env python3
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from datetime import date
import requests
import datetime
from google.cloud import monitoring_v3
BASE_URL = 'http://lab.llvm.org:8011/json/builders'
GCP_PROJECT_ID = 'llvm-premerge-checks'
class BuildStats:
"""Build statistics.
Plain data object.
"""
def __init__(self, successful:int = 0, failed:int = 0):
self.successful = successful # type: int
self.failed = failed # type: int
def add(self, success: bool):
if success:
self.successful += 1
else:
self.failed += 1
@property
def total(self) -> int:
return self.successful + self.failed
@property
def percent_failed(self) -> float:
return 100.0 * self.failed / self.total
def __add__(self, other: "BuildStats") -> "BuildStats":
return BuildStats(
self.successful + other.successful,
self.failed + other.failed)
def __str__(self) -> str:
result = [
'successful: {}'.format(self.successful),
'failed: {}'.format(self.failed),
'total: {}'.format(self.total),
'% failed: {:0.1f}'.format(self.percent_failed),
]
return '\n'.join(result)
def get_buildbot_stats(time_window : datetime.datetime) -> BuildStats:
"""Get the statistics for the all builders."""
print('getting list of builders...')
stats = BuildStats()
for builder in requests.get(BASE_URL).json().keys():
# TODO: maybe filter the builds to the ones we care about
stats += get_builder_stats(builder, time_window )
return stats
def get_builder_stats(builder: str, time_window: datetime.datetime) -> BuildStats:
"""Get the statistics for one builder."""
print('Gettings builds for {}...'.format(builder))
# TODO: can we limit the data we're requesting?
url = '{}/{}/builds/_all'.format(BASE_URL, builder)
stats = BuildStats()
for build, results in requests.get(url).json().items():
start_time = datetime.datetime.fromtimestamp(float(results['times'][0]))
if start_time < time_window:
continue
successful = results['text'] == ['build', 'successful']
stats.add(successful)
return stats
def gcp_create_metric_descriptor(project_id: str):
"""Create metric descriptors on Stackdriver.
Re-creating these with every call is fine."""
client = monitoring_v3.MetricServiceClient()
project_name = client.project_path(project_id)
for desc_type, desc_desc in [
["buildbots_percent_failed", "Percentage of failed builds"],
["buildbots_builds_successful", "Number of successful builds in the last 24h."],
["buildbots_builds_failed", "Number of failed builds in the last 24h."],
["buildbots_builds_total", "Total number of builds in the last 24h."],
]:
descriptor = monitoring_v3.types.MetricDescriptor()
descriptor.type = 'custom.googleapis.com/buildbots_{}'.format(desc_type)
descriptor.metric_kind = (
monitoring_v3.enums.MetricDescriptor.MetricKind.GAUGE)
descriptor.value_type = (
monitoring_v3.enums.MetricDescriptor.ValueType.DOUBLE)
descriptor.description = desc_desc
descriptor = client.create_metric_descriptor(project_name, descriptor)
print('Created {}.'.format(descriptor.name))
def gcp_write_data(project_id: str, stats: BuildStats):
"""Upload metrics to Stackdriver."""
client = monitoring_v3.MetricServiceClient()
project_name = client.project_path(project_id)
now = datetime.datetime.now()
for desc_type, value in [
["buildbots_percent_failed", stats.percent_failed],
["buildbots_builds_successful", stats.successful],
["buildbots_builds_failed", stats.failed],
["buildbots_builds_total", stats.total],
]:
series = monitoring_v3.types.TimeSeries()
series.metric.type = 'custom.googleapis.com/buildbots_{}'.format(desc_type)
series.resource.type = 'global'
point = series.points.add()
point.value.double_value = value
point.interval.end_time.seconds = int(now.timestamp())
client.create_time_series(project_name, [series])
if __name__ == '__main__':
gcp_create_metric_descriptor(GCP_PROJECT_ID)
stats = get_buildbot_stats(
datetime.datetime.now() - datetime.timedelta(hours=24))
gcp_write_data(GCP_PROJECT_ID, stats)
print(stats)

145
scripts/metrics/buildkite_master_stats.py
View file

@ -1,145 +0,0 @@
#!/usr/bin/env python3
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# -----------------------------------------------------------------------------
# This script will collect all breakages of the main branch builds from
# buildkite and format the results nicely.
# Arguments:
# llvm-path : folder where the LLVM checkout is kept
# token : Access token for buildkite
# -----------------------------------------------------------------------------
import requests
import git
import argparse
import json
import os
import datetime
class Build:
def __init__(self, json_dict):
self._json_dict = json_dict
@property
def number(self) -> int:
return self._json_dict['number']
@property
def web_url(self) -> str:
return self._json_dict['web_url']
@property
def state(self) -> str:
return self._json_dict['state']
@property
def has_passed(self) -> bool:
return self.state == 'passed'
@property
def commit(self) -> str:
return self._json_dict['commit']
@property
def created_at(self) -> datetime.datetime:
#example: 2019-11-07T14:13:07.942Z
return datetime.datetime.fromisoformat(self._json_dict['created_at'].rstrip('Z'))
class BuildKiteMasterStats:
def __init__(self, llvm_repo: str, token_path: str):
self._llvm_repo = llvm_repo
with open(token_path, 'r') as token_file:
self._token = token_file.read().strip()
def get_stats(self, organisation: str, pipeline: str):
url = "https://api.buildkite.com/v2/organizations/{}/pipelines/{}/builds".format(organisation, pipeline)
return self._get_url(url)
def _get_url(self, url: str):
"""Get paginated results from server."""
page = 1
results = []
while True:
page_url = url + f"?api_key={self._token}&page={page}&per_page=100"
new_results = requests.get(page_url).json()
results.extend(new_results)
print(len(new_results))
if len(new_results) < 100:
break
page += 1
return results
def save_results(self, output_path:str, data):
with open(output_path, 'w') as output_file:
json.dump(data, output_file)
def get_builds(self, json_path: str):
with open(json_path) as json_file:
builds = json.load(json_file)
build_dict = {}
for b in builds:
build = Build(b)
build_dict[build.number] = build
return build_dict
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('llvm_path')
parser.add_argument('token')
args = parser.parse_args()
CACHE_FILE = 'tmp/bklogs.json'
bk = BuildKiteMasterStats(args.llvm_path, args.token)
if not os.path.exists(CACHE_FILE):
results = bk.get_stats('llvm-project','llvm-main-build')
bk.save_results(CACHE_FILE, results)
builds = bk.get_builds(CACHE_FILE)
last_fail = None
fail_count = 0
start_time = None
for build_number in sorted(builds.keys()):
if build_number < 50:
# skip the first builds as they might not be mature enough
continue
build = builds[build_number]
if build.has_passed:
if last_fail is not None:
print(f'* ends with [build {build.number}]({build.web_url})')
print(f'* ends with commit {build.commit}')
print(f'* ends on {build.created_at}')
duration = build.created_at - start_time
print(f'* duration: {duration} [h:m:s]')
print('* cause: # TODO')
print()
last_fail = None
else:
if last_fail is None:
print(f'# Outage {fail_count}')
print()
print(f'* starts with [build {build.number}]({build.web_url})')
print(f'* starts with commit {build.commit}')
print(f'* starts on {build.created_at}')
fail_count += 1
start_time = build.created_at
else:
pass
last_fail = build.number

7
scripts/metrics/download_buildkite_builds_pg.sh
View file

@ -1,7 +0,0 @@
#!/usr/bin/env bash
set -eu
set -o pipefail
echo "downloading buildkite builds..."
env

541
scripts/metrics/pull_phab_build_stats.py
View file

@ -1,541 +0,0 @@
#!/usr/bin/env python3
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Get data on Revisions and builds from Phabricator
import phabricator
import json
import os
import datetime
from typing import Dict, List, Optional
import csv
import time
import socket
import git
import argparse
# PHIDs of build plans used for pre-merge testing
# FIXME: how do you get these?
_PRE_MERGE_PHIDs = ['PHID-HMCP-bfkbtacsszhg3feydpo6', # beta testers
'PHID-HMCP-qbviyekvgirhhhvkzpsn', # public pre-merge tests
'PHID-HMCP-p2oc4ocen3l2yzymvg2l',
]
# query all data after this date
START_DATE = datetime.date(year=2019, month=10, day=1)
class PhabResponse:
def __init__(self, revision_dict: Dict):
self.revision_dict = revision_dict
@property
def id(self) -> str:
return self.revision_dict['id']
@property
def phid(self) -> str:
return self.revision_dict['phid']
def __str__(self):
return str(self.revision_dict)
class Revision(PhabResponse):
def __init__(self, revision_dict):
super().__init__(revision_dict)
self.buildables = [] # type: List['Buildable']
self.diffs = [] # type: List['Diff']
@property
def status(self) -> str:
return self.revision_dict['fields']['status']['value']
@property
def builds(self) -> List['Build']:
builds = []
for b in self.buildables:
builds.extend(b.builds)
return builds
@property
def created_date(self):
return self.revision_dict['fields']['dateCreated']
@property
def was_premerge_tested(self) -> bool:
return any((b.was_premerge_tested for b in self.builds))
@property
def repository_phid(self) -> str:
return self.revision_dict['fields']['repositoryPHID']
@property
def diff_phid(self) -> str:
return self.revision_dict['fields']['diffPHID']
@property
def all_diffs_have_refs(self) -> bool:
return not any(not d.has_refs for d in self.diffs)
@property
def day(self) -> datetime.date:
return datetime.date.fromtimestamp(self.created_date)
@property
def week(self) -> str:
day = self.day
return'{}-w{:02d}'.format(day.year, day.isocalendar()[1])
@property
def published(self) -> bool:
return self.status == 'published'
@property
def build_status_list(self) -> List[bool]:
return [b.passed for b in self.builds if b.was_premerge_tested]
@property
def builds_finally_succeeded(self) -> bool:
"""Return true iff one of the builds failed and the last build passed."""
return self.has_failed_builds and self.build_status_list[-1]
@property
def has_failed_builds(self) -> bool:
"""Return true iff one of the builds failed."""
return False in self.build_status_list
@property
def published_failing(self) -> bool:
"""Return true iff published and the last build failed."""
return self.was_premerge_tested and self.published \
and not self.build_status_list[-1]
@property
def all_builds_passed(self) -> bool:
return self.was_premerge_tested and all(self.build_status_list)
@property
def all_builds_failed(self) -> bool:
return self.was_premerge_tested and all(not b for b in self.build_status_list)
@property
def dateModified(self) -> datetime.datetime:
return datetime.datetime.fromtimestamp(self.revision_dict['fields']['dateModified'])
class Buildable(PhabResponse):
def __init__(self, revision_dict):
super().__init__(revision_dict)
self.builds = [] # type: List[Build]
self.revision = None # type: Optional[Revision]
@property
def diff_phid(self) -> str:
return self.revision_dict['buildablePHID']
@property
def revison_phid(self) -> str:
return self.revision_dict['containerPHID']
class Build(PhabResponse):
def __init__(self, revision_dict):
super().__init__(revision_dict)
self.buildable = None # type: Optional[Buildable]
@property
def buildable_phid(self) -> str:
return self.revision_dict['fields']['buildablePHID']
@property
def buildplan_phid(self) -> str:
return self.revision_dict['fields']['buildPlanPHID']
@property
def was_premerge_tested(self) -> bool:
result = self.buildplan_phid in _PRE_MERGE_PHIDs
return result
@property
def passed(self) -> bool:
"""Returns true, if the build "passed" """
return self.was_premerge_tested and self.revision_dict['fields']['buildStatus']['value'] == 'passed'
@property
def dateModified(self) ->datetime.datetime:
return datetime.datetime.fromtimestamp(self.revision_dict['fields']['dateModified'])
class Diff(PhabResponse):
def __init__(self, revision_dict):
super().__init__(revision_dict)
self.revision = None # type: Optional[Revision]
@property
def revison_phid(self) -> str:
return self.revision_dict['fields']['revisionPHID']
@property
def _refs(self) -> List:
return self.revision_dict['fields']['refs']
@property
def has_refs(self) -> bool:
return len(self._refs) > 0
@property
def base_revision(self) -> str:
for ref in self._refs:
if ref['type'] == 'base':
return ref['identifier']
return None
@property
def base_branch(self) -> str:
for ref in self._refs:
if ref['type'] == 'branch':
return ref['name']
return None
@property
def dateCreated(self) -> datetime.datetime:
return datetime.datetime.fromtimestamp(self.revision_dict['fields']['dateCreated'])
class PhabBuildPuller:
# files/folder for sotring temporary results
_TMP_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), 'tmp'))
_REVISION_FILE = os.path.join(_TMP_DIR, 'phab-revisions.json')
_BUILDABLE_FILE = os.path.join(_TMP_DIR, 'phab-buildables.json')
_BUILD_FILE = os.path.join(_TMP_DIR, 'phab-build.json')
_DIFF_FILE = os.path.join(_TMP_DIR, 'phab-diffs.json')
_PHAB_WEEKLY_METRICS_FILE = os.path.join(_TMP_DIR, 'phabricator_{}.csv')
def __init__(self, repo_path: str):
self.conduit_token = None
self.host = None
self.phab = self._create_phab()
self._repo_path = repo_path # type: str
self.revisions = {} # type: Dict[str, Revision]
self.buildables = {} # type: Dict[str, Buildable]
self.builds = {} # type: Dict[str, Build]
self.diffs = {} # type: Dict[str, Diff]
def _create_phab(self) -> phabricator.Phabricator:
phab = phabricator.Phabricator(token=self.conduit_token, host=self.host)
phab.update_interfaces()
return phab
def _load_arcrc(self):
"""Load arc configuration from file if not set."""
if self.conduit_token is not None or self.host is not None:
return
print('Loading configuration from ~/.arcrc file')
with open(os.path.expanduser('~/.arcrc'), 'r') as arcrc_file:
arcrc = json.load(arcrc_file)
# use the first host configured in the file
self.host = next(iter(arcrc['hosts']))
self.conduit_token = arcrc['hosts'][self.host]['token']
def run(self):
if not os.path.exists(self._TMP_DIR):
os.mkdir(self._TMP_DIR)
if not os.path.isfile(self._REVISION_FILE):
self.get_revisions()
self.parse_revisions()
if not os.path.isfile(self._BUILDABLE_FILE):
self.get_buildables()
self.parse_buildables()
if not os.path.isfile(self._BUILD_FILE):
self.get_builds()
self.parse_builds()
if not os.path.isfile(self._DIFF_FILE):
self.get_diffs()
self.parse_diffs()
self.link_objects()
self.compute_metrics('day', lambda r: r.day)
self.compute_metrics('week', lambda r: r.week)
self.count_base_revisions()
self.revision_statistics()
self.match_base_revisions_with_repo(self._repo_path)
def get_revisions(self):
print('Downloading revisions starting...')
from_date = int(START_DATE.strftime('%s'))
data = []
constraints = {
'createdStart': from_date
}
# FIXME: lots of code duplication around pagination and error handling.
# find a way to separate this into a function.
after = None
while True:
revisions = self.phab.differential.revision.search(
constraints=constraints, after=after)
data.extend(revisions.response['data'])
print('{} revisions...'.format(len(data)))
after = revisions.response['cursor']['after']
if after is None:
break
print('Number of revisions:', len(data))
with open(self._REVISION_FILE, 'w') as json_file:
json.dump(data, json_file)
def get_buildables(self):
print('Downloading buildables...')
data = []
after = None
while True:
revisions = self.phab.harbormaster.querybuildables(
containerPHIDs=[r.phid for r in self.revisions.values()], after=after)
data.extend(revisions.response['data'])
print('{} buildables...'.format(len(data)))
after = revisions.response['cursor']['after']
if after is None:
break
print('Number of buildables:', len(data))
with open(self._BUILDABLE_FILE, 'w') as json_file:
json.dump(data, json_file)
def get_builds(self):
print('Downloading builds...')
data = []
constraints = {
'buildables': [r.phid for r in self.buildables.values()]
}
after = None
while True:
# retry on timeouts
fail_count = 0
while True:
try:
revisions = self.phab.harbormaster.build.search(
constraints=constraints, after=after)
except socket.timeout:
fail_count +=1
if fail_count > 5:
raise
time.sleep(10)
continue
break
data.extend(revisions.response['data'])
print('{} builds...'.format(len(data)))
after = revisions.response['cursor']['after']
if after is None:
break
print('Number of buildables:', len(data))
with open(self._BUILD_FILE, 'w') as json_file:
json.dump(data, json_file)
def get_diffs(self):
print('Downloading diffs...')
data = []
constraints = {
'revisionPHIDs': [r.phid for r in self.revisions.values()]
}
after = None
while True:
# retry on timeouts
fail_count = 0
while True:
try:
diffs = self.phab.differential.diff.search(
constraints=constraints, after=after)
except socket.timeout:
fail_count +=1
if fail_count > 5:
raise
time.sleep(10)
continue
break
data.extend(diffs.response['data'])
print('{} diffs...'.format(len(data)))
after = diffs.response['cursor']['after']
if after is None:
break
print('Number of diffs:', len(data))
with open(self._DIFF_FILE, 'w') as json_file:
json.dump(data, json_file)
def parse_revisions(self):
with open(self._REVISION_FILE) as revision_file:
revision_dict = json.load(revision_file)
self.revisions = {r.phid: r for r in (Revision(x) for x in revision_dict)}
print('Parsed {} revisions.'.format(len(self.revisions)))
def parse_buildables(self):
with open(self._BUILDABLE_FILE) as buildables_file:
buildable_dict = json.load(buildables_file)
self.buildables = {b.phid: b for b in (Buildable(x) for x in buildable_dict)}
print('Parsed {} buildables.'.format(len(self.buildables)))
def parse_builds(self):
with open(self._BUILD_FILE) as build_file:
build_dict = json.load(build_file)
self.builds = {b.phid: b for b in (Build(x) for x in build_dict)}
print('Parsed {} builds.'.format(len(self.builds)))
def parse_diffs(self):
with open(self._DIFF_FILE) as diff_file:
diff_dict = json.load(diff_file)
self.diffs = {d.phid: d for d in (Diff(x) for x in diff_dict)}
print('Parsed {} diffs.'.format(len(self.diffs)))
def link_objects(self):
for build in (b for b in self.builds.values()):
buildable = self.buildables[build.buildable_phid]
build.buildable = buildable
buildable.builds.append(build)
for buildable in self.buildables.values():
revision = self.revisions[buildable.revison_phid]
revision.buildables.append(buildable)
buildable.revision = revision
for diff in self.diffs.values():
revision = self.revisions[diff.revison_phid]
revision.diffs.append(diff)
diff.revision = revision
def compute_metrics(self, name: str, group_function):
print('Creating metrics for {}...'.format(name))
group_dict = {}
for revision in self.revisions.values():
group_dict.setdefault(group_function(revision), []).append(revision)
csv_file = open(self._PHAB_WEEKLY_METRICS_FILE.format(name), 'w')
fieldnames = [name, '# revisions', '# tested revisions', '% tested revisions', '# untested revisions',
'# revisions without builds', '% revisions without builds', '# no repository set',
'# had failed builds', '% had failed builds', '# failed first then passed',
'% failed first then passed', '# published failing', '% published failing',
'# all passed', '% all passed']
writer = csv.DictWriter(csv_file, fieldnames=fieldnames, dialect=csv.excel)
writer.writeheader()
for group in sorted(group_dict.keys()):
revisions = group_dict[group] # type: List[Revision]
num_revisions = len(revisions)
num_premt_revisions = len([r for r in revisions if r.was_premerge_tested])
precentage_premt_revisions = 100.0 * num_premt_revisions / num_revisions
num_no_build_triggered = len([r for r in revisions if len(r.builds) == 0])
percent_no_build_triggered = 100.0 * num_no_build_triggered / num_revisions
num_no_repo = len([r for r in revisions if r.repository_phid is None])
num_had_failed_builds =len([r for r in revisions if r.has_failed_builds])
num_failed_first_then_passed = len([r for r in revisions if r.builds_finally_succeeded])
num_published_failing = len([r for r in revisions if r.published_failing])
num_all_passed = len([r for r in revisions if r.all_builds_passed])
writer.writerow({
name: group,
'# revisions': num_revisions,
'# tested revisions': num_premt_revisions,
'% tested revisions': precentage_premt_revisions,
'# untested revisions': num_revisions - num_premt_revisions,
'# revisions without builds': num_no_build_triggered,
'% revisions without builds': percent_no_build_triggered,
'# no repository set': num_no_repo,
'# had failed builds': num_had_failed_builds,
'% had failed builds': 100 * num_had_failed_builds / num_revisions,
'# failed first then passed': num_failed_first_then_passed,
'% failed first then passed': 100 * num_failed_first_then_passed / num_revisions,
'# published failing': num_published_failing,
'% published failing': 100 * num_published_failing / num_revisions,
'# all passed': num_all_passed,
'% all passed': 100*num_all_passed / num_revisions,
})
def count_base_revisions(self):
base_revisions = {}
base_branches = {}
for diff in self.diffs.values():
base_revisions.setdefault(diff.base_revision, 0)
base_revisions[diff.base_revision] += 1
base_branches.setdefault(diff.base_branch, 0)
base_branches[diff.base_branch] +=1
print(f'{len(self.diffs)} diffs are using {len(base_revisions)} different git base revisions.')
print('The top 10 revisions and their usages are:')
revisions = sorted( base_revisions.items(), key=lambda x: x[1] , reverse=True)
for i in revisions[:10]:
print(f' commit {i[0]} was used {i[1]} times')
print()
print(f'{len(self.diffs)} diffs are using {len(base_branches)} different git base branches')
branches = sorted( base_branches.items(), key=lambda x: x[1] , reverse=True)
print('The top 10 branches and their usages are:')
for i in branches[:10]:
print(f' branch {i[0]} was used {i[1]} times')
print()
def match_base_revisions_with_repo(self, repo_path: str):
repo = git.Repo(repo_path)
not_found = 0
invalid_date = 0
has_base_revision = 0
for diff in self.diffs.values():
revision = diff.base_revision
if revision is None:
continue
has_base_revision += 1
try:
commit = repo.commit(revision)
except (ValueError, git.BadName):
not_found += 1
continue
commited_date = datetime.datetime.fromtimestamp(commit.committed_date)
if commited_date > diff.dateCreated:
invalid_date += 1
print()
print(f'Of the {has_base_revision} Diffs with base revision, the base revision was NOT found in the repo for {not_found} and ')
print(f'{invalid_date} base revisions were used before being available upstream.')
print(f'So {(not_found+invalid_date)/has_base_revision*100:0.2f} % of specified the base revisions were unusable.')
def revision_statistics(self):
no_builds = 0
has_failed = 0
fail_then_pass = 0
all_passed = 0
fail_last = 0
for revision in self.revisions.values():
build_status = [b.passed for b in revision.builds]
if len(revision.builds) == 0:
no_builds += 1
continue
if False in build_status:
has_failed += 1
if build_status[-1] == True:
fail_then_pass +=1
else:
all_passed += 1
if revision.published and build_status[-1] == False:
fail_last += 1
print()
print(f'Out of the {len(self.revisions)} Revisions:')
print(f' {no_builds} had no builds.')
print(f' {has_failed} had failed builds.')
print(f' {fail_then_pass} had failed builds, but the last build passed.')
print(f' {all_passed} had only successful builds.')
print(f' {fail_last} were published with a failing build.')
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('repo_path')
args = parser.parse_args()
puller = PhabBuildPuller(args.repo_path)
puller.run()

116
scripts/metrics/repo.py
View file

@ -1,116 +0,0 @@
#!/usr/bin/env python3
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import datetime
from time import timezone
import git
from typing import Dict, Optional
from google.cloud import monitoring_v3
import re
from datetime import tzinfo
GCP_PROJECT_ID = 'llvm-premerge-checks'
class RepoStats:
def __init__(self):
self.commits = 0 # type: int
self.reverts = 0 # type: int
self.reviewed = 0 # type: int
@property
def percent_reverted(self) -> Optional[float]:
try:
return 100.0 * self.reverts / self.commits
except ZeroDivisionError:
return None
@property
def percent_reviewed(self) -> Optional[float]:
try:
return 100.0 * self.reviewed / (self.commits - self.reverts)
except ZeroDivisionError:
return None
def __str__(self):
results = [
"commits: {}".format(self.commits),
"reverts: {}".format(self.reverts),
"reviewed: {}".format(self.reviewed),
]
try:
results.append("percent reverted: {:0.1f}".format(self.percent_reverted))
except TypeError:
pass
try:
results.append("percent reverted: {:0.1f}".format(self.percent_reverted))
except TypeError:
pass
return "\n".join(results)
def get_reverts_per_day(repo_path: str, max_age: datetime.datetime) -> RepoStats:
stats = RepoStats()
repo = git.Repo(repo_path)
repo.git.fetch()
diff_regex = re.compile(r'^Differential Revision: https:\/\/reviews\.llvm\.org\/(.*)$', re.MULTILINE)
for commit in repo.iter_commits('main'):
if commit.committed_datetime < max_age:
break
stats.commits += 1
if commit.message.startswith('Revert'):
stats.reverts += 1
if diff_regex.search(commit.message) is not None:
stats.reviewed += 1
return stats
def gcp_write_data(project_id: str, stats: RepoStats, now:datetime.datetime):
"""Upload metrics to Stackdriver."""
client = monitoring_v3.MetricServiceClient()
project_name = client.project_path(project_id)
for desc_type, value in [
["reverts", stats.reverts],
["commits", stats.commits],
["percent_reverted", stats.percent_reverted],
["reviewed", stats.reviewed],
["percent_reviewed", stats.percent_reviewed],
]:
if value is None:
continue
series = monitoring_v3.types.TimeSeries()
series.metric.type = 'custom.googleapis.com/repository_{}'.format(desc_type)
series.resource.type = 'global'
point = series.points.add()
point.value.double_value = value
point.interval.end_time.seconds = int(now.timestamp())
client.create_time_series(project_name, [series])
if __name__ == '__main__':
now = datetime.datetime.now(tz=datetime.timezone.utc)
max_age = now - datetime.timedelta(days=1)
# TODO: make path configurable
stats = get_reverts_per_day('~/git/llvm-project', max_age)
print(stats)
# TODO: add `dryrun` parameter
gcp_write_data(GCP_PROJECT_ID, stats, now)

463
scripts/metrics/repo_hist.py
View file

@ -1,463 +0,0 @@
#!/usr/bin/env python3
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# generate statistics on the llvm github repository
import csv
import datetime
import git
import re
import os
from typing import Dict, Optional, List, Set
import random
import string
REVISION_REGEX = re.compile(
r"^Differential Revision: https://reviews\.llvm\.org/(.*)$", re.MULTILINE
)
REVERT_REGEX = re.compile(r'^Revert "(.+)"')
REVERT_HASH_REGEX = re.compile("This reverts commit (\w+)", re.MULTILINE)
class MyCommit:
SALT = "".join(
random.choices(
string.ascii_lowercase + string.ascii_uppercase + string.digits, k=16
)
)
def __init__(self, commit: git.Commit):
self.commit = commit
self.chash = commit.hexsha # type: str
self.author = hash(commit.author.email + MyCommit.SALT) # type: int
self.author_domain = commit.author.email.rsplit("@")[-1].lower() # type: str
self.commiter = hash(commit.committer.email.lower() + MyCommit.SALT) # type:int
self.summary = commit.summary # type: str
self.date = datetime.datetime.fromtimestamp(
commit.committed_date
) # type: datetime.datetime
self.phab_revision = self._get_revision(commit) # type: Optional[str]
self.reverts = None # type: Optional[MyCommit]
self.reverted_by = None # type: Optional[MyCommit]
self._diff_index = None # type: Optional[git.DiffIndex]
@staticmethod
def _get_revision(commit: git.Commit) -> Optional[str]:
m = REVISION_REGEX.search(commit.message)
if m is None:
return None
return m.group(1)
@property
def day(self) -> datetime.date:
return self.date.date()
def reverts_summary(self) -> Optional[str]:
m = REVERT_REGEX.search(self.summary)
if m is None:
return None
return m.group(1)
def __str__(self):
return self.chash
@property
def was_reverted(self) -> bool:
return self.reverted_by is not None
@property
def was_reviewed(self) -> bool:
return self.phab_revision is not None
@property
def is_revert(self) -> bool:
return self.reverts is not None
@property
def week(self) -> str:
return "{}-w{:02d}".format(self.date.year, self.date.isocalendar()[1])
@property
def diff_index(self) -> git.DiffIndex:
# expensive operation, cache the results
if self._diff_index is None:
self._diff_index = self.commit.diff(
self.commit.parents[0], create_patch=True
)
return self._diff_index
@property
def num_loc(self) -> int:
nloc = 0
for diff in self.diff_index:
nloc += str(diff.diff, encoding="utf8").count("\n")
return nloc
@property
def modified_paths(self) -> Set[str]:
result = set(d.b_path for d in self.diff_index if d.b_path is not None)
result.update(d.a_path for d in self.diff_index if d.a_path is not None)
return result
@property
def modified_projects(self) -> Set[str]:
return set(p.split("/")[0] for p in self.modified_paths)
@property
def reverts_commit_hash(self) -> Optional[str]:
m = REVERT_HASH_REGEX.search(self.commit.message)
if m is None:
if self.reverts_summary() is None:
return None
# there was a revert, but we do not know the commit hash
return "unknown"
return m.group(1)
class RepoStats:
def __init__(self, git_dir: str):
self.repo = git.Repo(git_dir)
self.commit_by_hash = dict() # type: Dict[str, MyCommit]
self.commit_by_summary = dict() # type: Dict[str, List[MyCommit]]
self.commit_by_week = dict() # type: Dict[str, List[MyCommit]]
self.commit_by_author = dict() # type: Dict[int, List[MyCommit]]
self.commit_by_author_domain = dict() # type: Dict[str, List[MyCommit]]
def parse_repo(self, maxage: datetime.datetime):
for commit in self.repo.iter_commits("main"):
if commit.committed_datetime < maxage:
break
mycommit = MyCommit(commit)
self.commit_by_hash[mycommit.chash] = mycommit
self.commit_by_summary.setdefault(mycommit.summary, []).append(mycommit)
self.commit_by_week.setdefault(mycommit.week, []).append(mycommit)
self.commit_by_author.setdefault(mycommit.author, []).append(mycommit)
self.commit_by_author_domain.setdefault(mycommit.author_domain, []).append(
mycommit
)
print("Read {} commits".format(len(self.commit_by_hash)))
def find_reverts(self):
reverts = 0
for commit in self.commit_by_hash.values():
summary = commit.reverts_summary()
if summary is None:
continue
if summary not in self.commit_by_summary:
print("summary not found: {}".format(summary))
continue
reverting_commit = self.commit_by_summary[summary][-1]
commit.reverted_by = reverting_commit
reverting_commit.reverts = commit
reverts += 1
print("Found {} reverts".format(reverts))
# https://stackoverflow.com/questions/2600775/how-to-get-week-number-in-python
def dump_daily_stats(self):
fieldnames = [
"week",
"num_commits",
"num_reverts",
"percentage_reverts",
"num_reviewed",
"percentage_reviewed",
"# reviewed & revert",
"# !reviewed & !revert",
"# !reviewed & revert",
"# reviewed & !revert",
]
csvfile = open("tmp/llvm-project-weekly.csv", "w")
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
writer.writeheader()
for week in sorted(self.commit_by_week.keys()):
commits = self.commit_by_week[week]
num_commits = len(commits)
num_reverts = len([c for c in commits if c.is_revert])
percentage_reverts = 100.0 * num_reverts / num_commits
num_reviewed = len([c for c in commits if c.was_reviewed])
percentage_reviewed = 100 * num_reviewed / (num_commits - num_reverts)
num_reviewed_revert = len(
[c for c in commits if c.was_reviewed and c.is_revert]
)
num_reviewed_nrevert = len(
[c for c in commits if c.was_reviewed and not c.is_revert]
)
num_nreviewed_nrevert = len(
[c for c in commits if not c.was_reviewed and not c.is_revert]
)
num_nreviewed_revert = len(
[c for c in commits if not c.was_reviewed and c.is_revert]
)
writer.writerow(
{
"week": week,
"num_commits": num_commits,
"num_reverts": num_reverts,
"percentage_reverts": percentage_reverts,
"num_reviewed": num_reviewed,
"percentage_reviewed": percentage_reviewed,
"# reviewed & revert": num_reviewed_revert,
"# !reviewed & !revert": num_nreviewed_nrevert,
"# !reviewed & revert": num_nreviewed_revert,
"# reviewed & !revert": num_reviewed_nrevert,
}
)
def dump_overall_stats(self):
num_commits = len(self.commit_by_hash)
num_reverts = len([c for c in self.commit_by_hash.values() if c.is_revert])
print("Number of commits: {}".format(num_commits))
print("Number of reverts: {}".format(num_reverts))
print("percentage of reverts: {:0.2f}".format(100 * num_reverts / num_commits))
num_reviewed = len([c for c in self.commit_by_hash.values() if c.was_reviewed])
print("Number of reviewed commits: {}".format(num_reviewed))
print(
"percentage of reviewed commits: {:0.2f}".format(
100 * num_reviewed / num_commits
)
)
num_reviewed_reverted = len(
[
c
for c in self.commit_by_hash.values()
if c.was_reviewed and c.was_reverted
]
)
num_not_reviewed_reverted = len(
[
c
for c in self.commit_by_hash.values()
if not c.was_reviewed and c.was_reverted
]
)
print("Number of reviewed that were reverted: {}".format(num_reviewed_reverted))
print(
"Number of NOT reviewed that were reverted: {}".format(
num_not_reviewed_reverted
)
)
print(
"percentage of reviewed that were reverted: {:0.2f}".format(
100 * num_reviewed_reverted / num_reviewed
)
)
print(
"percentage of NOT reviewed that were reverted: {:0.2f}".format(
100 * num_not_reviewed_reverted / (num_commits - num_reviewed)
)
)
num_foreign_committer = len(
[c for c in self.commit_by_hash.values() if c.author != c.commiter]
)
print(
"Number of commits where author != committer: {}".format(
num_foreign_committer
)
)
print(
"Percentage of commits where author != committer: {:0.2f}".format(
100 * num_foreign_committer / num_commits
)
)
def dump_author_stats(self):
print("Number of authors: {}".format(len(self.commit_by_author)))
fieldnames = [
"author",
"num_commits",
"num_reverts",
"percentage_reverts",
"num_reviewed",
"percentage_reviewed",
]
csvfile = open("tmp/llvm-project-authors.csv", "w")
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
writer.writeheader()
for author, commits in self.commit_by_author.items():
num_commits = len(commits)
num_reverts = len([c for c in commits if c.was_reverted])
percentage_reverts = 100 * num_reverts / num_commits
num_reviewed = len([c for c in commits if c.was_reviewed])
percentage_reviewed = 100 * num_reviewed / num_commits
writer.writerow(
{
"author": author,
"num_commits": num_commits,
"num_reverts": num_reverts,
"percentage_reverts": percentage_reverts,
"num_reviewed": num_reviewed,
"percentage_reviewed": percentage_reviewed,
}
)
def dump_author_domain_stats(self):
print("Number of authors: {}".format(len(self.commit_by_author)))
fieldnames = ["author_domain", "num_commits", "num_committers"]
csvfile = open("tmp/llvm-project-author_domains.csv", "w")
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
writer.writeheader()
for author_domain, commits in self.commit_by_author_domain.items():
num_commits = len(commits)
committers = set(c.author for c in commits)
writer.writerow(
{
"author_domain": author_domain,
"num_commits": num_commits,
"num_committers": len(committers),
}
)
def dump_unreviewed_paths(self, maxage: datetime.datetime):
# TODO: this is really slow. Maybe parallelize?
path_count = {
True: {},
False: {},
} # type: Dict[bool, Dict[str, int]]
for commit in self.repo.iter_commits("main"):
if commit.committed_datetime < maxage:
break
mycommit = MyCommit(commit)
for prefix in set(p.split("/")[0] for p in mycommit.modified_paths):
path_count[mycommit.was_reviewed].setdefault(prefix, 0)
path_count[mycommit.was_reviewed][prefix] += 1
fieldnames = ["was_reviewed"]
all_paths = set(path_count[True].keys())
all_paths.update(path_count[False].keys())
fieldnames.extend(sorted(all_paths))
csvfile = open("tmp/llvm-project-unreviewed-paths.csv", "w")
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
writer.writeheader()
for reviewed in [True, False]:
row = {"was_reviewed": reviewed}
for path, count in path_count[reviewed].items():
row[path] = count
writer.writerow(row)
csvfile.close()
def dump_loc_commits(self, maxage: datetime.datetime):
# TODO: this is really slow. Maybe parallelize?
buckets = list(range(0, 2001, 100))
review_dict = {
True: {b: 0 for b in buckets},
False: {b: 0 for b in buckets},
} # type: Dict[bool, Dict[int, int]]
reverted_dict = {
True: {b: 0 for b in buckets},
False: {b: 0 for b in buckets},
} # type: Dict[bool, Dict[int, int]]
for commit in self.repo.iter_commits("main"):
if commit.committed_datetime < maxage:
break
mycommit = self.commit_by_hash[commit.hexsha]
review_dict[mycommit.was_reviewed][
self._find_bucket(mycommit.num_loc, buckets)
] += 1
reverted_dict[mycommit.was_reverted][
self._find_bucket(mycommit.num_loc, buckets)
] += 1
fieldnames = ["was_reviewed"]
for i in range(0, len(buckets) - 1):
fieldnames.append("{}-{}".format(buckets[i], buckets[i + 1] - 1))
fieldnames.append(">={}".format(buckets[-1]))
csvfile = open("tmp/llvm-project-unreviewed-loc.csv", "w")
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
writer.writeheader()
for reviewed in [True, False]:
row = {"was_reviewed": reviewed}
for i in range(0, len(buckets)):
row[fieldnames[i + 1]] = review_dict[reviewed][buckets[i]]
writer.writerow(row)
writer.writerow({"was_reviewed": "reverted"})
for reverted in [True, False]:
row = {"was_reviewed": reverted}
for i in range(0, len(buckets)):
row[fieldnames[i + 1]] = reverted_dict[reverted][buckets[i]]
writer.writerow(row)
csvfile.close()
@staticmethod
def _find_bucket(number: int, buckets: List[int]) -> int:
for bucket in buckets:
if number < bucket:
return bucket
return buckets[-1]
def export_commits(self):
print("starting export...")
csvfile = open("tmp/llvm-project-export.csv", "w")
fieldnames = [
"timestamp",
"hash",
"reviewed",
"was_reverted",
"is_revert",
"# LOC changed",
"modified projects",
"author domain",
"revision",
]
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
writer.writeheader()
# did not work with multiprocessing.map, gave recursion error from gitpython...
# so using normal map function
for row in map(_create_row, self.commit_by_hash.values()):
writer.writerow(row)
csvfile.close()
def _create_row(mycommit: MyCommit) -> Dict:
try:
return {
"timestamp": mycommit.date.isoformat(),
"hash": mycommit.chash,
"reviewed": mycommit.was_reviewed,
"was_reverted": mycommit.was_reverted,
"is_revert": mycommit.is_revert,
"# LOC changed": mycommit.num_loc,
"modified projects": (";".join(mycommit.modified_projects)),
"author domain": mycommit.author_domain,
"revision": mycommit.phab_revision
if mycommit.phab_revision is not None
else "",
}
except Exception as e:
print(e)
return {}
if __name__ == "__main__":
max_age = datetime.datetime(
year=2019, month=10, day=1, tzinfo=datetime.timezone.utc
)
now = datetime.datetime.now(tz=datetime.timezone.utc)
rs = RepoStats(os.path.expanduser("~/git/llvm-project"))
# TODO: make the path configurable, and `git clone/pull`
rs.parse_repo(max_age)
rs.find_reverts()
rs.dump_daily_stats()
rs.dump_overall_stats()
rs.dump_author_stats()
rs.dump_author_domain_stats()
# disabled as it's quite slow
# rs.dump_unreviewed_paths(now - datetime.timedelta(days=100))
# rs.dump_loc_commits(now - datetime.timedelta(days=100))
rs.export_commits()
print("Done.")

174
scripts/metrics/repo_hist_db.py
View file

@ -1,174 +0,0 @@
#!/usr/bin/env python3
import os
import psycopg2
import git
from repo_hist import MyCommit
import datetime
import csv
from typing import Set
# TODO: make his path configurable for use on the server
REPO_DIR = "tmp/llvm-project"
GIT_URL = "https://github.com/llvm/llvm-project.git"
GIT_BRANCH = "main"
OUTPUT_PATH = "tmp"
# this was the start of using git as primary repo
MAX_AGE = datetime.datetime(year=2019, month=10, day=1, tzinfo=datetime.timezone.utc)
def connect_to_db() -> psycopg2.extensions.connection:
"""Connect to the database, return connection object."""
conn = psycopg2.connect(
"host=127.0.0.1 sslmode=disable dbname=stats user={} password={}".format(
os.environ["PGUSER"], os.environ["PGPASSWORD"]
)
)
return conn
def create_tables(conn: psycopg2.extensions.connection):
"""Create database tables if needed."""
# TODO: add more attributes as needed
# TODO: add all projects as columns
print("Creating tables as needed...")
cur = conn.cursor()
# mod_<project> column: files in the subfolder (=project) <project> were
# modified by this commit.
# git hashes are 40 characters long, so using char(40) data type here
cur.execute(
""" CREATE TABLE IF NOT EXISTS git_commits (
hash char(40) PRIMARY KEY,
commit_time timestamp,
phab_id text,
reverts_hash char(40),
mod_llvm boolean,
mod_clang boolean,
mod_libcxx boolean,
mod_mlir boolean
); """
)
conn.commit()
def get_existing_hashes(conn: psycopg2.extensions.connection) -> Set[str]:
"""Fetch all stored git hashes from the database."""
print("Fetching known git hashes from the database...")
cur = conn.cursor()
cur.execute("SELECT hash from git_commits;")
return set((row[0] for row in cur.fetchall()))
def update_repo(repo_dir: str) -> git.Repo:
"""Clone or fetch local copy of the git repository."""
if os.path.isdir(repo_dir):
print("Fetching git repo...")
repo = git.Repo(repo_dir)
repo.remotes.origin.fetch(GIT_BRANCH)
else:
print("Cloning git repo...")
git.Repo.clone_from(GIT_URL, repo_dir, bare=True)
repo = git.Repo(repo_dir)
print("repo update done.")
return repo
def parse_commits(
conn: psycopg2.extensions.connection, repo: git.Repo, max_age: datetime.datetime
):
"""Parse the git repo history and upload it to the database."""
sql_insert_commit = """ INSERT INTO
git_commits (hash, commit_time, phab_id, reverts_hash)
values (%s,%s,%s,%s);
"""
sql_update_commit_project = (
""" UPDATE git_commits SET mod_{} = %s where hash = %s;"""
)
known_hashes = get_existing_hashes(conn)
day = None
cur = conn.cursor()
for commit in repo.iter_commits(GIT_BRANCH):
# TODO: This takes a couple of minutes, maybe try using multithreading
# Only store new/unknown hashes
if commit.hexsha in known_hashes:
continue
if commit.committed_datetime < max_age:
break
mycommit = MyCommit(commit)
if mycommit.date.day != day:
# take a snapshot commit, nice to see progress while updating the
# database
day = mycommit.date.day
print(mycommit.date)
conn.commit()
cur.execute(
sql_insert_commit,
(
mycommit.chash,
mycommit.date,
mycommit.phab_revision,
mycommit.reverts_commit_hash,
),
)
# Note: prasing the patches is quite slow
for project in mycommit.modified_projects:
# TODO find a way to make this generic for all projects, maybe user
# "ALTER TABLE" to add columns as they appear
# TODO: modifying the commited row is expensive, maybe find something faster
if project in ["llvm", "libcxx", "mlir", "clang"]:
cur.execute(
sql_update_commit_project.format(project), (True, mycommit.chash)
)
conn.commit()
def create_csv_report(title: str, query: str, output_path: str):
cursor = conn.cursor()
data = cursor.execute(query)
with open(os.path.join(output_path, title + ".csv"), "w") as csv_file:
writer = csv.writer(csv_file)
# write column headers
writer.writerow([description[0] for description in cursor.description])
for row in data:
writer.writerow(row)
def run_queries(conn: psycopg2.extensions.connection, output_path: str):
print("running queries...")
create_csv_report("full_db_dump", "select * from commits;", output_path)
query = """SELECT strftime('%Y-%m',commit_time) as month, count(hash) as num_commits, count(phab_id) as num_reviewed,
(100.0*count(phab_id)/count(hash)) as percent_reviewed, count(reverts_hash) as num_reverted,
(100.0*count(reverts_hash)/count(hash)) as percent_reverted
FROM commits
WHERE mod_{}
GROUP BY month;
"""
create_csv_report("libcxx_stats", query.format("libcxx"), output_path)
create_csv_report("mlir_stats", query.format("mlir"), output_path)
query = """SELECT strftime('%Y-%m',commit_time) as month, count(hash) as num_commits, count(phab_id) as num_reviewed,
(100.0*count(phab_id)/count(hash)) as percent_reviewed, count(reverts_hash) as num_reverted,
(100.0*count(reverts_hash)/count(hash)) as percent_reverted
FROM commits
GROUP BY month;
"""
create_csv_report("all_projects_stats", query, output_path)
def update_comits():
"""Update the git commits in the database from the git repository."""
repo = update_repo(REPO_DIR)
conn = connect_to_db()
create_tables(conn)
parse_commits(conn, repo, MAX_AGE)
if __name__ == "__main__":
update_comits()
# TODO: add argparse to switch between import and query mode or
# move queries to another file
# run_queries(conn)

76
scripts/metrics/server_monitoring.py
View file

@ -1,76 +0,0 @@
#!/usr/bin/env python3
import traceback
import psycopg2
from phabricator import Phabricator
import os
from typing import Optional
import datetime
import requests
import logging
PHABRICATOR_URL = "https://reviews.llvm.org/api/"
BUILDBOT_URL = "https://lab.llvm.org/buildbot/api/v2"
def phab_up() -> Optional[Phabricator]:
"""Try to connect to phabricator to see if the server is up.
Returns None if server is down.
"""
logging.info("Checking Phabricator status...")
try:
phab = Phabricator(token=os.getenv('CONDUIT_TOKEN'), host=PHABRICATOR_URL)
phab.update_interfaces()
logging.info("Phabricator is up.")
return phab
except Exception as ex:
logging.error(ex)
logging.error(traceback.format_exc())
logging.warning("Phabricator is down.")
return None
def buildbot_up() -> bool:
"""Check if buildbot server is up"""
logging.info("Checking Buildbot status...")
try:
response = requests.get(BUILDBOT_URL)
logging.info(f'{response.status_code} {BUILDBOT_URL}')
logging.info(response.content)
return response.status_code == 200
except Exception as ex:
logging.error(ex)
logging.error(traceback.format_exc())
logging.warning("Buildbot is down.")
return False
def log_server_status(phab: bool, buildbot: bool, conn: psycopg2.extensions.connection):
"""log the phabricator status to the database."""
logging.info("Writing Phabricator status to database...")
cur = conn.cursor()
cur.execute(
"INSERT INTO server_status (timestamp, phabricator, buildbot) VALUES (%s,%s,%s);",
(datetime.datetime.now(), phab, buildbot),
)
conn.commit()
def connect_to_db() -> psycopg2.extensions.connection:
"""Connect to the database, create tables as needed."""
conn = psycopg2.connect(
f"host=127.0.0.1 sslmode=disable dbname=stats user=stats password={os.getenv('DB_PASSWORD')}")
cur = conn.cursor()
cur.execute(
"CREATE TABLE IF NOT EXISTS server_status (timestamp timestamp, phabricator boolean, buildbot boolean);"
)
conn.commit()
return conn
if __name__ == "__main__":
logging.basicConfig(level='INFO', format='%(levelname)-7s %(message)s')
conn = connect_to_db()
phab = phab_up()
buildbot = buildbot_up()
log_server_status(phab is not None, buildbot, conn)

326
scripts/patch_diff.py
View file

@ -1,326 +0,0 @@
#!/usr/bin/env python3
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse
import datetime
import logging
import os
import re
import subprocess
import sys
from typing import List, Optional, Tuple, Dict
import backoff
from buildkite_utils import annotate, feedback_url, upload_file
import git
from phabricator import Phabricator
"""URL of upstream LLVM repository."""
LLVM_GITHUB_URL = 'ssh://git@github.com/llvm/llvm-project'
FORK_REMOTE_URL = 'ssh://git@github.com/llvm-premerge-tests/llvm-project'
class ApplyPatch:
"""Apply a diff from Phabricator on local working copy.
This script is a rewrite of `arc patch` to accommodate for dependencies
that have already landed, but could not be identified by `arc patch`.
For a given diff_id, this class will get the dependencies listed on Phabricator.
For each dependency D it will check the it's status:
- if D is closed, skip it.
- If D is not closed, it will download the patch for D and try to apply it locally.
Once this class has applied all dependencies, it will apply the original diff.
This script must be called from the root folder of a local checkout of
https://github.com/llvm/llvm-project or given a path to clone into.
"""
def __init__(self, path: str, diff_id: int, token: str, url: str, git_hash: str,
phid: str, push_branch: bool = False):
self.push_branch = push_branch # type: bool
self.conduit_token = token # type: Optional[str]
self.host = url # type: Optional[str]
self.diff_id = diff_id # type: int
self.phid = phid # type: str
if not self.host.endswith('/api/'):
self.host += '/api/'
self.phab = self.create_phab()
self.base_revision = git_hash # type: str
self.branch_base_hexsha = ''
self.apply_diff_counter = 0
self.build_dir = os.getcwd()
self.revision_id = ''
if not os.path.isdir(path):
logging.info(f'{path} does not exist, cloning repository...')
self.repo = git.Repo.clone_from(FORK_REMOTE_URL, path)
else:
logging.info('repository exist, will reuse')
self.repo = git.Repo(path) # type: git.Repo
self.repo.remote('origin').set_url(FORK_REMOTE_URL)
os.chdir(path)
logging.info(f'working dir {os.getcwd()}')
@property
def branch_name(self):
"""Name used for the git branch."""
return f'phab-diff-{self.diff_id}'
def run(self):
"""try to apply the patch from phabricator
"""
try:
diff = self.get_diff(self.diff_id)
revision = self.get_revision(diff.revisionID)
url = f"https://reviews.llvm.org/D{revision['id']}?id={diff['id']}"
annotate(f"Patching changes [{url}]({url})", style='info', context='patch_diff')
self.reset_repository()
self.revision_id = revision['id']
dependencies = self.get_dependencies(revision)
dependencies.reverse() # Now revisions will be from oldest to newest.
if len(dependencies) > 0:
logging.info('This diff depends on: {}'.format(revision_list_to_str(dependencies)))
plan = []
for r in dependencies:
if r['statusName'] == 'Closed':
logging.info(f'skipping revision {r["id"]} - it is closed, assuming it has landed')
continue
d = self.get_diff(r['diffs'][0])
plan.append((r, d))
plan.append((revision, diff))
logging.info('Planning to apply in order:')
for (r, d) in plan:
logging.info(f"https://reviews.llvm.org/D{r['id']}?id={d['id']}")
# Pick the newest known commit as a base for patches.
base_commit = None
for (r, d) in plan:
c = self.find_commit(d['sourceControlBaseRevision'])
if c is None:
logging.warning(f"D{r['id']}#{d['id']} commit {d['sourceControlBaseRevision']} does not exist")
continue
if base_commit is None:
logging.info(f"D{r['id']}#{d['id']} commit {c.hexsha} exists")
base_commit = c
elif c.committed_datetime > base_commit.committed_datetime:
logging.info(f"D{r['id']}#{d['id']} commit {c.hexsha} has a later commit date then"
f"{base_commit.hexsha}")
base_commit = c
if self.base_revision != 'auto':
logging.info(f'Base revision "{self.base_revision}" is set by command argument. Will use '
f'instead of resolved "{base_commit}"')
base_commit = self.find_commit(self.base_revision)
if base_commit is None:
base_commit = self.repo.heads['main'].commit
annotate(f"Cannot find a base git revision. Will use current HEAD.",
style='warning', context='patch_diff')
self.create_branch(base_commit)
for (r, d) in plan:
if not self.apply_diff(d, r):
return 1
if self.push_branch:
self.repo.git.push('--force', 'origin', self.branch_name)
annotate(f"Created branch [{self.branch_name}]"
f"(https://github.com/llvm-premerge-tests/llvm-project/tree/{self.branch_name}).\n\n"
f"To checkout locally, run in your copy of llvm-project directory:\n\n"
"```shell\n"
"git remote add premerge git@github.com:llvm-premerge-tests/llvm-project.git #first time\n"
f"git fetch premerge {self.branch_name}\n"
f"git checkout -b {self.branch_name} --track premerge/{self.branch_name}\n"
"```",
style='success',
context='patch_diff')
logging.info('Branch {} has been pushed'.format(self.branch_name))
return 0
except Exception as e:
annotate(f":bk-status-failed: Unexpected error. Consider [creating a bug]({feedback_url()}).",
style='error', context='patch_diff')
logging.error(f'exception: {e}')
return 1
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def reset_repository(self):
"""Update local git repo and origin.
As origin is disjoint from upstream, it needs to be updated by this script.
"""
# Remove index lock just in case.
lock_file = f"{self.repo.working_tree_dir}/.git/index.lock"
try:
os.remove(lock_file)
logging.info(f"removed {lock_file}")
except FileNotFoundError:
logging.info(f"{lock_file} does not exist")
logging.info('Syncing local, origin and upstream...')
if 'upstream' not in self.repo.remotes:
self.repo.create_remote('upstream', url=LLVM_GITHUB_URL)
self.repo.remotes.upstream.fetch()
self.repo.git.clean('-ffxdq')
self.repo.git.reset('--hard')
self.repo.git.fetch('--all')
if self.find_commit('main') is None:
origin = self.repo.remotes.origin
self.repo.create_head('main', origin.refs.main)
self.repo.heads.main.set_tracking_branch(origin.refs.main)
self.repo.heads.main.checkout()
self.repo.git.pull('origin', 'main')
self.repo.git.pull('upstream', 'main')
if self.push_branch:
self.repo.git.push('origin', 'main')
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def find_commit(self, rev):
try:
return self.repo.commit(rev)
except:
return None
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def create_branch(self, base_commit: git.Commit):
if self.branch_name in self.repo.heads:
self.repo.delete_head('--force', self.branch_name)
logging.info(f'creating branch {self.branch_name} at {base_commit.hexsha}')
new_branch = self.repo.create_head(self.branch_name, base_commit.hexsha)
self.repo.head.reference = new_branch
self.repo.head.reset(index=True, working_tree=True)
self.branch_base_hexsha = self.repo.head.commit.hexsha
logging.info('Base branch revision is {}'.format(self.repo.head.commit.hexsha))
annotate(f"Branch {self.branch_name} base revision is `{self.branch_base_hexsha}`.",
style='info', context='patch_diff')
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def commit(self, revision: Dict, diff: Dict):
"""Commit the current state and annotates with the revision info."""
self.repo.git.add('-A')
diff.setdefault('authorName', 'unknown')
diff.setdefault('authorEmail', 'unknown')
author = git.Actor(name=diff['authorName'], email=diff['authorEmail'])
message = (f"{revision['title']}\n\n"
f"Automated commit created by applying diff {self.diff_id}\n"
f"\n"
f"Phabricator-ID: {self.phid}\n"
f"Review-ID: {diff_to_str(revision['id'])}\n")
self.repo.index.commit(message=message, author=author)
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def create_phab(self):
phab = Phabricator(token=self.conduit_token, host=self.host)
phab.update_interfaces()
return phab
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def get_diff(self, diff_id: int):
"""Get a diff from Phabricator based on its diff id."""
return self.phab.differential.getdiff(diff_id=diff_id)
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def get_revision(self, revision_id: int):
"""Get a revision from Phabricator based on its revision id."""
return self.phab.differential.query(ids=[revision_id])[0]
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def get_revisions(self, *, phids: List[str] = None):
"""Get a list of revisions from Phabricator based on their PH-IDs."""
if phids is None:
raise Exception('_get_revisions phids is None')
if not phids:
# Handle an empty query locally. Otherwise the connection
# will time out.
return []
return self.phab.differential.query(phids=phids)
def get_dependencies(self, revision: Dict) -> List[Dict]:
"""Recursively resolves dependencies of the given revision.
They are listed in reverse chronological order - from most recent to least recent."""
dependency_ids = revision['auxiliary']['phabricator:depends-on']
revisions = self.get_revisions(phids=dependency_ids)
result = []
for r in revisions:
result.append(r)
sub = self.get_dependencies(r)
result.extend(sub)
return result
def apply_diff(self, diff: Dict, revision: Dict) -> bool:
"""Download and apply a diff to the local working copy."""
logging.info(f"Applying {diff['id']} for revision {revision['id']}...")
patch = self.get_raw_diff(str(diff['id']))
self.apply_diff_counter += 1
patch_file = f"{self.apply_diff_counter}_{diff['id']}.patch"
with open(os.path.join(self.build_dir, patch_file), 'wt') as f:
f.write(patch)
# For annotate to properly link this file it must exist before the upload.
upload_file(self.build_dir, patch_file)
logging.debug(f'raw patch:\n{patch}')
proc = subprocess.run('git apply -', input=patch, shell=True, text=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if proc.returncode != 0:
logging.info(proc.stdout)
logging.error(proc.stderr)
message = f":bk-status-failed: Failed to apply [{patch_file}](artifact://{patch_file}).\n\n"
if self.revision_id != revision['id']:
message += f"**Attention! D{revision['id']} is one of the dependencies of the target " \
f"revision D{self.revision_id}.**\n\n"
message += (f"No testing is possible because we couldn't apply the patch.\n\n"
f"---\n\n"
'### Troubleshooting\n\n'
'More information is available in the log of of *create branch* step. '
f"All patches applied are available as *Artifacts*.\n\n"
f":bulb: The patch may not apply if it includes only the most recent of "
f"multiple local commits. Try to upload a patch with\n"
f"```shell\n"
f"arc diff `git merge-base HEAD origin` --update D{revision['id']}\n"
f"```\n\n"
f"to include all local changes.\n\n"
'---\n\n'
f"If this case could have been handled better, please [create a bug]({feedback_url()}).")
annotate(message,
style='error',
context='patch_diff')
return False
self.commit(revision, diff)
return True
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def get_raw_diff(self, diff_id: str) -> str:
return self.phab.differential.getrawdiff(diffID=diff_id).response
def diff_to_str(diff: int) -> str:
"""Convert a diff id to a string with leading "D"."""
return 'D{}'.format(diff)
def revision_list_to_str(diffs: List[Dict]) -> str:
"""Convert list of diff ids to a comma separated list, prefixed with "D"."""
return ', '.join([diff_to_str(d['id']) for d in diffs])
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Apply Phabricator patch to working directory.')
parser.add_argument('diff_id', type=int)
parser.add_argument('--path', type=str, help='repository path', default=os.getcwd())
parser.add_argument('--token', type=str, default=None, help='Conduit API token')
parser.add_argument('--url', type=str, default='https://reviews.llvm.org', help='Phabricator URL')
parser.add_argument('--commit', dest='commit', type=str, default='auto',
help='Use this commit as a base. For "auto" tool tries to pick the base commit itself')
parser.add_argument('--push-branch', action='store_true', dest='push_branch',
help='choose if branch shall be pushed to origin')
parser.add_argument('--phid', type=str, default=None, help='Phabricator ID of the review this commit pertains to')
parser.add_argument('--log-level', type=str, default='INFO')
args = parser.parse_args()
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
patcher = ApplyPatch(args.path, args.diff_id, args.token, args.url, args.commit, args.phid, args.push_branch)
sys.exit(patcher.run())

1
scripts/phab2github/.gitignore vendored
View file

@ -1 +0,0 @@
/tmp

70
scripts/phab2github/README.md
View file

@ -1,70 +0,0 @@
This folder contains a set of scripts to mirror Phabricator *Revisions* to GitHub *Pull Requests*.
# Problem statement
The LLVM project performs code reviews on [Phabricator](https://reviews.llvm.org) and the source
code is stored on [GitHub](https://github.com/llvm/llvm-project). While there are many nice CI
system integrations available for GitHub (e.g. buildkite, CircleCI, GitHub Actions), there are not
that many for Phabricator. The current setup of the pre-merge tests is quite some effort to maintain
and does not scale well with additional build agents. User do not have access to the bulid status.
One of the challenges is that Phabricator maintains patches, that might not be based on a git
commit. And even if they are based on a git commit, that commit might be local to the user and
not available in the public repository.
# Proposal
Instead of integrating a CI system with Phabricator, move all the information to Github and trigger
the builds from there.
1. Create a service that mirrors all Revisions form Phabricator into GitHub Pull Requests.
1. Then use the usual CI integration for GitHub to build and test these revisions.
1. On Phabricator, add links to the builds.
1. At the end of the build, a script uploads the results to Phabricator (as we have it now).
If that works well, we can also use this to demonstrate the workflow of Github Pull Requests to the
LLVM community.
# current status
The script currently does:
* Get the latest 10 Revisions from Phabricator for testing.
* For each revision:
* Create a local git branch `phab-Dxxxxx`
* Create a local git branch `phab-diff-yyyyy` for every Diff of the Revision using the raw patch of each Diff.
* Layer the diffs on top of each other in the `phab-Dxxxxxx` branch.
* Push the `phab-Dxxxxxx` branch to the [LLVM fork](https://github.com/ChristianKuehnel/llvm-project).
* Create a [pull request](https://github.com/ChristianKuehnel/llvm-project/pulls) for each branch.
* The GitHub/Buildkite
[CI integration](https://buildkite.com/llvm-project/phabricator-pull-request-integration-prototype)
then triggers the builds for the PRs.
As this is a prototype, it currently has some shortcommings:
* I'm only looking at the 10 latest Revisions for testing.
* If a patch does not apply: just ignore it
* The Revision branches are always created from scratch, there is no incremental update.
* I run the script manually for testing, There are no automatic pull/push updates
# Work items
This is the list (and order) of the work items for this idea.
We will start with this on a fork of the LLVM repository running on my machine as cron job.
Cancel the effort if some thinks would not work or the community objects.
* [x] For each Revision create a branch. Create a PR from that.
* [x] Layer the diffs on top of each other in the git log.
* [ ] Figure out what to do with chained Revisions.
* [ ] Find a scalable way to reports results back to Phabricator.
* [ ] Clean up old branches and PRs.
* [ ] Create pre-merge testing service to the cloud, trigger it with every change on Phabricator.
Also do regular full scans of Phabricator to catch everything lost in the triggers.
* [ ] Add more build machines to buildkite and move beta testers to new infrastructure.
* [ ] Move all pre-merge tests to new infrastructure.
* [ ] Find way to test new scripts/builders/configuration/... without annoying the users.
* [ ] Enable others to add more checks/builders as needed.
Optional:
* [ ] Also sync Revisions that do not trigger Harbormaster at the moment and provide feedback as
well.
* [ ] Find a way to copy the description and the comments from the Revision to the PR.
* [ ] Create branches and PRs on LLVM repository, allow people to click the "Merge" button on the PR,
reducing the manual work to land a patch.

0
scripts/phab2github/__init__.py
View file

216
scripts/phab2github/phab2github.py
View file

@ -1,216 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
from typing import Optional, Union
import git
import logging
from phab_wrapper import PhabWrapper, Revision
import subprocess
import github
import json
import sys
# TODO: move to config file
LLVM_GITHUB_URL = 'ssh://git@github.com/llvm/llvm-project'
MY_GITHUB_URL = 'ssh://git@github.com/christiankuehnel/llvm-project'
MY_REPO = 'christiankuehnel/llvm-project'
_LOGGER = logging.getLogger()
class ApplyPatchException(Exception):
"""A patch could not be applied."""
pass
class Phab2Github:
def __init__(self, workdir: str):
self.workdir = workdir
self.llvm_dir = os.path.join(self.workdir, 'llvm-project')
self.repo = None # type: Optional[git.Repo]
self.phab_wrapper = PhabWrapper()
self.github = self._create_github()
self.github_repo = self.github.get_repo(MY_REPO) # type: github.Repository
def sync(self):
"""Sync Phabricator to Github."""
_LOGGER.info('Starting sync...')
self._refresh_main()
self._delete_phab_branches()
revisions = self.phab_wrapper.get_revisions()
pull_requests = {p.title: p for p in self.github_repo.get_pulls(state='open')}
for revision in revisions:
self.create_branches_for_revision(revision)
if self._has_branch(revision):
if self._branches_identical(revision.branch_name, 'origin/{}'.format(revision.branch_name)):
_LOGGER.info('Branch {} is identical to upstream. Not pushing.'.format(revision.branch_name))
else:
_LOGGER.info('Pushing branch {} to github...'.format(revision.branch_name))
# TODO: do we sill need to force-push?
self.repo.git.push('--force', 'origin', revision.branch_name)
if revision.pr_title in pull_requests:
_LOGGER.info('Pull request already exists: {}'.format(pull_requests[revision.pr_title].html_url))
else:
_LOGGER.info('Creating pull-request for branch {}...'.format(revision.branch_name))
pr = self.github_repo.create_pull(title=revision.pr_title,
body=revision.pr_summary,
head=revision.branch_name,
base='main')
_LOGGER.info(pr.html_url)
_LOGGER.info('Sync completed.')
def _refresh_main(self):
"""Clone/update local git repo."""
if not os.path.exists(self.workdir):
os.mkdir(self.workdir)
if os.path.exists(self.llvm_dir):
# TODO: in case of errors: delete and clone
_LOGGER.info('pulling origin and upstream...')
self.repo = git.Repo(self.llvm_dir)
self.repo.git.fetch('--all')
self.repo.git.checkout('main')
self.repo.git.pull('upstream', 'main')
self.repo.git.push('origin', 'main')
else:
_LOGGER.info('cloning repository...')
git.Repo.clone_from(MY_GITHUB_URL, self.llvm_dir)
self.repo = git.Repo(self.llvm_dir)
self.repo.create_remote('upstream', url=LLVM_GITHUB_URL)
self.repo.remotes.upstream.fetch()
_LOGGER.info('refresh of main branch completed')
def create_branches_for_revision(self, revision: Revision):
"""Create branches for a Revision and it's Diffs.
Apply changes from Phabricator to these branches.
"""
name = revision.branch_name
# TODO: only look at diffs that were not yet applied
# TODO: can diffs be modified on Phabricator and keep their ID?
for diff in revision.sorted_diffs:
if self._has_branch(diff):
continue
self.create_branch_for_diff(diff)
patch = self.phab_wrapper.get_raw_patch(diff)
try:
self.apply_patch(diff, patch)
except ApplyPatchException as e:
# TODO: retry on main if this fails
_LOGGER.error('Could not apply patch for Diff {}. Deleting branch'.format(diff.id))
_LOGGER.exception(e)
self.repo.heads['main'].checkout()
self.repo.delete_head(diff.branch_name)
diffs = [d for d in revision.sorted_diffs if self._has_branch(d)]
if len(diffs) == 0:
# TODO: handle error
_LOGGER.error('Could not create branch for Revision D{}'.format(revision.id))
return
new_branch = self.repo.create_head(revision.branch_name, diffs[0].branch_name)
new_branch.checkout()
for diff in diffs[1:]:
_LOGGER.info('Applying Diff {} onto branch {}'.format(diff.branch_name, revision.branch_name))
patch = self._create_patch(revision.branch_name, diff.branch_name)
if len(patch) == 0:
_LOGGER.warning('Diff {} is identical to last one.'.format(diff.id))
else:
try:
self.apply_patch(diff, patch)
except ApplyPatchException:
_LOGGER.error('Applying patch failed, but should not:')
_LOGGER.error(patch)
raise
_LOGGER.info('Created branch for Revision D{}'.format(revision.id))
def create_branch_for_diff(self, diff: "Diff"):
"""Create a branch for diff."""
base_hash = diff.base_hash
if base_hash is None:
base_hash = 'upstream/main'
_LOGGER.info('creating branch {} based on {}...'.format(diff.branch_name, base_hash))
try:
new_branch = self.repo.create_head(diff.branch_name, base_hash)
except ValueError:
# commit hash not found, try again with main
_LOGGER.warning('commit hash {} not found in upstream repository. '
'Trying main instead...'.format(diff.branch_name, base_hash))
base_hash = 'upstream/main'
new_branch = self.repo.create_head(diff.branch_name, base_hash)
self.repo.head.reference = new_branch
self.repo.head.reset(index=True, working_tree=True)
def apply_patch(self, diff: "Diff", raw_patch: str):
"""Apply a patch to the working copy."""
# TODO: impersonate the original author of the patch
proc = subprocess.run('git apply --ignore-whitespace --whitespace=fix -', input=raw_patch, shell=True,
text=True, cwd=self.repo.working_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if proc.returncode != 0:
raise ApplyPatchException('Applying patch failed:\n{}'.format(proc.stdout + proc.stderr))
self.repo.git.add('-A')
self.repo.index.commit(message='applying Diff {} for Revision D{}\n\n Diff: {}'.format(
diff.id, diff.revision.id, diff.id))
@staticmethod
def _create_github() -> github.Github:
"""Create instance of Github client.
Reads access token from a file.
"""
with open(os.path.expanduser('~/.llvm-premerge-checks/github-token.json')) as json_file:
token = json.load(json_file)['token']
return github.Github(token)
def _has_branch(self, item: Union["Diff", "Revision"]) -> bool:
"""Check if the Diff/Revision has a local branch."""
return item.branch_name in self.repo.heads
def _delete_phab_branches(self):
"""Delete all branches sarting with 'phab-'."""
_LOGGER.info('Deleting local Phabricator-relates branches...')
self.repo.git.checkout('main')
for branch in [b for b in self.repo.heads if b.name.startswith('phab-')]:
_LOGGER.info('Deleding branch {}'.format(branch))
self.repo.git.branch('-D', branch.name)
def _create_patch(self, base: str, changes: str) -> str:
"""Create patch from two commits."""
proc = subprocess.run('git diff {} {}'.format(base, changes), shell=True, text=True,
cwd=self.repo.working_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if proc.returncode != 0:
raise ApplyPatchException('Applying patch failed:\n{}'.format(proc.stdout + proc.stderr))
return proc.stdout
def _branches_identical(self, left, right) -> bool:
"""Check if two branches are identical."""
try:
patch = self.repo.git.diff(left, right)
except git.GitCommandError:
return False
if len(patch) == 0:
return True
return False
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
rootdir = os.path.dirname(os.path.abspath(__file__))
tmpdir = os.path.join(rootdir, 'tmp')
p2g = Phab2Github(tmpdir)
p2g.sync()

175
scripts/phab2github/phab_wrapper.py
View file

@ -1,175 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
import os
import json
from phabricator import Phabricator
from typing import List, Optional, Dict
import datetime
_BASE_URL = 'https://reviews.llvm.org'
_LOGGER = logging.getLogger()
class Revision:
"""A Revision on Phabricator"""
def __init__(self, revision_dict: Dict):
self.revision_dict = revision_dict
self.diffs = [] # type : "Diff"
self.phab_url = '{}/D{}'.format(_BASE_URL, self.id) # type: str
@property
def id(self) -> str:
return self.revision_dict['id']
@property
def phid(self) -> str:
return self.revision_dict['phid']
@property
def status(self) -> str:
return self.revision_dict['fields']['status']['value']
def __str__(self):
return 'Revision {}: {} - ({})'.format(self.id, self.status,
','.join([str(d.id) for d in self.diffs]))
@property
def branch_name(self) -> str:
return 'phab-D{}'.format(self.id)
@property
def title(self) -> str:
return self.revision_dict['fields']['title']
@property
def pr_title(self) -> str:
return 'D{}: {}'.format(self.id, self.title)
@property
def summary(self) -> str:
return self.revision_dict['fields']['summary']
@property
def pr_summary(self) -> str:
return '{}\n\n{}'.format(self.summary, self.phab_url)
@property
def sorted_diffs(self) -> List["Diff"]:
return sorted(self.diffs, key=lambda d: d.id)
class Diff:
"""A Phabricator diff."""
def __init__(self, diff_dict: Dict, revision: Revision):
self.diff_dict = diff_dict
self.revision = revision
# TODO: check in git repo instead of using a flag
@property
def id(self) -> str:
return self.diff_dict['id']
@property
def phid(self) -> str:
return self.diff_dict['phid']
def __str__(self):
return 'Diff {}'.format(self.id)
@property
def base_hash(self) -> Optional[str]:
for ref in self.diff_dict['fields']['refs']:
if ref['type'] == 'base':
return ref['identifier']
return None
@property
def branch_name(self) -> str:
return 'phab-diff-{}'.format(self.id)
class PhabWrapper:
"""
Wrapper around the interactions with Phabricator.
Conduit API documentation: https://reviews.llvm.org/conduit/
"""
def __init__(self):
self.conduit_token = None # type: Optional[str]
self.host = None # type: Optional[str]
self._load_arcrc()
self.phab = self._create_phab() # type: Phabricator
def _load_arcrc(self):
"""Load arc configuration from file if not set."""
_LOGGER.info('Loading configuration from ~/.arcrc file')
with open(os.path.expanduser('~/.arcrc'), 'r') as arcrc_file:
arcrc = json.load(arcrc_file)
# use the first host configured in the file
self.host = next(iter(arcrc['hosts']))
self.conduit_token = arcrc['hosts'][self.host]['token']
def _create_phab(self) -> Phabricator:
"""Create Phabricator API instance and update it."""
phab = Phabricator(token=self.conduit_token, host=self.host)
# TODO: retry on communication error
phab.update_interfaces()
return phab
def get_revisions(self) -> List[Revision]:
"""Get relevant revisions."""
# TODO: figure out which revisions we really need to pull in
_LOGGER.info('Getting revisions from Phabricator')
start_date = datetime.datetime.now() - datetime.timedelta(days=3)
constraints = {
'createdStart': int(start_date.timestamp())
#'ids': [76120]
}
# TODO: handle > 100 responses
revision_response = self.phab.differential.revision.search(
constraints=constraints)
revisions = [Revision(r) for r in revision_response.response['data']]
# TODO: only taking the first 10 to speed things up
revisions = revisions[0:10]
_LOGGER.info('Got {} revisions from the server'.format(len(revisions)))
for revision in revisions:
# TODO: batch-query diffs for all revisions, reduce number of
# API calls this would be much faster. But then we need to locally
# map the diffs to the right revisions
self._get_diffs(revision)
return revisions
def _get_diffs(self, revision: Revision):
"""Get diffs for a revision from Phabricator."""
_LOGGER.info('Downloading diffs for Revision D{}...'.format(revision.id))
constraints = {
'revisionPHIDs': [revision.phid]
}
diff_response = self.phab.differential.diff.search(
constraints=constraints)
revision.diffs = [Diff(d, revision) for d in diff_response.response['data']]
_LOGGER.info(', '.join([str(d.id) for d in revision.diffs]))
def get_raw_patch(self, diff: Diff) -> str:
"""Get raw patch for diff from Phabricator."""
_LOGGER.info('Downloading patch for Diff {}...'.format(diff.id))
return self.phab.differential.getrawdiff(diffID=str(diff.id)).response

4
scripts/phab2github/requirements.txt
View file

@ -1,4 +0,0 @@
phabricator==0.7.0
gitpython==3.0.6
retrying==1.3.3
PyGitHub=1.46

2
scripts/phabtalk/.gitignore vendored
View file

@ -1,2 +0,0 @@
venv
test-data/

4
scripts/phabtalk/README.md
View file

@ -1,4 +0,0 @@
This folder contains Python scripts that talk to Phabricator.
They require a few libraries listed in `requirements.txt`.
To install the requirements locally run `pip3 install -r requirements.txt`.

0
scripts/phabtalk/__init__.py
View file

138
scripts/phabtalk/apply_patch.py
View file

@ -1,138 +0,0 @@
#!/usr/bin/env python3
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse
import json
import os
import subprocess
import sys
from typing import List, Optional
import backoff
from phabricator import Phabricator
class ApplyPatch:
def __init__(self, comment_file_path: str, git_hash: str):
# TODO: turn os.environ parameter into command line arguments
# this would be much clearer and easier for testing
self.comment_file_path = comment_file_path
self.conduit_token = os.environ.get('CONDUIT_TOKEN') # type: Optional[str]
self.host = os.environ.get('PHABRICATOR_HOST') # type: Optional[str]
self._load_arcrc()
self.diff_id = os.environ['DIFF_ID'] # type: str
self.diff_json_path = os.environ['DIFF_JSON'] # type: str
if not self.host.endswith('/api/'):
self.host += '/api/'
self.phab = Phabricator(token=self.conduit_token, host=self.host)
self.git_hash = git_hash # type: Optional[str]
self.msg = [] # type: List[str]
def _load_arcrc(self):
"""Load arc configuration from file if not set."""
if self.conduit_token is not None or self.host is not None:
return
print('Loading configuration from ~/.arcrc file')
with open(os.path.expanduser('~/.arcrc'), 'r') as arcrc_file:
arcrc = json.load(arcrc_file)
# use the first host configured in the file
self.host = next(iter(arcrc['hosts']))
self.conduit_token = arcrc['hosts'][self.host]['token']
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def update_interfaces(self):
self.phab.update_interfaces()
def run(self):
"""try to apply the patch from phabricator
"""
self.update_interfaces()
try:
if self.git_hash is None:
self._get_parent_hash()
else:
print('Use provided commit {}'.format(self.git_hash))
self._git_checkout()
self._apply_patch()
finally:
self._write_error_message()
def _get_parent_hash(self):
diff = self.phab.differential.getdiff(diff_id=self.diff_id)
# Keep a copy of the Phabricator answer for later usage in a json file
try:
with open(self.diff_json_path, 'w') as json_file:
json.dump(diff.response, json_file, sort_keys=True, indent=4)
print('Wrote diff details to "{}".'.format(self.diff_json_path))
except Exception:
print('WARNING: could not write build/diff.json log file')
self.git_hash = diff['sourceControlBaseRevision']
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def _git_checkout(self):
try:
print('Checking out git hash {}'.format(self.git_hash))
subprocess.check_call('git reset --hard {}'.format(self.git_hash),
stdout=sys.stdout, stderr=sys.stderr, shell=True)
except subprocess.CalledProcessError:
print('WARNING: checkout of hash failed, using main branch instead.')
self.msg += [
'Could not check out parent git hash "{}". It was not found in '
'the repository. Did you configure the "Parent Revision" in '
'Phabricator properly? Trying to apply the patch to the '
'main branch instead...'.format(self.git_hash)]
subprocess.check_call('git checkout main', stdout=sys.stdout,
stderr=sys.stderr, shell=True)
subprocess.check_call('git show -s', stdout=sys.stdout,
stderr=sys.stderr, shell=True)
print('git checkout completed.')
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def _apply_patch(self):
print('running arc patch...')
cmd = 'arc patch --force --nobranch --no-ansi --diff "{}" --nocommit ' \
'--conduit-token "{}" --conduit-uri "{}"'.format(
self.diff_id, self.conduit_token, self.host)
result = subprocess.run(cmd, capture_output=True, shell=True, text=True)
print(result.stdout + result.stderr)
if result.returncode != 0:
msg = (
'ERROR: arc patch failed with error code {}. '
'Check build log for details.'.format(result.returncode))
self.msg += [msg]
raise Exception(msg)
print('Patching completed.')
def _write_error_message(self):
"""Write the log message to a file."""
if self.comment_file_path is None:
return
if len(self.msg) == 0:
return
print('writing error message to {}'.format(self.comment_file_path))
with open(self.comment_file_path, 'a') as comment_file:
text = '\n\n'.join(self.msg)
comment_file.write(text)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Apply Phabricator patch to working directory.')
parser.add_argument('--comment-file', type=str, dest='comment_file_path', default=None)
parser.add_argument('--commit', type=str, dest='commit', default=None, help='use explicitly specified base commit')
args = parser.parse_args()
patcher = ApplyPatch(args.comment_file_path, args.commit)
patcher.run()

226
scripts/phabtalk/phabtalk.py
View file

@ -1,226 +0,0 @@
#!/usr/bin/env python3
# Copyright 2019 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Interactions with Phabricator.
"""
import logging
from typing import Optional, List, Dict
import uuid
import argparse
import backoff
from phabricator import Phabricator
from benedict import benedict
class PhabTalk:
"""Talk to Phabricator to upload build results.
See https://secure.phabricator.com/conduit/method/harbormaster.sendmessage/
You might want to use it as it provides retries on most of the calls.
"""
def __init__(self, token: Optional[str], host: Optional[str] = 'https://reviews.llvm.org/api/',
dry_run_updates: bool = False):
self._phab = None # type: Optional[Phabricator]
self.dry_run_updates = dry_run_updates
self._phab = Phabricator(token=token, host=host)
self.update_interfaces()
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def update_interfaces(self):
self._phab.update_interfaces()
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def get_revision_id(self, diff: str) -> Optional[str]:
"""Get the revision ID for a diff from Phabricator."""
result = self._phab.differential.querydiffs(ids=[diff])
return 'D' + result[diff]['revisionID']
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def get_diff(self, diff_id: int):
"""Get a diff from Phabricator based on its diff id."""
return self._phab.differential.getdiff(diff_id=diff_id)
def comment_on_diff(self, diff_id: str, text: str):
"""Add a comment to a differential based on the diff_id"""
logging.info('Sending comment to diff {}:'.format(diff_id))
logging.info(text)
revision_id = self.get_revision_id(diff_id)
if revision_id is not None:
self._comment_on_revision(revision_id, text)
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def _comment_on_revision(self, revision: str, text: str):
"""Add comment on a differential based on the revision id."""
transactions = [{
'type': 'comment',
'value': text
}]
if self.dry_run_updates:
logging.info('differential.revision.edit =================')
logging.info('Transactions: {}'.format(transactions))
return
# API details at
# https://secure.phabricator.com/conduit/method/differential.revision.edit/
self._phab.differential.revision.edit(objectIdentifier=revision,
transactions=transactions)
logging.info('Uploaded comment to Revision D{}:{}'.format(revision, text))
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def update_build_status(self, phid: str, working: bool, success: bool, lint: {}, unit: []):
"""Submit collected report to Phabricator.
"""
result_type = 'pass'
if working:
result_type = 'working'
elif not success:
result_type = 'fail'
# Group lint messages by file and line.
lint_messages = []
for v in lint.values():
path = ''
line = 0
descriptions = []
for e in v:
path = e['path']
line = e['line']
descriptions.append('{}: {}'.format(e['name'], e['description']))
lint_message = {
'name': 'Pre-merge checks',
'severity': 'warning',
'code': 'llvm-premerge-checks',
'path': path,
'line': line,
'description': '\n'.join(descriptions),
}
lint_messages.append(lint_message)
if self.dry_run_updates:
logging.info('harbormaster.sendmessage =================')
logging.info('type: {}'.format(result_type))
logging.info('unit: {}'.format(unit))
logging.info('lint: {}'.format(lint_messages))
return
self._phab.harbormaster.sendmessage(
buildTargetPHID=phid,
type=result_type,
unit=unit,
lint=lint_messages)
logging.info('Uploaded build status {}, {} test results and {} lint results'.format(
result_type, len(unit), len(lint_messages)))
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def create_artifact(self, phid, artifact_key, artifact_type, artifact_data):
if self.dry_run_updates:
logging.info('harbormaster.createartifact =================')
logging.info('artifactKey: {}'.format(artifact_key))
logging.info('artifactType: {}'.format(artifact_type))
logging.info('artifactData: {}'.format(artifact_data))
return
self._phab.harbormaster.createartifact(
buildTargetPHID=phid,
artifactKey=artifact_key,
artifactType=artifact_type,
artifactData=artifact_data)
def maybe_add_url_artifact(self, phid: str, url: str, name: str):
if self.dry_run_updates:
logging.info(f'add ULR artifact "{name}" {url}')
return
if phid is None:
logging.warning('PHID is not provided, cannot create URL artifact')
return
self.create_artifact(phid, str(uuid.uuid4()), 'uri', {'uri': url, 'ui.external': True, 'name': name})
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def user_projects(self, user_phid: str) -> List[str]:
"""Returns slugs of all projects user has a membership."""
projects = benedict(self._phab.project.search(constraints={'members': [user_phid]}))
slugs = []
for p in projects.get('data', []):
slug = benedict(p).get('fields.slug')
if slug:
slugs.append(p['fields']['slug'])
return slugs
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def get_revision(self, revision_id: int):
"""Get a revision from Phabricator based on its revision id."""
return self._phab.differential.query(ids=[revision_id])[0]
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
def get_diff(self, diff_id: int):
"""Get a diff from Phabricator based on its diff id."""
return self._phab.differential.getdiff(diff_id=diff_id)
class Step:
def __init__(self, name=''):
self.name = name
self.success = True
self.duration = 0.0
self.reproduce_commands = []
def set_status_from_exit_code(self, exit_code: int):
if exit_code != 0:
self.success = False
class Report:
def __init__(self):
self.os = ''
self.name = ''
self.comments = []
self.success = True
self.working = False
self.unit = [] # type: List
self.lint = {}
self.test_stats = {
'pass': 0,
'fail': 0,
'skip': 0
} # type: Dict[str, int]
self.steps = [] # type: List[Step]
self.artifacts = [] # type: List
def __str__(self):
return str(self.__dict__)
def add_lint(self, m):
key = '{}:{}'.format(m['path'], m['line'])
if key not in self.lint:
self.lint[key] = []
self.lint[key].append(m)
def add_artifact(self, dir: str, file: str, name: str):
self.artifacts.append({'dir': dir, 'file': file, 'name': name})
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Sample interaction with Phabricator')
parser.add_argument('--url', type=str, dest='url', default='https://reviews.llvm.org/api/')
parser.add_argument('--token', type=str, dest='token', default=None, required=True)
parser.add_argument('--diff_id', type=str, dest='diff_id', default=None, required=True)
args = parser.parse_args()
phabtalk = PhabTalk(args.token, args.url, False)
print(phabtalk.get_revision_id(args.diff_id))

101
scripts/pipeline_create_branch.py
View file

@ -1,101 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import sys
import yaml
import logging
from buildkite_utils import set_metadata, BuildkiteApi
from phabtalk.phabtalk import PhabTalk
if __name__ == '__main__':
diff_id = os.getenv("ph_buildable_diff")
revision_id = os.getenv("ph_buildable_revision", '')
log_level = os.getenv('ph_log_level', 'INFO')
base_commit = os.getenv('ph_base_commit', 'auto')
run_build = os.getenv('ph_skip_build') is None
trigger = os.getenv('ph_trigger_pipeline')
logging.basicConfig(level=log_level, format='%(levelname)-7s %(message)s')
phabtalk = PhabTalk(os.getenv('CONDUIT_TOKEN'), dry_run_updates=(os.getenv('ph_dry_run_report') is not None))
rev = phabtalk.get_revision(int(revision_id))
user_id = rev.get('authorPHID')
logging.debug(f'authorPHID {user_id}')
if user_id is None:
logging.error('cannot find author of the revision')
sys.exit(1)
projects = phabtalk.user_projects(user_id)
logging.info(f'user projects: {", ".join(projects)}')
# Cancel any existing builds.
# Do this before setting own 'ph_buildable_revision'.
try:
bk = BuildkiteApi(os.getenv("BUILDKITE_API_TOKEN"), os.getenv("BUILDKITE_ORGANIZATION_SLUG"))
for b in bk.list_running_revision_builds(os.getenv("BUILDKITE_PIPELINE_SLUG"), os.getenv('ph_buildable_revision')):
logging.info(f'cancelling build {b.get("web_url")}')
bk.cancel_build(b)
except Exception as e:
logging.error(e)
set_metadata('ph_buildable_diff', os.getenv("ph_buildable_diff"))
set_metadata('ph_buildable_revision', os.getenv('ph_buildable_revision'))
set_metadata('ph_build_id', os.getenv("ph_build_id"))
if trigger is None:
trigger = 'premerge-checks'
env = {
'ph_scripts_refspec': '${BUILDKITE_BRANCH}',
'ph_user_project_slugs': ",".join(projects),
# TODO: those two are for "apply_patch.sh". Maybe just look for "ph_" env in patch_diff.py?
'LOG_LEVEL': log_level,
'BASE_COMMIT': base_commit,
}
for e in os.environ:
if e.startswith('ph_'):
env[e] = os.getenv(e)
steps = [{
'label': 'create branch',
'key': 'create-branch',
'commands': [
'pip install -q -r scripts/requirements.txt',
'scripts/apply_patch.sh'
],
'agents': {'queue': 'service'},
'timeout_in_minutes': 20,
'env': env
}]
if run_build:
# trigger_build_step = {
# 'trigger': trigger,
# 'label': ':rocket: build and test',
# 'async': False,
# 'depends_on': 'create-branch',
# 'build': {
# 'branch': f'phab-diff-{diff_id}',
# 'env': env,
# },
# }
# steps.append(trigger_build_step)
steps.append({
'trigger': 'phabricator-build',
'label': ':rocket: build and test',
'async': False,
'depends_on': 'create-branch',
'build': {
'branch': f'phab-diff-{diff_id}',
'env': env,
},
})
print(yaml.dump({'steps': steps}))

67
scripts/pipeline_main.py
View file

@ -1,67 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Script runs in checked out llvm-project directory.
from choose_projects import ChooseProjects
from steps import generic_linux, generic_windows, from_shell_output, extend_steps_env, bazel
from typing import Dict
import git
import git_utils
import os
import yaml
steps_generators = [
'${BUILDKITE_BUILD_CHECKOUT_PATH}/.ci/generate-buildkite-pipeline-scheduled',
]
if __name__ == '__main__':
scripts_refspec = os.getenv("ph_scripts_refspec", "main")
no_cache = os.getenv('ph_no_cache') is not None
log_level = os.getenv('ph_log_level', 'WARNING')
notify_emails = list(filter(None, os.getenv('ph_notify_emails', '').split(',')))
# Syncing LLVM fork so any pipelines started from upstream llvm-project
# but then triggered a build on fork will observe the commit.
repo = git_utils.initLlvmFork(os.path.join(os.getenv('BUILDKITE_BUILD_PATH', ''), 'llvm-project-fork'))
git_utils.syncRemotes(repo, 'upstream', 'origin')
steps = []
env: Dict[str, str] = {}
for e in os.environ:
if e.startswith('ph_'):
env[e] = os.getenv(e, '')
repo = git.Repo('.')
cp = ChooseProjects(None)
linux_projects = cp.get_all_enabled_projects('linux')
steps.extend(generic_linux(os.getenv('ph_projects', ';'.join(linux_projects)), check_diff=False))
windows_projects = cp.get_all_enabled_projects('windows')
steps.extend(generic_windows(os.getenv('ph_projects', ';'.join(windows_projects))))
steps.extend(bazel([], force=True))
if os.getenv('ph_skip_generated') is None:
# BUILDKITE_COMMIT might be an alias, e.g. "HEAD". Resolve it to make the build hermetic.
env['BUILDKITE_COMMIT'] = os.getenv('BUILDKITE_COMMIT')
if env['BUILDKITE_COMMIT'] == "HEAD":
env['BUILDKITE_COMMIT'] = repo.head.commit.hexsha
env['BUILDKITE_BRANCH'] = os.getenv('BUILDKITE_BRANCH')
env['BUILDKITE_MESSAGE'] = os.getenv('BUILDKITE_MESSAGE')
for gen in steps_generators:
steps.extend(from_shell_output(gen, env=env))
notify = []
for e in notify_emails:
notify.append({'email': e})
extend_steps_env(steps, env)
print(yaml.dump({'steps': steps, 'notify': notify}))

101
scripts/pipeline_premerge.py
View file

@ -1,101 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Script runs in checked out llvm-project directory.
import logging
import os
from typing import Dict
from buildkite_utils import annotate, feedback_url, set_metadata
from choose_projects import ChooseProjects
import git
from steps import generic_linux, generic_windows, from_shell_output, checkout_scripts, bazel, extend_steps_env
import yaml
steps_generators = [
'${BUILDKITE_BUILD_CHECKOUT_PATH}/.ci/generate-buildkite-pipeline-premerge',
]
if __name__ == '__main__':
scripts_refspec = os.getenv("ph_scripts_refspec", "main")
diff_id = os.getenv("ph_buildable_diff", "")
no_cache = os.getenv('ph_no_cache') is not None
projects = os.getenv('ph_projects', 'detect')
log_level = os.getenv('ph_log_level', 'INFO')
logging.basicConfig(level=log_level, format='%(levelname)-7s %(message)s')
phid = os.getenv('ph_target_phid')
url = f"https://reviews.llvm.org/D{os.getenv('ph_buildable_revision')}?id={diff_id}"
annotate(f"Build for [D{os.getenv('ph_buildable_revision')}#{diff_id}]({url}). "
f"[Harbormaster build](https://reviews.llvm.org/harbormaster/build/{os.getenv('ph_build_id')}).\n"
f"If there is a build infrastructure issue, please [create a bug]({feedback_url()}).")
set_metadata('ph_buildable_diff', os.getenv("ph_buildable_diff"))
set_metadata('ph_buildable_revision', os.getenv('ph_buildable_revision'))
set_metadata('ph_build_id', os.getenv("ph_build_id"))
env: Dict[str, str] = {}
for e in os.environ:
if e.startswith('ph_'):
env[e] = os.getenv(e, '')
repo = git.Repo('.')
steps = []
# List all affected projects.
patch = repo.git.diff("HEAD~1")
cp = ChooseProjects('.')
linux_projects = cp.choose_projects(patch = patch, os_name = "linux")
logging.info(f'linux_projects: {linux_projects}')
# It's now generated by .ci/generate-buildkite-pipeline-premerge
# if len(linux_projects) > 0:
# steps.extend(generic_linux(';'.join(linux_projects), check_diff=True))
windows_projects = cp.choose_projects(patch = patch, os_name = "windows")
logging.info(f'windows_projects: {windows_projects}')
# Generated by .ci/generate-buildkite-pipeline-premerge.
# if len(windows_projects) > 0:
# steps.extend(generic_windows(';'.join(windows_projects)))
# Add custom checks.
if os.getenv('ph_skip_generated') is None:
if os.getenv('BUILDKITE_COMMIT', 'HEAD') == "HEAD":
env['BUILDKITE_COMMIT'] = repo.head.commit.hexsha
for gen in steps_generators:
steps.extend(from_shell_output(gen, env=env))
modified_files = cp.get_changed_files(patch)
steps.extend(bazel(modified_files))
if phid is None:
logging.warning('ph_target_phid is not specified. Skipping "Report" step')
else:
steps.append({
'wait': '~',
'continue_on_failure': True,
})
report_step = {
'label': ':phabricator: update build status on Phabricator',
'commands': [
*checkout_scripts('linux', scripts_refspec),
'$${SRC}/scripts/summary.py',
],
'artifact_paths': ['artifacts/**/*'],
'agents': {'queue': 'service'},
'timeout_in_minutes': 10,
}
steps.append(report_step)
extend_steps_env(steps, env)
print(yaml.dump({'steps': steps}))

196
scripts/premerge_checks.py
View file

@ -1,196 +0,0 @@
#!/usr/bin/env python3
# Copyright 2020 Google LLC
#
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://llvm.org/LICENSE.txt
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Runs all check on buildkite agent.
import argparse
import json
import logging
import os
import pathlib
import re
import shutil
import sys
import time
from typing import Callable, List, Type
import clang_format_report
import clang_tidy_report
import run_cmake
from buildkite_utils import upload_file, annotate, strip_emojis
from exec_utils import watch_shell, if_not_matches, tee
from phabtalk.phabtalk import Report, PhabTalk, Step
from choose_projects import ChooseProjects
def ninja_all_report(step: Step, _: Report):
step.reproduce_commands.append('ninja all')
rc = watch_shell(
sys.stdout.buffer.write,
sys.stderr.buffer.write,
'ninja all', cwd=build_dir)
logging.debug(f'ninja all: returned {rc}')
step.set_status_from_exit_code(rc)
def ninja_check_all_report(step: Step, _: Report):
print('Full log will be available in Artifacts "ninja-check-all.log"', flush=True)
step.reproduce_commands.append('ninja check-all')
rc = watch_shell(
sys.stdout.buffer.write,
sys.stderr.buffer.write,
'ninja check-all', cwd=build_dir)
logging.debug(f'ninja check-all: returned {rc}')
step.set_status_from_exit_code(rc)
def ninja_check_projects_report(step: Step, _: Report, checks: str):
print('Full log will be available in Artifacts "ninja-check.log"', flush=True)
step.reproduce_commands.append(f'ninja {checks}')
rc = watch_shell(
sys.stdout.buffer.write,
sys.stderr.buffer.write,
f'ninja {checks}', cwd=build_dir)
logging.debug(f'ninja {checks}: returned {rc}')
step.set_status_from_exit_code(rc)
def run_step(name: str, report: Report, thunk: Callable[[Step, Report], None]) -> Step:
start = time.time()
print(f'--- {name}', flush=True) # New section in Buildkite log.
step = Step()
step.name = name
thunk(step, report)
step.duration = time.time() - start
# Expand section if step has failed.
if not step.success:
print('^^^ +++', flush=True)
if step.success:
annotate(f"{name}: OK")
else:
annotate(f"{name}: FAILED", style='error')
report.success = False
report.steps.append(step)
return step
def cmake_report(projects: str, step: Step, _: Report):
global build_dir
cmake_result, build_dir, cmake_artifacts, commands = run_cmake.run(projects, repo_path=os.getcwd())
for file in cmake_artifacts:
if os.path.exists(file):
shutil.copy2(file, artifacts_dir)
step.set_status_from_exit_code(cmake_result)
step.reproduce_commands = commands
def as_dict(obj):
try:
return obj.toJSON()
except:
return obj.__dict__
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Runs premerge checks8')
parser.add_argument('--log-level', type=str, default='WARNING')
parser.add_argument('--build-and-test-all', action='store_true',
help="Run `ninja all` and `ninja check-all`, instead of "
"the default of running `ninja check-{project}`.")
parser.add_argument('--check-clang-format', action='store_true')
parser.add_argument('--check-clang-tidy', action='store_true')
parser.add_argument('--projects', type=str, default='detect',
help="Projects to test as a list of projects like 'clang;libc'."
" Dependent projects are automatically added to the CMake invocation.")
args = parser.parse_args()
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
ctx = strip_emojis(os.getenv('BUILDKITE_LABEL', 'default'))
annotate(os.getenv('BUILDKITE_LABEL', 'default'), context=ctx)
build_dir = ''
step_key = os.getenv("BUILDKITE_STEP_KEY")
scripts_dir = pathlib.Path(__file__).parent.absolute()
artifacts_dir = os.path.join(os.getcwd(), 'artifacts')
os.makedirs(artifacts_dir, exist_ok=True)
report_path = f'{step_key}_result.json'
report = Report()
report.os = f'{os.getenv("BUILDKITE_AGENT_META_DATA_OS")}'
report.name = step_key
report.success = True
projects = set(args.projects.split(";"))
cp = ChooseProjects(None)
dependencies = cp.get_dependencies(projects)
logging.info(f"Dependencies: {dependencies}")
enabled_projects = ";".join(dependencies.union(projects))
cmake = run_step('cmake', report, lambda s, r: cmake_report(enabled_projects, s, r))
commands_in_build = True
if cmake.success:
if args.build_and_test_all:
ninja_all = run_step('ninja all', report, ninja_all_report)
if ninja_all.success:
run_step('ninja check-all', report, ninja_check_all_report)
else:
checks = " ".join(cp.get_check_targets(projects))
logging.info(f"Running checks: {checks}")
report_lambda: Callable[[Step, Report], None] = lambda s, r: ninja_check_projects_report(s, r, checks)
run_step(f"ninja {checks}", report, report_lambda)
if args.check_clang_tidy:
if commands_in_build:
s = Step('')
s.reproduce_commands.append('cd ..')
commands_in_build = False
report.steps.append(s)
run_step('clang-tidy', report,
lambda s, r: clang_tidy_report.run('HEAD~1', os.path.join(scripts_dir, 'clang-tidy.ignore'), s, r))
if args.check_clang_format:
if commands_in_build:
s = Step('')
s.reproduce_commands.append('cd ..')
commands_in_build = False
report.steps.append(s)
run_step('clang-format', report,
lambda s, r: clang_format_report.run('HEAD~1', os.path.join(scripts_dir, 'clang-format.ignore'), s, r))
logging.debug(report)
summary = []
summary.append('''
<details>
<summary>Reproduce build locally</summary>
```''')
summary.append(f'git clone {os.getenv("BUILDKITE_REPO")} llvm-project')
summary.append('cd llvm-project')
summary.append(f'git checkout {os.getenv("BUILDKITE_COMMIT")}')
for s in report.steps:
if len(s.reproduce_commands) == 0:
continue
summary.append('\n'.join(s.reproduce_commands))
summary.append('```\n</details>')
annotate('\n'.join(summary), style='success')
ph_target_phid = os.getenv('ph_target_phid')
if ph_target_phid is not None:
phabtalk = PhabTalk(os.getenv('CONDUIT_TOKEN'), dry_run_updates=(os.getenv('ph_dry_run_report') is not None))
phabtalk.update_build_status(ph_target_phid, True, report.success, report.lint, [])
for a in report.artifacts:
url = upload_file(a['dir'], a['file'])
if url is not None:
phabtalk.maybe_add_url_artifact(ph_target_phid, url, f'{a["name"]} ({step_key})')
else:
logging.warning('ph_target_phid is not specified. Will not update the build status in Phabricator')
with open(report_path, 'w') as f:
json.dump(report.__dict__, f, default=as_dict)
if not report.success:
print('Build completed with failures', flush=True)
exit(1)

Some files were not shown because too many files have changed in this diff Show more