remove obsolete code
most of the scripts were used for Phabricator integration that is now disabled and current scripts are located in `llvm-project/.ci` . Also removed obsolete playbooks and scripts that were not used and likely no longer work. Updated wording and minor typos. I have tried to look through potential references to the scripts and disabled some of buildkite pipelines that referred to them. That is not a guarantee that this will not break anything as I have not tried to rebuild containers and redeploy cluster.
This commit is contained in:
Mikhail Goncharov
parent
2d073b0307
commit
7f0564d978
114 changed files with 3 additions and 9405 deletions
87
README.md
87
README.md
|
|
@ -1,6 +1,6 @@
|
|||
This repo is holding VM configurations for machine cluster and scripts to run pre-merge tests triggered by http://reviews.llvm.org.
|
||||
|
||||
As LLVM project has moved to Pull Requests and Phabricator will no longer trigger builds, this repository will likely be gone.
|
||||
As LLVM project has moved to Pull Requests and Phabricator no longer triggers builds, this repository will likely be gone.
|
||||
|
||||
[Pull request migration schedule](https://discourse.llvm.org/t/pull-request-migration-schedule/71595).
|
||||
|
||||
|
|
@ -10,100 +10,17 @@ Presentation by Louis Dione on LLVM devmtg 2021 https://youtu.be/B7gB6van7Bw
|
|||
|
||||
[](https://www.youtube.com/watch?v=B7gB6van7Bw)
|
||||
|
||||
The *pre-merge checks* for the [LLVM project](http://llvm.org/) are a
|
||||
[continuous integration
|
||||
(CI)](https://en.wikipedia.org/wiki/Continuous_integration) workflow. The
|
||||
workflow checks the patches the developers upload to the [LLVM
|
||||
Phabricator](https://reviews.llvm.org) instance.
|
||||
|
||||
*Phabricator* (https://reviews.llvm.org) is the code review tool in the LLVM
|
||||
project.
|
||||
|
||||
The workflow checks the patches before a user merges them to the main branch -
|
||||
thus the term *pre-merge testing**. When a user uploads a patch to the LLVM
|
||||
Phabricator, Phabricator triggers the checks and then displays the results.
|
||||
|
||||
The CI system checks the patches **before** a user merges them to the main
|
||||
branch. This way bugs in a patch are contained during the code review stage and
|
||||
do not pollute the main branch. The more bugs the CI system can catch during
|
||||
the code review phase, the more stable and bug-free the main branch will
|
||||
become. <sup>[citation needed]()</sup>
|
||||
|
||||
This repository contains the configurations and script to run pre-merge checks
|
||||
for the LLVM project.
|
||||
|
||||
## Feedback
|
||||
|
||||
If you notice issues or have an idea on how to improve pre-merge checks, please
|
||||
create a [new issue](https://github.com/google/llvm-premerge-checks/issues/new)
|
||||
or give a :heart: to an existing one.
|
||||
|
||||
## Sign up for beta-test
|
||||
|
||||
To get the latest features and help us developing the project, sign up for the
|
||||
pre-merge beta testing by adding yourself to the ["pre-merge beta testing"
|
||||
project](https://reviews.llvm.org/project/members/78/) on Phabricator.
|
||||
|
||||
## Opt-out
|
||||
|
||||
In case you want to opt-out entirely of pre-merge testing, add yourself to the
|
||||
[OPT OUT project](https://reviews.llvm.org/project/view/83/).
|
||||
|
||||
If you decide to opt-out, please let us know why, so we might be able to improve
|
||||
in the future.
|
||||
|
||||
# Requirements
|
||||
|
||||
The builds are only triggered if the Revision in Phabricator is created/updated
|
||||
via `arc diff`. If you update a Revision via the Web UI it will [not
|
||||
trigger](https://secure.phabricator.com/Q447) a build.
|
||||
|
||||
To get a patch on Phabricator tested the build server must be able to apply the
|
||||
patch to the checked out git repository. If you want to get your patch tested,
|
||||
please make sure that either:
|
||||
|
||||
* You set a git hash as `sourceControlBaseRevision` in Phabricator which is
|
||||
* available on the Github repository, **or** you define the dependencies of your
|
||||
* patch in Phabricator, **or** your patch can be applied to the main branch.
|
||||
|
||||
Only then can the build server apply the patch locally and run the builds and
|
||||
tests.
|
||||
|
||||
# Accessing results on Phabricator
|
||||
|
||||
Phabricator will automatically trigger a build for every new patch you upload or
|
||||
modify. Phabricator shows the build results at the top of the entry: 
|
||||
|
||||
The CI will compile and run tests, run clang-format and
|
||||
[clang-tidy](docs/clang_tidy.md) on lines changed.
|
||||
|
||||
If a unit test failed, this is shown below the build status. You can also expand
|
||||
the unit test to see the details: .
|
||||
|
||||
# Contributing
|
||||
|
||||
We're happy to get help on improving the infrastructure and workflows!
|
||||
|
||||
Please check [contibuting](docs/contributing.md) first.
|
||||
|
||||
[Development](docs/development.md) gives an overview how different parts
|
||||
interact together.
|
||||
|
||||
[Playbooks](docs/playbooks.md) shows concrete examples how to, for example,
|
||||
build and run agents locally.
|
||||
|
||||
If you have any questions please contact by [mail](mailto:goncahrov@google.com)
|
||||
or find user "goncharov" on [LLVM Discord](https://discord.gg/xS7Z362).
|
||||
|
||||
# Additional Information
|
||||
|
||||
- [Playbooks](docs/playbooks.md) for installing/upgrading agents and testing
|
||||
changes.
|
||||
|
||||
- [Log of the service
|
||||
operations](https://github.com/google/llvm-premerge-checks/wiki/LLVM-pre-merge-tests-operations-blog)
|
||||
Please check [contributing](docs/contributing.md).
|
||||
|
||||
# License
|
||||
|
||||
|
|
|
|||
|
|
@ -1,29 +0,0 @@
|
|||
# Installation
|
||||
|
||||
Install helm (along with other tools in `local_setup.sh`).
|
||||
|
||||
Once per cluster:
|
||||
|
||||
`helm install arc --namespace "arc-system" --create-namespace oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller`
|
||||
|
||||
## Add new set of runners
|
||||
|
||||
Create runner set first time:
|
||||
|
||||
- copy 'values.yml` and updata parameters: APP ID, target node set, repo etc
|
||||
- run
|
||||
|
||||
```
|
||||
helm install <runner-set> --namespace <k8s-namespace> --create-namespace \
|
||||
--values=$(readlink -f <values.yml>)
|
||||
--set-file=githubConfigSecret.github_app_private_key=$(readlink -f <pem>) \
|
||||
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
|
||||
```
|
||||
|
||||
After that update values.yml to use `githubConfigSecret: arc-runner-set-gha-rs-github-secret`.
|
||||
|
||||
## Update
|
||||
|
||||
Example command for linux set:
|
||||
|
||||
`helm upgrade arc-google-linux --namespace arc-linux-prod -f $(readlink -f ~/src/merge-checks/actions-runner-controller/values-llvm.yaml) oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set`
|
||||
|
|
@ -1,194 +0,0 @@
|
|||
# See options doc in https://github.com/actions/actions-runner-controller/tree/master/charts/actions-runner-controller
|
||||
|
||||
## githubConfigUrl is the GitHub url for where you want to configure runners
|
||||
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
|
||||
githubConfigUrl: "https://github.com/llvm/llvm-project"
|
||||
|
||||
## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
|
||||
githubConfigSecret:
|
||||
### GitHub Apps Configuration
|
||||
## NOTE: IDs MUST be strings, use quotes
|
||||
github_app_id: "418336"
|
||||
github_app_installation_id: "43821912"
|
||||
## Pass --set-file=githubConfigSecret.github_app_private_key=<path to pem>
|
||||
# First installation creates this secret.
|
||||
# githubConfigSecret: arc-runner-set-gha-rs-github-secret
|
||||
|
||||
## proxy can be used to define proxy settings that will be used by the
|
||||
## controller, the listener and the runner of this scale set.
|
||||
#
|
||||
# proxy:
|
||||
# http:
|
||||
# url: http://proxy.com:1234
|
||||
# credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
|
||||
# https:
|
||||
# url: http://proxy.com:1234
|
||||
# credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
|
||||
# noProxy:
|
||||
# - example.com
|
||||
# - example.org
|
||||
|
||||
## maxRunners is the max number of runners the autoscaling runner set will scale up to.
|
||||
maxRunners: 3
|
||||
|
||||
## minRunners is the min number of runners the autoscaling runner set will scale down to.
|
||||
minRunners: 1
|
||||
|
||||
runnerGroup: "generic-google-cloud-2"
|
||||
|
||||
## name of the runner scale set to create. Defaults to the helm release name
|
||||
# runnerScaleSetName: ""
|
||||
|
||||
## A self-signed CA certificate for communication with the GitHub server can be
|
||||
## provided using a config map key selector. If `runnerMountPath` is set, for
|
||||
## each runner pod ARC will:
|
||||
## - create a `github-server-tls-cert` volume containing the certificate
|
||||
## specified in `certificateFrom`
|
||||
## - mount that volume on path `runnerMountPath`/{certificate name}
|
||||
## - set NODE_EXTRA_CA_CERTS environment variable to that same path
|
||||
## - set RUNNER_UPDATE_CA_CERTS environment variable to "1" (as of version
|
||||
## 2.303.0 this will instruct the runner to reload certificates on the host)
|
||||
##
|
||||
## If any of the above had already been set by the user in the runner pod
|
||||
## template, ARC will observe those and not overwrite them.
|
||||
## Example configuration:
|
||||
#
|
||||
# githubServerTLS:
|
||||
# certificateFrom:
|
||||
# configMapKeyRef:
|
||||
# name: config-map-name
|
||||
# key: ca.crt
|
||||
# runnerMountPath: /usr/local/share/ca-certificates/
|
||||
|
||||
## Container mode is an object that provides out-of-box configuration
|
||||
## for dind and kubernetes mode. Template will be modified as documented under the
|
||||
## template object.
|
||||
##
|
||||
## If any customization is required for dind or kubernetes mode, containerMode should remain
|
||||
## empty, and configuration should be applied to the template.
|
||||
# containerMode:
|
||||
# type: "dind" ## type can be set to dind or kubernetes
|
||||
# ## the following is required when containerMode.type=kubernetes
|
||||
# kubernetesModeWorkVolumeClaim:
|
||||
# accessModes: ["ReadWriteOnce"]
|
||||
# # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
|
||||
# storageClassName: "dynamic-blob-storage"
|
||||
# resources:
|
||||
# requests:
|
||||
# storage: 1Gi
|
||||
# kubernetesModeServiceAccount:
|
||||
# annotations:
|
||||
|
||||
## template is the PodSpec for each listener Pod
|
||||
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
|
||||
# listenerTemplate:
|
||||
# spec:
|
||||
# containers:
|
||||
# # Use this section to append additional configuration to the listener container.
|
||||
# # If you change the name of the container, the configuration will not be applied to the listener,
|
||||
# # and it will be treated as a side-car container.
|
||||
# - name: listener
|
||||
# securityContext:
|
||||
# runAsUser: 1000
|
||||
# # Use this section to add the configuration of a side-car container.
|
||||
# # Comment it out or remove it if you don't need it.
|
||||
# # Spec for this container will be applied as is without any modifications.
|
||||
# - name: side-car
|
||||
# image: example-sidecar
|
||||
|
||||
## template is the PodSpec for each runner Pod
|
||||
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
|
||||
## template.spec will be modified if you change the container mode
|
||||
## with containerMode.type=dind, we will populate the template.spec with following pod spec
|
||||
## template:
|
||||
## spec:
|
||||
## initContainers:
|
||||
## - name: init-dind-externals
|
||||
## image: ghcr.io/actions/actions-runner:latest
|
||||
## command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
|
||||
## volumeMounts:
|
||||
## - name: dind-externals
|
||||
## mountPath: /home/runner/tmpDir
|
||||
## containers:
|
||||
## - name: runner
|
||||
## image: ghcr.io/actions/actions-runner:latest
|
||||
## command: ["/home/runner/run.sh"]
|
||||
## env:
|
||||
## - name: DOCKER_HOST
|
||||
## value: unix:///run/docker/docker.sock
|
||||
## volumeMounts:
|
||||
## - name: work
|
||||
## mountPath: /home/runner/_work
|
||||
## - name: dind-sock
|
||||
## mountPath: /run/docker
|
||||
## readOnly: true
|
||||
## - name: dind
|
||||
## image: docker:dind
|
||||
## args:
|
||||
## - dockerd
|
||||
## - --host=unix:///run/docker/docker.sock
|
||||
## - --group=$(DOCKER_GROUP_GID)
|
||||
## env:
|
||||
## - name: DOCKER_GROUP_GID
|
||||
## value: "123"
|
||||
## securityContext:
|
||||
## privileged: true
|
||||
## volumeMounts:
|
||||
## - name: work
|
||||
## mountPath: /home/runner/_work
|
||||
## - name: dind-sock
|
||||
## mountPath: /run/docker
|
||||
## - name: dind-externals
|
||||
## mountPath: /home/runner/externals
|
||||
## volumes:
|
||||
## - name: work
|
||||
## emptyDir: {}
|
||||
## - name: dind-sock
|
||||
## emptyDir: {}
|
||||
## - name: dind-externals
|
||||
## emptyDir: {}
|
||||
######################################################################################################
|
||||
## with containerMode.type=kubernetes, we will populate the template.spec with following pod spec
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: runner
|
||||
image: us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux:latest
|
||||
command: ["/bin/bash"]
|
||||
args: ["-c", "/entrypoint.sh /home/runner/run.sh"]
|
||||
env:
|
||||
- name: ACTIONS_RUNNER_CONTAINER_HOOKS
|
||||
value: /home/runner/k8s/index.js
|
||||
- name: ACTIONS_RUNNER_POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
|
||||
value: "false"
|
||||
- name: WORKDIR
|
||||
value: "/home/runner/_work"
|
||||
resources:
|
||||
limits:
|
||||
cpu: 31
|
||||
memory: 80Gi
|
||||
requests:
|
||||
cpu: 31
|
||||
memory: 80Gi
|
||||
volumeMounts:
|
||||
- name: work
|
||||
mountPath: /home/runner/_work
|
||||
volumes:
|
||||
- name: work
|
||||
emptyDir: {}
|
||||
nodeSelector:
|
||||
cloud.google.com/gke-nodepool: linux-agents-2
|
||||
|
||||
## Optional controller service account that needs to have required Role and RoleBinding
|
||||
## to operate this gha-runner-scale-set installation.
|
||||
## The helm chart will try to find the controller deployment and its service account at installation time.
|
||||
## In case the helm chart can't find the right service account, you can explicitly pass in the following value
|
||||
## to help it finish RoleBinding with the right service account.
|
||||
## Note: if your controller is installed to only watch a single namespace, you have to pass these values explicitly.
|
||||
# controllerServiceAccount:
|
||||
# namespace: arc-system
|
||||
# name: test-arc-gha-runner-scale-set-controller
|
||||
|
|
@ -1,194 +0,0 @@
|
|||
# See options doc in https://github.com/actions/actions-runner-controller/tree/master/charts/actions-runner-controller
|
||||
|
||||
## githubConfigUrl is the GitHub url for where you want to configure runners
|
||||
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
|
||||
githubConfigUrl: "https://github.com/metafloworg/llvm-project"
|
||||
|
||||
## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
|
||||
#githubConfigSecret:
|
||||
### GitHub Apps Configuration
|
||||
## NOTE: IDs MUST be strings, use quotes
|
||||
#github_app_id: "427039"
|
||||
#github_app_installation_id: "43840704"
|
||||
## Pass --set-file=githubConfigSecret.github_app_private_key=<path to pem>
|
||||
# First installation creates this secret.
|
||||
githubConfigSecret: arc-runner-set-gha-rs-github-secret
|
||||
|
||||
## proxy can be used to define proxy settings that will be used by the
|
||||
## controller, the listener and the runner of this scale set.
|
||||
#
|
||||
# proxy:
|
||||
# http:
|
||||
# url: http://proxy.com:1234
|
||||
# credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
|
||||
# https:
|
||||
# url: http://proxy.com:1234
|
||||
# credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
|
||||
# noProxy:
|
||||
# - example.com
|
||||
# - example.org
|
||||
|
||||
## maxRunners is the max number of runners the autoscaling runner set will scale up to.
|
||||
maxRunners: 3
|
||||
|
||||
## minRunners is the min number of runners the autoscaling runner set will scale down to.
|
||||
minRunners: 0
|
||||
|
||||
# runnerGroup: "default"
|
||||
|
||||
## name of the runner scale set to create. Defaults to the helm release name
|
||||
# runnerScaleSetName: ""
|
||||
|
||||
## A self-signed CA certificate for communication with the GitHub server can be
|
||||
## provided using a config map key selector. If `runnerMountPath` is set, for
|
||||
## each runner pod ARC will:
|
||||
## - create a `github-server-tls-cert` volume containing the certificate
|
||||
## specified in `certificateFrom`
|
||||
## - mount that volume on path `runnerMountPath`/{certificate name}
|
||||
## - set NODE_EXTRA_CA_CERTS environment variable to that same path
|
||||
## - set RUNNER_UPDATE_CA_CERTS environment variable to "1" (as of version
|
||||
## 2.303.0 this will instruct the runner to reload certificates on the host)
|
||||
##
|
||||
## If any of the above had already been set by the user in the runner pod
|
||||
## template, ARC will observe those and not overwrite them.
|
||||
## Example configuration:
|
||||
#
|
||||
# githubServerTLS:
|
||||
# certificateFrom:
|
||||
# configMapKeyRef:
|
||||
# name: config-map-name
|
||||
# key: ca.crt
|
||||
# runnerMountPath: /usr/local/share/ca-certificates/
|
||||
|
||||
## Container mode is an object that provides out-of-box configuration
|
||||
## for dind and kubernetes mode. Template will be modified as documented under the
|
||||
## template object.
|
||||
##
|
||||
## If any customization is required for dind or kubernetes mode, containerMode should remain
|
||||
## empty, and configuration should be applied to the template.
|
||||
# containerMode:
|
||||
# type: "dind" ## type can be set to dind or kubernetes
|
||||
# ## the following is required when containerMode.type=kubernetes
|
||||
# kubernetesModeWorkVolumeClaim:
|
||||
# accessModes: ["ReadWriteOnce"]
|
||||
# # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
|
||||
# storageClassName: "dynamic-blob-storage"
|
||||
# resources:
|
||||
# requests:
|
||||
# storage: 1Gi
|
||||
# kubernetesModeServiceAccount:
|
||||
# annotations:
|
||||
|
||||
## template is the PodSpec for each listener Pod
|
||||
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
|
||||
# listenerTemplate:
|
||||
# spec:
|
||||
# containers:
|
||||
# # Use this section to append additional configuration to the listener container.
|
||||
# # If you change the name of the container, the configuration will not be applied to the listener,
|
||||
# # and it will be treated as a side-car container.
|
||||
# - name: listener
|
||||
# securityContext:
|
||||
# runAsUser: 1000
|
||||
# # Use this section to add the configuration of a side-car container.
|
||||
# # Comment it out or remove it if you don't need it.
|
||||
# # Spec for this container will be applied as is without any modifications.
|
||||
# - name: side-car
|
||||
# image: example-sidecar
|
||||
|
||||
## template is the PodSpec for each runner Pod
|
||||
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
|
||||
## template.spec will be modified if you change the container mode
|
||||
## with containerMode.type=dind, we will populate the template.spec with following pod spec
|
||||
## template:
|
||||
## spec:
|
||||
## initContainers:
|
||||
## - name: init-dind-externals
|
||||
## image: ghcr.io/actions/actions-runner:latest
|
||||
## command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
|
||||
## volumeMounts:
|
||||
## - name: dind-externals
|
||||
## mountPath: /home/runner/tmpDir
|
||||
## containers:
|
||||
## - name: runner
|
||||
## image: ghcr.io/actions/actions-runner:latest
|
||||
## command: ["/home/runner/run.sh"]
|
||||
## env:
|
||||
## - name: DOCKER_HOST
|
||||
## value: unix:///run/docker/docker.sock
|
||||
## volumeMounts:
|
||||
## - name: work
|
||||
## mountPath: /home/runner/_work
|
||||
## - name: dind-sock
|
||||
## mountPath: /run/docker
|
||||
## readOnly: true
|
||||
## - name: dind
|
||||
## image: docker:dind
|
||||
## args:
|
||||
## - dockerd
|
||||
## - --host=unix:///run/docker/docker.sock
|
||||
## - --group=$(DOCKER_GROUP_GID)
|
||||
## env:
|
||||
## - name: DOCKER_GROUP_GID
|
||||
## value: "123"
|
||||
## securityContext:
|
||||
## privileged: true
|
||||
## volumeMounts:
|
||||
## - name: work
|
||||
## mountPath: /home/runner/_work
|
||||
## - name: dind-sock
|
||||
## mountPath: /run/docker
|
||||
## - name: dind-externals
|
||||
## mountPath: /home/runner/externals
|
||||
## volumes:
|
||||
## - name: work
|
||||
## emptyDir: {}
|
||||
## - name: dind-sock
|
||||
## emptyDir: {}
|
||||
## - name: dind-externals
|
||||
## emptyDir: {}
|
||||
######################################################################################################
|
||||
## with containerMode.type=kubernetes, we will populate the template.spec with following pod spec
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: runner
|
||||
image: us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux:latest
|
||||
command: ["/bin/bash"]
|
||||
args: ["-c", "/entrypoint.sh /home/runner/run.sh"]
|
||||
env:
|
||||
- name: ACTIONS_RUNNER_CONTAINER_HOOKS
|
||||
value: /home/runner/k8s/index.js
|
||||
- name: ACTIONS_RUNNER_POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
|
||||
value: "false"
|
||||
- name: WORKDIR
|
||||
value: "/home/runner/_work"
|
||||
resources:
|
||||
limits:
|
||||
cpu: 31
|
||||
memory: 80Gi
|
||||
requests:
|
||||
cpu: 31
|
||||
memory: 80Gi
|
||||
volumeMounts:
|
||||
- name: work
|
||||
mountPath: /home/runner/_work
|
||||
volumes:
|
||||
- name: work
|
||||
emptyDir: {}
|
||||
nodeSelector:
|
||||
cloud.google.com/gke-nodepool: linux-agents-2
|
||||
|
||||
## Optional controller service account that needs to have required Role and RoleBinding
|
||||
## to operate this gha-runner-scale-set installation.
|
||||
## The helm chart will try to find the controller deployment and its service account at installation time.
|
||||
## In case the helm chart can't find the right service account, you can explicitly pass in the following value
|
||||
## to help it finish RoleBinding with the right service account.
|
||||
## Note: if your controller is installed to only watch a single namespace, you have to pass these values explicitly.
|
||||
# controllerServiceAccount:
|
||||
# namespace: arc-system
|
||||
# name: test-arc-gha-runner-scale-set-controller
|
||||
|
|
@ -5,9 +5,6 @@ for the merge guards.
|
|||
## Scripts
|
||||
The scripts are written in bash (for Linux) and powershell (for Windows).
|
||||
|
||||
### build_run.(sh|ps1)
|
||||
Build the docker image and run it locally. This is useful for testing it.
|
||||
|
||||
### build_deploy.(sh|ps1)
|
||||
Build the docker and deploy it to the GCP registry. This is useful for deploying
|
||||
it in the Kubernetes cluster.
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
cloudbuild.yaml
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
FROM debian:stable
|
||||
|
||||
RUN echo 'intall packages'; \
|
||||
apt-get update; \
|
||||
apt-get install -y --no-install-recommends \
|
||||
locales openssh-client gnupg ca-certificates \
|
||||
zip wget git \
|
||||
gdb build-essential \
|
||||
ninja-build \
|
||||
libelf-dev libffi-dev gcc-multilib \
|
||||
# for llvm-libc tests that build mpfr and gmp from source
|
||||
autoconf automake libtool \
|
||||
ccache \
|
||||
python3 python3-psutil \
|
||||
python3-pip python3-setuptools \
|
||||
lsb-release software-properties-common \
|
||||
swig python3-dev libedit-dev libncurses5-dev libxml2-dev liblzma-dev golang rsync jq \
|
||||
# for llvm installation script
|
||||
sudo;
|
||||
|
||||
# debian stable cmake is 3.18, we need to install a more recent version.
|
||||
RUN wget --no-verbose -O /cmake.sh https://github.com/Kitware/CMake/releases/download/v3.23.3/cmake-3.23.3-linux-x86_64.sh; \
|
||||
chmod +x /cmake.sh; \
|
||||
mkdir -p /etc/cmake; \
|
||||
/cmake.sh --prefix=/etc/cmake --skip-license; \
|
||||
ln -s /etc/cmake/bin/cmake /usr/bin/cmake; \
|
||||
cmake --version; \
|
||||
rm /cmake.sh
|
||||
|
||||
# LLVM must be installed after prerequsite packages.
|
||||
ENV LLVM_VERSION=16
|
||||
RUN echo 'install llvm ${LLVM_VERSION}'; \
|
||||
wget --no-verbose https://apt.llvm.org/llvm.sh; \
|
||||
chmod +x llvm.sh; \
|
||||
./llvm.sh ${LLVM_VERSION};\
|
||||
apt-get update; \
|
||||
apt install -y clang-${LLVM_VERSION} clang-format-${LLVM_VERSION} clang-tidy-${LLVM_VERSION} lld-${LLVM_VERSION}; \
|
||||
ln -s /usr/bin/clang-${LLVM_VERSION} /usr/bin/clang;\
|
||||
ln -s /usr/bin/clang++-${LLVM_VERSION} /usr/bin/clang++;\
|
||||
ln -s /usr/bin/clang-tidy-${LLVM_VERSION} /usr/bin/clang-tidy;\
|
||||
ln -s /usr/bin/clang-tidy-diff-${LLVM_VERSION}.py /usr/bin/clang-tidy-diff;\
|
||||
ln -s /usr/bin/clang-format-${LLVM_VERSION} /usr/bin/clang-format;\
|
||||
ln -s /usr/bin/clang-format-diff-${LLVM_VERSION} /usr/bin/clang-format-diff;\
|
||||
ln -s /usr/bin/lld-${LLVM_VERSION} /usr/bin/lld;\
|
||||
ln -s /usr/bin/lldb-${LLVM_VERSION} /usr/bin/lldb;\
|
||||
ln -s /usr/bin/ld.lld-${LLVM_VERSION} /usr/bin/ld.lld
|
||||
|
||||
RUN echo 'configure locale'; \
|
||||
sed --in-place '/en_US.UTF-8/s/^#//' /etc/locale.gen ;\
|
||||
locale-gen ;\
|
||||
echo 'make python 3 default'; \
|
||||
rm -f /usr/bin/python && ln -s /usr/bin/python3 /usr/bin/python; \
|
||||
pip3 install wheel
|
||||
|
||||
# Configure locale
|
||||
ENV LANG en_US.UTF-8
|
||||
ENV LANGUAGE en_US:en
|
||||
ENV LC_ALL en_US.UTF-8
|
||||
|
||||
RUN wget --no-verbose -O /usr/bin/bazelisk https://github.com/bazelbuild/bazelisk/releases/download/v1.17.0/bazelisk-linux-amd64; \
|
||||
chmod +x /usr/bin/bazelisk; \
|
||||
bazelisk --version
|
||||
|
||||
RUN echo 'install buildkite' ;\
|
||||
apt-get install -y apt-transport-https gnupg;\
|
||||
sh -c 'echo deb https://apt.buildkite.com/buildkite-agent stable main > /etc/apt/sources.list.d/buildkite-agent.list' ;\
|
||||
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 32A37959C2FA5C3C99EFBC32A79206696452D198 ;\
|
||||
apt-get update ;\
|
||||
apt-get install -y buildkite-agent tini gosu; \
|
||||
apt-get clean;
|
||||
COPY *.sh /usr/local/bin/
|
||||
RUN chmod og+rx /usr/local/bin/*.sh
|
||||
COPY --chown=buildkite-agent:buildkite-agent pre-checkout /etc/buildkite-agent/hooks
|
||||
COPY --chown=buildkite-agent:buildkite-agent post-checkout /etc/buildkite-agent/hooks
|
||||
|
||||
# buildkite working directory
|
||||
VOLUME /var/lib/buildkite-agent
|
||||
|
||||
ENTRYPOINT ["entrypoint.sh"]
|
||||
CMD ["gosu", "buildkite-agent", "buildkite-agent", "start", "--no-color"]
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# Copyright 2021 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
set -eo pipefail
|
||||
|
||||
USER=buildkite-agent
|
||||
P="${BUILDKITE_BUILD_PATH:-/var/lib/buildkite-agent}"
|
||||
set -u
|
||||
mkdir -p "$P"
|
||||
chown -R ${USER}:${USER} "$P"
|
||||
|
||||
export CCACHE_DIR="${P}"/ccache
|
||||
export CCACHE_MAXSIZE=20G
|
||||
mkdir -p "${CCACHE_DIR}"
|
||||
chown -R ${USER}:${USER} "${CCACHE_DIR}"
|
||||
|
||||
# /mnt/ssh should contain known_hosts, id_rsa and id_rsa.pub .
|
||||
mkdir -p /var/lib/buildkite-agent/.ssh
|
||||
if [ -d /mnt/ssh ]; then
|
||||
cp /mnt/ssh/* /var/lib/buildkite-agent/.ssh || echo "no "
|
||||
chmod 700 /var/lib/buildkite-agent/.ssh
|
||||
chmod 600 /var/lib/buildkite-agent/.ssh/*
|
||||
chown -R buildkite-agent:buildkite-agent /var/lib/buildkite-agent/.ssh/
|
||||
else
|
||||
echo "/mnt/ssh is not mounted"
|
||||
fi
|
||||
exec /usr/bin/tini -g -- $@
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
# Copyright 2021 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# The `post-checkout` hook runs after checkout.
|
||||
|
||||
set -eu
|
||||
|
||||
# Run prune as git will start to complain "warning: There are too many unreachable loose objects; run 'git prune' to remove them.".
|
||||
# That happens as we routinely drop old branches.
|
||||
git prune
|
||||
if [ -f ".git/gc.log" ]; then
|
||||
echo ".git/gc.log exist"
|
||||
cat ./.git/gc.log
|
||||
rm -f ./.git/gc.log
|
||||
fi
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# The `pre-checkout` hook will run just before your pipelines source code is
|
||||
# checked out from your SCM provider
|
||||
|
||||
set -e
|
||||
|
||||
# Convert https://github.com/llvm-premerge-tests/llvm-project.git -> llvm-project
|
||||
# to use the same directory for fork and origin.
|
||||
BUILDKITE_BUILD_CHECKOUT_PATH="${BUILDKITE_BUILD_PATH}/$(echo $BUILDKITE_REPO | sed -E "s#.*/([^/]*)#\1#" | sed "s/.git$//")"
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# define command line arguments
|
||||
param(
|
||||
[Parameter(Mandatory=$true)][string]$IMAGE_NAME,
|
||||
[Parameter(Mandatory=$false)][string]$CMD="",
|
||||
[string]$token
|
||||
)
|
||||
|
||||
# set script to stop on first error
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
# some docs recommend setting 2GB memory limit
|
||||
docker build `
|
||||
--memory 2GB `
|
||||
-t $IMAGE_NAME `
|
||||
--build-arg token=$token `
|
||||
"$PSScriptRoot\$IMAGE_NAME"
|
||||
If ($LastExitCode -ne 0) {
|
||||
exit
|
||||
}
|
||||
|
||||
$DIGEST=$(docker image inspect --format "{{range .RepoDigests}}{{.}}{{end}}" $IMAGE_NAME) -replace ".*@sha256:(.{6})(.*)$","`$1"
|
||||
# mount a persistent workspace for experiments
|
||||
docker run -it `
|
||||
-v C:/ws:C:/ws `
|
||||
-v C:/credentials:C:/credentials `
|
||||
-e BUILDKITE_BUILD_PATH=C:\ws `
|
||||
-e IMAGE_DIGEST=${DIGEST} `
|
||||
-e PARENT_HOSTNAME=$env:computername `
|
||||
$IMAGE_NAME $CMD
|
||||
If ($LastExitCode -ne 0) {
|
||||
exit
|
||||
}
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Starts a new instances of a docker image. Example:
|
||||
# sudo build_run.sh buildkite-premerge-debian /bin/bash
|
||||
|
||||
set -eux
|
||||
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
|
||||
|
||||
IMAGE_NAME="${1%/}"
|
||||
|
||||
cd "${DIR}/${IMAGE_NAME}"
|
||||
docker build -t ${IMAGE_NAME} .
|
||||
docker run -i -t -v ~/.llvm-premerge-checks:/credentials -v ${DIR}/workspace:/workspace ${IMAGE_NAME} ${2}
|
||||
|
|
@ -1,5 +1,3 @@
|
|||
# TODO: remove after migrating scripts to LLVM repo.
|
||||
git clone https://github.com/google/llvm-premerge-checks.git c:\llvm-premerge-checks
|
||||
# Checkout path is set to a fixed short value (e.g. c:\ws\src) to keep paths
|
||||
# short as many tools break on Windows with paths longer than 250.
|
||||
if ($null -eq $env:BUILDKITE_BUILD_PATH) { $env:BUILDKITE_BUILD_PATH = 'c:\ws' }
|
||||
|
|
|
|||
|
|
@ -1,105 +0,0 @@
|
|||
FROM ubuntu:latest
|
||||
|
||||
RUN echo 'intall packages'; \
|
||||
apt-get update; \
|
||||
apt-get upgrade; \
|
||||
apt-get install -y --no-install-recommends \
|
||||
gosu \
|
||||
locales openssh-client gnupg ca-certificates \
|
||||
zip unzip wget curl git \
|
||||
gdb build-essential \
|
||||
ninja-build \
|
||||
libelf-dev libffi-dev gcc-multilib libmpfr-dev libpfm4-dev \
|
||||
# for llvm-libc tests that build mpfr and gmp from source
|
||||
autoconf automake libtool \
|
||||
python3 python3-psutil python3-pip python3-setuptools \
|
||||
lsb-release software-properties-common \
|
||||
swig python3-dev libedit-dev libncurses5-dev libxml2-dev liblzma-dev golang rsync jq \
|
||||
# for llvm installation script
|
||||
sudo \
|
||||
# build scripts
|
||||
nodejs \
|
||||
# shell users
|
||||
less vim
|
||||
|
||||
# debian stable cmake is 3.18, we need to install a more recent version.
|
||||
RUN wget --no-verbose -O /cmake.sh https://github.com/Kitware/CMake/releases/download/v3.23.3/cmake-3.23.3-linux-x86_64.sh; \
|
||||
chmod +x /cmake.sh; \
|
||||
mkdir -p /etc/cmake; \
|
||||
/cmake.sh --prefix=/etc/cmake --skip-license; \
|
||||
ln -s /etc/cmake/bin/cmake /usr/bin/cmake; \
|
||||
cmake --version; \
|
||||
rm /cmake.sh
|
||||
|
||||
# LLVM must be installed after prerequsite packages.
|
||||
ENV LLVM_VERSION=16
|
||||
RUN echo 'install llvm ${LLVM_VERSION}' && \
|
||||
wget --no-verbose https://apt.llvm.org/llvm.sh && \
|
||||
chmod +x llvm.sh && \
|
||||
./llvm.sh ${LLVM_VERSION} && \
|
||||
apt-get update && \
|
||||
apt-get install -y clang-${LLVM_VERSION} clang-format-${LLVM_VERSION} clang-tidy-${LLVM_VERSION} lld-${LLVM_VERSION} && \
|
||||
ln -s /usr/bin/clang-${LLVM_VERSION} /usr/bin/clang && \
|
||||
ln -s /usr/bin/clang++-${LLVM_VERSION} /usr/bin/clang++ && \
|
||||
ln -s /usr/bin/clang-tidy-${LLVM_VERSION} /usr/bin/clang-tidy && \
|
||||
ln -s /usr/bin/clang-tidy-diff-${LLVM_VERSION}.py /usr/bin/clang-tidy-diff && \
|
||||
ln -s /usr/bin/clang-format-${LLVM_VERSION} /usr/bin/clang-format && \
|
||||
ln -s /usr/bin/clang-format-diff-${LLVM_VERSION} /usr/bin/clang-format-diff && \
|
||||
ln -s /usr/bin/lld-${LLVM_VERSION} /usr/bin/lld && \
|
||||
ln -s /usr/bin/lldb-${LLVM_VERSION} /usr/bin/lldb && \
|
||||
ln -s /usr/bin/ld.lld-${LLVM_VERSION} /usr/bin/ld.lld && \
|
||||
clang --version
|
||||
|
||||
RUN echo 'configure locale' && \
|
||||
sed --in-place '/en_US.UTF-8/s/^#//' /etc/locale.gen && \
|
||||
locale-gen
|
||||
ENV LANG en_US.UTF-8
|
||||
ENV LANGUAGE en_US:en
|
||||
ENV LC_ALL en_US.UTF-8
|
||||
|
||||
RUN curl -o sccache-v0.5.4-x86_64-unknown-linux-musl.tar.gz -L https://github.com/mozilla/sccache/releases/download/v0.5.4/sccache-v0.5.4-x86_64-unknown-linux-musl.tar.gz \
|
||||
&& echo "4bf3ce366aa02599019093584a5cbad4df783f8d6e3610548c2044daa595d40b sccache-v0.5.4-x86_64-unknown-linux-musl.tar.gz" | shasum -a 256 -c \
|
||||
&& tar xzf ./sccache-v0.5.4-x86_64-unknown-linux-musl.tar.gz \
|
||||
&& mv sccache-v0.5.4-x86_64-unknown-linux-musl/sccache /usr/bin \
|
||||
&& chown root:root /usr/bin/sccache \
|
||||
&& ls -la /usr/bin/sccache \
|
||||
&& sccache --version
|
||||
|
||||
RUN groupadd -g 121 runner \
|
||||
&& useradd -mr -d /home/runner -u 1001 -g 121 runner \
|
||||
&& mkdir -p /home/runner/_work
|
||||
|
||||
WORKDIR /home/runner
|
||||
|
||||
# https://github.com/actions/runner/releases
|
||||
ARG RUNNER_VERSION="2.311.0"
|
||||
ARG RUNNER_ARCH="x64"
|
||||
|
||||
RUN curl -f -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz \
|
||||
&& tar xzf ./runner.tar.gz \
|
||||
&& rm runner.tar.gz
|
||||
|
||||
# https://github.com/actions/runner-container-hooks/releases
|
||||
ARG RUNNER_CONTAINER_HOOKS_VERSION="0.4.0"
|
||||
RUN curl -f -L -o runner-container-hooks.zip https://github.com/actions/runner-container-hooks/releases/download/v${RUNNER_CONTAINER_HOOKS_VERSION}/actions-runner-hooks-k8s-${RUNNER_CONTAINER_HOOKS_VERSION}.zip \
|
||||
&& unzip ./runner-container-hooks.zip -d ./k8s \
|
||||
&& rm runner-container-hooks.zip
|
||||
|
||||
RUN chown -R runner:runner /home/runner
|
||||
|
||||
# Copy entrypoint but script. There is no point of adding
|
||||
# `ENTRYPOINT ["/entrypoint.sh"]` as k8s runs it with explicit command and
|
||||
# entrypoint is ignored.
|
||||
COPY entrypoint.sh /entrypoint.sh
|
||||
RUN chmod +x /entrypoint.sh
|
||||
# USER runner
|
||||
|
||||
# We don't need entry point and token logic as it's all orchestrated by action runner controller.
|
||||
# BUT we need to start server etc
|
||||
|
||||
|
||||
# RUN chmod +x /entrypoint.sh /token.sh
|
||||
# # try: USER runner instead of gosu
|
||||
# ENTRYPOINT ["/entrypoint.sh"]
|
||||
# CMD ["sleep", "infinity"]
|
||||
# CMD ["./bin/Runner.Listener", "run", "--startuptype", "service"]
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
steps:
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: [ 'build', '-t', 'us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux', '.' ]
|
||||
images:
|
||||
- 'us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux:latest'
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# Copyright 2023 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
set -ueo pipefail
|
||||
|
||||
export PATH=${PATH}:/home/runner
|
||||
|
||||
USER=runner
|
||||
WORKDIR=${WORKDIR:-/home/runner/_work}
|
||||
|
||||
export SCCACHE_DIR="${WORKDIR}/sccache"
|
||||
mkdir -p "${SCCACHE_DIR}"
|
||||
chown -R ${USER}:${USER} "${SCCACHE_DIR}"
|
||||
chmod oug+rw "${SCCACHE_DIR}"
|
||||
gosu runner bash -c 'SCCACHE_DIR="${SCCACHE_DIR}" SCCACHE_IDLE_TIMEOUT=0 SCCACHE_CACHE_SIZE=20G sccache --start-server'
|
||||
sccache --show-stats
|
||||
|
||||
[[ ! -d "${WORKDIR}" ]] && mkdir -p "${WORKDIR}"
|
||||
|
||||
# exec /usr/bin/tini -g -- $@
|
||||
gosu runner "$@"
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# https://github.com/myoung34/docker-github-actions-runner/blob/master/token.sh
|
||||
# Licensed under MIT
|
||||
# https://github.com/myoung34/docker-github-actions-runner/blob/master/LICENSE
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
_GITHUB_HOST=${GITHUB_HOST:="github.com"}
|
||||
|
||||
# If URL is not github.com then use the enterprise api endpoint
|
||||
if [[ ${GITHUB_HOST} = "github.com" ]]; then
|
||||
URI="https://api.${_GITHUB_HOST}"
|
||||
else
|
||||
URI="https://${_GITHUB_HOST}/api/v3"
|
||||
fi
|
||||
|
||||
API_VERSION=v3
|
||||
API_HEADER="Accept: application/vnd.github.${API_VERSION}+json"
|
||||
AUTH_HEADER="Authorization: token ${ACCESS_TOKEN}"
|
||||
CONTENT_LENGTH_HEADER="Content-Length: 0"
|
||||
|
||||
case ${RUNNER_SCOPE} in
|
||||
org*)
|
||||
_FULL_URL="${URI}/orgs/${ORG_NAME}/actions/runners/registration-token"
|
||||
;;
|
||||
|
||||
ent*)
|
||||
_FULL_URL="${URI}/enterprises/${ENTERPRISE_NAME}/actions/runners/registration-token"
|
||||
;;
|
||||
|
||||
*)
|
||||
_PROTO="https://"
|
||||
# shellcheck disable=SC2116
|
||||
_URL="$(echo "${REPO_URL/${_PROTO}/}")"
|
||||
_PATH="$(echo "${_URL}" | grep / | cut -d/ -f2-)"
|
||||
_ACCOUNT="$(echo "${_PATH}" | cut -d/ -f1)"
|
||||
_REPO="$(echo "${_PATH}" | cut -d/ -f2)"
|
||||
_FULL_URL="${URI}/repos/${_ACCOUNT}/${_REPO}/actions/runners/registration-token"
|
||||
;;
|
||||
esac
|
||||
|
||||
RUNNER_TOKEN="$(curl -XPOST -fsSL \
|
||||
-H "${CONTENT_LENGTH_HEADER}" \
|
||||
-H "${AUTH_HEADER}" \
|
||||
-H "${API_HEADER}" \
|
||||
"${_FULL_URL}" \
|
||||
| jq -r '.token')"
|
||||
|
||||
echo "{\"token\": \"${RUNNER_TOKEN}\", \"full_url\": \"${_FULL_URL}\"}"
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
FROM ubuntu:latest
|
||||
|
||||
RUN apt-get update ;\
|
||||
apt-get upgrade -y;\
|
||||
apt-get install -y --no-install-recommends \
|
||||
locales openssh-client gnupg ca-certificates \
|
||||
build-essential \
|
||||
zip wget git less vim \
|
||||
python3 python3-psutil python3-pip python3-setuptools pipenv \
|
||||
# PostgreSQL
|
||||
libpq-dev \
|
||||
# debugging
|
||||
less vim wget curl \
|
||||
rsync jq tini gosu;
|
||||
|
||||
RUN echo 'configure locale' && \
|
||||
sed --in-place '/en_US.UTF-8/s/^#//' /etc/locale.gen && \
|
||||
locale-gen
|
||||
ENV LANG en_US.UTF-8
|
||||
ENV LANGUAGE en_US:en
|
||||
ENV LC_ALL en_US.UTF-8
|
||||
|
||||
COPY *.sh /usr/local/bin/
|
||||
# requirements.txt generated by running `pipenv lock -r > ../../containers/stats/requirements.txt` in `scripts/metrics`.
|
||||
# COPY requirements.txt /tmp/
|
||||
# RUN pip install -q -r /tmp/requirements.txt
|
||||
RUN chmod og+rx /usr/local/bin/*.sh
|
||||
RUN wget -q https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O /usr/local/bin/cloud_sql_proxy;\
|
||||
chmod +x /usr/local/bin/cloud_sql_proxy
|
||||
|
||||
ENTRYPOINT ["entrypoint.sh"]
|
||||
CMD ["/bin/bash"]
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# Copyright 2021 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
set -euo pipefail
|
||||
|
||||
git clone --depth 1 https://github.com/google/llvm-premerge-checks.git ~/llvm-premerge-checks
|
||||
cd ~/llvm-premerge-checks
|
||||
git fetch origin "${SCRIPTS_REFSPEC:=main}":x
|
||||
git checkout x
|
||||
exec /usr/bin/tini -g -- $@
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
backoff==1.10.0
|
||||
certifi==2023.7.22
|
||||
chardet==4.0.0
|
||||
ftfy==6.0.1; python_version >= '3.6'
|
||||
gitdb==4.0.7
|
||||
gitpython==3.1.41
|
||||
idna==2.10
|
||||
lxml==4.9.1
|
||||
mailchecker==4.0.7
|
||||
pathspec==0.8.1
|
||||
phabricator==0.8.1
|
||||
phonenumbers==8.12.23
|
||||
psycopg2-binary==2.8.6
|
||||
pyaml==20.4.0
|
||||
python-benedict==0.24.0
|
||||
python-dateutil==2.8.1
|
||||
python-fsutil==0.5.0
|
||||
python-slugify==5.0.2
|
||||
pyyaml==5.4.1
|
||||
requests==2.31.0
|
||||
six==1.16.0
|
||||
smmap==4.0.0
|
||||
text-unidecode==1.3
|
||||
toml==0.10.2
|
||||
urllib3==1.26.18
|
||||
wcwidth==0.2.5
|
||||
xmltodict==0.12.0
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
# clang-tidy checks
|
||||
## Warning is not useful
|
||||
If you found that a warning produced by clang-tidy is not useful:
|
||||
|
||||
- If clang-tidy must not run for some files at all (e.g. lit test), please
|
||||
[add files to ignorelist](../scripts/clang-tidy.ignore).
|
||||
|
||||
- Consider fixing or [suppressing diagnostic](https://clang.llvm.org/extra/clang-tidy/#suppressing-undesired-diagnostics)
|
||||
if there is a good reason.
|
||||
|
||||
- [File a bug](https://github.com/google/llvm-premerge-checks/issues/new?assignees=&labels=bug&template=bug_report.md&title=)
|
||||
if build process should be improved.
|
||||
|
||||
- If you believe that you found a clang-tidy bug then please keep in mind that clang-tidy version run by bot
|
||||
might not be most recent. Please reproduce your issue on current version before submitting a bug to clang-tidy.
|
||||
|
||||
## Review comments
|
||||
|
||||
Build bot leaves inline comments only for a small subset of files that are not blacklisted for analysis (see above) *and*
|
||||
specifically whitelisted for comments.
|
||||
|
||||
That is done to avoid potential noise when a file already contains a number of warnings.
|
||||
|
||||
If your are confident that some files are in good shape already, please
|
||||
[whitelist them](../scripts/clang-tidy-comments.ignore).
|
||||
|
||||
----
|
||||
|
||||
[about pre-merge checks](../README.md)
|
||||
|
|
@ -1,69 +0,0 @@
|
|||
## Goal
|
||||
|
||||
Migrate from the current build infrastructure on Jenkins. It works but:
|
||||
|
||||
- due to security concerns we cannot give users access to build interface and has to generate reports,copy artifacts etc. to mimic that.
|
||||
|
||||
- creating builds with new configurations, e.g. libc++ tests, requires a lot of cooperation between us and interested user.
|
||||
|
||||
- it's not possible for contributors to add new agents with different configurations. It also not clear how to make jenkins auto-scale number of agents.
|
||||
|
||||
- intial idea of "look we have a plugin to integrate Jenkins with Phabricator, that will make everything easy" does not hold: we have to interact with Conduit API and understand Phabricator quirks anyway to provide experience we want.
|
||||
|
||||
- it's hard to reproduce builds locally.
|
||||
|
||||
- relying only on Herald notifications does not seems to work ([#96](https://github.com/google/llvm-premerge-checks/issues/96))
|
||||
|
||||
**Non goal**
|
||||
|
||||
No changes to the current review workflow, e.g. run tests only if commit if is LGTM-ed [#60](https://github.com/google/llvm-premerge-checks/issues/60)).
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
I have looked at [GitHub actions](https://help.github.com/en/actions) and [Buildkite](https://buildkite.com/docs/tutorials/getting-started) as alternatives to current setup.
|
||||
|
||||
While GitHub actions might be providing some interesting integration features with GitHub (maybe) I found it lacks some nice features that Buildkite have. Namely actions have:
|
||||
|
||||
- a limited support for shell and proposes javascript or docker containers as an "executor". That might not be as easy to debug or use to reproduce locally;
|
||||
|
||||
- a workflows that are more or less predefined, so to start specific steps for, e.g. 'clang-extra', we will likely need to start a job and detect within it that build is not needed. Buildkite give ability to dynamically generate steps as we go;
|
||||
|
||||
- very limited support for scaling / monitoring self-hosted agents ([issue](https://github.community/t5/GitHub-Actions/Self-hosted-runner-autoscaling/td-p/38416)) as well as [security issues](https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories).
|
||||
|
||||
## Proposed setup
|
||||
|
||||
Use buildkite pipelines and define them here, in llvm-premerge-checks repo. Diffs are going to be reflected in llvm-project fork.
|
||||
Hopefully buildkite alone should be enought for the common workflow. If it's not enough and/or there are some issues with phabricator-buildkite itegration that require client update (Phabricator, like we had with [Jenkins CSRF](https://github.com/google/llvm-premerge-checks/issues/163) we can introduce a separate "conrol" service. This service can also have a DB to collect stats, maintain cohesive history of every revisions, manage tricky workflow cases, and manage scaling of buildkite agents. As much as I don't wan't to maintain a new custom service it seems to be a better idea on a long run to work around missing features or having an ability to maintain a complex state or history.
|
||||
|
||||
### Common workflow
|
||||
In the most simple case build will be triggered directly by the Harbormaster "buildkite" rule. Build pipeline will receive PHID / DiffID and:
|
||||
|
||||
- using the ID, communicate with the Phabricator and decide what diffs have to be applied to represent the change
|
||||
- report back to Phabricator a link to the build in buildkite
|
||||
- create a new branch in llvm-project fork and apply changes as a separate commits
|
||||
- run "build planning" script that will generate further steps based on e.g. author or the diff (if she participates in beta-testing), affected paths, trottling...; there might be multiple steps like this further down the line.
|
||||
- complete every build and attach relevant artifacts to the build; add linter comments
|
||||
- report final build result back.
|
||||
|
||||
Later this workflow can be split into two steps: one will setup github branch and create a PR. Another one will be triggered by GitHub and report PR build status to it. There are challenges on who and when is going to communicate back to Phabricator results of the build. Alternatively we can create a separate pipeline for the reviews on GitHub.
|
||||
|
||||
### Cleaning up branches / PR
|
||||
|
||||
Branches / PR are going to be kept for some time (for example month) after review is completed of diff seems to be abandoned.
|
||||
That can intially be done by running a scheduled build on buildkite that will list branches and communicate with Phabricator to understand what to be deleted. Alternatively we can use "control" as can easily maintain state and, thus can do less work and even get notifications from Phabricator about revisions being closed.
|
||||
|
||||
### Missed diffs
|
||||
|
||||
We see that in some cases [#96](https://github.com/google/llvm-premerge-checks/issues/96) phabricator does not trigger a build. But we still want to run them. We can periodically (hourly) list recent reviews in phabricator and figure out if builds already being run for them. I see two possibilites here:
|
||||
|
||||
1. (preferred) find a way to trigger a build for such missed revision. It's definitely possible to do from the UI / command line but missing from the conduit API. This way "Common workflow" will kick in and we will get a build.
|
||||
|
||||
2. start a build "out of bound" and attach a comment to the review, describing that situation and providing a link to the build status. Update comment when build is done.
|
||||
|
||||
### Custom worflows / build agents
|
||||
|
||||
This setup allows community to relatively easily add own agents with relevant "purpose-tag". E.g. libc++ contributors can add a new condition into the "build planning" step that will add new steps (including ones that target their own agents) if relevant code was modified.
|
||||
|
||||
### Autoscaling
|
||||
|
||||
Buildkite provides a [number of ways](https://buildkite.com/docs/tutorials/parallel-builds#auto-scaling-your-build-agents) to collect agent stats and decide if we should up or downscale the number of agents. Technically that can be probably be done as a separate build pipeline though for security reasons and ease of implementation we might want to do that either in "control" service or in a completely separate one that is not accessible publicly.
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
@startuml
|
||||
component phabricator
|
||||
component buildkite
|
||||
|
||||
node github {
|
||||
component "llvm-project"
|
||||
component "llvm-project fork"
|
||||
component "llvm-premerge-checks"
|
||||
}
|
||||
|
||||
node "GCP" {
|
||||
component "buildkite agents"
|
||||
component "phab-proxy"
|
||||
}
|
||||
|
||||
|
||||
[phabricator] --> [phab-proxy]
|
||||
[phab-proxy] --> [buildkite]
|
||||
[buildkite] <-- [buildkite agents]
|
||||
[buildkite agents] <-- [llvm-project]
|
||||
[buildkite agents] <--> [llvm-project fork]
|
||||
[buildkite agents] --> [phabricator]
|
||||
[buildkite agents] <-- [llvm-premerge-checks]
|
||||
@enduml
|
||||
|
|
@ -1,189 +0,0 @@
|
|||
- [Overview](#overview)
|
||||
- [Buildkite agents](#buildkite-agents)
|
||||
- [Build steps](#build-steps)
|
||||
- [Phabricator integration](#phabricator-integration)
|
||||
- [Life of a pre-merge check](#life-of-a-pre-merge-check)
|
||||
- [Cluster parts](#cluster-parts)
|
||||
- [Ingress and public addresses](#ingress-and-public-addresses)
|
||||
- [Enabled projects and project detection](#enabled-projects-and-project-detection)
|
||||
- [Agent machines](#agent-machines)
|
||||
- [Compilation caching](#compilation-caching)
|
||||
- [Buildkite monitoring](#buildkite-monitoring)
|
||||
|
||||
# Overview
|
||||
|
||||
- [Buildkite](https://buildkite.com/llvm-project) orchestrates each build.
|
||||
|
||||
- multiple Linux and windows agents connected to Buildkite. Agents are run at
|
||||
Google Cloud Platform.
|
||||
|
||||
- [small proxy service](/phabricator-proxy) that takes build requests from
|
||||
[reviews.llvm.org](http://reviews.llvm.org) and converts them into Buildkite
|
||||
build request. Buildkite job sends build results directly to Phabricator.
|
||||
|
||||
- every review creates a new branch in [fork of
|
||||
llvm-project](https://github.com/llvm-premerge-tests/llvm-project).
|
||||
|
||||
|
||||
|
||||
# Buildkite agents
|
||||
|
||||
Agents are deployed in two clusters `llvm-premerge-checks` and `windows-cluster`.
|
||||
The latter is for windows machines (as it is controlled on cluster level if
|
||||
machines can run windows containers).
|
||||
|
||||
Container configurations are in ./containers and deployment configurations are
|
||||
in ./kubernetes. Most important ones are:
|
||||
|
||||
- Windows agents: container `containers/buildkite-windows`, deployment `kubernetes/buildkite/windows.yaml`. TODO: at the moment Docker image is created and uploaded
|
||||
from a windows machine (e.g. win-dev). It would be great to setup a cloudbuild.
|
||||
|
||||
- Linux agents: run tests for linux x86 config, container `containers/buildkite-linux`, deployment `kubernetes/buildkite/linux.yaml`.
|
||||
|
||||
- Service agents: run low CPU build steps (e.g. generate pipeline steps) container `containers/buildkite-linux`, deployment `kubernetes/buildkite/service.yaml`.
|
||||
|
||||
All deployments have a copy `..-test` to be used as a test playground to check
|
||||
container / deployment changes before modifying "prod" setup.
|
||||
|
||||
# Build steps
|
||||
|
||||
Buildkite allows [dynamically define pipelines as the output of a
|
||||
command](https://buildkite.com/docs/pipelines/defining-steps#dynamic-pipelines).
|
||||
And most of pipelines use this pattern of running a script and using the
|
||||
resulting yaml. E.g. script to run pull-request checks is llvm-project [.ci/generate-buildkite-pipeline-premerge](https://github.com/llvm/llvm-project/blob/main/.ci/generate-buildkite-pipeline-premerge). Thus any changes to steps to run should
|
||||
go into that script.
|
||||
|
||||
We have a legacy set of scripts in `/scripts` in this repo but discourage new
|
||||
use and development of them - they are mostly kept to make Phabricator integration
|
||||
to function.
|
||||
|
||||
# Phabricator integration
|
||||
|
||||
Note: this section is about integrating with Phabricator that is now discouraged,
|
||||
some things might already be renamed or straight broken as we moving to Pull Requests.
|
||||
|
||||
- [Harbormaster build plan](https://reviews.llvm.org/harbormaster/plan/5) the
|
||||
Phabricator side these things were configured
|
||||
|
||||
- Herald [rule for everyone](https://reviews.llvm.org/H576) and for [beta
|
||||
testers](https://reviews.llvm.org/H511). Note that right now there is no
|
||||
difference between beta and "normal" builds.
|
||||
|
||||
- the [merge_guards_bot user](https://reviews.llvm.org/p/merge_guards_bot/)
|
||||
account for writing comments.
|
||||
|
||||
## Life of a pre-merge check
|
||||
|
||||
When new diff arrives for review it triggers a Herald rule ("everyone" or "beta
|
||||
testers").
|
||||
|
||||
That in sends an HTTP POST request to [**phab-proxy**](../phabricator-proxy)
|
||||
that submits a new buildkite job **diff-checks**. All parameters from the
|
||||
original request are put in the build's environment with `ph_` prefix (to avoid
|
||||
shadowing any Buildkite environment variable). "ph_scripts_refspec" parameter
|
||||
defines refspec of llvm-premerge-checks to use ("main" by default).
|
||||
|
||||
**diff-checks** pipeline
|
||||
([create_branch_pipeline.py](../scripts/create_branch_pipeline.py))
|
||||
downloads a patch (or series of patches) and applies it to a fork of the
|
||||
llvm-project repository. Then it pushes a new state as a new branch (e.g.
|
||||
"phab-diff-288211") and triggers "premerge-checks" on it (all "ph_" env
|
||||
variables are passed to it). This new branch can now be used to reproduce the
|
||||
build or by another tooling. Periodical **cleanup-branches** pipeline deletes
|
||||
branches older than 30 days.
|
||||
|
||||
**premerge-checks** pipeline
|
||||
([build_branch_pipeline.py](../scripts/build_branch_pipeline.py))
|
||||
builds and tests changes on Linux and Windows agents. Then it uploads a
|
||||
combined result to Phabricator.
|
||||
|
||||
## Cluster parts
|
||||
|
||||
### Ingress and public addresses
|
||||
|
||||
We use NGINX ingress for Kubernetes. Right now it's only used to provide basic
|
||||
HTTP authentication and forwards all requests from load balancer to
|
||||
[phabricator proxy](../phabricator-proxy) application.
|
||||
|
||||
Follow up to date docs to install [reverse
|
||||
proxy](https://kubernetes.github.io/ingress-nginx/deploy/#gce-gke).
|
||||
|
||||
[cert-manager] is installed with helm https://cert-manager.io/docs/installation/helm/
|
||||
|
||||
helm install \
|
||||
cert-manager jetstack/cert-manager \
|
||||
--namespace cert-manager \
|
||||
--create-namespace \
|
||||
--version v1.9.1 \
|
||||
--set installCRDs=true
|
||||
|
||||
We also have [certificate manager](https://cert-manager.io/docs/) and
|
||||
[lets-encrypt configuration](../kubernetes/cert-issuer.yaml) in place, but they are
|
||||
not used at the moment and should be removed if we decide to live with static IP.
|
||||
|
||||
HTTP auth is configured with k8s secret 'http-auth' in 'buildkite' namespace
|
||||
(see [how to update auth](playbooks.md#update-http-auth-credentials)).
|
||||
|
||||
## Enabled projects and project detection
|
||||
|
||||
To reduce build times and mask unrelated problems, we're only building and
|
||||
testing the projects that were modified by a patch.
|
||||
[choose_projects.py](../scripts/choose_projects.py) uses manually maintained
|
||||
[config file](../scripts/llvm-dependencies.yaml) to define inter-project
|
||||
dependencies and exclude projects:
|
||||
|
||||
1. Get prefix (e.g. "llvm", "clang") of all paths modified by a patch.
|
||||
|
||||
1. Add all dependant projects.
|
||||
|
||||
1. Add all projects that this extended list depends on, completing the
|
||||
dependency subtree.
|
||||
|
||||
1. Remove all disabled projects.
|
||||
|
||||
## Agent machines
|
||||
|
||||
All build machines are running from Docker containers so that they can be
|
||||
debugged, updated, and scaled easily:
|
||||
|
||||
- [Linux](../containers/buildkite-premerge-debian/Dockerfile). We use
|
||||
[Kubernetes deployment](../kubernetes/buildkite) to manage these agents.
|
||||
|
||||
- [Windows](../containers/agent-windows-buildkite/Dockerfile). At the moment they are run as
|
||||
multiple individual VM instances.
|
||||
|
||||
See [playbooks](playbooks.md) how to manage and set up machines.
|
||||
|
||||
## Compilation caching
|
||||
|
||||
Each build is performed on a clean copy of the git repository. To speed up the
|
||||
builds [ccache](https://ccache.dev/) is used on Linux and
|
||||
[sccache](https://github.com/mozilla/sccache) on Windows.
|
||||
|
||||
# Buildkite monitoring
|
||||
|
||||
FIXME: does not work as of 2023-09-11. Those metrics could allow
|
||||
us to setup auto-scaling of machines to the current demend.
|
||||
|
||||
VM instance `buildkite-monitoring` exposes Buildkite metrics to GCP.
|
||||
To set up a new instance:
|
||||
|
||||
1. Create as small Linux VM with full access to *Stackdriver Monitoring API*.
|
||||
|
||||
1. Follow instructions to [install monitoring
|
||||
agent](https://cloud.google.com/monitoring/agent/install-agent) and [enable
|
||||
statsd plugin](https://cloud.google.com/monitoring/agent/plugins/statsd).
|
||||
|
||||
1. Download recent release of
|
||||
[buildkite-agent-metrics](https://github.com/buildkite/buildkite-agent-metrics/releases).
|
||||
|
||||
1. Run in SSH session:
|
||||
```bash
|
||||
chmod +x buildkite-agent-metrics-linux-amd64
|
||||
nohup ./buildkite-agent-metrics-linux-amd64 -token XXXX -interval 30s -backend statsd &
|
||||
```
|
||||
|
||||
Metrics are exported as "custom/statsd/gauge".
|
||||
|
||||
TODO: update "Testing scripts locally" playbook on how to run Linux build locally with Docker.
|
||||
TODO: migrate 'builkite-monitoring' to k8s deployment.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 22 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 25 KiB |
|
|
@ -1,74 +0,0 @@
|
|||
# Monitoring the LLVM main branch
|
||||
|
||||
This document proposes a design to improve the stability of the LLVM main branch
|
||||
for use in pre-merge testing.
|
||||
|
||||
## Background
|
||||
|
||||
The stability of the pre-merge testing largely depends on the stability of the
|
||||
LLVM main branch: Whenever something is broken on the main branch, also
|
||||
pre-merge testing will fail: The patches from Phabricator are applied to some
|
||||
revision on the main branch before they can be built. So the fewer revisions of
|
||||
the main branch being broken, the more stable pre-merge testing will be.
|
||||
|
||||
## High-level Design
|
||||
|
||||
We propose to run a Buildbot worker on the main branch with the same
|
||||
Docker image we're using for pre-merge testing. That worker shall check all
|
||||
commits to main for build and test failures with regards to the configuration
|
||||
we're using for pre-merge testing. Whenever these builds fails, Buildbot
|
||||
notifies the commiters and gives them the opportunity to fix or revert their
|
||||
patch.
|
||||
|
||||
This is much faster than a having a human investigate the issue and notify the
|
||||
committers. By having faster feedback the main branch is broken for fewer
|
||||
revisions and this the probability of a false-positive pre-merge test is lower.
|
||||
|
||||
## Machine setup
|
||||
|
||||
We would deploy another container as part of the existing Kubernetes cluster for
|
||||
pre-merge testing. The image would be based on
|
||||
[buildkite-premerge-debian](https://github.com/google/llvm-premerge-checks/blob/main/containers/buildkite-premerge-debian/Dockerfile)
|
||||
, and we would just add the things needed for the Buildbot agent by resuing the
|
||||
setup of an existing worker (e.g.
|
||||
[clangd-ubuntu-clang](https://github.com/llvm/llvm-zorg/blob/main/Buildbot/google/docker/Buildbot-clangd-ubuntu-clang/Dockerfile.)
|
||||
).
|
||||
|
||||
## Build configuration
|
||||
|
||||
The Buildbot worker should run as much of the pre-merge testing scripts as
|
||||
possible to get the results as close as we can to that of what we would see
|
||||
happen in pre-merge testing.
|
||||
|
||||
Currently we're running these steps on Buildkite to check the main branch:
|
||||
|
||||
``` bash
|
||||
export SRC=${BUILDKITE_BUILD_PATH}/llvm-premerge-checks
|
||||
rm -rf ${SRC}
|
||||
git clone --depth 1 https://github.com/google/llvm-premerge-checks.git "${SRC}"
|
||||
cd "${SRC}"
|
||||
git fetch origin "${ph_scripts_refspec:-master}":x
|
||||
git checkout x
|
||||
cd "$BUILDKITE_BUILD_CHECKOUT_PATH"
|
||||
pip install -q -r ${SRC}/scripts/requirements.txt
|
||||
${SRC}/scripts/pipeline_main.py | tee /dev/tty | buildkite-agent pipeline upload
|
||||
```
|
||||
|
||||
We should run similar steps on Buildbot by creating a new `BuilderFactory` to
|
||||
the [existing
|
||||
list](https://github.com/llvm/llvm-zorg/tree/main/zorg/buildbot/builders) and
|
||||
then configuring it [as
|
||||
usual](https://github.com/llvm/llvm-zorg/blob/main/buildbot/osuosl/master/config/builders.py).
|
||||
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Do we actually need to create a `BuilderFactory` or can we also just run a
|
||||
shell script?
|
||||
2. How can we quickly test and iterate on the Python scripting? How can we run
|
||||
our own Buildbot master for testing?
|
||||
3. The current solution by only wrapping the existing pre-merge testing scripts
|
||||
will not result in nice build steps as the other builders are producing as
|
||||
buildkite does not have that concept. Is that an issue?
|
||||
4. We do need to check out the pre-merge testing config and scripts form another
|
||||
git repo. Is that an issue?
|
||||
|
|
@ -1,258 +0,0 @@
|
|||
- [Playbooks](#playbooks)
|
||||
* [Testing scripts locally](#testing-scripts-locally)
|
||||
* [Testing changes before merging](#testing-changes-before-merging)
|
||||
* [Deployment to a clean infrastructure](#deployment-to-a-clean-infrastructure)
|
||||
* [Creating docker containers on Windows](#creating-docker-containers-on-windows)
|
||||
* [Spawning a new windows agent](#spawning-a-new-windows-agent)
|
||||
+ [Buildkite](#buildkite)
|
||||
* [Custom environment variables](#custom-environment-variables)
|
||||
* [Update HTTP auth credentials](#update-http-auth-credentials)
|
||||
|
||||
# Playbooks
|
||||
|
||||
## Development environment
|
||||
|
||||
You need will need recent python 3 installed, e.g. follow this
|
||||
[installation guide](https://cloud.google.com/python/docs/setup?hl=en).
|
||||
To install required packages run:
|
||||
|
||||
```shell script
|
||||
pip install -r ./scripts/requirements.txt
|
||||
```
|
||||
optional:
|
||||
```shell script
|
||||
pip install jupyterlab pandas seaborn # for jupyter labs.
|
||||
```
|
||||
|
||||
## Testing scripts locally
|
||||
|
||||
Build and run agent docker image `sudo ./containers/build_run.sh buildkite-premerge-debian /bin/bash`.
|
||||
|
||||
Set `CONDUIT_TOKEN` with your personal one from `https://reviews.llvm.org/settings/user/<USERNAME>/page/apitokens/`.
|
||||
|
||||
## Testing changes before merging
|
||||
|
||||
It's recommended to test even smallest changes before committing them to the `main` branch.
|
||||
|
||||
1. Create a pull request here.
|
||||
1. Manually create a buildkite build in the pipeline you are updating and specify
|
||||
environment variable `ph_scripts_refspec="pull/123/head"`. Replace `123`
|
||||
with your PR number. If you don't have access to create buildkite builds,
|
||||
please ask a reviewer to do that.
|
||||
|
||||
To test "premerge-tests" pipeline pick an existing build and copy "ph_"
|
||||
parameters from it, omitting "ph_target_phid" to skip updating an existing
|
||||
review.
|
||||
|
||||
See also [custom environment variables](#custom-environment-variables).
|
||||
1. Wait for build to complete and maybe attach a link to it to your PR.
|
||||
|
||||
To test changes for the pipeline "setup" step please experiment on a copy first.
|
||||
|
||||
## Deployment to a clean infrastructure
|
||||
|
||||
General remarks:
|
||||
* GCP does not route any traffic to your services unless the service is
|
||||
"healthy". It might take a few minutes after startup before the services is
|
||||
classified as healthy. Until then, you will only see some generic error
|
||||
message.
|
||||
|
||||
These are the steps to set up the build server on a clean infrastructure:
|
||||
1. Configure the tools on your local machine:
|
||||
```bash
|
||||
./local_setup.sh
|
||||
```
|
||||
If you not running docker under your user, you might need to
|
||||
`sudo gcloud auth login --no-launch-browser && sudo gcloud auth configure-docker`
|
||||
before running other commands under sudo.
|
||||
1. Delete the old cluster, if it still exists:
|
||||
```bash
|
||||
cd kubernetes/cluster
|
||||
./cluster_delete.sh
|
||||
```
|
||||
1. Create the cluster:
|
||||
```bash
|
||||
cd kubernetes/cluster
|
||||
./cluster_create.sh
|
||||
```
|
||||
1. Push the docker images to gcr.io:
|
||||
```bash
|
||||
cd containers
|
||||
#for each subfolder:
|
||||
./build_deploy.sh <foldername>
|
||||
```
|
||||
1. Deploy the stack:
|
||||
```bash
|
||||
cd kubernetes
|
||||
./deploy.sh
|
||||
```
|
||||
1. Configure it
|
||||
|
||||
## Running Docker containers
|
||||
|
||||
`./containers/build_run.sh` provides a good way to run a container once to check that it builds and contains good package versions. To work with a container and
|
||||
persist it state do (taking 'base-debian' as example):
|
||||
|
||||
```
|
||||
cd ./containers/base-debian
|
||||
sudo docker build -t base-debian .
|
||||
sudo docker run -it base-debian /bin/bash
|
||||
```
|
||||
|
||||
to resume it from where you left
|
||||
|
||||
```
|
||||
sudo docker start -a -i $(sudo docker ps -a -f ancestor=base-debian -q -l)
|
||||
```
|
||||
|
||||
|
||||
## Creating docker containers on Windows
|
||||
|
||||
If you want to build/update/test docker container for Windows, you need to do this on a Windows machine.
|
||||
|
||||
**Note**: There is an existing *windows-development* machine that you can resume and use for development. Please stop it after use.
|
||||
|
||||
To setup new machine in GCP:
|
||||
|
||||
1. Pick a GCP Windows image with Desktop Support.
|
||||
* pick a "persistent SSD" as boot Disk. This is much faster.
|
||||
* make sure that you give enough permissions in "Identity and API access" to be able to e.g. push new docker images to GCR.
|
||||
|
||||
1. Format the local SSD partition and use it as workspace.
|
||||
1. install [Chocolately](https://chocolatey.org/docs/installation):
|
||||
```powershell
|
||||
iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
|
||||
```
|
||||
1. Install development tools: `choco install -y git googlechrome vscode`
|
||||
1. (optionally) If you want to be able to push changes to github, you need to set up your github SSH keys and user name:
|
||||
```powershell
|
||||
ssh-keygen
|
||||
git config --global user.name <your name>
|
||||
git config --global user.email <your email>
|
||||
```
|
||||
1. Clone premerge checks sources:
|
||||
```powershell
|
||||
cd c:\
|
||||
git clone https://github.com/google/llvm-premerge-checks
|
||||
```
|
||||
1. Install [Docker Enterprise](https://docs.docker.com/ee/docker-ee/windows/docker-ee/) and reboot:
|
||||
```powershell
|
||||
Install-Module DockerMsftProvider -Force
|
||||
Install-Package Docker -ProviderName DockerMsftProvider -Force
|
||||
Restart-Computer
|
||||
```
|
||||
1. Configure the Docker credentials for GCP:
|
||||
```powershell
|
||||
gcloud init # set options according to ./k8s_config here
|
||||
gcloud components install docker-credential-gcr
|
||||
docker-credential-gcr configure-docker
|
||||
```
|
||||
|
||||
## Build / test docker for Windows.
|
||||
|
||||
1. To build and run a dockerfile:
|
||||
```powershell
|
||||
cd llvm-premerge-checks\containers
|
||||
.\build_deploy.ps1 agent-windows-buildkite
|
||||
1. To deploy container:
|
||||
```powershell
|
||||
cd llvm-premerge-checks\containers
|
||||
.\build_deploy.ps1 agent-windows-buildkite
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```powershell
|
||||
cd llvm-premerge-checks\containers
|
||||
.\build_run.ps1 agent-windows-buildkite cmd
|
||||
```
|
||||
|
||||
Test this newly uploaded image:
|
||||
|
||||
```powershell
|
||||
c:\llvm-premerge-check\scripts\windows_agent_start_buildkite.ps1
|
||||
```
|
||||
|
||||
## Spawning a new windows agent
|
||||
|
||||
To spawn a new Windows agent:
|
||||
|
||||
1. Go to the [GCP page](https://pantheon.corp.google.com/compute/instances?project=llvm-premerge-checks&instancessize=50).
|
||||
1. Add new windows machine wih OS "Windows Server" and version with "desktop experience" (so you can RDP) and boot disk size ~500 Gb. There is a "windows-agent-template" that might not be up to date.
|
||||
1. Go to the [GCP page](https://pantheon.corp.google.com/compute/instances?project=llvm-premerge-checks&instancessize=50) again
|
||||
1. Login to the new machine via RDP (you will need a RDP client, e.g. Chrome app).
|
||||
1. (optional, quality of life) Add a powershell shortcut at desktop with "run as admin" flag. Create a folder with machine name (e.g "w16c2-2") somewhere and click "add new toolbar" on windows toolbar: this way it will be easier to identify which machine you are working with later.
|
||||
1. Run these commands in the power shell under admin to bootstrap the Windows machine:
|
||||
```powershell
|
||||
Invoke-WebRequest -uri 'https://raw.githubusercontent.com/google/llvm-premerge-checks/main/scripts/windows_agent_bootstrap.ps1' -OutFile c:\windows_agent_bootstrap.ps1
|
||||
c:/windows_agent_bootstrap.ps1
|
||||
```
|
||||
VM will be restarted after a prompt.
|
||||
|
||||
To start agent manually:
|
||||
```powershell
|
||||
C:\llvm-premerge-checks\scripts\windows_agent_start_buildkite.ps1 [-workdir D:\] [-testing] [-version latest]
|
||||
```
|
||||
|
||||
## Custom environment variables
|
||||
|
||||
Buildkite pipelines have a number of custom environment variables one can set to change their behavior. That is useful to debug issues
|
||||
or test changes. They are mostly used by pipleine generators, e.g. [pipeline_main](../scripts/pipeline_main.py),
|
||||
please refer to the source code for the details. These variables have `ph_` prefix and can be set with URL parameters in Harbormaster build.
|
||||
|
||||
Most commonly used are:
|
||||
|
||||
- `ph_scripts_refspec`: ("main" by default): refspec branch of llvm-premerge-checks to use. This variable is also used in pipeline "bootstrap" in Buildkite interface. Use "branch-name" for branches and "pull/123/head" for Pull Requests.
|
||||
- `ph_dry_run_report`: do not report any results back to Phabricator.
|
||||
- `ph_no_cache`: (if set to any value) clear compilation cache before the build.
|
||||
- `ph_projects`: which projects to use (semicolon separated), "detect" will look on diff to infer the projects, "default" selects all projects.
|
||||
- `ph_notify_email`: comma-separated list of email addresses to be notified when build is complete.
|
||||
- `ph_log_level` ("DEBUG", "INFO", "WARNING" (default) or "ERROR"): log level for build scripts.
|
||||
- `ph_linux_agents`, `ph_windows_agents`: custom JSON constraints on agents. For example, you might put one machine to a custom queue if it's errornous and send jobs to it with `ph_windows_agents={"queue": "custom"}`.
|
||||
- `ph_skip_linux`, `ph_skip_windows` (if set to any value): skip build on this OS.
|
||||
- `ph_skip_generated`: don't run custom steps generated from within llvm-project.
|
||||
|
||||
While trying a new patch for premerge scripts it's typical to start a new build by copying "ph_"
|
||||
env variables from one of the recent builds and appending
|
||||
|
||||
```shell
|
||||
ph_dry_run_report=yes
|
||||
ph_skip_windows=yes
|
||||
ph_skip_generated=yes
|
||||
ph_scripts_refspec="<branch name>"
|
||||
ph_log_level=DEBUG
|
||||
```
|
||||
|
||||
## Update HTTP auth credentials
|
||||
|
||||
To update e.g. buildkite http-auth:
|
||||
```shell script
|
||||
kubectl get secret http-auth -n buildkite -o yaml
|
||||
# base64 decode it's data to 'auth'.
|
||||
echo <data/auth from yaml> | base64 --decode > auth
|
||||
# add / update passwords
|
||||
htpasswd -b auth <user> <pass>
|
||||
# update secret
|
||||
kubectl delete secret http-auth -n buildkite
|
||||
kubectl create secret generic http-auth -n buildkite --from-file=./auth
|
||||
```
|
||||
|
||||
## Update github-ssh secret
|
||||
|
||||
This Kubernetes stores github ssh access files for llvm.premerge.tests@gmail.com user.
|
||||
|
||||
To access it run `kubectl get secret github-ssh -n buildkite -o yaml | tee github-ssh.yam`.
|
||||
It has 3 entries: "id_rsa", "id_rsa.pub", "known_hosts".
|
||||
If you need to updated it, edit resulting yaml above with updated base 64 values.
|
||||
|
||||
To updated "known_hosts" run `ssh-keyscan github.com` and store base64 encoded output in "known_hosts".
|
||||
Then run `kubectl apply -f github-ssh.yaml`.
|
||||
|
||||
To create this secret from scratch run
|
||||
|
||||
`kubectl create secret generic github-ssh --namespace buildkite --from-file ./id_rsa --from-file ./id_rsa.pub --from-file ./known_hosts`
|
||||
|
||||
providing all files needed.
|
||||
|
||||
Note that containers have to be restarted as ssh settings are copied at startup
|
||||
(entrypoint.sh).
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# based on documentation on
|
||||
# https://github.com/jetstack/cert-manager/blob/master/docs/tutorials/acme/quick-start/index.rst
|
||||
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: letsencrypt-staging
|
||||
spec:
|
||||
acme:
|
||||
server: https://acme-staging-v02.api.letsencrypt.org/directory
|
||||
email: goncharov@google.com
|
||||
privateKeySecretRef:
|
||||
name: letsencrypt-staging
|
||||
solvers:
|
||||
- dns01:
|
||||
cloudDNS:
|
||||
# The ID of the GCP project
|
||||
project: "llvm-premerge-checks"
|
||||
# This is the secret used to access the service account
|
||||
serviceAccountSecretRef:
|
||||
name: clouddns-dns01-solver-svc-acct
|
||||
key: key.json
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: letsencrypt-prod
|
||||
spec:
|
||||
acme:
|
||||
server: https://acme-v02.api.letsencrypt.org/directory
|
||||
email: goncharov@google.com
|
||||
privateKeySecretRef:
|
||||
name: letsencrypt-prod
|
||||
solvers:
|
||||
- dns01:
|
||||
cloudDNS:
|
||||
project: "llvm-premerge-checks"
|
||||
serviceAccountSecretRef:
|
||||
name: clouddns-dns01-solver-svc-acct
|
||||
key: key.json
|
||||
# - http01:
|
||||
# ingress:
|
||||
# class: gce
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
set -eux
|
||||
|
||||
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
|
||||
ROOT_DIR="$(dirname ${DIR})"
|
||||
|
||||
# get config options
|
||||
source "${ROOT_DIR}/k8s_config"
|
||||
|
||||
# create the cluster
|
||||
gcloud container clusters create $GCP_CLUSTER --zone $GCP_ZONE \
|
||||
--machine-type=n1-standard-4 --num-nodes=1
|
||||
|
||||
# Linux agents node pool with local ssd.
|
||||
# https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd
|
||||
gcloud container node-pools create linux-agents --cluster $GCP_CLUSTER --zone $GCP_ZONE \
|
||||
--machine-type=n1-standard-32 --num-nodes=2 --local-ssd-count=1
|
||||
|
||||
# created separate cluster for windows, as we need "ip-alias" enabled
|
||||
# this can't be changed in a running cluster...
|
||||
gcloud beta container clusters create $GCP_CLUSTER_WINDOWS \
|
||||
--enable-ip-alias \
|
||||
--num-nodes=1 \
|
||||
--release-channel=rapid \
|
||||
--enable-private-nodes
|
||||
|
||||
# Windows agents with local ssd
|
||||
gcloud container node-pools create windows-pool --cluster $GCP_CLUSTER_WINDOWS \
|
||||
--image-type=WINDOWS_SAC --no-enable-autoupgrade \
|
||||
--machine-type=n1-standard-16 --local-ssd-count=1
|
||||
|
||||
# create static IP address
|
||||
# IP can be created, but not used in Ingress. Not sure why
|
||||
gcloud compute addresses create web-static-ip --zone=$GCP_ZONE
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
set -eux
|
||||
|
||||
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
|
||||
ROOT_DIR="$(dirname ${DIR})"
|
||||
|
||||
# get config options
|
||||
source "${ROOT_DIR}/k8s_config"
|
||||
|
||||
#TODO: check gcloud auth login
|
||||
|
||||
# create the cluster
|
||||
gcloud container clusters delete $GCP_CLUSTER
|
||||
|
|
@ -1,63 +0,0 @@
|
|||
# Copyright 2021 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
apiVersion: batch/v1beta1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: buildbot-stats
|
||||
namespace: buildkite
|
||||
spec:
|
||||
schedule: "40 * * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 24
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: collect-buildkite-stats
|
||||
image: gcr.io/llvm-premerge-checks/stats:latest
|
||||
args: ["/root/llvm-premerge-checks/scripts/metrics/connect_db.sh python3 buildbot_monitoring.py"]
|
||||
env:
|
||||
- name: BUILDKITE_AGENT_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: buildkite-agent-token
|
||||
key: token
|
||||
- name: POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: CONDUIT_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: conduit-api-token
|
||||
key: token
|
||||
- name: BUILDKITE_API_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: buildkite-api-token
|
||||
key: token
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: db-stats
|
||||
key: password
|
||||
- name: SCRIPTS_REFSPEC
|
||||
value: "main"
|
||||
restartPolicy: Never
|
||||
nodeSelector:
|
||||
cloud.google.com/gke-nodepool: service
|
||||
|
|
@ -1,63 +0,0 @@
|
|||
# Copyright 2021 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: buildkite-stats
|
||||
namespace: buildkite
|
||||
spec:
|
||||
schedule: "0 * * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 24
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: collect-buildkite-stats
|
||||
image: gcr.io/llvm-premerge-checks/stats:latest
|
||||
args: ["/root/llvm-premerge-checks/scripts/metrics/connect_db.sh download_buildkite_builds_pg.sh"]
|
||||
env:
|
||||
- name: BUILDKITE_AGENT_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: buildkite-agent-token
|
||||
key: token
|
||||
- name: POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: CONDUIT_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: conduit-api-token
|
||||
key: token
|
||||
- name: BUILDKITE_API_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: buildkite-api-token
|
||||
key: token
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: db-stats
|
||||
key: password
|
||||
- name: SCRIPTS_REFSPEC
|
||||
value: "main"
|
||||
restartPolicy: Never
|
||||
nodeSelector:
|
||||
cloud.google.com/gke-nodepool: default-pool
|
||||
|
|
@ -1,63 +0,0 @@
|
|||
# Copyright 2021 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
apiVersion: batch/v1beta1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: uptime-stats
|
||||
namespace: buildkite
|
||||
spec:
|
||||
schedule: "20 * * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 24
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: collect-buildkite-stats
|
||||
image: gcr.io/llvm-premerge-checks/stats:latest
|
||||
args: ["/root/llvm-premerge-checks/scripts/metrics/connect_db.sh python3 server_monitoring.py"]
|
||||
env:
|
||||
- name: BUILDKITE_AGENT_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: buildkite-agent-token
|
||||
key: token
|
||||
- name: POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: CONDUIT_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: conduit-api-token
|
||||
key: token
|
||||
- name: BUILDKITE_API_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: buildkite-api-token
|
||||
key: token
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: db-stats
|
||||
key: password
|
||||
- name: SCRIPTS_REFSPEC
|
||||
value: "main"
|
||||
restartPolicy: Never
|
||||
nodeSelector:
|
||||
cloud.google.com/gke-nodepool: service
|
||||
|
|
@ -1,72 +0,0 @@
|
|||
# Copyright 2023 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: github-linux-test
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
rollingUpdate:
|
||||
maxUnavailable: 1
|
||||
maxSurge: 0
|
||||
type: RollingUpdate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: github-linux-test
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: github-linux-test
|
||||
spec:
|
||||
containers:
|
||||
- name: runner
|
||||
image: us-central1-docker.pkg.dev/llvm-premerge-checks/docker/github-linux:latest
|
||||
resources:
|
||||
limits:
|
||||
cpu: 31
|
||||
memory: 80Gi
|
||||
requests:
|
||||
cpu: 31
|
||||
memory: 80Gi
|
||||
volumeMounts:
|
||||
- name: workdir
|
||||
mountPath: /work
|
||||
env:
|
||||
- name: WORKDIR
|
||||
value: "/work"
|
||||
- name: ACCESS_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: github-register-agent-pat
|
||||
key: token
|
||||
- name: POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: RUNNER_NAME
|
||||
value: "$(POD_NAME)"
|
||||
- name: RUNNER_SCOPE
|
||||
value: "org"
|
||||
- name: ORG_NAME
|
||||
value: "metafloworg"
|
||||
- name: LABELS
|
||||
value: "linux"
|
||||
volumes:
|
||||
- name: workdir
|
||||
emptyDir: {}
|
||||
nodeSelector:
|
||||
cloud.google.com/gke-nodepool: linux-agents-2
|
||||
terminationGracePeriodSeconds: 30
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: ingress-llvm-premerge
|
||||
annotations:
|
||||
# nginx.ingress.kubernetes.io/auth-type: basic
|
||||
# nginx.ingress.kubernetes.io/auth-secret: http-auth
|
||||
# nginx.ingress.kubernetes.io/auth-realm: "LLVM pre-merge checks"
|
||||
# nginx.ingress.kubernetes.io/ssl-redirect: "true"
|
||||
kubernetes.io/ingress.class: "nginx"
|
||||
# cert-manager
|
||||
cert-manager.io/cluster-issuer: "letsencrypt-prod"
|
||||
# kubernetes.io/ingress.global-static-ip-name: "llvm-premerge"
|
||||
acme.cert-manager.io/http01-edit-in-place: "true"
|
||||
# ^ cert-manager
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
# cert-manager
|
||||
tls:
|
||||
- hosts:
|
||||
- llvm-premerge.org
|
||||
secretName: llvm-premerge-org-cert
|
||||
# ^ cert-manager
|
||||
defaultBackend:
|
||||
service:
|
||||
name: phabricator-proxy
|
||||
port:
|
||||
number: 8080
|
||||
rules:
|
||||
- host: llvm-premerge.org
|
||||
http:
|
||||
paths:
|
||||
- pathType: Prefix
|
||||
path: /
|
||||
backend:
|
||||
service:
|
||||
name: phabricator-proxy
|
||||
port:
|
||||
number: 8080
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: phabricator-proxy
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: phabricator-proxy
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: phabricator-proxy
|
||||
spec:
|
||||
containers:
|
||||
- name: phabricator-proxy
|
||||
image: gcr.io/llvm-premerge-checks/phabricator-proxy:latest
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
protocol: TCP
|
||||
env:
|
||||
- name: BUILDKITE_API_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: buildkite-api-token
|
||||
key: token
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
successThreshold: 2
|
||||
failureThreshold: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 1500Mi
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 1500Mi
|
||||
nodeSelector:
|
||||
cloud.google.com/gke-nodepool: default-pool
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: buildkite
|
||||
resources:
|
||||
- Deployment.yaml
|
||||
- Services.yaml
|
||||
- Ingress.yaml
|
||||
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
kind: Service
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: phabricator-proxy
|
||||
spec:
|
||||
selector:
|
||||
app: phabricator-proxy
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#-------------------------------------------------------------------------------
|
||||
# create a new windows agent
|
||||
|
||||
set -eux
|
||||
|
||||
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
|
||||
ROOT_DIR="$(dirname ${DIR})"
|
||||
|
||||
# get config options
|
||||
source "${ROOT_DIR}/k8s_config"
|
||||
NAME=$1
|
||||
|
||||
gcloud beta compute instances create "${NAME}" \
|
||||
--project="${GCP_PROJECT}" \
|
||||
--zone="${GCP_ZONE}" \
|
||||
--machine-type=n2d-standard-16 \
|
||||
--image=windows-server-2019-dc-for-containers-v20200714 \
|
||||
--image-project=windows-cloud \
|
||||
--boot-disk-size=200GB --boot-disk-type=pd-ssd
|
||||
1
scripts/.gitignore
vendored
1
scripts/.gitignore
vendored
|
|
@ -1 +0,0 @@
|
|||
benchmark/
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
|
||||
from phabtalk.phabtalk import PhabTalk
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='Runs premerge checks8')
|
||||
parser.add_argument('--url', type=str)
|
||||
parser.add_argument('--name', type=str)
|
||||
parser.add_argument('--phid', type=str)
|
||||
parser.add_argument('--log-level', type=str, default='WARNING')
|
||||
args = parser.parse_args()
|
||||
|
||||
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
|
||||
dry = os.getenv('ph_dry_run_report') is not None
|
||||
PhabTalk(os.getenv('CONDUIT_TOKEN'), dry_run_updates=dry).maybe_add_url_artifact(args.phid, args.url, args.name)
|
||||
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Scripts that use secrets has to be a separate files, not inlined in pipeline:
|
||||
# https://buildkite.com/docs/pipelines/secrets#anti-pattern-referencing-secrets-in-your-pipeline-yaml
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
scripts/patch_diff.py $ph_buildable_diff \
|
||||
--path "${BUILDKITE_BUILD_PATH}"/llvm-project-fork \
|
||||
--token $CONDUIT_TOKEN \
|
||||
--url $PHABRICATOR_HOST \
|
||||
--log-level $LOG_LEVEL \
|
||||
--commit $BASE_COMMIT \
|
||||
--phid ${ph_target_phid} \
|
||||
--push-branch
|
||||
|
||||
EXIT_STATUS=$?
|
||||
|
||||
if [ $EXIT_STATUS -ne 0 ]; then
|
||||
scripts/add_phabricator_artifact.py --phid="$ph_target_phid" --url="$BUILDKITE_BUILD_URL" --name="patch application failed"
|
||||
scripts/set_build_status.py
|
||||
echo failed
|
||||
fi
|
||||
|
||||
exit $EXIT_STATUS
|
||||
|
|
@ -1,150 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Run benchmark of the various steps in the pre-merge tests.
|
||||
|
||||
This can be used to tune the build times.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import datetime
|
||||
import multiprocessing
|
||||
import os
|
||||
import platform
|
||||
import psutil
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import Optional, Dict, List
|
||||
|
||||
|
||||
class Cmd:
|
||||
"""Command to be executed as part of the benchmark.
|
||||
|
||||
If name is not set, the results will not be logged to the result file.
|
||||
"""
|
||||
|
||||
def __init__(self, cmd: str, title: str = None):
|
||||
self.cmd = cmd # type: str
|
||||
self.title = title # type: Optional[str]
|
||||
self.execution_time = None # type: Optional[datetime.timedelta]
|
||||
|
||||
@property
|
||||
def has_title(self) -> bool:
|
||||
return self.title is not None
|
||||
|
||||
@property
|
||||
def was_executed(self) -> bool:
|
||||
return self.execution_time is not None
|
||||
|
||||
|
||||
class Remove(Cmd):
|
||||
"""Remove command, sensitive to OS."""
|
||||
|
||||
def __init__(self, path: str):
|
||||
if platform.system() == 'Windows':
|
||||
cmd = 'cmd /c rd /s/q {}'.format(path)
|
||||
else:
|
||||
cmd = 'rm -rf {}'.format(path)
|
||||
super().__init__(cmd, "rm -rf {}".format(path))
|
||||
|
||||
|
||||
# commands for the benchmark
|
||||
COMMANDS = [
|
||||
Cmd('git clone https://github.com/llvm/llvm-project .', 'git clone'),
|
||||
Cmd('git checkout release/10.x', 'git checkout release'),
|
||||
Cmd('git pull', 'git pull'),
|
||||
Cmd('git checkout {commit}'),
|
||||
# clean build, make sure ccache (or similar) is empty
|
||||
Cmd('{pmt_root_path}/scripts/run_cmake.py', 'cmake 1st'),
|
||||
Cmd('{pmt_root_path}/scripts/run_ninja.py all', 'ninja all 1st'),
|
||||
Cmd('{pmt_root_path}/scripts/run_ninja.py check-all', 'ninja check-all 1st'),
|
||||
Remove('build'),
|
||||
# now rebuild from ccache (or similar)
|
||||
Cmd('{pmt_root_path}/scripts/run_cmake.py', 'cmake 2nd'),
|
||||
Cmd('{pmt_root_path}/scripts/run_ninja.py all', 'ninja all 2nd'),
|
||||
Cmd('{pmt_root_path}/scripts/run_ninja.py check-all', 'ninja check-all 2nd'),
|
||||
]
|
||||
|
||||
|
||||
def run_benchmark(commit: str, name: str, result_file_path: str, workdir: str, pmt_root_path: str):
|
||||
"""Tun the benchmark, write the results to a file."""
|
||||
print('Usingn workdir {}'.format(workdir))
|
||||
print('Using scripts from {}'.format(pmt_root_path))
|
||||
cmd_parameters = {
|
||||
'pmt_root_path': pmt_root_path,
|
||||
'commit': commit,
|
||||
}
|
||||
if os.path.exists(workdir):
|
||||
run_cmd(Remove(workdir), cmd_parameters, '.')
|
||||
os.makedirs(workdir)
|
||||
try:
|
||||
for command in COMMANDS:
|
||||
run_cmd(command, cmd_parameters, workdir)
|
||||
finally:
|
||||
write_results(COMMANDS, result_file_path, name)
|
||||
|
||||
|
||||
def write_results(commands: List[Cmd], result_file_path: str, name: str):
|
||||
fieldnames = ['name', 'cores', 'CPU', 'RAM', 'timestamp', 'OS']
|
||||
fieldnames.extend(cmd.title for cmd in COMMANDS if cmd.has_title)
|
||||
exists = os.path.exists(result_file_path)
|
||||
with open(result_file_path, 'a') as csv_file:
|
||||
writer = csv.DictWriter(csv_file, fieldnames=fieldnames, dialect=csv.excel)
|
||||
if not exists:
|
||||
writer.writeheader()
|
||||
row = {
|
||||
'name': name,
|
||||
'cores': multiprocessing.cpu_count(),
|
||||
'CPU': platform.processor(),
|
||||
'RAM': psutil.virtual_memory().total,
|
||||
'timestamp': datetime.datetime.now().timestamp(),
|
||||
'OS': platform.platform()
|
||||
}
|
||||
for command in (cmd for cmd in commands if (cmd.has_title and cmd.was_executed)):
|
||||
row[command.title] = command.execution_time.total_seconds()
|
||||
writer.writerow(row)
|
||||
print('Benchmark completed.')
|
||||
|
||||
|
||||
def run_cmd(command: Cmd, cmd_parameters: Dict[str, str], workdir: str):
|
||||
"""Run a single command."""
|
||||
cmdline = command.cmd.format(**cmd_parameters)
|
||||
print('Running: {}'.format(cmdline))
|
||||
start_time = datetime.datetime.now()
|
||||
proc = subprocess.Popen(cmdline, shell=True, cwd=workdir, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
stdout, stderr = proc.communicate()
|
||||
if proc.poll() != 0:
|
||||
print(stdout)
|
||||
print(stderr)
|
||||
print('Benchmark failed.')
|
||||
sys.exit(1)
|
||||
end_time = datetime.datetime.now()
|
||||
command.execution_time = end_time - start_time
|
||||
print(' Execution time was: {}'.format(command.execution_time))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
pmt_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Benchmark for LLVM pre-merge tests.')
|
||||
parser.add_argument('--commit', type=str, default='main', help="LLVM commit to run this benchmark on.")
|
||||
parser.add_argument('--result-file', type=str, default='pmt-benchmark.csv',
|
||||
help="path to CSV file where to store the benchmark results")
|
||||
parser.add_argument('--workdir', type=str, default=os.path.join(os.getcwd(), 'benchmark'),
|
||||
help='Folder to store the LLVM checkout.')
|
||||
parser.add_argument('--name', type=str, default=None, help="name for the benchmark")
|
||||
args = parser.parse_args()
|
||||
run_benchmark(args.commit, args.name, args.result_file, args.workdir, pmt_root)
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
import os
|
||||
import json
|
||||
import argparse
|
||||
import requests
|
||||
import time
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='Run sample build on buildkite.')
|
||||
parser.add_argument('--dryrun', action='store_true')
|
||||
parser.add_argument('--commit')
|
||||
parser.add_argument('--pipeline', default='llvm-main')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
pipeline=args.pipeline
|
||||
time.sleep(2)
|
||||
d = json.dumps({
|
||||
'branch': 'main',
|
||||
'commit': args.commit,
|
||||
'env': {
|
||||
'ph_log_level': 'DEBUG',
|
||||
#'ph_skip_linux': 'skip',
|
||||
'ph_linux_agents': '{"queue": "linux-test-google"}',
|
||||
#'ph_linux_agents': '{"queue": "linux-test"}',
|
||||
# 'ph_linux_agents': '{"queue": "linux-clang15-test"}',
|
||||
'ph_skip_windows': 'skip',
|
||||
#'ph_windows_agents': f'{{"name": "win-dev", "queue": "windows-test"}}',
|
||||
# 'ph_windows_agents': '{"queue": "windows-test"}',
|
||||
# 'ph_scripts_refspec': 'windows-vscmd',
|
||||
# 'ph_projects': 'all',
|
||||
'ph_skip_generated': 'skip',
|
||||
# 'ph_windows_agents': f'{{"name": "", "queue": "{queue}"}}',
|
||||
}})
|
||||
print(d)
|
||||
if (args.dryrun):
|
||||
exit(0)
|
||||
token = os.getenv('BUILDKITE_API_TOKEN')
|
||||
if token is None:
|
||||
print("'BUILDKITE_API_TOKEN' environment variable is not set")
|
||||
exit(1)
|
||||
re = requests.post(f'https://api.buildkite.com/v2/organizations/llvm-project/pipelines/{pipeline}/builds',
|
||||
data=d,
|
||||
headers={'Authorization': f'Bearer {token}'})
|
||||
print(re.status_code)
|
||||
j = re.json()
|
||||
print(j['web_url'])
|
||||
|
|
@ -1,109 +0,0 @@
|
|||
import logging
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import urllib.parse
|
||||
from typing import Optional
|
||||
from benedict import benedict
|
||||
|
||||
import backoff
|
||||
import requests
|
||||
|
||||
context_style = {}
|
||||
previous_context = 'default'
|
||||
styles = ['default', 'info', 'success', 'warning', 'error']
|
||||
|
||||
|
||||
def upload_file(base_dir: str, file: str):
|
||||
"""
|
||||
Uploads artifact to buildkite and returns URL to it
|
||||
"""
|
||||
r = subprocess.run(f'buildkite-agent artifact upload "{file}"', shell=True, capture_output=True, cwd=base_dir)
|
||||
logging.debug(f'upload-artifact {r}')
|
||||
match = re.search('Uploading artifact ([^ ]*) ', r.stderr.decode())
|
||||
logging.debug(f'match {match}')
|
||||
if match:
|
||||
url = f'https://buildkite.com/organizations/llvm-project/pipelines/premerge-checks/builds/{os.getenv("BUILDKITE_BUILD_NUMBER")}/jobs/{os.getenv("BUILDKITE_JOB_ID")}/artifacts/{match.group(1)}'
|
||||
logging.info(f'uploaded {file} to {url}')
|
||||
return url
|
||||
else:
|
||||
logging.warning(f'could not find artifact {base_dir}/{file}')
|
||||
return None
|
||||
|
||||
|
||||
def set_metadata(key: str, value: str):
|
||||
r = subprocess.run(f'buildkite-agent meta-data set "{key}" "{value}"', shell=True, capture_output=True)
|
||||
if r.returncode != 0:
|
||||
logging.warning(r)
|
||||
|
||||
|
||||
def annotate(message: str, style: str = 'default', context: Optional[str] = None, append: bool = True):
|
||||
"""
|
||||
Adds an annotation for that currently running build.
|
||||
Note that last `style` applied to the same `context` takes precedence.
|
||||
"""
|
||||
global previous_context, styles, context_style
|
||||
if style not in styles:
|
||||
style = 'default'
|
||||
if context is None:
|
||||
context = previous_context
|
||||
previous_context = context
|
||||
# Pick most severe style so far.
|
||||
context_style.setdefault(context, 0)
|
||||
context_style[context] = max(styles.index(style), context_style[context])
|
||||
style = styles[context_style[context]]
|
||||
if append:
|
||||
message += '\n\n'
|
||||
cmd = ['buildkite-agent', 'annotate', message, '--style', style, '--context', context]
|
||||
if append:
|
||||
cmd.append('--append')
|
||||
r = subprocess.run(cmd, capture_output=True)
|
||||
logging.debug(f'annotate call {r}')
|
||||
if r.returncode != 0:
|
||||
logging.warning(r)
|
||||
|
||||
|
||||
def feedback_url():
|
||||
title = f"buildkite build {os.getenv('BUILDKITE_PIPELINE_SLUG')} {os.getenv('BUILDKITE_BUILD_NUMBER')}"
|
||||
return f'https://github.com/google/llvm-premerge-checks/issues/new?assignees=&labels=bug' \
|
||||
f'&template=bug_report.md&title={urllib.parse.quote(title)}'
|
||||
|
||||
|
||||
class BuildkiteApi:
|
||||
def __init__(self, token: str, organization: str):
|
||||
self.token = token
|
||||
self.organization = organization
|
||||
|
||||
def get_build(self, pipeline: str, build_number: str):
|
||||
# https://buildkite.com/docs/apis/rest-api/builds#get-a-build
|
||||
return benedict(self.get(f'https://api.buildkite.com/v2/organizations/{self.organization}/pipelines/{pipeline}/builds/{build_number}').json())
|
||||
|
||||
def list_running_revision_builds(self, pipeline: str, rev: str):
|
||||
return self.get(f'https://api.buildkite.com/v2/organizations/{self.organization}/pipelines/{pipeline}/builds?state[]=scheduled&state[]=running&meta_data[ph_buildable_revision]={rev}').json()
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=3, logger='', factor=3)
|
||||
def get(self, url: str):
|
||||
authorization = f'Bearer {self.token}'
|
||||
response = requests.get(url, allow_redirects=True, headers={'Authorization': authorization})
|
||||
if response.status_code != 200:
|
||||
raise Exception(f'Buildkite responded with non-OK status: {response.status_code}')
|
||||
return response
|
||||
|
||||
# cancel a build. 'build' is a json object returned by API.
|
||||
def cancel_build(self, build):
|
||||
build = benedict(build)
|
||||
url = f'https://api.buildkite.com/v2/organizations/{self.organization}/pipelines/{build.get("pipeline.slug")}/builds/{build.get("number")}/cancel'
|
||||
authorization = f'Bearer {self.token}'
|
||||
response = requests.put(url, headers={'Authorization': authorization})
|
||||
if response.status_code != 200:
|
||||
raise Exception(f'Buildkite responded with non-OK status: {response.status_code}')
|
||||
|
||||
|
||||
def format_url(url: str, name: Optional[str] = None):
|
||||
if name is None:
|
||||
name = url
|
||||
return f"\033]1339;url='{url}';content='{name}'\a\n"
|
||||
|
||||
|
||||
def strip_emojis(s: str) -> str:
|
||||
return re.sub(r':[^:]+:', '', s).strip()
|
||||
|
|
@ -1,215 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Compute the LLVM_ENABLE_PROJECTS for cmake from diff.
|
||||
|
||||
This script will compute which projects are affected by the diff provided via STDIN.
|
||||
It gets the modified files in the patch, assigns them to projects and based on a
|
||||
project dependency graph it will get the transitively affected projects.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import platform
|
||||
import sys
|
||||
from typing import Any, Dict, List, Set, TextIO, Tuple, Optional, Union
|
||||
from unidiff import PatchSet # type: ignore
|
||||
import yaml
|
||||
|
||||
class ChooseProjects:
|
||||
# file where dependencies are defined
|
||||
SCRIPT_DIR = os.path.dirname(__file__)
|
||||
DEPENDENCIES_FILE = os.path.join(SCRIPT_DIR, 'llvm-dependencies.yaml')
|
||||
|
||||
def __init__(self, llvm_dir: Optional[str]):
|
||||
self.llvm_dir = llvm_dir
|
||||
self.defaultProjects: Dict[str, Dict[str, str]] = {}
|
||||
# List of projects this project depends on, transitive closure.
|
||||
# E.g. compiler-rt -> [llvm, clang].
|
||||
self.dependencies: Dict[str,Set[str]] = {}
|
||||
# List of projects that depends on this project. It's a full closure.
|
||||
# E.g. llvm -> [clang, libcxx, ...]
|
||||
self.usages: Dict[str, Set[str]] = dict()
|
||||
self.all_projects: List[str] = ['all']
|
||||
self.config: Dict[str, Any] = {}
|
||||
self._load_config()
|
||||
|
||||
def _load_config(self):
|
||||
logging.info('loading project config from {}'.format(self.DEPENDENCIES_FILE))
|
||||
with open(self.DEPENDENCIES_FILE) as dependencies_file:
|
||||
self.config = yaml.load(dependencies_file, Loader=yaml.SafeLoader)
|
||||
for k, v in self.config['dependencies'].items():
|
||||
self.dependencies[k] = set(v)
|
||||
# Closure of dependencies.
|
||||
while True:
|
||||
updated = False
|
||||
for s in self.dependencies.values():
|
||||
n = len(s)
|
||||
extend = set()
|
||||
for d in s:
|
||||
extend.update(self.dependencies.get(d, set()))
|
||||
s.update(extend)
|
||||
if len(s) > n:
|
||||
updated = True
|
||||
if not updated:
|
||||
break
|
||||
# Usages don't need to be closed as dependencies already are.
|
||||
for project, deps in self.dependencies.items():
|
||||
for d in deps:
|
||||
self.usages.setdefault(d, set()).add(project)
|
||||
logging.info(f'computed dependencies: {self.dependencies}')
|
||||
logging.info(f'computed usages: {self.usages}')
|
||||
self.all_projects = self.config['allprojects'].keys()
|
||||
|
||||
def get_excluded(self, os: str) -> Set[str]:
|
||||
"""Returns transitive closure for excluded projects"""
|
||||
return self.get_affected_projects(set(self.config['excludedProjects'].get(os, [])))
|
||||
|
||||
def get_check_targets(self, projects: Set[str]) -> Set[str]:
|
||||
"""Return the `check-xxx` targets to pass to ninja for the given list of projects"""
|
||||
if 'all' in projects:
|
||||
return set(["check-all"])
|
||||
targets = set()
|
||||
all_projects = self.config['allprojects']
|
||||
for project in projects:
|
||||
targets.update(set(all_projects.get(project, [])))
|
||||
return targets
|
||||
|
||||
@staticmethod
|
||||
def _detect_os() -> str:
|
||||
"""Detect the current operating system."""
|
||||
if platform.system() == 'Windows':
|
||||
return 'windows'
|
||||
return 'linux'
|
||||
|
||||
def choose_projects(self, patch: str = None, os_name: Optional[str] = None) -> List[str]:
|
||||
"""List all touched project with all projects that they depend on and also
|
||||
all projects that depend on them"""
|
||||
if self.llvm_dir is None:
|
||||
raise ValueError('path to llvm folder must be set in ChooseProject.')
|
||||
llvm_dir = os.path.abspath(os.path.expanduser(self.llvm_dir))
|
||||
logging.info('Scanning LLVM in {}'.format(llvm_dir))
|
||||
if not self.match_projects_dirs():
|
||||
logging.warning(f'{llvm_dir} does not look like a llvm-project directory')
|
||||
return self.get_all_enabled_projects(os_name)
|
||||
changed_files = self.get_changed_files(patch)
|
||||
changed_projects, unmapped_changes = self.get_changed_projects(changed_files)
|
||||
if unmapped_changes:
|
||||
logging.warning('There were changes that could not be mapped to a project.'
|
||||
'Building all projects instead!')
|
||||
return self.get_all_enabled_projects(os_name)
|
||||
return self.extend_projects(changed_projects, os_name)
|
||||
|
||||
def extend_projects(self, projects: Set[str], os_name : Optional[str] = None) -> List[str]:
|
||||
"""Given a set of projects returns a set of projects to be tested taking
|
||||
in account exclusions from llvm-dependencies.yaml.
|
||||
"""
|
||||
logging.info(f'projects: {projects}')
|
||||
if not os_name:
|
||||
os_name = self._detect_os()
|
||||
# Find all affected by current set.
|
||||
affected_projects = self.get_affected_projects(projects)
|
||||
logging.info(f'all affected projects(*) {affected_projects}')
|
||||
# Exclude everything that is affected by excluded.
|
||||
excluded_projects = self.get_excluded(os_name)
|
||||
logging.info(f'all excluded projects(*) {excluded_projects}')
|
||||
affected_projects = affected_projects - excluded_projects
|
||||
logging.info(f'effective projects list {affected_projects}')
|
||||
return sorted(affected_projects)
|
||||
|
||||
def run(self):
|
||||
affected_projects = self.choose_projects()
|
||||
print("Affected:", ';'.join(affected_projects))
|
||||
print("Dependencies:", ';'.join(self.get_dependencies(affected_projects)))
|
||||
print("Check targets:", ';'.join(self.get_check_targets(affected_projects)))
|
||||
return 0
|
||||
|
||||
def match_projects_dirs(self) -> bool:
|
||||
"""Make sure that all projects are folders in the LLVM dir.
|
||||
"""
|
||||
subdirs = os.listdir(self.llvm_dir)
|
||||
for project in self.all_projects:
|
||||
if project not in subdirs:
|
||||
logging.error('Project not found in LLVM root folder: {}'.format(project))
|
||||
return False
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def get_changed_files(patch_str: Union[str, TextIO, None] = None) -> Set[str]:
|
||||
"""get list of changed files from the patch or from STDIN.
|
||||
e.g. ['compiler-rt/lib/tsan/CMakeLists.txt']"""
|
||||
if patch_str is None:
|
||||
patch_str = sys.stdin
|
||||
patch = PatchSet(patch_str)
|
||||
|
||||
changed_files = set({f.path for f in patch.modified_files + patch.added_files + patch.removed_files})
|
||||
|
||||
logging.info('Files modified by this patch:\n ' + '\n '.join(sorted(changed_files)))
|
||||
return changed_files
|
||||
|
||||
def get_changed_projects(self, changed_files: Set[str]) -> Tuple[Set[str], bool]:
|
||||
"""Get list of projects affected by the change."""
|
||||
logging.info("Get list of projects affected by the change.")
|
||||
changed_projects = set()
|
||||
unmapped_changes = False
|
||||
for changed_file in changed_files:
|
||||
project = changed_file.split('/', maxsplit=1)
|
||||
# There is no utils project.
|
||||
if project[0] == 'utils':
|
||||
continue
|
||||
if (project is None) or (project[0] not in self.all_projects):
|
||||
unmapped_changes = True
|
||||
logging.warning('Could not map file to project: {}'.format(changed_file))
|
||||
else:
|
||||
changed_projects.add(project[0])
|
||||
|
||||
logging.info('Projects directly modified by this patch:\n ' + '\n '.join(sorted(changed_projects)))
|
||||
return changed_projects, unmapped_changes
|
||||
|
||||
def get_affected_projects(self, changed_projects: Set[str]) -> Set[str]:
|
||||
"""Compute transitive closure of affected projects based on the
|
||||
dependencies between the projects (including initially passed)."""
|
||||
affected: Set[str] = set(changed_projects)
|
||||
for p in changed_projects:
|
||||
affected.update(self.usages.get(p, set()))
|
||||
logging.info(f'added {affected - changed_projects} projects as they are affected')
|
||||
return affected
|
||||
|
||||
def get_dependencies(self, projects: Set[str]) -> Set[str]:
|
||||
"""Return transitive dependencies for a given projects (including the projects themself).
|
||||
|
||||
These are the required dependencies for given `projects` so that they can be built.
|
||||
"""
|
||||
affected: Set[str] = set(projects)
|
||||
for p in projects:
|
||||
affected.update(self.dependencies.get(p, set()))
|
||||
return affected
|
||||
|
||||
def get_all_enabled_projects(self, os_name: Optional[str] = None) -> List[str]:
|
||||
"""Get list of all not-excluded projects for current platform."""
|
||||
return self.extend_projects(set(self.all_projects), os_name)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Compute the projects affected by a change. A patch file is expected on stdin.')
|
||||
parser.add_argument('--llvmdir', type=str, default='.')
|
||||
parser.add_argument('--log-level', type=str, default='INFO')
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
|
||||
logging.info(f'checking changes in {args.llvmdir}')
|
||||
chooser = ChooseProjects(args.llvmdir)
|
||||
sys.exit(chooser.run())
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
# Patterns for clang-format to ignore.
|
||||
**/test
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
# Files that are allowed by clang-tidy.ignore but should not receive inline review comments.
|
||||
# Right now it works in whitelist mode and only some files / directories are whitelisted.
|
||||
# FIXME: is is used?
|
||||
*
|
||||
!clang-tools-extra/clangd/**
|
||||
|
|
@ -1,56 +0,0 @@
|
|||
# Files to be ignored by clang-tidy. Will not appear in short report and inline comments.
|
||||
debuginfo-tests/
|
||||
libcxxabi/test
|
||||
libcxxabi/test/libcxxabi/test
|
||||
libcxx/test
|
||||
libcxx/utils/libcxx/test
|
||||
libcxx/utils/google-benchmark/test
|
||||
llvm/test
|
||||
llvm/utils/gn/secondary/llvm/test
|
||||
llvm/utils/gn/secondary/lld/test
|
||||
llvm/utils/gn/secondary/clang-tools-extra/test
|
||||
llvm/utils/gn/secondary/clang-tools-extra/clangd/test
|
||||
llvm/utils/gn/secondary/compiler-rt/test
|
||||
llvm/utils/gn/secondary/clang/test
|
||||
llvm/utils/benchmark/test
|
||||
polly/test
|
||||
lldb/examples/test
|
||||
lldb/test
|
||||
lldb/utils/test
|
||||
lldb/tools/intel-features/intel-mpx/test
|
||||
lldb/packages/Python/lldbsuite/test
|
||||
lldb/packages/Python/lldbsuite/test/tools/lldb-server/test
|
||||
lldb/packages/Python/lldbsuite/test/commands/expression/test
|
||||
lldb/packages/Python/lldbsuite/test/test_runner/test
|
||||
lldb/third_party/Python/module/unittest2/unittest2/test
|
||||
lld/test
|
||||
clang-tools-extra/test
|
||||
clang-tools-extra/clangd/clients/clangd-vscode/test
|
||||
clang-tools-extra/clangd/test
|
||||
pstl/test
|
||||
libc/test
|
||||
llgo/test
|
||||
compiler-rt/test
|
||||
compiler-rt/test/builtins/Unit/ppc/test
|
||||
compiler-rt/test/builtins/Unit/test
|
||||
compiler-rt/lib/sanitizer_common/sanitizer.*.inc
|
||||
compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp
|
||||
compiler-rt/lib/sanitizer_common/sanitizer_common_syscalls.inc
|
||||
debuginfo-tests/dexter/dex/tools/test
|
||||
debuginfo-tests/dexter/feature_tests/subtools/test
|
||||
clang/lib/Headers/__clang_hip_*.h
|
||||
clang/lib/Headers/wasm_simd128.h
|
||||
clang/test
|
||||
libclc
|
||||
mlir/examples/standalone
|
||||
mlir/include/mlir-c/Bindings
|
||||
mlir/include/mlir/Bindings
|
||||
mlir/lib/Bindings
|
||||
mlir/test
|
||||
mlir/tools/mlir-cuda-runner
|
||||
mlir/tools/mlir-rocm-runner
|
||||
openmp/libomptarget/test
|
||||
openmp/libomptarget/deviceRTLs/nvptx/test
|
||||
openmp/runtime/test
|
||||
libunwind/test
|
||||
libunwind/test/libunwind/test
|
||||
|
|
@ -1,111 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
from typing import Tuple, Optional
|
||||
import pathspec
|
||||
import unidiff
|
||||
|
||||
from phabtalk.phabtalk import Report, Step
|
||||
from buildkite_utils import annotate
|
||||
|
||||
|
||||
def get_diff(base_commit) -> Tuple[bool, str]:
|
||||
r = subprocess.run(f'python3 clang/tools/clang-format/git-clang-format {base_commit}', shell=True)
|
||||
logging.debug(f'git-clang-format {r}')
|
||||
if r.returncode != 0:
|
||||
logging.error(f'git-clang-format returned an non-zero exit code {r.returncode}')
|
||||
r = subprocess.run(f'git checkout -- .', stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
|
||||
logging.debug(f'git reset {r}')
|
||||
return False, ''
|
||||
diff_run = subprocess.run(f'git diff -U0 --no-prefix --exit-code', capture_output=True, shell=True)
|
||||
logging.debug(f'git diff {diff_run}')
|
||||
r = subprocess.run(f'git checkout -- .', stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
|
||||
logging.debug(f'git reset {r}')
|
||||
return True, diff_run.stdout.decode()
|
||||
|
||||
|
||||
def run(base_commit, ignore_config, step: Optional[Step], report: Optional[Report]):
|
||||
"""Apply clang-format and return if no issues were found."""
|
||||
if report is None:
|
||||
report = Report() # For debugging.
|
||||
if step is None:
|
||||
step = Step() # For debugging.
|
||||
step.reproduce_commands.append(f'git-clang-format {base_commit}')
|
||||
r, patch = get_diff(base_commit)
|
||||
if not r:
|
||||
step.success = False
|
||||
return
|
||||
add_artifact = False
|
||||
patches = unidiff.PatchSet(patch)
|
||||
ignore_lines = []
|
||||
if ignore_config is not None and os.path.exists(ignore_config):
|
||||
ignore_lines = open(ignore_config, 'r').readlines()
|
||||
logging.debug("ignore patterns: " + str(';'.join(ignore_lines)))
|
||||
ignore = pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern, ignore_lines)
|
||||
patched_file: unidiff.PatchedFile
|
||||
success = True
|
||||
for patched_file in patches:
|
||||
add_artifact = True
|
||||
if ignore.match_file(patched_file.source_file) or ignore.match_file(patched_file.target_file):
|
||||
logging.info(f'patch of {patched_file.patch_info} is ignored')
|
||||
continue
|
||||
hunk: unidiff.Hunk
|
||||
for hunk in patched_file:
|
||||
lines = [str(x) for x in hunk]
|
||||
success = False
|
||||
m = 10 # max number of lines to report.
|
||||
description = 'please reformat the code\n```\n'
|
||||
n = len(lines)
|
||||
cut = n > m + 1
|
||||
if cut:
|
||||
lines = lines[:m]
|
||||
description += ''.join(lines) + '\n```'
|
||||
if cut:
|
||||
description += f'\n{n - m} diff lines are omitted. See full path.'
|
||||
report.add_lint({
|
||||
'name': 'clang-format',
|
||||
'severity': 'autofix',
|
||||
'code': 'clang-format',
|
||||
'path': patched_file.source_file,
|
||||
'line': hunk.source_start,
|
||||
'char': 1,
|
||||
'description': description,
|
||||
})
|
||||
if add_artifact:
|
||||
patch_file = 'clang-format.patch'
|
||||
with open(patch_file, 'w') as f:
|
||||
f.write(patch)
|
||||
report.add_artifact(os.getcwd(), patch_file, 'clang-format')
|
||||
if not success:
|
||||
step.success = False
|
||||
annotate(f'clang-format: Please format your changes with clang-format by running `git-clang-format HEAD^`'
|
||||
f'or applying the attached patch.', style='error')
|
||||
logging.debug(f'report: {report}')
|
||||
logging.debug(f'step: {step}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='Runs clang-format against given diff with given commit. '
|
||||
'Produces patch and attaches linter comments to a review.')
|
||||
parser.add_argument('--base', default='HEAD~1')
|
||||
parser.add_argument('--ignore-config', default=None, help='path to file with patters of files to ignore')
|
||||
parser.add_argument('--log-level', type=str, default='INFO')
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=args.log_level)
|
||||
run(args.base, args.ignore_config, None)
|
||||
|
|
@ -1,118 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from typing import Optional
|
||||
import pathspec
|
||||
|
||||
import ignore_diff
|
||||
from buildkite_utils import annotate
|
||||
from phabtalk.phabtalk import Report, Step
|
||||
|
||||
|
||||
def run(base_commit, ignore_config, step: Optional[Step], report: Optional[Report]):
|
||||
"""Apply clang-tidy and return if no issues were found."""
|
||||
if report is None:
|
||||
report = Report() # For debugging.
|
||||
if step is None:
|
||||
step = Step() # For debugging.
|
||||
r = subprocess.run(f'git diff -U0 --no-prefix {base_commit}', shell=True, capture_output=True)
|
||||
logging.debug(f'git diff {r}')
|
||||
diff = r.stdout.decode("utf-8", "replace")
|
||||
if ignore_config is not None and os.path.exists(ignore_config):
|
||||
ignore = pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern,
|
||||
open(ignore_config, 'r').readlines())
|
||||
diff = ignore_diff.remove_ignored(diff.splitlines(keepends=True), open(ignore_config, 'r'))
|
||||
logging.debug(f'filtered diff: {diff}')
|
||||
else:
|
||||
ignore = pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern, [])
|
||||
p = subprocess.Popen(['clang-tidy-diff', '-p0', '-quiet'], stdout=subprocess.PIPE, stdin=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE)
|
||||
step.reproduce_commands.append(f'git diff -U0 --no-prefix {base_commit} | clang-tidy-diff -p0')
|
||||
a = ''.join(diff)
|
||||
logging.info(f'clang-tidy input: {a}')
|
||||
out = p.communicate(input=a.encode())[0].decode()
|
||||
logging.debug(f'clang-tidy-diff {p}: {out}')
|
||||
# Typical finding looks like:
|
||||
# [cwd/]clang/include/clang/AST/DeclCXX.h:3058:20: error: ... [clang-diagnostic-error]
|
||||
pattern = '^([^:]*):(\\d+):(\\d+): (.*): (.*)'
|
||||
add_artifact = False
|
||||
logging.debug("cwd", os.getcwd())
|
||||
errors_count = 0
|
||||
warn_count = 0
|
||||
inline_comments = 0
|
||||
for line in out.splitlines(keepends=False):
|
||||
line = line.strip()
|
||||
line = line.replace(os.getcwd() + os.sep, '')
|
||||
logging.debug(line)
|
||||
if len(line) == 0 or line == 'No relevant changes found.':
|
||||
continue
|
||||
add_artifact = True
|
||||
match = re.search(pattern, line)
|
||||
if match:
|
||||
file_name = match.group(1)
|
||||
line_pos = match.group(2)
|
||||
char_pos = match.group(3)
|
||||
severity = match.group(4)
|
||||
text = match.group(5)
|
||||
text += '\n[[{} | not useful]] '.format(
|
||||
'https://github.com/google/llvm-premerge-checks/blob/main/docs/clang_tidy.md#warning-is-not-useful')
|
||||
if severity in ['warning', 'error']:
|
||||
if severity == 'warning':
|
||||
warn_count += 1
|
||||
if severity == 'error':
|
||||
errors_count += 1
|
||||
if ignore.match_file(file_name):
|
||||
print('{} is ignored by pattern and no comment will be added'.format(file_name))
|
||||
else:
|
||||
inline_comments += 1
|
||||
report.add_lint({
|
||||
'name': 'clang-tidy',
|
||||
'severity': 'warning',
|
||||
'code': 'clang-tidy',
|
||||
'path': file_name,
|
||||
'line': int(line_pos),
|
||||
'char': int(char_pos),
|
||||
'description': '{}: {}'.format(severity, text),
|
||||
})
|
||||
else:
|
||||
logging.debug('does not match pattern')
|
||||
if add_artifact:
|
||||
p = 'clang-tidy.txt'
|
||||
with open(p, 'w') as f:
|
||||
f.write(out)
|
||||
report.add_artifact(os.getcwd(), p, 'clang-tidy')
|
||||
if errors_count + warn_count != 0:
|
||||
step.success = False
|
||||
url = "https://github.com/google/llvm-premerge-checks/blob/main/docs/clang_tidy.md#review-comments."
|
||||
annotate(f'clang-tidy found {errors_count} errors and {warn_count} warnings. {inline_comments} of them were '
|
||||
f'added as review comments [why?]({url})', style='error')
|
||||
logging.debug(f'report: {report}')
|
||||
logging.debug(f'step: {step}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='Runs clang-format against given diff with given commit. '
|
||||
'Produces patch and attaches linter comments to a review.')
|
||||
parser.add_argument('--base', default='HEAD~1')
|
||||
parser.add_argument('--ignore-config', default=None, help='path to file with patters of files to ignore')
|
||||
parser.add_argument('--log-level', type=str, default='INFO')
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
|
||||
run(args.base, args.ignore_config, None, None)
|
||||
|
|
@ -1,85 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""The script will delete old git branches."""
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import functools
|
||||
import git
|
||||
import operator
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from typing import List
|
||||
|
||||
|
||||
def delete_old_branches(repo_path: str, max_age: datetime.datetime, branch_patterns: List[re.Pattern],
|
||||
*, dry_run: bool = True, remote_name: str = 'origin') -> bool:
|
||||
"""Deletes 'old' branches from a git repo.
|
||||
|
||||
This script assumes that $repo_path contains a current checkout of the repository ot be cleaned up.
|
||||
:retrun True IFF branches could be deleted successfully.
|
||||
"""
|
||||
repo = git.Repo(repo_path)
|
||||
remote = repo.remote(name=remote_name)
|
||||
repo.git.fetch('--prune')
|
||||
refs = remote.refs
|
||||
print('Found {} branches at {} in total.'.format(len(refs), remote_name))
|
||||
del_count = 0
|
||||
fail_count = 0
|
||||
if dry_run:
|
||||
print('DRY RUN. NO BRANCHES WILL BE DELETED', flush=True)
|
||||
print('Deleting: \n', flush=True)
|
||||
for reference in refs:
|
||||
committed_date = datetime.datetime.fromtimestamp(reference.commit.committed_date)
|
||||
if committed_date < max_age and _has_pattern_match(reference.name, branch_patterns):
|
||||
print(reference.name, flush=True)
|
||||
if not dry_run:
|
||||
try:
|
||||
remote.push(refspec=':{}'.format(reference.remote_head))
|
||||
del_count += 1
|
||||
except git.GitCommandError as err:
|
||||
print('ERROR: Failed to delete "{}": {}'.format(reference.name, err), flush=True)
|
||||
fail_count += 1
|
||||
print('Deleted {} branches.'.format(del_count))
|
||||
if fail_count > 0:
|
||||
print('Failed to delete {} branches.'.format(fail_count))
|
||||
return fail_count == 0
|
||||
|
||||
|
||||
def _has_pattern_match(name: str, patterns) -> bool:
|
||||
"""Check if name matches any of the patterns"""
|
||||
return functools.reduce(
|
||||
operator.or_,
|
||||
map(lambda r: r.search(name) is not None, patterns),
|
||||
False)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='Clean a git repository')
|
||||
parser.add_argument('repo_path', type=str, nargs='?', default=os.getcwd())
|
||||
parser.add_argument('--days', type=int, default=30)
|
||||
parser.add_argument('--pattern', action='append', type=str)
|
||||
parser.add_argument('--dryrun', action='store_true')
|
||||
args = parser.parse_args()
|
||||
|
||||
max_age = datetime.datetime.now() - datetime.timedelta(days=args.days)
|
||||
branch_pattern = [re.compile(r) for r in args.pattern]
|
||||
|
||||
success = delete_old_branches(args.repo_path, max_age, branch_pattern, dry_run=args.dryrun)
|
||||
|
||||
if not success:
|
||||
sys.exit(1)
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
def get_env_or_die(name: str):
|
||||
v = os.environ.get(name)
|
||||
if not v:
|
||||
sys.stderr.write(f"Error: '{name}' environment variable is not set.\n")
|
||||
exit(2)
|
||||
return v
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# common header file for all .ps1 scripts
|
||||
|
||||
# stop script on errors
|
||||
Set-StrictMode -Version Latest
|
||||
$PSDefaultParameterValues['*:ErrorAction']='Stop'
|
||||
|
||||
|
||||
# Invokes a Cmd.exe shell script and updates the environment.
|
||||
# from: https://stackoverflow.com/questions/41399692/running-a-build-script-after-calling-vcvarsall-bat-from-powershell
|
||||
function Invoke-CmdScript {
|
||||
param(
|
||||
[String] $scriptName
|
||||
)
|
||||
$cmdLine = """$scriptName"" $args & set"
|
||||
& $Env:SystemRoot\system32\cmd.exe /c $cmdLine |
|
||||
select-string '^([^=]*)=(.*)$' | foreach-object {
|
||||
$varName = $_.Matches[0].Groups[1].Value
|
||||
$varValue = $_.Matches[0].Groups[2].Value
|
||||
set-item Env:$varName $varValue
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
# call an executable and check the error code
|
||||
# from: https://stackoverflow.com/questions/9948517/how-to-stop-a-powershell-script-on-the-first-error
|
||||
function Invoke-Call {
|
||||
param (
|
||||
[scriptblock]$ScriptBlock,
|
||||
[string]$ErrorAction = "Stop"
|
||||
)
|
||||
& @ScriptBlock *>&1 | ForEach-Object { "$_" }
|
||||
if (($lastexitcode -ne 0) -and $ErrorAction -eq "Stop") {
|
||||
Write-Error "Command $ScriptBlock exited with $lastexitcode."
|
||||
exit $lastexitcode
|
||||
}
|
||||
}
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
# Sample script that you can run in docker container to build vllm
|
||||
#!/usr/bin/env bash
|
||||
su buildkite-agent
|
||||
cd /var/lib/buildkite-agent
|
||||
git clone https://github.com/llvm-premerge-tests/llvm-project.git llvm-project
|
||||
cd llvm-project
|
||||
rm -rf build
|
||||
mkdir build
|
||||
cd build
|
||||
export CC="clang"
|
||||
export CXX="clang++"
|
||||
export LD="LLD"
|
||||
cmake ../llvm -D LLVM_ENABLE_PROJECTS="clang;mlir;lldb;llvm" -G Ninja -D CMAKE_BUILD_TYPE=Release -D LLVM_ENABLE_ASSERTIONS=ON -D LLVM_BUILD_EXAMPLES=ON -D LLVM_LIT_ARGS="-v --xunit-xml-output test-results.xml" -D LLVM_ENABLE_LLD=ON -D CMAKE_CXX_FLAGS=-gmlt -DBOLT_CLANG_EXE=/usr/bin/clang
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
[ ! -d llvm-project ] && git clone https://github.com/llvm/llvm-project.git llvm-project
|
||||
cd llvm-project
|
||||
git fetch --all
|
||||
git checkout ${1:-20ba079dda7be1a72d64cebc9f55d909bf29f6c1}
|
||||
rm -rf build
|
||||
mkdir build
|
||||
cd build
|
||||
export CC="clang"
|
||||
export CXX="clang++"
|
||||
export LD="LLD"
|
||||
cmake ../llvm -D LLVM_ENABLE_PROJECTS="bolt;compiler-rt;libc;pstl;polly;llvm;libclc;clang;mlir;clang-tools-extra;flang;lld" -G Ninja -D CMAKE_BUILD_TYPE=Release -D LLVM_ENABLE_ASSERTIONS=ON -D LLVM_BUILD_EXAMPLES=ON -D LLVM_LIT_ARGS="-v --xunit-xml-output test-results.xml --resultdb-output resultdb.json" -D LLVM_ENABLE_LLD=ON -D CMAKE_CXX_FLAGS=-gmlt -DBOLT_CLANG_EXE=/usr/bin/clang
|
||||
# ^note that compiler cache arguments are omitted
|
||||
ln -s $PWD/compile_commands.json ../compile_commands.json
|
||||
ninja check-all
|
||||
|
|
@ -1,64 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from asyncio.subprocess import PIPE
|
||||
from typing import Callable, AnyStr
|
||||
|
||||
|
||||
async def read_stream_and_display(stream, display):
|
||||
while True:
|
||||
line = await stream.readline()
|
||||
if not line:
|
||||
break
|
||||
display(line) # assume it doesn't block
|
||||
|
||||
|
||||
async def read_and_display(write_stdout, write_stderr, *cmd, **kwargs):
|
||||
logging.debug(f'subprocess called with {cmd}; {kwargs}')
|
||||
process = await asyncio.create_subprocess_shell(*cmd, stdout=PIPE, stderr=PIPE, **kwargs)
|
||||
try:
|
||||
await asyncio.gather(
|
||||
read_stream_and_display(process.stdout, write_stdout),
|
||||
read_stream_and_display(process.stderr, write_stderr))
|
||||
except Exception:
|
||||
process.kill()
|
||||
raise
|
||||
finally:
|
||||
return await process.wait()
|
||||
|
||||
|
||||
def tee(s: AnyStr, write1: Callable[[AnyStr], None], write2: Callable[[AnyStr], None]):
|
||||
write1(s)
|
||||
write2(s)
|
||||
|
||||
|
||||
def if_not_matches(s: AnyStr, regexp, write: Callable[[AnyStr], None]):
|
||||
x = s
|
||||
if isinstance(s, (bytes, bytearray)):
|
||||
x = s.decode()
|
||||
if regexp.match(x) is None:
|
||||
write(s)
|
||||
|
||||
|
||||
def watch_shell(write_stdout, write_stderr, *cmd, **kwargs):
|
||||
if os.name == 'nt':
|
||||
loop = asyncio.ProactorEventLoop() # Windows
|
||||
asyncio.set_event_loop(loop)
|
||||
else:
|
||||
loop = asyncio.get_event_loop()
|
||||
rc = loop.run_until_complete(read_and_display(write_stdout, write_stderr, *cmd, **kwargs))
|
||||
return rc
|
||||
|
|
@ -1,58 +0,0 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import git
|
||||
import os
|
||||
import logging
|
||||
|
||||
"""URL of upstream LLVM repository."""
|
||||
LLVM_GITHUB_URL = 'ssh://git@github.com/llvm/llvm-project'
|
||||
FORK_REMOTE_URL = 'ssh://git@github.com/llvm-premerge-tests/llvm-project'
|
||||
|
||||
def initLlvmFork(path: str) -> git.Repo:
|
||||
if not os.path.isdir(path):
|
||||
logging.info(f'{path} does not exist, cloning repository...')
|
||||
git.Repo.clone_from(FORK_REMOTE_URL, path)
|
||||
repo = git.Repo(path)
|
||||
# Remove index lock just in case.
|
||||
lock_file = f"{repo.working_tree_dir}/.git/index.lock"
|
||||
try:
|
||||
os.remove(lock_file)
|
||||
logging.info(f"removed {lock_file}")
|
||||
except FileNotFoundError:
|
||||
logging.info(f"{lock_file} does not exist")
|
||||
repo.remote('origin').set_url(FORK_REMOTE_URL)
|
||||
if 'upstream' not in repo.remotes:
|
||||
repo.create_remote('upstream', url=LLVM_GITHUB_URL)
|
||||
else:
|
||||
repo.remote('upstream').set_url(LLVM_GITHUB_URL)
|
||||
repo.git.fetch('--all')
|
||||
repo.git.clean('-ffxdq')
|
||||
repo.git.reset('--hard')
|
||||
repo.heads.main.checkout()
|
||||
repo.git.pull('origin', 'main')
|
||||
return repo
|
||||
|
||||
def syncLlvmFork(repo: git.Repo):
|
||||
repo.remote('origin').set_url(FORK_REMOTE_URL)
|
||||
if 'upstream' not in repo.remotes:
|
||||
repo.create_remote('upstream', url=LLVM_GITHUB_URL)
|
||||
repo.remote('upstream').set_url(LLVM_GITHUB_URL)
|
||||
syncRemotes(repo, 'upstream', 'origin')
|
||||
pass
|
||||
|
||||
def syncRemotes(repo: git.Repo, fromRemote, toRemote):
|
||||
"""sync one remote from another"""
|
||||
repo.remotes[fromRemote].fetch()
|
||||
repo.git.push(toRemote, '-f', f'refs/remotes/{fromRemote}/*:refs/heads/*')
|
||||
|
|
@ -1,109 +0,0 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import git
|
||||
import os
|
||||
import scripts.git_utils as git_utils
|
||||
|
||||
def assertForkIsSynced(upstreamPath, forkPath):
|
||||
upstream = git.Repo(path=upstreamPath)
|
||||
fork = git.Repo(path=forkPath)
|
||||
forkBranches = {}
|
||||
for b in fork.branches:
|
||||
forkBranches[b.name] = b
|
||||
for b in upstream.branches:
|
||||
assert b.name in forkBranches
|
||||
assert b.commit.hexsha == forkBranches[b.name].commit.hexsha, f'branch {b.name} head'
|
||||
|
||||
def forkIsSynced(upstreamPath, forkPath) -> bool:
|
||||
upstream = git.Repo(path=upstreamPath)
|
||||
fork = git.Repo(path=forkPath)
|
||||
forkBranches = {}
|
||||
for b in fork.branches:
|
||||
forkBranches[b.name] = b
|
||||
for b in upstream.branches:
|
||||
if b.name not in forkBranches:
|
||||
return False
|
||||
if b.commit.hexsha != forkBranches[b.name].commit.hexsha:
|
||||
return False
|
||||
return True
|
||||
|
||||
def add_simple_commit(remote, name: str):
|
||||
with open(os.path.join(remote.working_tree_dir, name), 'wt') as f:
|
||||
f.write('first line\n')
|
||||
remote.index.add([os.path.join(remote.working_tree_dir, name)])
|
||||
remote.index.commit(name)
|
||||
|
||||
def test_sync_branches(tmp_path):
|
||||
upstreamRemote = os.path.join(tmp_path, 'upstreamBare')
|
||||
forkRemote = os.path.join(tmp_path, 'forkBare')
|
||||
git.Repo.init(path=upstreamRemote, bare=True)
|
||||
git.Repo.init(path=forkRemote, bare=True)
|
||||
upstreamPath = os.path.join(tmp_path, 'upstream')
|
||||
forkPath = os.path.join(tmp_path, 'fork')
|
||||
upstream = git.Repo.clone_from(url=upstreamRemote, to_path=upstreamPath)
|
||||
add_simple_commit(upstream, '1')
|
||||
upstream.git.push('origin', 'main')
|
||||
fork = git.Repo.clone_from(url=forkRemote, to_path=forkPath)
|
||||
fork.create_remote('upstream', url=upstreamRemote)
|
||||
git_utils.syncRemotes(fork, 'upstream', 'origin')
|
||||
fork.remotes.upstream.fetch()
|
||||
fork.create_head('main', fork.remotes.upstream.refs.main)
|
||||
fork.heads.main.checkout()
|
||||
|
||||
# Sync init commit.
|
||||
git_utils.syncRemotes(fork, 'upstream', 'origin')
|
||||
assertForkIsSynced(upstreamRemote, forkRemote)
|
||||
|
||||
# Add new change upstream.
|
||||
add_simple_commit(upstream, '2')
|
||||
upstream.git.push('--all')
|
||||
|
||||
git_utils.syncRemotes(fork, 'upstream', 'origin')
|
||||
assertForkIsSynced(upstreamRemote, forkRemote)
|
||||
|
||||
# Add new branch.
|
||||
upstream.create_head('branch1')
|
||||
upstream.heads['branch1'].checkout()
|
||||
add_simple_commit(upstream, '3')
|
||||
upstream.git.push('--all')
|
||||
|
||||
git_utils.syncRemotes(fork, 'upstream', 'origin')
|
||||
assertForkIsSynced(upstreamRemote, forkRemote)
|
||||
|
||||
# Add another branch commit.
|
||||
add_simple_commit(upstream, '4')
|
||||
upstream.git.push('--all')
|
||||
git_utils.syncRemotes(fork, 'upstream', 'origin')
|
||||
assertForkIsSynced(upstreamRemote, forkRemote)
|
||||
|
||||
# Discard changes in fork.
|
||||
fork.remotes.origin.pull()
|
||||
fork.heads.main.checkout()
|
||||
add_simple_commit(fork, '5')
|
||||
fork.remotes.origin.push()
|
||||
|
||||
upstream.remotes.origin.pull('main')
|
||||
upstream.heads.main.checkout()
|
||||
add_simple_commit(upstream, '6')
|
||||
upstream.remotes.origin.push()
|
||||
|
||||
assert not forkIsSynced(upstreamRemote, forkRemote)
|
||||
assert os.path.isfile(os.path.join(fork.working_tree_dir, '5'))
|
||||
git_utils.syncRemotes(fork, 'upstream', 'origin')
|
||||
assertForkIsSynced(upstreamRemote, forkRemote)
|
||||
fork.git.pull('origin', 'main')
|
||||
fork.heads.main.checkout()
|
||||
assert not os.path.isfile(os.path.join(fork.working_tree_dir, '5'))
|
||||
assert os.path.isfile(os.path.join(fork.working_tree_dir, '6'))
|
||||
|
|
@ -1,49 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import argparse
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
import pathspec
|
||||
|
||||
|
||||
def remove_ignored(diff_lines, ignore_patterns_lines):
|
||||
logging.debug(f'ignore pattern {ignore_patterns_lines}')
|
||||
ignore = pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern, ignore_patterns_lines)
|
||||
good = True
|
||||
result = []
|
||||
for line in diff_lines:
|
||||
match = re.search(r'^diff --git (.*) (.*)$', line)
|
||||
if match:
|
||||
good = not (ignore.match_file(match.group(1)) and ignore.match_file(match.group(2)))
|
||||
if not good:
|
||||
logging.debug(f'skip {line.rstrip()}')
|
||||
continue
|
||||
result.append(line)
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Maybe FIXME: Replace this tool usage with flags for tidy/format, use paths relative to `__file__`
|
||||
parser = argparse.ArgumentParser(description='Takes an output of git diff and removes files ignored by patten '
|
||||
'specified by ignore file')
|
||||
parser.add_argument('ignore_config', default=None,
|
||||
help='path to file with patters of files to ignore')
|
||||
parser.add_argument('--log-level', type=str, default='WARNING')
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=args.log_level)
|
||||
filtered = remove_ignored([x for x in sys.stdin], open(args.ignore_config, 'r').readlines())
|
||||
for x in filtered:
|
||||
sys.stdout.write(x)
|
||||
|
|
@ -1,90 +0,0 @@
|
|||
# This mapping is only used to determine which projects need to be rebuild.
|
||||
# E.g. all builds are still in-tree, so 'llvm' will always be included in the
|
||||
# built projects.
|
||||
|
||||
dependencies:
|
||||
llvm: []
|
||||
bolt:
|
||||
- llvm
|
||||
- lld
|
||||
clang:
|
||||
- llvm
|
||||
clang-tools-extra:
|
||||
- clang
|
||||
- llvm
|
||||
# FIXME: "compiler-rt" depends on "llvm" only for configuration, right?
|
||||
# it means we can miss breakages in configuration changes.
|
||||
# Same for libcxx, libc and other projects that don't have 'llvm'
|
||||
# as a dependency.
|
||||
compiler-rt:
|
||||
- clang
|
||||
flang:
|
||||
- llvm
|
||||
- mlir
|
||||
- clang
|
||||
libc:
|
||||
- clang
|
||||
- clang-tools-extra
|
||||
libcxx:
|
||||
- clang
|
||||
- lldb
|
||||
libcxxabi:
|
||||
- clang
|
||||
- lldb
|
||||
libclc: []
|
||||
lld:
|
||||
- llvm
|
||||
lldb:
|
||||
- clang
|
||||
- llvm
|
||||
mlir:
|
||||
- llvm
|
||||
openmp:
|
||||
- clang
|
||||
polly:
|
||||
- llvm
|
||||
pstl: []
|
||||
cross-project-tests:
|
||||
- clang
|
||||
- lld
|
||||
|
||||
# List of all projects in the LLVM monorepository. This list is taken from
|
||||
# llvm/CMakeLists.txt in "set(LLVM_ALL_PROJECTS ..."
|
||||
# The value for all project is the list of ninja targets to run.
|
||||
allprojects:
|
||||
bolt: ["check-bolt"]
|
||||
clang: ["check-clang"]
|
||||
clang-tools-extra: ["check-clang-tools"]
|
||||
compiler-rt: ["check-all"] # check-compiler-rt seems to exist only in standalone builds.
|
||||
cross-project-tests: ["check-cross-project"]
|
||||
flang: ["check-flang"]
|
||||
libc: ["check-libc"]
|
||||
libclc: ["check-all"] # There does not seem to be a more specific target.
|
||||
libcxx: ["check-cxx"]
|
||||
libcxxabi: ["check-cxxabi"]
|
||||
lld: ["check-lld"]
|
||||
lldb: ["check-all"] # FIXME: `check-lldb` may not include every lldb tests?
|
||||
mlir: ["check-mlir"]
|
||||
openmp: ["check-openmp"]
|
||||
polly: ["check-polly"]
|
||||
pstl: ["check-all"] # There does not seem to be a more specific target.
|
||||
llvm: ["check-llvm"]
|
||||
|
||||
# projects excluded from automatic configuration as they could not be built
|
||||
excludedProjects:
|
||||
windows:
|
||||
- bolt # tests are not supported yet
|
||||
- lldb # failing tests
|
||||
- libcxx # no windows support
|
||||
- libcxxabi # no windows support
|
||||
- libc # no windows support
|
||||
- openmp # TODO: check: kuhnel has trouble with the Perl installation
|
||||
- cross-project-tests # test failing
|
||||
- check-cxxabi
|
||||
- compiler-rt # tests taking too long
|
||||
linux:
|
||||
- libcxx # has custom pipeline
|
||||
- libcxxabi # has custom pipeline
|
||||
- cross-project-tests # tests failing
|
||||
- lldb # tests failing
|
||||
- openmp # https://github.com/google/llvm-premerge-checks/issues/410
|
||||
1
scripts/metrics/.gitignore
vendored
1
scripts/metrics/.gitignore
vendored
|
|
@ -1 +0,0 @@
|
|||
tmp
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
[[source]]
|
||||
url = "https://pypi.python.org/simple"
|
||||
verify_ssl = true
|
||||
name = "pypi"
|
||||
|
||||
[packages]
|
||||
backoff = "*"
|
||||
GitPython = "*"
|
||||
lxml = "*"
|
||||
pathspec = "*"
|
||||
phabricator = "==0.8.1"
|
||||
pyaml = "*"
|
||||
requests = "*"
|
||||
python-benedict = "*"
|
||||
psycopg2-binary = "*"
|
||||
chardet = "*"
|
||||
|
||||
[dev-packages]
|
||||
|
||||
[requires]
|
||||
python_version = "3.9"
|
||||
386
scripts/metrics/Pipfile.lock
generated
386
scripts/metrics/Pipfile.lock
generated
|
|
@ -1,386 +0,0 @@
|
|||
{
|
||||
"_meta": {
|
||||
"hash": {
|
||||
"sha256": "e3acba0f267d3eda606997edc26c93b7ce252026303108a9f1ae7d354c616aed"
|
||||
},
|
||||
"pipfile-spec": 6,
|
||||
"requires": {
|
||||
"python_version": "3.9"
|
||||
},
|
||||
"sources": [
|
||||
{
|
||||
"name": "pypi",
|
||||
"url": "https://pypi.python.org/simple",
|
||||
"verify_ssl": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"default": {
|
||||
"backoff": {
|
||||
"hashes": [
|
||||
"sha256:5e73e2cbe780e1915a204799dba0a01896f45f4385e636bcca7a0614d879d0cd",
|
||||
"sha256:b8fba021fac74055ac05eb7c7bfce4723aedde6cd0a504e5326bcb0bdd6d19a4"
|
||||
],
|
||||
"index": "pypi",
|
||||
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
|
||||
"version": "==1.10.0"
|
||||
},
|
||||
"certifi": {
|
||||
"hashes": [
|
||||
"sha256:539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082",
|
||||
"sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9"
|
||||
],
|
||||
"markers": "python_version >= '3.6'",
|
||||
"version": "==2023.7.22"
|
||||
},
|
||||
"chardet": {
|
||||
"hashes": [
|
||||
"sha256:0d6f53a15db4120f2b08c94f11e7d93d2c911ee118b6b30a04ec3ee8310179fa",
|
||||
"sha256:f864054d66fd9118f2e67044ac8981a54775ec5b67aed0441892edb553d21da5"
|
||||
],
|
||||
"index": "pypi",
|
||||
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
|
||||
"version": "==4.0.0"
|
||||
},
|
||||
"ftfy": {
|
||||
"hashes": [
|
||||
"sha256:0ffd33fce16b54cccaec78d6ec73d95ad370e5df5a25255c8966a6147bd667ca",
|
||||
"sha256:bfc2019f84fcd851419152320a6375604a0f1459c281b5b199b2cd0d2e727f8f"
|
||||
],
|
||||
"markers": "python_version >= '3.6'",
|
||||
"version": "==6.1.1"
|
||||
},
|
||||
"gitdb": {
|
||||
"hashes": [
|
||||
"sha256:6eb990b69df4e15bad899ea868dc46572c3f75339735663b81de79b06f17eb9a",
|
||||
"sha256:c286cf298426064079ed96a9e4a9d39e7f3e9bf15ba60701e95f5492f28415c7"
|
||||
],
|
||||
"markers": "python_version >= '3.7'",
|
||||
"version": "==4.0.10"
|
||||
},
|
||||
"gitpython": {
|
||||
"hashes": [
|
||||
"sha256:5f4c4187de49616d710a77e98ddf17b4782060a1788df441846bddefbb89ab33",
|
||||
"sha256:f9b9ddc0761c125d5780eab2d64be4873fc6817c2899cbcb34b02344bdc7bc54"
|
||||
],
|
||||
"markers": "python_version >= '3.7'",
|
||||
"version": "==3.1.37"
|
||||
},
|
||||
"idna": {
|
||||
"hashes": [
|
||||
"sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6",
|
||||
"sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0"
|
||||
],
|
||||
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
|
||||
"version": "==2.10"
|
||||
},
|
||||
"lxml": {
|
||||
"hashes": [
|
||||
"sha256:04da965dfebb5dac2619cb90fcf93efdb35b3c6994fea58a157a834f2f94b318",
|
||||
"sha256:0538747a9d7827ce3e16a8fdd201a99e661c7dee3c96c885d8ecba3c35d1032c",
|
||||
"sha256:0645e934e940107e2fdbe7c5b6fb8ec6232444260752598bc4d09511bd056c0b",
|
||||
"sha256:079b68f197c796e42aa80b1f739f058dcee796dc725cc9a1be0cdb08fc45b000",
|
||||
"sha256:0f3f0059891d3254c7b5fb935330d6db38d6519ecd238ca4fce93c234b4a0f73",
|
||||
"sha256:10d2017f9150248563bb579cd0d07c61c58da85c922b780060dcc9a3aa9f432d",
|
||||
"sha256:1355755b62c28950f9ce123c7a41460ed9743c699905cbe664a5bcc5c9c7c7fb",
|
||||
"sha256:13c90064b224e10c14dcdf8086688d3f0e612db53766e7478d7754703295c7c8",
|
||||
"sha256:1423631e3d51008871299525b541413c9b6c6423593e89f9c4cfbe8460afc0a2",
|
||||
"sha256:1436cf0063bba7888e43f1ba8d58824f085410ea2025befe81150aceb123e345",
|
||||
"sha256:1a7c59c6ffd6ef5db362b798f350e24ab2cfa5700d53ac6681918f314a4d3b94",
|
||||
"sha256:1e1cf47774373777936c5aabad489fef7b1c087dcd1f426b621fda9dcc12994e",
|
||||
"sha256:206a51077773c6c5d2ce1991327cda719063a47adc02bd703c56a662cdb6c58b",
|
||||
"sha256:21fb3d24ab430fc538a96e9fbb9b150029914805d551deeac7d7822f64631dfc",
|
||||
"sha256:27e590352c76156f50f538dbcebd1925317a0f70540f7dc8c97d2931c595783a",
|
||||
"sha256:287605bede6bd36e930577c5925fcea17cb30453d96a7b4c63c14a257118dbb9",
|
||||
"sha256:2aaf6a0a6465d39b5ca69688fce82d20088c1838534982996ec46633dc7ad6cc",
|
||||
"sha256:32a73c53783becdb7eaf75a2a1525ea8e49379fb7248c3eeefb9412123536387",
|
||||
"sha256:41fb58868b816c202e8881fd0f179a4644ce6e7cbbb248ef0283a34b73ec73bb",
|
||||
"sha256:4780677767dd52b99f0af1f123bc2c22873d30b474aa0e2fc3fe5e02217687c7",
|
||||
"sha256:4878e667ebabe9b65e785ac8da4d48886fe81193a84bbe49f12acff8f7a383a4",
|
||||
"sha256:487c8e61d7acc50b8be82bda8c8d21d20e133c3cbf41bd8ad7eb1aaeb3f07c97",
|
||||
"sha256:4beea0f31491bc086991b97517b9683e5cfb369205dac0148ef685ac12a20a67",
|
||||
"sha256:4cfbe42c686f33944e12f45a27d25a492cc0e43e1dc1da5d6a87cbcaf2e95627",
|
||||
"sha256:4d5bae0a37af799207140652a700f21a85946f107a199bcb06720b13a4f1f0b7",
|
||||
"sha256:4e285b5f2bf321fc0857b491b5028c5f276ec0c873b985d58d7748ece1d770dd",
|
||||
"sha256:57e4d637258703d14171b54203fd6822fda218c6c2658a7d30816b10995f29f3",
|
||||
"sha256:5974895115737a74a00b321e339b9c3f45c20275d226398ae79ac008d908bff7",
|
||||
"sha256:5ef87fca280fb15342726bd5f980f6faf8b84a5287fcc2d4962ea8af88b35130",
|
||||
"sha256:603a464c2e67d8a546ddaa206d98e3246e5db05594b97db844c2f0a1af37cf5b",
|
||||
"sha256:6653071f4f9bac46fbc30f3c7838b0e9063ee335908c5d61fb7a4a86c8fd2036",
|
||||
"sha256:6ca2264f341dd81e41f3fffecec6e446aa2121e0b8d026fb5130e02de1402785",
|
||||
"sha256:6d279033bf614953c3fc4a0aa9ac33a21e8044ca72d4fa8b9273fe75359d5cca",
|
||||
"sha256:6d949f53ad4fc7cf02c44d6678e7ff05ec5f5552b235b9e136bd52e9bf730b91",
|
||||
"sha256:6daa662aba22ef3258934105be2dd9afa5bb45748f4f702a3b39a5bf53a1f4dc",
|
||||
"sha256:6eafc048ea3f1b3c136c71a86db393be36b5b3d9c87b1c25204e7d397cee9536",
|
||||
"sha256:830c88747dce8a3e7525defa68afd742b4580df6aa2fdd6f0855481e3994d391",
|
||||
"sha256:86e92728ef3fc842c50a5cb1d5ba2bc66db7da08a7af53fb3da79e202d1b2cd3",
|
||||
"sha256:8caf4d16b31961e964c62194ea3e26a0e9561cdf72eecb1781458b67ec83423d",
|
||||
"sha256:8d1a92d8e90b286d491e5626af53afef2ba04da33e82e30744795c71880eaa21",
|
||||
"sha256:8f0a4d179c9a941eb80c3a63cdb495e539e064f8054230844dcf2fcb812b71d3",
|
||||
"sha256:9232b09f5efee6a495a99ae6824881940d6447debe272ea400c02e3b68aad85d",
|
||||
"sha256:927a9dd016d6033bc12e0bf5dee1dde140235fc8d0d51099353c76081c03dc29",
|
||||
"sha256:93e414e3206779ef41e5ff2448067213febf260ba747fc65389a3ddaa3fb8715",
|
||||
"sha256:98cafc618614d72b02185ac583c6f7796202062c41d2eeecdf07820bad3295ed",
|
||||
"sha256:9c3a88d20e4fe4a2a4a84bf439a5ac9c9aba400b85244c63a1ab7088f85d9d25",
|
||||
"sha256:9f36de4cd0c262dd9927886cc2305aa3f2210db437aa4fed3fb4940b8bf4592c",
|
||||
"sha256:a60f90bba4c37962cbf210f0188ecca87daafdf60271f4c6948606e4dabf8785",
|
||||
"sha256:a614e4afed58c14254e67862456d212c4dcceebab2eaa44d627c2ca04bf86837",
|
||||
"sha256:ae06c1e4bc60ee076292e582a7512f304abdf6c70db59b56745cca1684f875a4",
|
||||
"sha256:b122a188cd292c4d2fcd78d04f863b789ef43aa129b233d7c9004de08693728b",
|
||||
"sha256:b570da8cd0012f4af9fa76a5635cd31f707473e65a5a335b186069d5c7121ff2",
|
||||
"sha256:bcaa1c495ce623966d9fc8a187da80082334236a2a1c7e141763ffaf7a405067",
|
||||
"sha256:bd34f6d1810d9354dc7e35158aa6cc33456be7706df4420819af6ed966e85448",
|
||||
"sha256:be9eb06489bc975c38706902cbc6888f39e946b81383abc2838d186f0e8b6a9d",
|
||||
"sha256:c4b2e0559b68455c085fb0f6178e9752c4be3bba104d6e881eb5573b399d1eb2",
|
||||
"sha256:c62e8dd9754b7debda0c5ba59d34509c4688f853588d75b53c3791983faa96fc",
|
||||
"sha256:c852b1530083a620cb0de5f3cd6826f19862bafeaf77586f1aef326e49d95f0c",
|
||||
"sha256:d9fc0bf3ff86c17348dfc5d322f627d78273eba545db865c3cd14b3f19e57fa5",
|
||||
"sha256:dad7b164905d3e534883281c050180afcf1e230c3d4a54e8038aa5cfcf312b84",
|
||||
"sha256:e5f66bdf0976ec667fc4594d2812a00b07ed14d1b44259d19a41ae3fff99f2b8",
|
||||
"sha256:e8f0c9d65da595cfe91713bc1222af9ecabd37971762cb830dea2fc3b3bb2acf",
|
||||
"sha256:edffbe3c510d8f4bf8640e02ca019e48a9b72357318383ca60e3330c23aaffc7",
|
||||
"sha256:eea5d6443b093e1545ad0210e6cf27f920482bfcf5c77cdc8596aec73523bb7e",
|
||||
"sha256:ef72013e20dd5ba86a8ae1aed7f56f31d3374189aa8b433e7b12ad182c0d2dfb",
|
||||
"sha256:f05251bbc2145349b8d0b77c0d4e5f3b228418807b1ee27cefb11f69ed3d233b",
|
||||
"sha256:f1be258c4d3dc609e654a1dc59d37b17d7fef05df912c01fc2e15eb43a9735f3",
|
||||
"sha256:f9ced82717c7ec65a67667bb05865ffe38af0e835cdd78728f1209c8fffe0cad",
|
||||
"sha256:fe17d10b97fdf58155f858606bddb4e037b805a60ae023c009f760d8361a4eb8",
|
||||
"sha256:fe749b052bb7233fe5d072fcb549221a8cb1a16725c47c37e42b0b9cb3ff2c3f"
|
||||
],
|
||||
"index": "pypi",
|
||||
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
|
||||
"version": "==4.9.1"
|
||||
},
|
||||
"mailchecker": {
|
||||
"hashes": [
|
||||
"sha256:8b083907da42daa448d93eb19f0c978dde2e8565617a5e76271b35a5f7285ccd"
|
||||
],
|
||||
"version": "==5.0.9"
|
||||
},
|
||||
"pathspec": {
|
||||
"hashes": [
|
||||
"sha256:86379d6b86d75816baba717e64b1a3a3469deb93bb76d613c9ce79edc5cb68fd",
|
||||
"sha256:aa0cb481c4041bf52ffa7b0d8fa6cd3e88a2ca4879c533c9153882ee2556790d"
|
||||
],
|
||||
"index": "pypi",
|
||||
"version": "==0.8.1"
|
||||
},
|
||||
"phabricator": {
|
||||
"hashes": [
|
||||
"sha256:a276fb41eb91b550c46612aba874160807664ff7fe430adaa0ffa319249df981",
|
||||
"sha256:fabf0f63f020256bb44bb9ea0eba82a1d9493b2c2c5cf7a37fcdab5d4f9404e2"
|
||||
],
|
||||
"index": "pypi",
|
||||
"version": "==0.8.1"
|
||||
},
|
||||
"phonenumbers": {
|
||||
"hashes": [
|
||||
"sha256:34d6cb279dd4a64714e324c71350f96e5bda3237be28d11b4c555c44701544cd",
|
||||
"sha256:869e44fcaaf276eca6b953a401e2b27d57461f3a18a66cf5f13377e7bb0e228c"
|
||||
],
|
||||
"version": "==8.13.23"
|
||||
},
|
||||
"psycopg2-binary": {
|
||||
"hashes": [
|
||||
"sha256:0deac2af1a587ae12836aa07970f5cb91964f05a7c6cdb69d8425ff4c15d4e2c",
|
||||
"sha256:0e4dc3d5996760104746e6cfcdb519d9d2cd27c738296525d5867ea695774e67",
|
||||
"sha256:11b9c0ebce097180129e422379b824ae21c8f2a6596b159c7659e2e5a00e1aa0",
|
||||
"sha256:15978a1fbd225583dd8cdaf37e67ccc278b5abecb4caf6b2d6b8e2b948e953f6",
|
||||
"sha256:1fabed9ea2acc4efe4671b92c669a213db744d2af8a9fc5d69a8e9bc14b7a9db",
|
||||
"sha256:2dac98e85565d5688e8ab7bdea5446674a83a3945a8f416ad0110018d1501b94",
|
||||
"sha256:42ec1035841b389e8cc3692277a0bd81cdfe0b65d575a2c8862cec7a80e62e52",
|
||||
"sha256:6422f2ff0919fd720195f64ffd8f924c1395d30f9a495f31e2392c2efafb5056",
|
||||
"sha256:6a32f3a4cb2f6e1a0b15215f448e8ce2da192fd4ff35084d80d5e39da683e79b",
|
||||
"sha256:7312e931b90fe14f925729cde58022f5d034241918a5c4f9797cac62f6b3a9dd",
|
||||
"sha256:7d92a09b788cbb1aec325af5fcba9fed7203897bbd9269d5691bb1e3bce29550",
|
||||
"sha256:833709a5c66ca52f1d21d41865a637223b368c0ee76ea54ca5bad6f2526c7679",
|
||||
"sha256:89705f45ce07b2dfa806ee84439ec67c5d9a0ef20154e0e475e2b2ed392a5b83",
|
||||
"sha256:8cd0fb36c7412996859cb4606a35969dd01f4ea34d9812a141cd920c3b18be77",
|
||||
"sha256:950bc22bb56ee6ff142a2cb9ee980b571dd0912b0334aa3fe0fe3788d860bea2",
|
||||
"sha256:a0c50db33c32594305b0ef9abc0cb7db13de7621d2cadf8392a1d9b3c437ef77",
|
||||
"sha256:a0eb43a07386c3f1f1ebb4dc7aafb13f67188eab896e7397aa1ee95a9c884eb2",
|
||||
"sha256:aaa4213c862f0ef00022751161df35804127b78adf4a2755b9f991a507e425fd",
|
||||
"sha256:ac0c682111fbf404525dfc0f18a8b5f11be52657d4f96e9fcb75daf4f3984859",
|
||||
"sha256:ad20d2eb875aaa1ea6d0f2916949f5c08a19c74d05b16ce6ebf6d24f2c9f75d1",
|
||||
"sha256:b4afc542c0ac0db720cf516dd20c0846f71c248d2b3d21013aa0d4ef9c71ca25",
|
||||
"sha256:b8a3715b3c4e604bcc94c90a825cd7f5635417453b253499664f784fc4da0152",
|
||||
"sha256:ba28584e6bca48c59eecbf7efb1576ca214b47f05194646b081717fa628dfddf",
|
||||
"sha256:ba381aec3a5dc29634f20692349d73f2d21f17653bda1decf0b52b11d694541f",
|
||||
"sha256:bd1be66dde2b82f80afb9459fc618216753f67109b859a361cf7def5c7968729",
|
||||
"sha256:c2507d796fca339c8fb03216364cca68d87e037c1f774977c8fc377627d01c71",
|
||||
"sha256:cec7e622ebc545dbb4564e483dd20e4e404da17ae07e06f3e780b2dacd5cee66",
|
||||
"sha256:d14b140a4439d816e3b1229a4a525df917d6ea22a0771a2a78332273fd9528a4",
|
||||
"sha256:d1b4ab59e02d9008efe10ceabd0b31e79519da6fb67f7d8e8977118832d0f449",
|
||||
"sha256:d5227b229005a696cc67676e24c214740efd90b148de5733419ac9aaba3773da",
|
||||
"sha256:e1f57aa70d3f7cc6947fd88636a481638263ba04a742b4a37dd25c373e41491a",
|
||||
"sha256:e74a55f6bad0e7d3968399deb50f61f4db1926acf4a6d83beaaa7df986f48b1c",
|
||||
"sha256:e82aba2188b9ba309fd8e271702bd0d0fc9148ae3150532bbb474f4590039ffb",
|
||||
"sha256:ee69dad2c7155756ad114c02db06002f4cded41132cc51378e57aad79cc8e4f4",
|
||||
"sha256:f5ab93a2cb2d8338b1674be43b442a7f544a0971da062a5da774ed40587f18f5"
|
||||
],
|
||||
"index": "pypi",
|
||||
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
|
||||
"version": "==2.8.6"
|
||||
},
|
||||
"pyaml": {
|
||||
"hashes": [
|
||||
"sha256:29a5c2a68660a799103d6949167bd6c7953d031449d08802386372de1db6ad71",
|
||||
"sha256:67081749a82b72c45e5f7f812ee3a14a03b3f5c25ff36ec3b290514f8c4c4b99"
|
||||
],
|
||||
"index": "pypi",
|
||||
"version": "==20.4.0"
|
||||
},
|
||||
"python-benedict": {
|
||||
"hashes": [
|
||||
"sha256:1aa65ea78bd0bfd269e9289edb05d6f4e82ce62669ad0604a27d80db00a1a575",
|
||||
"sha256:af30f2a93aa15c0136c13bb528ded18b0e23171e8c567e968ee522a2b1cfa3b9"
|
||||
],
|
||||
"index": "pypi",
|
||||
"version": "==0.24.0"
|
||||
},
|
||||
"python-dateutil": {
|
||||
"hashes": [
|
||||
"sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86",
|
||||
"sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9"
|
||||
],
|
||||
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
|
||||
"version": "==2.8.2"
|
||||
},
|
||||
"python-fsutil": {
|
||||
"hashes": [
|
||||
"sha256:5f02ed7764cac27633b4aaaa9cf56af2245a64a6cc048866b6f02792a8c5bf86",
|
||||
"sha256:e0421440b5763d08129dcc469364105f3b434fc9736c3bc6ed181b1c623148ee"
|
||||
],
|
||||
"version": "==0.10.0"
|
||||
},
|
||||
"python-slugify": {
|
||||
"hashes": [
|
||||
"sha256:70ca6ea68fe63ecc8fa4fcf00ae651fc8a5d02d93dcd12ae6d4fc7ca46c4d395",
|
||||
"sha256:ce0d46ddb668b3be82f4ed5e503dbc33dd815d83e2eb6824211310d3fb172a27"
|
||||
],
|
||||
"markers": "python_version >= '3.7'",
|
||||
"version": "==8.0.1"
|
||||
},
|
||||
"pyyaml": {
|
||||
"hashes": [
|
||||
"sha256:04ac92ad1925b2cff1db0cfebffb6ffc43457495c9b3c39d3fcae417d7125dc5",
|
||||
"sha256:062582fca9fabdd2c8b54a3ef1c978d786e0f6b3a1510e0ac93ef59e0ddae2bc",
|
||||
"sha256:0d3304d8c0adc42be59c5f8a4d9e3d7379e6955ad754aa9d6ab7a398b59dd1df",
|
||||
"sha256:1635fd110e8d85d55237ab316b5b011de701ea0f29d07611174a1b42f1444741",
|
||||
"sha256:184c5108a2aca3c5b3d3bf9395d50893a7ab82a38004c8f61c258d4428e80206",
|
||||
"sha256:18aeb1bf9a78867dc38b259769503436b7c72f7a1f1f4c93ff9a17de54319b27",
|
||||
"sha256:1d4c7e777c441b20e32f52bd377e0c409713e8bb1386e1099c2415f26e479595",
|
||||
"sha256:1e2722cc9fbb45d9b87631ac70924c11d3a401b2d7f410cc0e3bbf249f2dca62",
|
||||
"sha256:1fe35611261b29bd1de0070f0b2f47cb6ff71fa6595c077e42bd0c419fa27b98",
|
||||
"sha256:28c119d996beec18c05208a8bd78cbe4007878c6dd15091efb73a30e90539696",
|
||||
"sha256:326c013efe8048858a6d312ddd31d56e468118ad4cdeda36c719bf5bb6192290",
|
||||
"sha256:40df9b996c2b73138957fe23a16a4f0ba614f4c0efce1e9406a184b6d07fa3a9",
|
||||
"sha256:42f8152b8dbc4fe7d96729ec2b99c7097d656dc1213a3229ca5383f973a5ed6d",
|
||||
"sha256:49a183be227561de579b4a36efbb21b3eab9651dd81b1858589f796549873dd6",
|
||||
"sha256:4fb147e7a67ef577a588a0e2c17b6db51dda102c71de36f8549b6816a96e1867",
|
||||
"sha256:50550eb667afee136e9a77d6dc71ae76a44df8b3e51e41b77f6de2932bfe0f47",
|
||||
"sha256:510c9deebc5c0225e8c96813043e62b680ba2f9c50a08d3724c7f28a747d1486",
|
||||
"sha256:5773183b6446b2c99bb77e77595dd486303b4faab2b086e7b17bc6bef28865f6",
|
||||
"sha256:596106435fa6ad000c2991a98fa58eeb8656ef2325d7e158344fb33864ed87e3",
|
||||
"sha256:6965a7bc3cf88e5a1c3bd2e0b5c22f8d677dc88a455344035f03399034eb3007",
|
||||
"sha256:69b023b2b4daa7548bcfbd4aa3da05b3a74b772db9e23b982788168117739938",
|
||||
"sha256:6c22bec3fbe2524cde73d7ada88f6566758a8f7227bfbf93a408a9d86bcc12a0",
|
||||
"sha256:704219a11b772aea0d8ecd7058d0082713c3562b4e271b849ad7dc4a5c90c13c",
|
||||
"sha256:7e07cbde391ba96ab58e532ff4803f79c4129397514e1413a7dc761ccd755735",
|
||||
"sha256:81e0b275a9ecc9c0c0c07b4b90ba548307583c125f54d5b6946cfee6360c733d",
|
||||
"sha256:855fb52b0dc35af121542a76b9a84f8d1cd886ea97c84703eaa6d88e37a2ad28",
|
||||
"sha256:8d4e9c88387b0f5c7d5f281e55304de64cf7f9c0021a3525bd3b1c542da3b0e4",
|
||||
"sha256:9046c58c4395dff28dd494285c82ba00b546adfc7ef001486fbf0324bc174fba",
|
||||
"sha256:9eb6caa9a297fc2c2fb8862bc5370d0303ddba53ba97e71f08023b6cd73d16a8",
|
||||
"sha256:a0cd17c15d3bb3fa06978b4e8958dcdc6e0174ccea823003a106c7d4d7899ac5",
|
||||
"sha256:afd7e57eddb1a54f0f1a974bc4391af8bcce0b444685d936840f125cf046d5bd",
|
||||
"sha256:b1275ad35a5d18c62a7220633c913e1b42d44b46ee12554e5fd39c70a243d6a3",
|
||||
"sha256:b786eecbdf8499b9ca1d697215862083bd6d2a99965554781d0d8d1ad31e13a0",
|
||||
"sha256:ba336e390cd8e4d1739f42dfe9bb83a3cc2e80f567d8805e11b46f4a943f5515",
|
||||
"sha256:baa90d3f661d43131ca170712d903e6295d1f7a0f595074f151c0aed377c9b9c",
|
||||
"sha256:bc1bf2925a1ecd43da378f4db9e4f799775d6367bdb94671027b73b393a7c42c",
|
||||
"sha256:bd4af7373a854424dabd882decdc5579653d7868b8fb26dc7d0e99f823aa5924",
|
||||
"sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34",
|
||||
"sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43",
|
||||
"sha256:c8098ddcc2a85b61647b2590f825f3db38891662cfc2fc776415143f599bb859",
|
||||
"sha256:d2b04aac4d386b172d5b9692e2d2da8de7bfb6c387fa4f801fbf6fb2e6ba4673",
|
||||
"sha256:d483d2cdf104e7c9fa60c544d92981f12ad66a457afae824d146093b8c294c54",
|
||||
"sha256:d858aa552c999bc8a8d57426ed01e40bef403cd8ccdd0fc5f6f04a00414cac2a",
|
||||
"sha256:e7d73685e87afe9f3b36c799222440d6cf362062f78be1013661b00c5c6f678b",
|
||||
"sha256:f003ed9ad21d6a4713f0a9b5a7a0a79e08dd0f221aff4525a2be4c346ee60aab",
|
||||
"sha256:f22ac1c3cac4dbc50079e965eba2c1058622631e526bd9afd45fedd49ba781fa",
|
||||
"sha256:faca3bdcf85b2fc05d06ff3fbc1f83e1391b3e724afa3feba7d13eeab355484c",
|
||||
"sha256:fca0e3a251908a499833aa292323f32437106001d436eca0e6e7833256674585",
|
||||
"sha256:fd1592b3fdf65fff2ad0004b5e363300ef59ced41c2e6b3a99d4089fa8c5435d",
|
||||
"sha256:fd66fc5d0da6d9815ba2cebeb4205f95818ff4b79c3ebe268e75d961704af52f"
|
||||
],
|
||||
"markers": "python_version >= '3.6'",
|
||||
"version": "==6.0.1"
|
||||
},
|
||||
"requests": {
|
||||
"hashes": [
|
||||
"sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804",
|
||||
"sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e"
|
||||
],
|
||||
"index": "pypi",
|
||||
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
|
||||
"version": "==2.25.1"
|
||||
},
|
||||
"six": {
|
||||
"hashes": [
|
||||
"sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
|
||||
"sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
|
||||
],
|
||||
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
|
||||
"version": "==1.16.0"
|
||||
},
|
||||
"smmap": {
|
||||
"hashes": [
|
||||
"sha256:dceeb6c0028fdb6734471eb07c0cd2aae706ccaecab45965ee83f11c8d3b1f62",
|
||||
"sha256:e6d8668fa5f93e706934a62d7b4db19c8d9eb8cf2adbb75ef1b675aa332b69da"
|
||||
],
|
||||
"markers": "python_version >= '3.7'",
|
||||
"version": "==5.0.1"
|
||||
},
|
||||
"text-unidecode": {
|
||||
"hashes": [
|
||||
"sha256:1311f10e8b895935241623731c2ba64f4c455287888b18189350b67134a822e8",
|
||||
"sha256:bad6603bb14d279193107714b288be206cac565dfa49aa5b105294dd5c4aab93"
|
||||
],
|
||||
"version": "==1.3"
|
||||
},
|
||||
"toml": {
|
||||
"hashes": [
|
||||
"sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b",
|
||||
"sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"
|
||||
],
|
||||
"markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
|
||||
"version": "==0.10.2"
|
||||
},
|
||||
"urllib3": {
|
||||
"hashes": [
|
||||
"sha256:34b97092d7e0a3a8cf7cd10e386f401b3737364026c45e622aa02903dffe0f07",
|
||||
"sha256:f8ecc1bba5667413457c529ab955bf8c67b45db799d159066261719e328580a0"
|
||||
],
|
||||
"index": "pypi",
|
||||
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'",
|
||||
"version": "==1.26.18"
|
||||
},
|
||||
"wcwidth": {
|
||||
"hashes": [
|
||||
"sha256:77f719e01648ed600dfa5402c347481c0992263b81a027344f3e1ba25493a704",
|
||||
"sha256:8705c569999ffbb4f6a87c6d1b80f324bd6db952f5eb0b95bc07517f4c1813d4"
|
||||
],
|
||||
"version": "==0.2.8"
|
||||
},
|
||||
"xmltodict": {
|
||||
"hashes": [
|
||||
"sha256:341595a488e3e01a85a9d8911d8912fd922ede5fecc4dce437eb4b6c8d037e56",
|
||||
"sha256:aa89e8fd76320154a40d19a0df04a4695fb9dc5ba977cbb68ab3e4eb225e7852"
|
||||
],
|
||||
"markers": "python_version >= '3.4'",
|
||||
"version": "==0.13.0"
|
||||
}
|
||||
},
|
||||
"develop": {}
|
||||
}
|
||||
|
|
@ -1,83 +0,0 @@
|
|||
# Metrics
|
||||
|
||||
To measure the impact and usefulness of the pre-merge checks, we want to collect
|
||||
a set of metrics. This doc will summarize the metrics and tools. All of the data
|
||||
shall be collected as time series, so that we can see changes over time.
|
||||
|
||||
* Impact - The metrics we ultimately want to improve
|
||||
* Percentage of [build-bot build](http://lab.llvm.org:8011/) on main
|
||||
failing. (Buildbot_percentage_failing)
|
||||
* Time to fix a broken main build: Time between start of failing builds
|
||||
until the build is fixed. (BuildBot_time_to_fix)
|
||||
* Percentage of Revisions on Phabricator where a broken build was fixed
|
||||
afterwards. This would indicate that a bug was found and fixed during
|
||||
the code review phase. (Premerge_fixes)
|
||||
* Number of reverts on main. This indicates that something was broken on
|
||||
main that slipped through the pre-merge tests or was submitted without
|
||||
any review. (Upstream_reverts)
|
||||
|
||||
* Users and behavior - Interesting to see and useful to adapt our approach.
|
||||
* Percentage of commits to main that went through Phabricator.
|
||||
* Number of participants in pre-merge tests.
|
||||
* Percentage of Revisions with pre-merge tests executed
|
||||
* Number of 30-day active committers on main and Phabricator.
|
||||
|
||||
* Builds - See how the infrastructure is doing.
|
||||
* Time between upload of diff until build results available.
|
||||
* Percentage of Revisions with successful/failed tests
|
||||
* Number of pre-merge builds/day.
|
||||
* Build queuing time.
|
||||
* Individual times for `cmake`, `ninja all`, `ninja check-all` per
|
||||
OS/architecture.
|
||||
* Result storage size.
|
||||
* Percentage of builds failing.
|
||||
|
||||
# Requirements
|
||||
|
||||
* Must:
|
||||
* Do not collect/store personal data.
|
||||
* Should:
|
||||
* Minimize the amount of additional tools/scripts we need to maintain.
|
||||
* Collect all metrics in a central location for easy evaluation (e.g.
|
||||
database, CSV files).
|
||||
* Nice to have:
|
||||
* As the data is from an open source project and available anyway, give
|
||||
public access to the metrics (numbers and charts).
|
||||
* Send out alerts/notifications.
|
||||
* Show live data in charts.
|
||||
|
||||
|
||||
# Data sources
|
||||
|
||||
This section will explain where we can get the data from.
|
||||
|
||||
* build bot statistics
|
||||
|
||||
# Solution
|
||||
|
||||
We need to find solutions for these parts:
|
||||
* Collect the data (regularly).
|
||||
* Store the time series somewhere.
|
||||
* Create & display charts.
|
||||
|
||||
Some ideas for this:
|
||||
* bunch of scripts:
|
||||
* Run a bunch of scripts manually to generate the metrics every now and
|
||||
then. Phabricator already has a database and most entries there have
|
||||
timestamps. So we could also reconstruct the history from that.
|
||||
* TODO: Figure out if we can collect the most important metrics this way.
|
||||
This requires that we can reconstruct historic values from the current
|
||||
logs/git/database/... entries.
|
||||
* Jenkins + CSV + Sheets:
|
||||
* collect data with jenkins
|
||||
* store numbers as CSV in this repo
|
||||
* Charts are created manually on Google Sheets
|
||||
* do it yourself:
|
||||
* Collect data with Jenkins jobs
|
||||
* Store the data on Prometheus
|
||||
* Visualize with Grafana
|
||||
* host all tools ourselves
|
||||
* Stackdriver on GCP:
|
||||
* TODO: figure out if we can get all the required data into Stackdriver
|
||||
* Jupyter notebooks:
|
||||
* TODO: figure out how that works
|
||||
|
|
@ -1,17 +0,0 @@
|
|||
/* list builders with success rate <70% over the last 7 days
|
||||
|
||||
these are probably worth investigating
|
||||
*/
|
||||
|
||||
Select *,
|
||||
format('https://lab.llvm.org/buildbot/#/builders/%s', builder_id) as link
|
||||
FROM (
|
||||
SELECT
|
||||
builder_id,
|
||||
count(*) as num_builds,
|
||||
(100.0*count(CASE WHEN buildbot_overview.result='build successful' THEN 1 END)/count(*)) as success_prct
|
||||
FROM buildbot_overview
|
||||
WHERE completed_at > current_date - interval '7' day
|
||||
GROUP BY builder_id
|
||||
) as builder_success
|
||||
WHERE success_prct < 70
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
/* list last 100 git commits with number of builds and success rate */
|
||||
SELECT git_commits.hash as revision,
|
||||
count(*) as num_builds,
|
||||
(100.0*count(CASE WHEN buildbot_overview.result='build successful' THEN 1 END)/count(*)) as success_prct
|
||||
FROM buildbot_overview, git_commits
|
||||
WHERE buildbot_overview.revision = git_commits.hash
|
||||
GROUP BY git_commits.hash
|
||||
ORDER BY git_commits.commit_time DesC
|
||||
LIMIT 100
|
||||
|
|
@ -1,7 +0,0 @@
|
|||
/* list number of builds and percentage of successful builds per day */
|
||||
SELECT to_date(cast(git_commits.commit_time as TEXT),'YYYY-MM-DD') as date,
|
||||
count(*) as num_builds,
|
||||
(100.0*count(CASE WHEN buildbot_overview.result='build successful' THEN 1 END)/count(*)) as success_prct
|
||||
FROM buildbot_overview, git_commits
|
||||
WHERE buildbot_overview.revision = git_commits.hash
|
||||
GROUP BY date ORDER BY date ASC;
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
/* Aggregated build information from multiple tables
|
||||
|
||||
also extracts additional columns from json data to make access easier. */
|
||||
|
||||
CREATE OR REPLACE VIEW buildbot_overview AS
|
||||
SELECT
|
||||
buildbot_buildsets.data -> 'sourcestamps' -> 0 ->> 'revision' AS revision,
|
||||
buildbot_builds.build_id,
|
||||
buildbot_builds.builder_id,
|
||||
buildbot_builds.build_number,
|
||||
buildbot_builds.build_data ->>'state_string' AS result,
|
||||
format('https://lab.llvm.org/buildbot/#/builders/%s/builds/%s', buildbot_builds.builder_id, buildbot_builds.build_number) as link,
|
||||
to_timestamp(CAST(build_data ->> 'complete_at' as int))::date as completed_at,
|
||||
to_timestamp(CAST(build_data ->> 'started_at' as int))::date as started_at
|
||||
FROM buildbot_buildsets, buildbot_buildrequests, buildbot_builds
|
||||
WHERE buildbot_buildrequests.buildset_id = buildbot_buildsets.buildset_id AND
|
||||
CAST(buildbot_builds.build_data ->> 'buildrequestid' AS int) = buildbot_buildrequests.buildrequest_id;
|
||||
|
||||
|
|
@ -1,7 +0,0 @@
|
|||
|
||||
/* get server uptime statistics in percent */
|
||||
SELECT DATE(timestamp) as date,
|
||||
count(*) as measurements,
|
||||
(100.0*count(CASE WHEN phabricator THEN 1 END)/count(*)) as phabricator_up_prcnt,
|
||||
(100.0*count(CASE WHEN buildbot THEN 1 END)/count(*)) as buildbot_up_prcnt
|
||||
FROM server_status GROUP BY date ORDER BY date ASC;
|
||||
|
|
@ -1,323 +0,0 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## License\n",
|
||||
"\n",
|
||||
"Copyright 2020 Google LLC\n",
|
||||
"\n",
|
||||
"Licensed under the the Apache License v2.0 with LLVM Exceptions (the \"License\");\n",
|
||||
"you may not use this file except in compliance with the License.\n",
|
||||
"You may obtain a copy of the License at\n",
|
||||
"\n",
|
||||
"https://llvm.org/LICENSE.txt\n",
|
||||
"\n",
|
||||
"Unless required by applicable law or agreed to in writing, software\n",
|
||||
"distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"See the License for the specific language governing permissions and\n",
|
||||
"limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"import os\n",
|
||||
"import logging\n",
|
||||
"import argparse\n",
|
||||
"import json\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import seaborn as sns"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cache_file = \"cache.json\"\n",
|
||||
"token = f'Bearer {os.getenv(\"BUILDKITE_API_TOKEN\")}'\n",
|
||||
"builds = []"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.path.exists(cache_file):\n",
|
||||
" with open(cache_file) as f:\n",
|
||||
" builds = json.load(f)\n",
|
||||
" print(f'loaded {len(builds)} entries') "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# load new jobs from Buildkite API\n",
|
||||
"max_pages = 2000\n",
|
||||
"if True:\n",
|
||||
" existing = set()\n",
|
||||
" for j in builds:\n",
|
||||
" existing.add(j['id'])\n",
|
||||
"\n",
|
||||
" # uncomment to reset\n",
|
||||
"# builds = []\n",
|
||||
"# existing = set()\n",
|
||||
" page = 1\n",
|
||||
" stop = False\n",
|
||||
" while page <= max_pages:\n",
|
||||
" print('loading page', page)\n",
|
||||
" re = requests.get('https://api.buildkite.com/v2/organizations/llvm-project/builds',\n",
|
||||
" params={'page': page},\n",
|
||||
" headers={'Authorization': token})\n",
|
||||
" if re.status_code != 200:\n",
|
||||
" print('response status', re.status_code, re)\n",
|
||||
" break\n",
|
||||
" x = re.json()\n",
|
||||
" if x == []:\n",
|
||||
" print('empty response')\n",
|
||||
" break\n",
|
||||
" for j in x:\n",
|
||||
" if j['id'] in existing:\n",
|
||||
"# print('found existing job', j['id'])\n",
|
||||
" # load some more pages as some of them might be running before and wasn't added. \n",
|
||||
" max_pages = min(page + 5, max_pages)\n",
|
||||
" else:\n",
|
||||
" # skip running jobs \n",
|
||||
" if (j['state'] == 'running') or (j['state'] == 'scheduled'):\n",
|
||||
" print(j['web_url'], 'is', j['state'], ', skipping')\n",
|
||||
" continue\n",
|
||||
" builds.append(j)\n",
|
||||
" page += 1\n",
|
||||
" print(len(builds), 'jobs in total') \n",
|
||||
" with open(cache_file, 'w') as f:\n",
|
||||
" json.dump(builds, f)\n",
|
||||
" print(f'saved {len(builds)} entries')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"d = {\n",
|
||||
" 'id': [],\n",
|
||||
" 'number': [],\n",
|
||||
" 'pipeline': [],\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"jobs = {\n",
|
||||
" 'pipeline': [],\n",
|
||||
" 'name': [],\n",
|
||||
" 'step_key': [],\n",
|
||||
" 'state': [],\n",
|
||||
" 'exit_status': [],\n",
|
||||
" 'agent_id': [],\n",
|
||||
" \n",
|
||||
" 'agent_name': [],\n",
|
||||
" 'runnable_at': [],\n",
|
||||
" 'started_at': [],\n",
|
||||
" 'wait_duration': [],\n",
|
||||
" 'finished_at': [],\n",
|
||||
" 'run_duration': [],\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"sec = np.timedelta64(1, 's')\n",
|
||||
"\n",
|
||||
"for b in builds:\n",
|
||||
" env = b['env']\n",
|
||||
"# if 'ph_windows_agents' not in env:\n",
|
||||
"# continue\n",
|
||||
"# if 'scripts_branch' not in env:\n",
|
||||
"# continue\n",
|
||||
" d['id'].append(b['id'])\n",
|
||||
" d['number'].append(b['number'])\n",
|
||||
" d['pipeline'].append(b['pipeline']['slug'])\n",
|
||||
" for x in b['jobs']:\n",
|
||||
" if x['state'] in ['waiting_failed', 'canceled', 'skipped', 'broken']:\n",
|
||||
" continue\n",
|
||||
" try:\n",
|
||||
" jobs['pipeline'].append(b['pipeline']['slug'])\n",
|
||||
" jobs['name'].append(x['name'])\n",
|
||||
" jobs['step_key'].append(x['step_key'] if 'step_key' in x else '')\n",
|
||||
" jobs['state'].append(x['state'] )\n",
|
||||
" jobs['exit_status'].append(x['exit_status'] if 'exit_status' in x else -1)\n",
|
||||
" jobs['agent_id'].append(x['agent']['id'] if 'agent' in x else '')\n",
|
||||
" jobs['agent_name'].append(x['agent']['name'] if 'agent' in x else '')\n",
|
||||
" runnable = np.datetime64(x['runnable_at'].replace('Z', ''))\n",
|
||||
" started = np.datetime64(x['started_at'].replace('Z', ''))\n",
|
||||
" finished = np.datetime64(x['finished_at'].replace('Z', ''))\n",
|
||||
" jobs['runnable_at'].append(runnable)\n",
|
||||
" jobs['started_at'].append(started)\n",
|
||||
" jobs['wait_duration'].append((started - runnable) / sec)\n",
|
||||
" jobs['finished_at'].append(finished)\n",
|
||||
" jobs['run_duration'].append((finished - started) / sec)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(x)\n",
|
||||
" raise e \n",
|
||||
"jobs = pd.DataFrame(jobs)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"jobs.pipeline.unique()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"jobs"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ds = jobs[jobs['pipeline'] == 'llvm-main-build'][jobs['step_key'] == 'windows'][jobs['state']=='passed'][~jobs['agent_name'].str.startswith('buildkite-')][jobs['started_at'] > np.datetime64('2020-01-22')]\n",
|
||||
"ds = jobs[jobs['pipeline'] == 'llvm-main-build'][jobs['step_key'] == 'windows'][jobs['state']=='passed'][~jobs['agent_name'].str.startswith('buildkite-')][jobs['started_at'] > np.datetime64('2020-01-22')]\n",
|
||||
"ds = ds.drop_duplicates()\n",
|
||||
"# remove one slowest run (repo checkout)\n",
|
||||
"# t = ds.loc[ds.groupby([\"agent_name\"])[\"run_duration\"].idxmax()]\n",
|
||||
"# ds = pd.concat([ds, t]).drop_duplicates(keep=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(10,10)) # size of the plot (width, height)\n",
|
||||
"\n",
|
||||
"sns.swarmplot(\n",
|
||||
" ax=ax,\n",
|
||||
" x='run_duration',\n",
|
||||
" y='agent_name',\n",
|
||||
" split=True,\n",
|
||||
" data=ds)\n",
|
||||
"\n",
|
||||
"plt.xlim(0, None)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"t = pd.pivot_table(ds, values=['run_duration'], index=['agent_name'],\n",
|
||||
" aggfunc=[np.median, np.mean, np.std, np.count_nonzero])\n",
|
||||
"t = t.reindex(t.sort_values(by=('median', 'run_duration'), ascending=False).index)\n",
|
||||
"t"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"// TODO: try with w64 w/o AV (temporary disabled)\n",
|
||||
"w1 - 16 - non ssd 100 + ssd\n",
|
||||
"w1a - 16 - non ssd 100 + ssd - AV (temp)\n",
|
||||
"w4a - 16 - non ssd 100 + ssd - AV\n",
|
||||
"w16-1 - 16 - non sdd 200 + ssd\n",
|
||||
"w32-1 - 32 - non ssd 100 + ssd\n",
|
||||
"w32-2 - 32 - non ssd 100 + ssd - AV\n",
|
||||
"w32-2d - 32 - non ssd 200 + ssd\n",
|
||||
"w64 - 64 - ssd 200 + ssd \n",
|
||||
"w64a - 64 - ssd 200 + ssd - AV (temp)\n",
|
||||
"wh63-3 - 64 low ram - non ssd 100 + ssd - AV\n",
|
||||
"\n",
|
||||
"buildkite-windows-16n1-exp\n",
|
||||
"gcloud beta compute instances create \"${NAME}\" \\\n",
|
||||
" --project=\"${GCP_PROJECT}\" \\\n",
|
||||
" --zone=\"${GCP_ZONE}\" \\\n",
|
||||
" --machine-type=n1-standard-16 \\\n",
|
||||
" --local-ssd=device-name=local-ssd-0 \\\n",
|
||||
" --image=windows-server-2019-dc-for-containers-v20200714 \\\n",
|
||||
" --image-project=windows-cloud \\\n",
|
||||
" --boot-disk-size=200GB --boot-disk-type=pd-ssd\n",
|
||||
"\n",
|
||||
"buildkite-32-experimental-1\n",
|
||||
"gcloud beta compute instances create \"${NAME}\" \\\n",
|
||||
" --project=\"${GCP_PROJECT}\" \\\n",
|
||||
" --zone=\"${GCP_ZONE}\" \\\n",
|
||||
" --machine-type=n1-standard-32 \\\n",
|
||||
" --local-ssd=device-name=local-ssd-0 \\\n",
|
||||
" --image=windows-server-2019-dc-for-containers-v20200714 \\\n",
|
||||
" --image-project=windows-cloud \\\n",
|
||||
" --boot-disk-size=200GB --boot-disk-type=pd-ssd\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"buildkite-16c2-exp\n",
|
||||
"gcloud beta compute instances create \"${NAME}\" \\\n",
|
||||
" --project=\"${GCP_PROJECT}\" \\\n",
|
||||
" --zone=\"${GCP_ZONE}\" \\\n",
|
||||
" --machine-type=c2-standard-16 \\\n",
|
||||
" --local-ssd=device-name=local-ssd-0 \\\n",
|
||||
" --local-ssd=interface=SCSI \\\n",
|
||||
" --image=windows-server-2019-dc-for-containers-v20200714 \\\n",
|
||||
" --image-project=windows-cloud \\\n",
|
||||
" --boot-disk-size=200GB --boot-disk-type=pd-ssd\n",
|
||||
" \n",
|
||||
"...a: as is D:/ is a workdir (ealy run with full cache build)\n",
|
||||
"...b: C:/ws as a workdir, no cache\n",
|
||||
"...c: c:/ws as workdir, no av\n",
|
||||
"```\n",
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.5rc1"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -1,322 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
import logging
|
||||
import psycopg2
|
||||
import os
|
||||
import datetime
|
||||
import requests
|
||||
from typing import Optional, Dict, List
|
||||
import json
|
||||
|
||||
PHABRICATOR_URL = "https://reviews.llvm.org/api/"
|
||||
BUILDBOT_URL = "https://lab.llvm.org/buildbot/api/v2/"
|
||||
|
||||
|
||||
# TODO(kuhnel): retry on connection issues, maybe resuse
|
||||
# https://github.com/google/llvm-premerge-checks/blob/main/scripts/phabtalk/phabtalk.py#L44
|
||||
|
||||
# TODO(kuhnel): Import the step data so we can figure out in which step a build fails
|
||||
# (e.g. compile vs. test)
|
||||
|
||||
|
||||
def connect_to_db() -> psycopg2.extensions.connection:
|
||||
"""Connect to the database."""
|
||||
conn = psycopg2.connect(
|
||||
f"host=127.0.0.1 sslmode=disable dbname=stats user=stats password={os.getenv('DB_PASSWORD')}")
|
||||
return conn
|
||||
|
||||
|
||||
def create_tables(conn: psycopg2.extensions.connection):
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS buildbot_workers (
|
||||
timestamp timestamp NOT NULL,
|
||||
worker_id integer NOT NULL,
|
||||
data jsonb NOT NULL
|
||||
);"""
|
||||
)
|
||||
cur.execute(
|
||||
"""CREATE INDEX IF NOT EXISTS buildbot_worker_ids
|
||||
ON buildbot_workers
|
||||
(worker_id);"""
|
||||
)
|
||||
cur.execute(
|
||||
"""CREATE INDEX IF NOT EXISTS buildbot_worker_timestamp
|
||||
ON buildbot_workers
|
||||
(timestamp);"""
|
||||
)
|
||||
# Note: step_data is not yet populated with data!
|
||||
cur.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS buildbot_builds (
|
||||
build_id integer PRIMARY KEY,
|
||||
builder_id integer NOT NULL,
|
||||
build_number integer NOT NULL,
|
||||
build_data jsonb NOT NULL,
|
||||
step_data jsonb
|
||||
);"""
|
||||
)
|
||||
cur.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS buildbot_buildsets (
|
||||
buildset_id integer PRIMARY KEY,
|
||||
data jsonb NOT NULL
|
||||
);"""
|
||||
)
|
||||
cur.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS buildbot_buildrequests (
|
||||
buildrequest_id integer PRIMARY KEY,
|
||||
buildset_id integer NOT NULL,
|
||||
data jsonb NOT NULL
|
||||
);"""
|
||||
)
|
||||
cur.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS buildbot_builders (
|
||||
builder_id integer PRIMARY KEY,
|
||||
timestamp timestamp NOT NULL,
|
||||
name text NOT NULL,
|
||||
data jsonb NOT NULL
|
||||
);"""
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def get_worker_status(
|
||||
worker_id: int, conn: psycopg2.extensions.connection
|
||||
) -> Optional[Dict]:
|
||||
"""Note: postgres returns a dict for a stored json object."""
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"SELECT data FROM buildbot_workers WHERE worker_id = %s ORDER BY timestamp DESC;",
|
||||
[worker_id],
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return row[0]
|
||||
|
||||
|
||||
def get_builder_status(
|
||||
builder_id: int, conn: psycopg2.extensions.connection
|
||||
) -> Optional[Dict]:
|
||||
"""Note: postgres returns a dict for a stored json object."""
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"""SELECT data FROM buildbot_builders WHERE builder_id = %s
|
||||
ORDER BY timestamp DESC;""",
|
||||
[builder_id],
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return row[0]
|
||||
|
||||
|
||||
def set_worker_status(
|
||||
timestamp: datetime.datetime,
|
||||
worker_id: int,
|
||||
data: str,
|
||||
conn: psycopg2.extensions.connection,
|
||||
):
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"""INSERT INTO buildbot_workers (timestamp, worker_id, data)
|
||||
values (%s,%s,%s);""",
|
||||
(timestamp, worker_id, data),
|
||||
)
|
||||
|
||||
|
||||
def update_workers(conn: psycopg2.extensions.connection):
|
||||
logging.info("Updating worker status...")
|
||||
response = requests.get(BUILDBOT_URL + "workers")
|
||||
timestamp = datetime.datetime.now()
|
||||
for worker in response.json()["workers"]:
|
||||
worker_id = worker["workerid"]
|
||||
data = json.dumps(worker)
|
||||
# TODO: It would be faster if request all worker info and cache it
|
||||
# locally
|
||||
old_data = get_worker_status(worker_id, conn)
|
||||
# only update worker information if it has changed as this data is quite
|
||||
# static
|
||||
if old_data is None or worker != old_data:
|
||||
set_worker_status(timestamp, worker_id, data, conn)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def update_builders(conn: psycopg2.extensions.connection):
|
||||
"""get list of all builder ids."""
|
||||
logging.info("Updating builder status...")
|
||||
response = requests.get(BUILDBOT_URL + "builders")
|
||||
timestamp = datetime.datetime.now()
|
||||
for builder in response.json()["builders"]:
|
||||
builder_id = builder["builderid"]
|
||||
data = json.dumps(builder)
|
||||
# TODO: It would be faster if request all builder info and cache it
|
||||
# locally
|
||||
old_data = get_builder_status(builder_id, conn)
|
||||
# only update worker information if it has changed as this data is quite
|
||||
# static
|
||||
if old_data is None or builder != old_data:
|
||||
set_worker_status(timestamp, builder_id, data, conn)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def get_last_build(conn: psycopg2.extensions.connection) -> int:
|
||||
"""Get the latest build number for a builder.
|
||||
|
||||
This is used to only get new builds."""
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT MAX(build_id) FROM buildbot_builds")
|
||||
row = cur.fetchone()
|
||||
if row is None or row[0] is None:
|
||||
return 0
|
||||
return row[0]
|
||||
|
||||
|
||||
def update_build_status(conn: psycopg2.extensions.connection):
|
||||
start_id = get_last_build(conn)
|
||||
logging.info("Updating build results, starting with {}...".format(start_id))
|
||||
url = BUILDBOT_URL + "builds"
|
||||
cur = conn.cursor()
|
||||
for result_set in rest_request_iterator(url, "builds", "buildid", start_id=start_id):
|
||||
args_str = b",".join(
|
||||
cur.mogrify(
|
||||
b" (%s,%s,%s,%s) ",
|
||||
(
|
||||
build["buildid"],
|
||||
build["builderid"],
|
||||
build["number"],
|
||||
json.dumps(build, sort_keys=True),
|
||||
),
|
||||
)
|
||||
for build in result_set
|
||||
if build["complete"]
|
||||
)
|
||||
cur.execute(
|
||||
b"INSERT INTO buildbot_builds (build_id, builder_id, build_number, build_data) values "
|
||||
+ args_str
|
||||
)
|
||||
logging.info("last build id: {}".format(result_set[-1]["buildid"]))
|
||||
conn.commit()
|
||||
|
||||
|
||||
def rest_request_iterator(
|
||||
url: str,
|
||||
array_field_name: str,
|
||||
id_field_name: str,
|
||||
start_id: int = 0,
|
||||
step: int = 1000,
|
||||
):
|
||||
"""Request paginated data from the buildbot master.
|
||||
|
||||
This returns a generator. Each call to it gives you shards of
|
||||
<=limit results. This can be used to do a mass-SQL insert of data.
|
||||
|
||||
Limiting the range of the returned IDs causes Buildbot to sort the data.
|
||||
This makes incremental imports much easier.
|
||||
"""
|
||||
while True:
|
||||
count = 0
|
||||
stop_id = start_id + step
|
||||
response = requests.get(
|
||||
url
|
||||
+ "?{id_field_name}__gt={start_id}&{id_field_name}__le={stop_id}&".format(
|
||||
**locals()
|
||||
)
|
||||
)
|
||||
if response.status_code != 200:
|
||||
raise Exception(
|
||||
"Got status code {} on request to {}".format(response.status_code, url)
|
||||
)
|
||||
results = response.json()[array_field_name]
|
||||
if len(results) == 0:
|
||||
return
|
||||
yield results
|
||||
start_id = stop_id
|
||||
|
||||
|
||||
def get_latest_buildset(conn: psycopg2.extensions.connection) -> int:
|
||||
"""Get the maximumg buildset id.
|
||||
|
||||
This is useful for incremental updates."""
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT MAX(buildset_id) from buildbot_buildsets;")
|
||||
row = cur.fetchone()
|
||||
if row[0] is None:
|
||||
return 0
|
||||
return row[0]
|
||||
|
||||
|
||||
def update_buildsets(conn: psycopg2.extensions.connection):
|
||||
start_id = get_latest_buildset(conn)
|
||||
logging.info("Getting buildsets, starting with {}...".format(start_id))
|
||||
url = BUILDBOT_URL + "buildsets"
|
||||
cur = conn.cursor()
|
||||
|
||||
for result_set in rest_request_iterator(
|
||||
url, "buildsets", "bsid", start_id=start_id
|
||||
):
|
||||
args_str = b",".join(
|
||||
cur.mogrify(
|
||||
b" (%s,%s) ",
|
||||
(buildset["bsid"], json.dumps(buildset, sort_keys=True)),
|
||||
)
|
||||
for buildset in result_set
|
||||
if buildset["complete"]
|
||||
)
|
||||
|
||||
if len(args_str) == 0:
|
||||
break
|
||||
cur.execute(
|
||||
b"INSERT INTO buildbot_buildsets (buildset_id, data) values " + args_str
|
||||
)
|
||||
logging.info("last id {}".format(result_set[-1]["bsid"]))
|
||||
conn.commit()
|
||||
|
||||
|
||||
def get_latest_buildrequest(conn: psycopg2.extensions.connection) -> int:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT MAX(buildrequest_id) from buildbot_buildrequests;")
|
||||
row = cur.fetchone()
|
||||
if row[0] is None:
|
||||
return 0
|
||||
return row[0]
|
||||
|
||||
|
||||
def update_buildrequests(conn: psycopg2.extensions.connection):
|
||||
start_id = get_latest_buildrequest(conn)
|
||||
logging.info("Getting buildrequests, starting with {}...".format(start_id))
|
||||
url = BUILDBOT_URL + "buildrequests"
|
||||
cur = conn.cursor()
|
||||
for result_set in rest_request_iterator(
|
||||
url, "buildrequests", "buildrequestid", start_id=start_id
|
||||
):
|
||||
# cur.mogrify returns a byte string, so we need to join on a byte string
|
||||
args_str = b",".join(
|
||||
cur.mogrify(
|
||||
" (%s,%s,%s) ",
|
||||
(
|
||||
buildrequest["buildrequestid"],
|
||||
buildrequest["buildsetid"],
|
||||
json.dumps(buildrequest),
|
||||
),
|
||||
)
|
||||
for buildrequest in result_set
|
||||
if buildrequest["complete"]
|
||||
)
|
||||
if len(args_str) == 0:
|
||||
break
|
||||
cur.execute(
|
||||
b"INSERT INTO buildbot_buildrequests (buildrequest_id, buildset_id, data) values "
|
||||
+ args_str
|
||||
)
|
||||
logging.info("{}".format(result_set[-1]["buildrequestid"]))
|
||||
conn.commit()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logging.basicConfig(level='INFO', format='%(levelname)-7s %(message)s')
|
||||
conn = connect_to_db()
|
||||
create_tables(conn)
|
||||
update_workers(conn)
|
||||
update_builders(conn)
|
||||
update_build_status(conn)
|
||||
update_buildsets(conn)
|
||||
update_buildrequests(conn)
|
||||
|
|
@ -1,141 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import csv
|
||||
import datetime
|
||||
import gzip
|
||||
import os
|
||||
import mailbox
|
||||
import requests
|
||||
import re
|
||||
from typing import List, Dict, Set
|
||||
|
||||
|
||||
EMAIL_ARCHIVE_URL = 'http://lists.llvm.org/pipermail/llvm-dev/{year}-{month}.txt.gz'
|
||||
TMP_DIR = os.path.join(os.path.dirname(__file__), 'tmp')
|
||||
|
||||
|
||||
class LLVMBotArchiveScanner:
|
||||
|
||||
def __init__(self):
|
||||
self._tmpdir = TMP_DIR
|
||||
|
||||
@staticmethod
|
||||
def _generate_archive_url(month: datetime.date) -> str:
|
||||
return EMAIL_ARCHIVE_URL.format(year=month.year, month=month.strftime('%B'))
|
||||
|
||||
def _download_archive(self, month: datetime.date):
|
||||
os.makedirs(self._tmpdir, exist_ok=True)
|
||||
filename = os.path.join(self._tmpdir, 'llvmdev-{year}-{month:02d}.txt'.format(year=month.year, month=month.month))
|
||||
url = self._generate_archive_url(month)
|
||||
# FIXME: decompress the files
|
||||
self.download(url, filename)
|
||||
|
||||
def get_archives(self, start_month: datetime.date):
|
||||
print('Downloading data...')
|
||||
month = start_month
|
||||
today = datetime.date.today()
|
||||
while month < today:
|
||||
self._download_archive(month)
|
||||
if month.month < 12:
|
||||
month = datetime.date(year=month.year, month=month.month+1, day=1)
|
||||
else:
|
||||
month = datetime.date(year=month.year+1, month=1, day=1)
|
||||
|
||||
def extract_emails(self) -> List[mailbox.Message]:
|
||||
result = []
|
||||
for archive_name in (d for d in os.listdir(self._tmpdir) if d.startswith('llvmdev-')):
|
||||
print('Scanning {}'.format(archive_name))
|
||||
mb = mailbox.mbox(os.path.join(self._tmpdir, archive_name), factory=mbox_reader)
|
||||
for mail in mb.values():
|
||||
subject = mail.get('subject')
|
||||
if subject is None:
|
||||
continue
|
||||
if 'Buildbot numbers' in mail['subject']:
|
||||
yield(mail)
|
||||
yield
|
||||
|
||||
def get_attachments(self, email: mailbox.Message):
|
||||
if email is None:
|
||||
return
|
||||
week_str = re.search(r'(\d+/\d+/\d+)', email['subject']).group(1)
|
||||
week = datetime.datetime.strptime(week_str, '%m/%d/%Y').date()
|
||||
attachment_url = re.search(r'Name: completed_failed_avr_time.csv[^<]*URL: <([^>]+)>', email.get_payload(), re.DOTALL).group(1)
|
||||
filename = os.path.join(self._tmpdir, 'buildbot_stats_{}.csv'.format(week.isoformat()))
|
||||
self.download(attachment_url, filename)
|
||||
|
||||
@staticmethod
|
||||
def download(url, filename):
|
||||
if os.path.exists(filename):
|
||||
return
|
||||
r = requests.get(url)
|
||||
print('Getting {}'.format(filename))
|
||||
with open(filename, 'wb') as f:
|
||||
f.write(r.content)
|
||||
|
||||
def merge_results(self):
|
||||
def _convert_int(s: str) -> int:
|
||||
if len(s) == 0:
|
||||
return 0
|
||||
return int(s)
|
||||
|
||||
bot_stats = {} # type: Dict[str, Dict[datetime.date, float]]
|
||||
weeks = set() # type: Set[datetime.date]
|
||||
for csv_filename in (d for d in os.listdir(self._tmpdir) if d.startswith('buildbot_stats_')):
|
||||
week_str = re.search(r'(\d+-\d+-\d+)', csv_filename).group(1)
|
||||
week = datetime.datetime.fromisoformat(week_str).date()
|
||||
weeks.add(week)
|
||||
with open(os.path.join(self._tmpdir, csv_filename)) as csv_file:
|
||||
reader = csv.DictReader(csv_file)
|
||||
for row in reader:
|
||||
name = row['name']
|
||||
red_build = _convert_int(row['red_builds'])
|
||||
all_builds = _convert_int(row['all_builds'])
|
||||
percentage = 100.0 * red_build / all_builds
|
||||
bot_stats.setdefault(name, {})
|
||||
bot_stats[name][week] = percentage
|
||||
|
||||
with open(os.path.join(self._tmpdir, 'buildbot_weekly.csv'), 'w') as csv_file:
|
||||
fieldnames = ['week']
|
||||
filtered_bots = sorted(b for b in bot_stats.keys()) # if len(bot_stats[b]) == len(weeks)
|
||||
fieldnames.extend(filtered_bots)
|
||||
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
|
||||
writer.writeheader()
|
||||
for week in sorted(weeks):
|
||||
row = {'week': week.isoformat()}
|
||||
for bot in filtered_bots:
|
||||
percentage = bot_stats[bot].get(week)
|
||||
if percentage is None:
|
||||
continue
|
||||
row[bot] = percentage
|
||||
writer.writerow(row)
|
||||
|
||||
|
||||
def mbox_reader(stream):
|
||||
"""Read a non-ascii message from mailbox.
|
||||
|
||||
Based on https://stackoverflow.com/questions/37890123/how-to-trap-an-exception-that-occurs-in-code-underlying-python-for-loop
|
||||
"""
|
||||
data = stream.read()
|
||||
text = data.decode(encoding="utf-8")
|
||||
return mailbox.mboxMessage(text)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
scanner = LLVMBotArchiveScanner()
|
||||
scanner.get_archives(datetime.date(year=2019, month=8, day=1))
|
||||
for message in scanner.extract_emails():
|
||||
scanner.get_attachments(message)
|
||||
scanner.merge_results()
|
||||
|
|
@ -1,138 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from datetime import date
|
||||
import requests
|
||||
import datetime
|
||||
from google.cloud import monitoring_v3
|
||||
|
||||
BASE_URL = 'http://lab.llvm.org:8011/json/builders'
|
||||
GCP_PROJECT_ID = 'llvm-premerge-checks'
|
||||
|
||||
class BuildStats:
|
||||
"""Build statistics.
|
||||
|
||||
Plain data object.
|
||||
"""
|
||||
|
||||
def __init__(self, successful:int = 0, failed:int = 0):
|
||||
self.successful = successful # type: int
|
||||
self.failed = failed # type: int
|
||||
|
||||
def add(self, success: bool):
|
||||
if success:
|
||||
self.successful += 1
|
||||
else:
|
||||
self.failed += 1
|
||||
|
||||
@property
|
||||
def total(self) -> int:
|
||||
return self.successful + self.failed
|
||||
|
||||
@property
|
||||
def percent_failed(self) -> float:
|
||||
return 100.0 * self.failed / self.total
|
||||
|
||||
def __add__(self, other: "BuildStats") -> "BuildStats":
|
||||
return BuildStats(
|
||||
self.successful + other.successful,
|
||||
self.failed + other.failed)
|
||||
|
||||
def __str__(self) -> str:
|
||||
result = [
|
||||
'successful: {}'.format(self.successful),
|
||||
'failed: {}'.format(self.failed),
|
||||
'total: {}'.format(self.total),
|
||||
'% failed: {:0.1f}'.format(self.percent_failed),
|
||||
]
|
||||
return '\n'.join(result)
|
||||
|
||||
|
||||
def get_buildbot_stats(time_window : datetime.datetime) -> BuildStats:
|
||||
"""Get the statistics for the all builders."""
|
||||
print('getting list of builders...')
|
||||
stats = BuildStats()
|
||||
for builder in requests.get(BASE_URL).json().keys():
|
||||
# TODO: maybe filter the builds to the ones we care about
|
||||
stats += get_builder_stats(builder, time_window )
|
||||
return stats
|
||||
|
||||
|
||||
def get_builder_stats(builder: str, time_window: datetime.datetime) -> BuildStats:
|
||||
"""Get the statistics for one builder."""
|
||||
print('Gettings builds for {}...'.format(builder))
|
||||
# TODO: can we limit the data we're requesting?
|
||||
url = '{}/{}/builds/_all'.format(BASE_URL, builder)
|
||||
stats = BuildStats()
|
||||
for build, results in requests.get(url).json().items():
|
||||
start_time = datetime.datetime.fromtimestamp(float(results['times'][0]))
|
||||
if start_time < time_window:
|
||||
continue
|
||||
successful = results['text'] == ['build', 'successful']
|
||||
stats.add(successful)
|
||||
return stats
|
||||
|
||||
|
||||
def gcp_create_metric_descriptor(project_id: str):
|
||||
"""Create metric descriptors on Stackdriver.
|
||||
|
||||
Re-creating these with every call is fine."""
|
||||
client = monitoring_v3.MetricServiceClient()
|
||||
project_name = client.project_path(project_id)
|
||||
|
||||
for desc_type, desc_desc in [
|
||||
["buildbots_percent_failed", "Percentage of failed builds"],
|
||||
["buildbots_builds_successful", "Number of successful builds in the last 24h."],
|
||||
["buildbots_builds_failed", "Number of failed builds in the last 24h."],
|
||||
["buildbots_builds_total", "Total number of builds in the last 24h."],
|
||||
]:
|
||||
|
||||
descriptor = monitoring_v3.types.MetricDescriptor()
|
||||
descriptor.type = 'custom.googleapis.com/buildbots_{}'.format(desc_type)
|
||||
descriptor.metric_kind = (
|
||||
monitoring_v3.enums.MetricDescriptor.MetricKind.GAUGE)
|
||||
descriptor.value_type = (
|
||||
monitoring_v3.enums.MetricDescriptor.ValueType.DOUBLE)
|
||||
descriptor.description = desc_desc
|
||||
descriptor = client.create_metric_descriptor(project_name, descriptor)
|
||||
print('Created {}.'.format(descriptor.name))
|
||||
|
||||
|
||||
def gcp_write_data(project_id: str, stats: BuildStats):
|
||||
"""Upload metrics to Stackdriver."""
|
||||
client = monitoring_v3.MetricServiceClient()
|
||||
project_name = client.project_path(project_id)
|
||||
now = datetime.datetime.now()
|
||||
|
||||
for desc_type, value in [
|
||||
["buildbots_percent_failed", stats.percent_failed],
|
||||
["buildbots_builds_successful", stats.successful],
|
||||
["buildbots_builds_failed", stats.failed],
|
||||
["buildbots_builds_total", stats.total],
|
||||
]:
|
||||
series = monitoring_v3.types.TimeSeries()
|
||||
series.metric.type = 'custom.googleapis.com/buildbots_{}'.format(desc_type)
|
||||
series.resource.type = 'global'
|
||||
point = series.points.add()
|
||||
point.value.double_value = value
|
||||
point.interval.end_time.seconds = int(now.timestamp())
|
||||
client.create_time_series(project_name, [series])
|
||||
|
||||
if __name__ == '__main__':
|
||||
gcp_create_metric_descriptor(GCP_PROJECT_ID)
|
||||
stats = get_buildbot_stats(
|
||||
datetime.datetime.now() - datetime.timedelta(hours=24))
|
||||
gcp_write_data(GCP_PROJECT_ID, stats)
|
||||
print(stats)
|
||||
|
|
@ -1,145 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# This script will collect all breakages of the main branch builds from
|
||||
# buildkite and format the results nicely.
|
||||
# Arguments:
|
||||
# llvm-path : folder where the LLVM checkout is kept
|
||||
# token : Access token for buildkite
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
import requests
|
||||
import git
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import datetime
|
||||
|
||||
class Build:
|
||||
|
||||
def __init__(self, json_dict):
|
||||
self._json_dict = json_dict
|
||||
|
||||
@property
|
||||
def number(self) -> int:
|
||||
return self._json_dict['number']
|
||||
|
||||
@property
|
||||
def web_url(self) -> str:
|
||||
return self._json_dict['web_url']
|
||||
|
||||
@property
|
||||
def state(self) -> str:
|
||||
return self._json_dict['state']
|
||||
|
||||
@property
|
||||
def has_passed(self) -> bool:
|
||||
return self.state == 'passed'
|
||||
|
||||
@property
|
||||
def commit(self) -> str:
|
||||
return self._json_dict['commit']
|
||||
|
||||
@property
|
||||
def created_at(self) -> datetime.datetime:
|
||||
#example: 2019-11-07T14:13:07.942Z
|
||||
return datetime.datetime.fromisoformat(self._json_dict['created_at'].rstrip('Z'))
|
||||
|
||||
|
||||
class BuildKiteMasterStats:
|
||||
|
||||
def __init__(self, llvm_repo: str, token_path: str):
|
||||
self._llvm_repo = llvm_repo
|
||||
with open(token_path, 'r') as token_file:
|
||||
self._token = token_file.read().strip()
|
||||
|
||||
def get_stats(self, organisation: str, pipeline: str):
|
||||
url = "https://api.buildkite.com/v2/organizations/{}/pipelines/{}/builds".format(organisation, pipeline)
|
||||
return self._get_url(url)
|
||||
|
||||
|
||||
def _get_url(self, url: str):
|
||||
"""Get paginated results from server."""
|
||||
page = 1
|
||||
results = []
|
||||
while True:
|
||||
page_url = url + f"?api_key={self._token}&page={page}&per_page=100"
|
||||
new_results = requests.get(page_url).json()
|
||||
results.extend(new_results)
|
||||
print(len(new_results))
|
||||
if len(new_results) < 100:
|
||||
break
|
||||
page += 1
|
||||
return results
|
||||
|
||||
def save_results(self, output_path:str, data):
|
||||
with open(output_path, 'w') as output_file:
|
||||
json.dump(data, output_file)
|
||||
|
||||
def get_builds(self, json_path: str):
|
||||
with open(json_path) as json_file:
|
||||
builds = json.load(json_file)
|
||||
build_dict = {}
|
||||
for b in builds:
|
||||
build = Build(b)
|
||||
build_dict[build.number] = build
|
||||
return build_dict
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='Process some integers.')
|
||||
parser.add_argument('llvm_path')
|
||||
parser.add_argument('token')
|
||||
args = parser.parse_args()
|
||||
CACHE_FILE = 'tmp/bklogs.json'
|
||||
bk = BuildKiteMasterStats(args.llvm_path, args.token)
|
||||
|
||||
if not os.path.exists(CACHE_FILE):
|
||||
results = bk.get_stats('llvm-project','llvm-main-build')
|
||||
bk.save_results(CACHE_FILE, results)
|
||||
|
||||
builds = bk.get_builds(CACHE_FILE)
|
||||
last_fail = None
|
||||
fail_count = 0
|
||||
start_time = None
|
||||
for build_number in sorted(builds.keys()):
|
||||
if build_number < 50:
|
||||
# skip the first builds as they might not be mature enough
|
||||
continue
|
||||
build = builds[build_number]
|
||||
if build.has_passed:
|
||||
if last_fail is not None:
|
||||
print(f'* ends with [build {build.number}]({build.web_url})')
|
||||
print(f'* ends with commit {build.commit}')
|
||||
print(f'* ends on {build.created_at}')
|
||||
duration = build.created_at - start_time
|
||||
print(f'* duration: {duration} [h:m:s]')
|
||||
print('* cause: # TODO')
|
||||
print()
|
||||
last_fail = None
|
||||
else:
|
||||
if last_fail is None:
|
||||
print(f'# Outage {fail_count}')
|
||||
print()
|
||||
print(f'* starts with [build {build.number}]({build.web_url})')
|
||||
print(f'* starts with commit {build.commit}')
|
||||
print(f'* starts on {build.created_at}')
|
||||
fail_count += 1
|
||||
start_time = build.created_at
|
||||
else:
|
||||
pass
|
||||
last_fail = build.number
|
||||
|
||||
|
||||
|
|
@ -1,7 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
set -eu
|
||||
set -o pipefail
|
||||
|
||||
echo "downloading buildkite builds..."
|
||||
env
|
||||
|
|
@ -1,541 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Get data on Revisions and builds from Phabricator
|
||||
|
||||
import phabricator
|
||||
import json
|
||||
import os
|
||||
import datetime
|
||||
from typing import Dict, List, Optional
|
||||
import csv
|
||||
import time
|
||||
import socket
|
||||
import git
|
||||
import argparse
|
||||
|
||||
# PHIDs of build plans used for pre-merge testing
|
||||
# FIXME: how do you get these?
|
||||
_PRE_MERGE_PHIDs = ['PHID-HMCP-bfkbtacsszhg3feydpo6', # beta testers
|
||||
'PHID-HMCP-qbviyekvgirhhhvkzpsn', # public pre-merge tests
|
||||
'PHID-HMCP-p2oc4ocen3l2yzymvg2l',
|
||||
]
|
||||
|
||||
# query all data after this date
|
||||
START_DATE = datetime.date(year=2019, month=10, day=1)
|
||||
|
||||
class PhabResponse:
|
||||
|
||||
def __init__(self, revision_dict: Dict):
|
||||
self.revision_dict = revision_dict
|
||||
|
||||
@property
|
||||
def id(self) -> str:
|
||||
return self.revision_dict['id']
|
||||
|
||||
@property
|
||||
def phid(self) -> str:
|
||||
return self.revision_dict['phid']
|
||||
|
||||
def __str__(self):
|
||||
return str(self.revision_dict)
|
||||
|
||||
|
||||
class Revision(PhabResponse):
|
||||
|
||||
def __init__(self, revision_dict):
|
||||
super().__init__(revision_dict)
|
||||
self.buildables = [] # type: List['Buildable']
|
||||
self.diffs = [] # type: List['Diff']
|
||||
|
||||
@property
|
||||
def status(self) -> str:
|
||||
return self.revision_dict['fields']['status']['value']
|
||||
|
||||
@property
|
||||
def builds(self) -> List['Build']:
|
||||
builds = []
|
||||
for b in self.buildables:
|
||||
builds.extend(b.builds)
|
||||
return builds
|
||||
|
||||
@property
|
||||
def created_date(self):
|
||||
return self.revision_dict['fields']['dateCreated']
|
||||
|
||||
@property
|
||||
def was_premerge_tested(self) -> bool:
|
||||
return any((b.was_premerge_tested for b in self.builds))
|
||||
|
||||
@property
|
||||
def repository_phid(self) -> str:
|
||||
return self.revision_dict['fields']['repositoryPHID']
|
||||
|
||||
@property
|
||||
def diff_phid(self) -> str:
|
||||
return self.revision_dict['fields']['diffPHID']
|
||||
|
||||
@property
|
||||
def all_diffs_have_refs(self) -> bool:
|
||||
return not any(not d.has_refs for d in self.diffs)
|
||||
|
||||
@property
|
||||
def day(self) -> datetime.date:
|
||||
return datetime.date.fromtimestamp(self.created_date)
|
||||
|
||||
@property
|
||||
def week(self) -> str:
|
||||
day = self.day
|
||||
return'{}-w{:02d}'.format(day.year, day.isocalendar()[1])
|
||||
|
||||
@property
|
||||
def published(self) -> bool:
|
||||
return self.status == 'published'
|
||||
|
||||
@property
|
||||
def build_status_list(self) -> List[bool]:
|
||||
return [b.passed for b in self.builds if b.was_premerge_tested]
|
||||
|
||||
@property
|
||||
def builds_finally_succeeded(self) -> bool:
|
||||
"""Return true iff one of the builds failed and the last build passed."""
|
||||
return self.has_failed_builds and self.build_status_list[-1]
|
||||
|
||||
@property
|
||||
def has_failed_builds(self) -> bool:
|
||||
"""Return true iff one of the builds failed."""
|
||||
return False in self.build_status_list
|
||||
|
||||
@property
|
||||
def published_failing(self) -> bool:
|
||||
"""Return true iff published and the last build failed."""
|
||||
return self.was_premerge_tested and self.published \
|
||||
and not self.build_status_list[-1]
|
||||
|
||||
@property
|
||||
def all_builds_passed(self) -> bool:
|
||||
return self.was_premerge_tested and all(self.build_status_list)
|
||||
|
||||
@property
|
||||
def all_builds_failed(self) -> bool:
|
||||
return self.was_premerge_tested and all(not b for b in self.build_status_list)
|
||||
|
||||
@property
|
||||
def dateModified(self) -> datetime.datetime:
|
||||
return datetime.datetime.fromtimestamp(self.revision_dict['fields']['dateModified'])
|
||||
|
||||
|
||||
class Buildable(PhabResponse):
|
||||
|
||||
def __init__(self, revision_dict):
|
||||
super().__init__(revision_dict)
|
||||
self.builds = [] # type: List[Build]
|
||||
self.revision = None # type: Optional[Revision]
|
||||
|
||||
@property
|
||||
def diff_phid(self) -> str:
|
||||
return self.revision_dict['buildablePHID']
|
||||
|
||||
@property
|
||||
def revison_phid(self) -> str:
|
||||
return self.revision_dict['containerPHID']
|
||||
|
||||
|
||||
class Build(PhabResponse):
|
||||
|
||||
def __init__(self, revision_dict):
|
||||
super().__init__(revision_dict)
|
||||
self.buildable = None # type: Optional[Buildable]
|
||||
|
||||
@property
|
||||
def buildable_phid(self) -> str:
|
||||
return self.revision_dict['fields']['buildablePHID']
|
||||
|
||||
@property
|
||||
def buildplan_phid(self) -> str:
|
||||
return self.revision_dict['fields']['buildPlanPHID']
|
||||
|
||||
@property
|
||||
def was_premerge_tested(self) -> bool:
|
||||
result = self.buildplan_phid in _PRE_MERGE_PHIDs
|
||||
return result
|
||||
|
||||
@property
|
||||
def passed(self) -> bool:
|
||||
"""Returns true, if the build "passed" """
|
||||
return self.was_premerge_tested and self.revision_dict['fields']['buildStatus']['value'] == 'passed'
|
||||
|
||||
@property
|
||||
def dateModified(self) ->datetime.datetime:
|
||||
return datetime.datetime.fromtimestamp(self.revision_dict['fields']['dateModified'])
|
||||
|
||||
class Diff(PhabResponse):
|
||||
|
||||
def __init__(self, revision_dict):
|
||||
super().__init__(revision_dict)
|
||||
self.revision = None # type: Optional[Revision]
|
||||
|
||||
@property
|
||||
def revison_phid(self) -> str:
|
||||
return self.revision_dict['fields']['revisionPHID']
|
||||
|
||||
@property
|
||||
def _refs(self) -> List:
|
||||
return self.revision_dict['fields']['refs']
|
||||
|
||||
@property
|
||||
def has_refs(self) -> bool:
|
||||
return len(self._refs) > 0
|
||||
|
||||
@property
|
||||
def base_revision(self) -> str:
|
||||
for ref in self._refs:
|
||||
if ref['type'] == 'base':
|
||||
return ref['identifier']
|
||||
return None
|
||||
|
||||
@property
|
||||
def base_branch(self) -> str:
|
||||
for ref in self._refs:
|
||||
if ref['type'] == 'branch':
|
||||
return ref['name']
|
||||
return None
|
||||
|
||||
@property
|
||||
def dateCreated(self) -> datetime.datetime:
|
||||
return datetime.datetime.fromtimestamp(self.revision_dict['fields']['dateCreated'])
|
||||
|
||||
class PhabBuildPuller:
|
||||
# files/folder for sotring temporary results
|
||||
_TMP_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), 'tmp'))
|
||||
_REVISION_FILE = os.path.join(_TMP_DIR, 'phab-revisions.json')
|
||||
_BUILDABLE_FILE = os.path.join(_TMP_DIR, 'phab-buildables.json')
|
||||
_BUILD_FILE = os.path.join(_TMP_DIR, 'phab-build.json')
|
||||
_DIFF_FILE = os.path.join(_TMP_DIR, 'phab-diffs.json')
|
||||
_PHAB_WEEKLY_METRICS_FILE = os.path.join(_TMP_DIR, 'phabricator_{}.csv')
|
||||
|
||||
def __init__(self, repo_path: str):
|
||||
self.conduit_token = None
|
||||
self.host = None
|
||||
self.phab = self._create_phab()
|
||||
self._repo_path = repo_path # type: str
|
||||
self.revisions = {} # type: Dict[str, Revision]
|
||||
self.buildables = {} # type: Dict[str, Buildable]
|
||||
self.builds = {} # type: Dict[str, Build]
|
||||
self.diffs = {} # type: Dict[str, Diff]
|
||||
|
||||
def _create_phab(self) -> phabricator.Phabricator:
|
||||
phab = phabricator.Phabricator(token=self.conduit_token, host=self.host)
|
||||
phab.update_interfaces()
|
||||
return phab
|
||||
|
||||
def _load_arcrc(self):
|
||||
"""Load arc configuration from file if not set."""
|
||||
if self.conduit_token is not None or self.host is not None:
|
||||
return
|
||||
print('Loading configuration from ~/.arcrc file')
|
||||
with open(os.path.expanduser('~/.arcrc'), 'r') as arcrc_file:
|
||||
arcrc = json.load(arcrc_file)
|
||||
# use the first host configured in the file
|
||||
self.host = next(iter(arcrc['hosts']))
|
||||
self.conduit_token = arcrc['hosts'][self.host]['token']
|
||||
|
||||
def run(self):
|
||||
if not os.path.exists(self._TMP_DIR):
|
||||
os.mkdir(self._TMP_DIR)
|
||||
if not os.path.isfile(self._REVISION_FILE):
|
||||
self.get_revisions()
|
||||
self.parse_revisions()
|
||||
if not os.path.isfile(self._BUILDABLE_FILE):
|
||||
self.get_buildables()
|
||||
self.parse_buildables()
|
||||
if not os.path.isfile(self._BUILD_FILE):
|
||||
self.get_builds()
|
||||
self.parse_builds()
|
||||
if not os.path.isfile(self._DIFF_FILE):
|
||||
self.get_diffs()
|
||||
self.parse_diffs()
|
||||
self.link_objects()
|
||||
self.compute_metrics('day', lambda r: r.day)
|
||||
self.compute_metrics('week', lambda r: r.week)
|
||||
self.count_base_revisions()
|
||||
self.revision_statistics()
|
||||
self.match_base_revisions_with_repo(self._repo_path)
|
||||
|
||||
def get_revisions(self):
|
||||
print('Downloading revisions starting...')
|
||||
from_date = int(START_DATE.strftime('%s'))
|
||||
data = []
|
||||
constraints = {
|
||||
'createdStart': from_date
|
||||
}
|
||||
# FIXME: lots of code duplication around pagination and error handling.
|
||||
# find a way to separate this into a function.
|
||||
after = None
|
||||
while True:
|
||||
revisions = self.phab.differential.revision.search(
|
||||
constraints=constraints, after=after)
|
||||
data.extend(revisions.response['data'])
|
||||
print('{} revisions...'.format(len(data)))
|
||||
after = revisions.response['cursor']['after']
|
||||
if after is None:
|
||||
break
|
||||
print('Number of revisions:', len(data))
|
||||
with open(self._REVISION_FILE, 'w') as json_file:
|
||||
json.dump(data, json_file)
|
||||
|
||||
def get_buildables(self):
|
||||
print('Downloading buildables...')
|
||||
data = []
|
||||
after = None
|
||||
while True:
|
||||
revisions = self.phab.harbormaster.querybuildables(
|
||||
containerPHIDs=[r.phid for r in self.revisions.values()], after=after)
|
||||
data.extend(revisions.response['data'])
|
||||
print('{} buildables...'.format(len(data)))
|
||||
after = revisions.response['cursor']['after']
|
||||
if after is None:
|
||||
break
|
||||
print('Number of buildables:', len(data))
|
||||
with open(self._BUILDABLE_FILE, 'w') as json_file:
|
||||
json.dump(data, json_file)
|
||||
|
||||
def get_builds(self):
|
||||
print('Downloading builds...')
|
||||
data = []
|
||||
constraints = {
|
||||
'buildables': [r.phid for r in self.buildables.values()]
|
||||
}
|
||||
after = None
|
||||
while True:
|
||||
# retry on timeouts
|
||||
fail_count = 0
|
||||
while True:
|
||||
try:
|
||||
revisions = self.phab.harbormaster.build.search(
|
||||
constraints=constraints, after=after)
|
||||
except socket.timeout:
|
||||
fail_count +=1
|
||||
if fail_count > 5:
|
||||
raise
|
||||
time.sleep(10)
|
||||
continue
|
||||
break
|
||||
data.extend(revisions.response['data'])
|
||||
print('{} builds...'.format(len(data)))
|
||||
after = revisions.response['cursor']['after']
|
||||
if after is None:
|
||||
break
|
||||
print('Number of buildables:', len(data))
|
||||
with open(self._BUILD_FILE, 'w') as json_file:
|
||||
json.dump(data, json_file)
|
||||
|
||||
def get_diffs(self):
|
||||
print('Downloading diffs...')
|
||||
data = []
|
||||
constraints = {
|
||||
'revisionPHIDs': [r.phid for r in self.revisions.values()]
|
||||
}
|
||||
after = None
|
||||
while True:
|
||||
# retry on timeouts
|
||||
fail_count = 0
|
||||
while True:
|
||||
try:
|
||||
diffs = self.phab.differential.diff.search(
|
||||
constraints=constraints, after=after)
|
||||
except socket.timeout:
|
||||
fail_count +=1
|
||||
if fail_count > 5:
|
||||
raise
|
||||
time.sleep(10)
|
||||
continue
|
||||
break
|
||||
data.extend(diffs.response['data'])
|
||||
print('{} diffs...'.format(len(data)))
|
||||
after = diffs.response['cursor']['after']
|
||||
if after is None:
|
||||
break
|
||||
print('Number of diffs:', len(data))
|
||||
with open(self._DIFF_FILE, 'w') as json_file:
|
||||
json.dump(data, json_file)
|
||||
|
||||
def parse_revisions(self):
|
||||
with open(self._REVISION_FILE) as revision_file:
|
||||
revision_dict = json.load(revision_file)
|
||||
self.revisions = {r.phid: r for r in (Revision(x) for x in revision_dict)}
|
||||
print('Parsed {} revisions.'.format(len(self.revisions)))
|
||||
|
||||
def parse_buildables(self):
|
||||
with open(self._BUILDABLE_FILE) as buildables_file:
|
||||
buildable_dict = json.load(buildables_file)
|
||||
self.buildables = {b.phid: b for b in (Buildable(x) for x in buildable_dict)}
|
||||
print('Parsed {} buildables.'.format(len(self.buildables)))
|
||||
|
||||
def parse_builds(self):
|
||||
with open(self._BUILD_FILE) as build_file:
|
||||
build_dict = json.load(build_file)
|
||||
self.builds = {b.phid: b for b in (Build(x) for x in build_dict)}
|
||||
print('Parsed {} builds.'.format(len(self.builds)))
|
||||
|
||||
def parse_diffs(self):
|
||||
with open(self._DIFF_FILE) as diff_file:
|
||||
diff_dict = json.load(diff_file)
|
||||
self.diffs = {d.phid: d for d in (Diff(x) for x in diff_dict)}
|
||||
print('Parsed {} diffs.'.format(len(self.diffs)))
|
||||
|
||||
def link_objects(self):
|
||||
for build in (b for b in self.builds.values()):
|
||||
buildable = self.buildables[build.buildable_phid]
|
||||
build.buildable = buildable
|
||||
buildable.builds.append(build)
|
||||
|
||||
for buildable in self.buildables.values():
|
||||
revision = self.revisions[buildable.revison_phid]
|
||||
revision.buildables.append(buildable)
|
||||
buildable.revision = revision
|
||||
|
||||
for diff in self.diffs.values():
|
||||
revision = self.revisions[diff.revison_phid]
|
||||
revision.diffs.append(diff)
|
||||
diff.revision = revision
|
||||
|
||||
def compute_metrics(self, name: str, group_function):
|
||||
print('Creating metrics for {}...'.format(name))
|
||||
group_dict = {}
|
||||
for revision in self.revisions.values():
|
||||
group_dict.setdefault(group_function(revision), []).append(revision)
|
||||
|
||||
csv_file = open(self._PHAB_WEEKLY_METRICS_FILE.format(name), 'w')
|
||||
fieldnames = [name, '# revisions', '# tested revisions', '% tested revisions', '# untested revisions',
|
||||
'# revisions without builds', '% revisions without builds', '# no repository set',
|
||||
'# had failed builds', '% had failed builds', '# failed first then passed',
|
||||
'% failed first then passed', '# published failing', '% published failing',
|
||||
'# all passed', '% all passed']
|
||||
|
||||
writer = csv.DictWriter(csv_file, fieldnames=fieldnames, dialect=csv.excel)
|
||||
writer.writeheader()
|
||||
for group in sorted(group_dict.keys()):
|
||||
revisions = group_dict[group] # type: List[Revision]
|
||||
num_revisions = len(revisions)
|
||||
num_premt_revisions = len([r for r in revisions if r.was_premerge_tested])
|
||||
precentage_premt_revisions = 100.0 * num_premt_revisions / num_revisions
|
||||
num_no_build_triggered = len([r for r in revisions if len(r.builds) == 0])
|
||||
percent_no_build_triggered = 100.0 * num_no_build_triggered / num_revisions
|
||||
num_no_repo = len([r for r in revisions if r.repository_phid is None])
|
||||
num_had_failed_builds =len([r for r in revisions if r.has_failed_builds])
|
||||
num_failed_first_then_passed = len([r for r in revisions if r.builds_finally_succeeded])
|
||||
num_published_failing = len([r for r in revisions if r.published_failing])
|
||||
num_all_passed = len([r for r in revisions if r.all_builds_passed])
|
||||
writer.writerow({
|
||||
name: group,
|
||||
'# revisions': num_revisions,
|
||||
'# tested revisions': num_premt_revisions,
|
||||
'% tested revisions': precentage_premt_revisions,
|
||||
'# untested revisions': num_revisions - num_premt_revisions,
|
||||
'# revisions without builds': num_no_build_triggered,
|
||||
'% revisions without builds': percent_no_build_triggered,
|
||||
'# no repository set': num_no_repo,
|
||||
'# had failed builds': num_had_failed_builds,
|
||||
'% had failed builds': 100 * num_had_failed_builds / num_revisions,
|
||||
'# failed first then passed': num_failed_first_then_passed,
|
||||
'% failed first then passed': 100 * num_failed_first_then_passed / num_revisions,
|
||||
'# published failing': num_published_failing,
|
||||
'% published failing': 100 * num_published_failing / num_revisions,
|
||||
'# all passed': num_all_passed,
|
||||
'% all passed': 100*num_all_passed / num_revisions,
|
||||
})
|
||||
|
||||
def count_base_revisions(self):
|
||||
base_revisions = {}
|
||||
base_branches = {}
|
||||
for diff in self.diffs.values():
|
||||
base_revisions.setdefault(diff.base_revision, 0)
|
||||
base_revisions[diff.base_revision] += 1
|
||||
|
||||
base_branches.setdefault(diff.base_branch, 0)
|
||||
base_branches[diff.base_branch] +=1
|
||||
|
||||
print(f'{len(self.diffs)} diffs are using {len(base_revisions)} different git base revisions.')
|
||||
print('The top 10 revisions and their usages are:')
|
||||
revisions = sorted( base_revisions.items(), key=lambda x: x[1] , reverse=True)
|
||||
for i in revisions[:10]:
|
||||
print(f' commit {i[0]} was used {i[1]} times')
|
||||
print()
|
||||
print(f'{len(self.diffs)} diffs are using {len(base_branches)} different git base branches')
|
||||
branches = sorted( base_branches.items(), key=lambda x: x[1] , reverse=True)
|
||||
print('The top 10 branches and their usages are:')
|
||||
for i in branches[:10]:
|
||||
print(f' branch {i[0]} was used {i[1]} times')
|
||||
print()
|
||||
|
||||
def match_base_revisions_with_repo(self, repo_path: str):
|
||||
repo = git.Repo(repo_path)
|
||||
not_found = 0
|
||||
invalid_date = 0
|
||||
has_base_revision = 0
|
||||
for diff in self.diffs.values():
|
||||
revision = diff.base_revision
|
||||
if revision is None:
|
||||
continue
|
||||
has_base_revision += 1
|
||||
try:
|
||||
commit = repo.commit(revision)
|
||||
except (ValueError, git.BadName):
|
||||
not_found += 1
|
||||
continue
|
||||
commited_date = datetime.datetime.fromtimestamp(commit.committed_date)
|
||||
if commited_date > diff.dateCreated:
|
||||
invalid_date += 1
|
||||
|
||||
print()
|
||||
print(f'Of the {has_base_revision} Diffs with base revision, the base revision was NOT found in the repo for {not_found} and ')
|
||||
print(f'{invalid_date} base revisions were used before being available upstream.')
|
||||
print(f'So {(not_found+invalid_date)/has_base_revision*100:0.2f} % of specified the base revisions were unusable.')
|
||||
|
||||
def revision_statistics(self):
|
||||
no_builds = 0
|
||||
has_failed = 0
|
||||
fail_then_pass = 0
|
||||
all_passed = 0
|
||||
fail_last = 0
|
||||
for revision in self.revisions.values():
|
||||
build_status = [b.passed for b in revision.builds]
|
||||
if len(revision.builds) == 0:
|
||||
no_builds += 1
|
||||
continue
|
||||
if False in build_status:
|
||||
has_failed += 1
|
||||
if build_status[-1] == True:
|
||||
fail_then_pass +=1
|
||||
else:
|
||||
all_passed += 1
|
||||
if revision.published and build_status[-1] == False:
|
||||
fail_last += 1
|
||||
print()
|
||||
print(f'Out of the {len(self.revisions)} Revisions:')
|
||||
print(f' {no_builds} had no builds.')
|
||||
print(f' {has_failed} had failed builds.')
|
||||
print(f' {fail_then_pass} had failed builds, but the last build passed.')
|
||||
print(f' {all_passed} had only successful builds.')
|
||||
print(f' {fail_last} were published with a failing build.')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('repo_path')
|
||||
args = parser.parse_args()
|
||||
puller = PhabBuildPuller(args.repo_path)
|
||||
puller.run()
|
||||
|
|
@ -1,116 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import datetime
|
||||
from time import timezone
|
||||
import git
|
||||
from typing import Dict, Optional
|
||||
from google.cloud import monitoring_v3
|
||||
import re
|
||||
|
||||
from datetime import tzinfo
|
||||
|
||||
GCP_PROJECT_ID = 'llvm-premerge-checks'
|
||||
|
||||
|
||||
class RepoStats:
|
||||
|
||||
def __init__(self):
|
||||
self.commits = 0 # type: int
|
||||
self.reverts = 0 # type: int
|
||||
self.reviewed = 0 # type: int
|
||||
|
||||
@property
|
||||
def percent_reverted(self) -> Optional[float]:
|
||||
try:
|
||||
return 100.0 * self.reverts / self.commits
|
||||
except ZeroDivisionError:
|
||||
return None
|
||||
|
||||
@property
|
||||
def percent_reviewed(self) -> Optional[float]:
|
||||
try:
|
||||
return 100.0 * self.reviewed / (self.commits - self.reverts)
|
||||
except ZeroDivisionError:
|
||||
return None
|
||||
|
||||
def __str__(self):
|
||||
results = [
|
||||
"commits: {}".format(self.commits),
|
||||
"reverts: {}".format(self.reverts),
|
||||
"reviewed: {}".format(self.reviewed),
|
||||
]
|
||||
try:
|
||||
results.append("percent reverted: {:0.1f}".format(self.percent_reverted))
|
||||
except TypeError:
|
||||
pass
|
||||
try:
|
||||
results.append("percent reverted: {:0.1f}".format(self.percent_reverted))
|
||||
except TypeError:
|
||||
pass
|
||||
|
||||
return "\n".join(results)
|
||||
|
||||
|
||||
def get_reverts_per_day(repo_path: str, max_age: datetime.datetime) -> RepoStats:
|
||||
stats = RepoStats()
|
||||
repo = git.Repo(repo_path)
|
||||
repo.git.fetch()
|
||||
diff_regex = re.compile(r'^Differential Revision: https:\/\/reviews\.llvm\.org\/(.*)$', re.MULTILINE)
|
||||
|
||||
for commit in repo.iter_commits('main'):
|
||||
if commit.committed_datetime < max_age:
|
||||
break
|
||||
stats.commits += 1
|
||||
if commit.message.startswith('Revert'):
|
||||
stats.reverts += 1
|
||||
if diff_regex.search(commit.message) is not None:
|
||||
stats.reviewed += 1
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def gcp_write_data(project_id: str, stats: RepoStats, now:datetime.datetime):
|
||||
"""Upload metrics to Stackdriver."""
|
||||
client = monitoring_v3.MetricServiceClient()
|
||||
project_name = client.project_path(project_id)
|
||||
|
||||
for desc_type, value in [
|
||||
["reverts", stats.reverts],
|
||||
["commits", stats.commits],
|
||||
["percent_reverted", stats.percent_reverted],
|
||||
["reviewed", stats.reviewed],
|
||||
["percent_reviewed", stats.percent_reviewed],
|
||||
]:
|
||||
if value is None:
|
||||
continue
|
||||
|
||||
series = monitoring_v3.types.TimeSeries()
|
||||
series.metric.type = 'custom.googleapis.com/repository_{}'.format(desc_type)
|
||||
series.resource.type = 'global'
|
||||
point = series.points.add()
|
||||
point.value.double_value = value
|
||||
point.interval.end_time.seconds = int(now.timestamp())
|
||||
client.create_time_series(project_name, [series])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
now = datetime.datetime.now(tz=datetime.timezone.utc)
|
||||
max_age = now - datetime.timedelta(days=1)
|
||||
# TODO: make path configurable
|
||||
stats = get_reverts_per_day('~/git/llvm-project', max_age)
|
||||
print(stats)
|
||||
# TODO: add `dryrun` parameter
|
||||
gcp_write_data(GCP_PROJECT_ID, stats, now)
|
||||
|
|
@ -1,463 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# generate statistics on the llvm github repository
|
||||
|
||||
import csv
|
||||
import datetime
|
||||
import git
|
||||
import re
|
||||
import os
|
||||
from typing import Dict, Optional, List, Set
|
||||
import random
|
||||
import string
|
||||
|
||||
REVISION_REGEX = re.compile(
|
||||
r"^Differential Revision: https://reviews\.llvm\.org/(.*)$", re.MULTILINE
|
||||
)
|
||||
REVERT_REGEX = re.compile(r'^Revert "(.+)"')
|
||||
REVERT_HASH_REGEX = re.compile("This reverts commit (\w+)", re.MULTILINE)
|
||||
|
||||
|
||||
class MyCommit:
|
||||
|
||||
SALT = "".join(
|
||||
random.choices(
|
||||
string.ascii_lowercase + string.ascii_uppercase + string.digits, k=16
|
||||
)
|
||||
)
|
||||
|
||||
def __init__(self, commit: git.Commit):
|
||||
self.commit = commit
|
||||
self.chash = commit.hexsha # type: str
|
||||
self.author = hash(commit.author.email + MyCommit.SALT) # type: int
|
||||
self.author_domain = commit.author.email.rsplit("@")[-1].lower() # type: str
|
||||
self.commiter = hash(commit.committer.email.lower() + MyCommit.SALT) # type:int
|
||||
self.summary = commit.summary # type: str
|
||||
self.date = datetime.datetime.fromtimestamp(
|
||||
commit.committed_date
|
||||
) # type: datetime.datetime
|
||||
self.phab_revision = self._get_revision(commit) # type: Optional[str]
|
||||
self.reverts = None # type: Optional[MyCommit]
|
||||
self.reverted_by = None # type: Optional[MyCommit]
|
||||
self._diff_index = None # type: Optional[git.DiffIndex]
|
||||
|
||||
@staticmethod
|
||||
def _get_revision(commit: git.Commit) -> Optional[str]:
|
||||
m = REVISION_REGEX.search(commit.message)
|
||||
if m is None:
|
||||
return None
|
||||
return m.group(1)
|
||||
|
||||
@property
|
||||
def day(self) -> datetime.date:
|
||||
return self.date.date()
|
||||
|
||||
def reverts_summary(self) -> Optional[str]:
|
||||
m = REVERT_REGEX.search(self.summary)
|
||||
if m is None:
|
||||
return None
|
||||
return m.group(1)
|
||||
|
||||
def __str__(self):
|
||||
return self.chash
|
||||
|
||||
@property
|
||||
def was_reverted(self) -> bool:
|
||||
return self.reverted_by is not None
|
||||
|
||||
@property
|
||||
def was_reviewed(self) -> bool:
|
||||
return self.phab_revision is not None
|
||||
|
||||
@property
|
||||
def is_revert(self) -> bool:
|
||||
return self.reverts is not None
|
||||
|
||||
@property
|
||||
def week(self) -> str:
|
||||
return "{}-w{:02d}".format(self.date.year, self.date.isocalendar()[1])
|
||||
|
||||
@property
|
||||
def diff_index(self) -> git.DiffIndex:
|
||||
# expensive operation, cache the results
|
||||
if self._diff_index is None:
|
||||
self._diff_index = self.commit.diff(
|
||||
self.commit.parents[0], create_patch=True
|
||||
)
|
||||
return self._diff_index
|
||||
|
||||
@property
|
||||
def num_loc(self) -> int:
|
||||
nloc = 0
|
||||
for diff in self.diff_index:
|
||||
nloc += str(diff.diff, encoding="utf8").count("\n")
|
||||
return nloc
|
||||
|
||||
@property
|
||||
def modified_paths(self) -> Set[str]:
|
||||
result = set(d.b_path for d in self.diff_index if d.b_path is not None)
|
||||
result.update(d.a_path for d in self.diff_index if d.a_path is not None)
|
||||
return result
|
||||
|
||||
@property
|
||||
def modified_projects(self) -> Set[str]:
|
||||
return set(p.split("/")[0] for p in self.modified_paths)
|
||||
|
||||
@property
|
||||
def reverts_commit_hash(self) -> Optional[str]:
|
||||
m = REVERT_HASH_REGEX.search(self.commit.message)
|
||||
if m is None:
|
||||
if self.reverts_summary() is None:
|
||||
return None
|
||||
# there was a revert, but we do not know the commit hash
|
||||
return "unknown"
|
||||
return m.group(1)
|
||||
|
||||
|
||||
class RepoStats:
|
||||
def __init__(self, git_dir: str):
|
||||
self.repo = git.Repo(git_dir)
|
||||
self.commit_by_hash = dict() # type: Dict[str, MyCommit]
|
||||
self.commit_by_summary = dict() # type: Dict[str, List[MyCommit]]
|
||||
self.commit_by_week = dict() # type: Dict[str, List[MyCommit]]
|
||||
self.commit_by_author = dict() # type: Dict[int, List[MyCommit]]
|
||||
self.commit_by_author_domain = dict() # type: Dict[str, List[MyCommit]]
|
||||
|
||||
def parse_repo(self, maxage: datetime.datetime):
|
||||
for commit in self.repo.iter_commits("main"):
|
||||
if commit.committed_datetime < maxage:
|
||||
break
|
||||
mycommit = MyCommit(commit)
|
||||
self.commit_by_hash[mycommit.chash] = mycommit
|
||||
self.commit_by_summary.setdefault(mycommit.summary, []).append(mycommit)
|
||||
self.commit_by_week.setdefault(mycommit.week, []).append(mycommit)
|
||||
self.commit_by_author.setdefault(mycommit.author, []).append(mycommit)
|
||||
self.commit_by_author_domain.setdefault(mycommit.author_domain, []).append(
|
||||
mycommit
|
||||
)
|
||||
print("Read {} commits".format(len(self.commit_by_hash)))
|
||||
|
||||
def find_reverts(self):
|
||||
reverts = 0
|
||||
for commit in self.commit_by_hash.values():
|
||||
summary = commit.reverts_summary()
|
||||
if summary is None:
|
||||
continue
|
||||
if summary not in self.commit_by_summary:
|
||||
print("summary not found: {}".format(summary))
|
||||
continue
|
||||
reverting_commit = self.commit_by_summary[summary][-1]
|
||||
commit.reverted_by = reverting_commit
|
||||
reverting_commit.reverts = commit
|
||||
reverts += 1
|
||||
print("Found {} reverts".format(reverts))
|
||||
|
||||
# https://stackoverflow.com/questions/2600775/how-to-get-week-number-in-python
|
||||
def dump_daily_stats(self):
|
||||
fieldnames = [
|
||||
"week",
|
||||
"num_commits",
|
||||
"num_reverts",
|
||||
"percentage_reverts",
|
||||
"num_reviewed",
|
||||
"percentage_reviewed",
|
||||
"# reviewed & revert",
|
||||
"# !reviewed & !revert",
|
||||
"# !reviewed & revert",
|
||||
"# reviewed & !revert",
|
||||
]
|
||||
csvfile = open("tmp/llvm-project-weekly.csv", "w")
|
||||
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
|
||||
writer.writeheader()
|
||||
for week in sorted(self.commit_by_week.keys()):
|
||||
commits = self.commit_by_week[week]
|
||||
num_commits = len(commits)
|
||||
num_reverts = len([c for c in commits if c.is_revert])
|
||||
percentage_reverts = 100.0 * num_reverts / num_commits
|
||||
num_reviewed = len([c for c in commits if c.was_reviewed])
|
||||
percentage_reviewed = 100 * num_reviewed / (num_commits - num_reverts)
|
||||
num_reviewed_revert = len(
|
||||
[c for c in commits if c.was_reviewed and c.is_revert]
|
||||
)
|
||||
num_reviewed_nrevert = len(
|
||||
[c for c in commits if c.was_reviewed and not c.is_revert]
|
||||
)
|
||||
num_nreviewed_nrevert = len(
|
||||
[c for c in commits if not c.was_reviewed and not c.is_revert]
|
||||
)
|
||||
num_nreviewed_revert = len(
|
||||
[c for c in commits if not c.was_reviewed and c.is_revert]
|
||||
)
|
||||
writer.writerow(
|
||||
{
|
||||
"week": week,
|
||||
"num_commits": num_commits,
|
||||
"num_reverts": num_reverts,
|
||||
"percentage_reverts": percentage_reverts,
|
||||
"num_reviewed": num_reviewed,
|
||||
"percentage_reviewed": percentage_reviewed,
|
||||
"# reviewed & revert": num_reviewed_revert,
|
||||
"# !reviewed & !revert": num_nreviewed_nrevert,
|
||||
"# !reviewed & revert": num_nreviewed_revert,
|
||||
"# reviewed & !revert": num_reviewed_nrevert,
|
||||
}
|
||||
)
|
||||
|
||||
def dump_overall_stats(self):
|
||||
num_commits = len(self.commit_by_hash)
|
||||
num_reverts = len([c for c in self.commit_by_hash.values() if c.is_revert])
|
||||
print("Number of commits: {}".format(num_commits))
|
||||
print("Number of reverts: {}".format(num_reverts))
|
||||
print("percentage of reverts: {:0.2f}".format(100 * num_reverts / num_commits))
|
||||
|
||||
num_reviewed = len([c for c in self.commit_by_hash.values() if c.was_reviewed])
|
||||
print("Number of reviewed commits: {}".format(num_reviewed))
|
||||
print(
|
||||
"percentage of reviewed commits: {:0.2f}".format(
|
||||
100 * num_reviewed / num_commits
|
||||
)
|
||||
)
|
||||
|
||||
num_reviewed_reverted = len(
|
||||
[
|
||||
c
|
||||
for c in self.commit_by_hash.values()
|
||||
if c.was_reviewed and c.was_reverted
|
||||
]
|
||||
)
|
||||
num_not_reviewed_reverted = len(
|
||||
[
|
||||
c
|
||||
for c in self.commit_by_hash.values()
|
||||
if not c.was_reviewed and c.was_reverted
|
||||
]
|
||||
)
|
||||
print("Number of reviewed that were reverted: {}".format(num_reviewed_reverted))
|
||||
print(
|
||||
"Number of NOT reviewed that were reverted: {}".format(
|
||||
num_not_reviewed_reverted
|
||||
)
|
||||
)
|
||||
print(
|
||||
"percentage of reviewed that were reverted: {:0.2f}".format(
|
||||
100 * num_reviewed_reverted / num_reviewed
|
||||
)
|
||||
)
|
||||
print(
|
||||
"percentage of NOT reviewed that were reverted: {:0.2f}".format(
|
||||
100 * num_not_reviewed_reverted / (num_commits - num_reviewed)
|
||||
)
|
||||
)
|
||||
|
||||
num_foreign_committer = len(
|
||||
[c for c in self.commit_by_hash.values() if c.author != c.commiter]
|
||||
)
|
||||
print(
|
||||
"Number of commits where author != committer: {}".format(
|
||||
num_foreign_committer
|
||||
)
|
||||
)
|
||||
print(
|
||||
"Percentage of commits where author != committer: {:0.2f}".format(
|
||||
100 * num_foreign_committer / num_commits
|
||||
)
|
||||
)
|
||||
|
||||
def dump_author_stats(self):
|
||||
print("Number of authors: {}".format(len(self.commit_by_author)))
|
||||
fieldnames = [
|
||||
"author",
|
||||
"num_commits",
|
||||
"num_reverts",
|
||||
"percentage_reverts",
|
||||
"num_reviewed",
|
||||
"percentage_reviewed",
|
||||
]
|
||||
csvfile = open("tmp/llvm-project-authors.csv", "w")
|
||||
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
|
||||
writer.writeheader()
|
||||
for author, commits in self.commit_by_author.items():
|
||||
num_commits = len(commits)
|
||||
num_reverts = len([c for c in commits if c.was_reverted])
|
||||
percentage_reverts = 100 * num_reverts / num_commits
|
||||
num_reviewed = len([c for c in commits if c.was_reviewed])
|
||||
percentage_reviewed = 100 * num_reviewed / num_commits
|
||||
writer.writerow(
|
||||
{
|
||||
"author": author,
|
||||
"num_commits": num_commits,
|
||||
"num_reverts": num_reverts,
|
||||
"percentage_reverts": percentage_reverts,
|
||||
"num_reviewed": num_reviewed,
|
||||
"percentage_reviewed": percentage_reviewed,
|
||||
}
|
||||
)
|
||||
|
||||
def dump_author_domain_stats(self):
|
||||
print("Number of authors: {}".format(len(self.commit_by_author)))
|
||||
fieldnames = ["author_domain", "num_commits", "num_committers"]
|
||||
csvfile = open("tmp/llvm-project-author_domains.csv", "w")
|
||||
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
|
||||
writer.writeheader()
|
||||
for author_domain, commits in self.commit_by_author_domain.items():
|
||||
num_commits = len(commits)
|
||||
committers = set(c.author for c in commits)
|
||||
writer.writerow(
|
||||
{
|
||||
"author_domain": author_domain,
|
||||
"num_commits": num_commits,
|
||||
"num_committers": len(committers),
|
||||
}
|
||||
)
|
||||
|
||||
def dump_unreviewed_paths(self, maxage: datetime.datetime):
|
||||
# TODO: this is really slow. Maybe parallelize?
|
||||
path_count = {
|
||||
True: {},
|
||||
False: {},
|
||||
} # type: Dict[bool, Dict[str, int]]
|
||||
for commit in self.repo.iter_commits("main"):
|
||||
if commit.committed_datetime < maxage:
|
||||
break
|
||||
mycommit = MyCommit(commit)
|
||||
for prefix in set(p.split("/")[0] for p in mycommit.modified_paths):
|
||||
path_count[mycommit.was_reviewed].setdefault(prefix, 0)
|
||||
path_count[mycommit.was_reviewed][prefix] += 1
|
||||
fieldnames = ["was_reviewed"]
|
||||
all_paths = set(path_count[True].keys())
|
||||
all_paths.update(path_count[False].keys())
|
||||
fieldnames.extend(sorted(all_paths))
|
||||
csvfile = open("tmp/llvm-project-unreviewed-paths.csv", "w")
|
||||
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
|
||||
writer.writeheader()
|
||||
for reviewed in [True, False]:
|
||||
row = {"was_reviewed": reviewed}
|
||||
for path, count in path_count[reviewed].items():
|
||||
row[path] = count
|
||||
writer.writerow(row)
|
||||
csvfile.close()
|
||||
|
||||
def dump_loc_commits(self, maxage: datetime.datetime):
|
||||
# TODO: this is really slow. Maybe parallelize?
|
||||
buckets = list(range(0, 2001, 100))
|
||||
review_dict = {
|
||||
True: {b: 0 for b in buckets},
|
||||
False: {b: 0 for b in buckets},
|
||||
} # type: Dict[bool, Dict[int, int]]
|
||||
reverted_dict = {
|
||||
True: {b: 0 for b in buckets},
|
||||
False: {b: 0 for b in buckets},
|
||||
} # type: Dict[bool, Dict[int, int]]
|
||||
for commit in self.repo.iter_commits("main"):
|
||||
if commit.committed_datetime < maxage:
|
||||
break
|
||||
mycommit = self.commit_by_hash[commit.hexsha]
|
||||
review_dict[mycommit.was_reviewed][
|
||||
self._find_bucket(mycommit.num_loc, buckets)
|
||||
] += 1
|
||||
reverted_dict[mycommit.was_reverted][
|
||||
self._find_bucket(mycommit.num_loc, buckets)
|
||||
] += 1
|
||||
fieldnames = ["was_reviewed"]
|
||||
for i in range(0, len(buckets) - 1):
|
||||
fieldnames.append("{}-{}".format(buckets[i], buckets[i + 1] - 1))
|
||||
fieldnames.append(">={}".format(buckets[-1]))
|
||||
csvfile = open("tmp/llvm-project-unreviewed-loc.csv", "w")
|
||||
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
|
||||
writer.writeheader()
|
||||
for reviewed in [True, False]:
|
||||
row = {"was_reviewed": reviewed}
|
||||
for i in range(0, len(buckets)):
|
||||
row[fieldnames[i + 1]] = review_dict[reviewed][buckets[i]]
|
||||
writer.writerow(row)
|
||||
|
||||
writer.writerow({"was_reviewed": "reverted"})
|
||||
for reverted in [True, False]:
|
||||
row = {"was_reviewed": reverted}
|
||||
for i in range(0, len(buckets)):
|
||||
row[fieldnames[i + 1]] = reverted_dict[reverted][buckets[i]]
|
||||
writer.writerow(row)
|
||||
csvfile.close()
|
||||
|
||||
@staticmethod
|
||||
def _find_bucket(number: int, buckets: List[int]) -> int:
|
||||
for bucket in buckets:
|
||||
if number < bucket:
|
||||
return bucket
|
||||
return buckets[-1]
|
||||
|
||||
def export_commits(self):
|
||||
|
||||
print("starting export...")
|
||||
csvfile = open("tmp/llvm-project-export.csv", "w")
|
||||
fieldnames = [
|
||||
"timestamp",
|
||||
"hash",
|
||||
"reviewed",
|
||||
"was_reverted",
|
||||
"is_revert",
|
||||
"# LOC changed",
|
||||
"modified projects",
|
||||
"author domain",
|
||||
"revision",
|
||||
]
|
||||
writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect=csv.excel)
|
||||
writer.writeheader()
|
||||
# did not work with multiprocessing.map, gave recursion error from gitpython...
|
||||
# so using normal map function
|
||||
for row in map(_create_row, self.commit_by_hash.values()):
|
||||
writer.writerow(row)
|
||||
csvfile.close()
|
||||
|
||||
|
||||
def _create_row(mycommit: MyCommit) -> Dict:
|
||||
try:
|
||||
return {
|
||||
"timestamp": mycommit.date.isoformat(),
|
||||
"hash": mycommit.chash,
|
||||
"reviewed": mycommit.was_reviewed,
|
||||
"was_reverted": mycommit.was_reverted,
|
||||
"is_revert": mycommit.is_revert,
|
||||
"# LOC changed": mycommit.num_loc,
|
||||
"modified projects": (";".join(mycommit.modified_projects)),
|
||||
"author domain": mycommit.author_domain,
|
||||
"revision": mycommit.phab_revision
|
||||
if mycommit.phab_revision is not None
|
||||
else "",
|
||||
}
|
||||
except Exception as e:
|
||||
print(e)
|
||||
return {}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
max_age = datetime.datetime(
|
||||
year=2019, month=10, day=1, tzinfo=datetime.timezone.utc
|
||||
)
|
||||
now = datetime.datetime.now(tz=datetime.timezone.utc)
|
||||
rs = RepoStats(os.path.expanduser("~/git/llvm-project"))
|
||||
# TODO: make the path configurable, and `git clone/pull`
|
||||
rs.parse_repo(max_age)
|
||||
rs.find_reverts()
|
||||
rs.dump_daily_stats()
|
||||
rs.dump_overall_stats()
|
||||
rs.dump_author_stats()
|
||||
rs.dump_author_domain_stats()
|
||||
# disabled as it's quite slow
|
||||
# rs.dump_unreviewed_paths(now - datetime.timedelta(days=100))
|
||||
# rs.dump_loc_commits(now - datetime.timedelta(days=100))
|
||||
rs.export_commits()
|
||||
print("Done.")
|
||||
|
|
@ -1,174 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
import os
|
||||
import psycopg2
|
||||
import git
|
||||
from repo_hist import MyCommit
|
||||
import datetime
|
||||
import csv
|
||||
from typing import Set
|
||||
|
||||
# TODO: make his path configurable for use on the server
|
||||
REPO_DIR = "tmp/llvm-project"
|
||||
GIT_URL = "https://github.com/llvm/llvm-project.git"
|
||||
GIT_BRANCH = "main"
|
||||
OUTPUT_PATH = "tmp"
|
||||
|
||||
# this was the start of using git as primary repo
|
||||
MAX_AGE = datetime.datetime(year=2019, month=10, day=1, tzinfo=datetime.timezone.utc)
|
||||
|
||||
|
||||
def connect_to_db() -> psycopg2.extensions.connection:
|
||||
"""Connect to the database, return connection object."""
|
||||
conn = psycopg2.connect(
|
||||
"host=127.0.0.1 sslmode=disable dbname=stats user={} password={}".format(
|
||||
os.environ["PGUSER"], os.environ["PGPASSWORD"]
|
||||
)
|
||||
)
|
||||
return conn
|
||||
|
||||
|
||||
def create_tables(conn: psycopg2.extensions.connection):
|
||||
"""Create database tables if needed."""
|
||||
# TODO: add more attributes as needed
|
||||
# TODO: add all projects as columns
|
||||
print("Creating tables as needed...")
|
||||
cur = conn.cursor()
|
||||
# mod_<project> column: files in the subfolder (=project) <project> were
|
||||
# modified by this commit.
|
||||
# git hashes are 40 characters long, so using char(40) data type here
|
||||
cur.execute(
|
||||
""" CREATE TABLE IF NOT EXISTS git_commits (
|
||||
hash char(40) PRIMARY KEY,
|
||||
commit_time timestamp,
|
||||
phab_id text,
|
||||
reverts_hash char(40),
|
||||
mod_llvm boolean,
|
||||
mod_clang boolean,
|
||||
mod_libcxx boolean,
|
||||
mod_mlir boolean
|
||||
); """
|
||||
)
|
||||
|
||||
conn.commit()
|
||||
|
||||
|
||||
def get_existing_hashes(conn: psycopg2.extensions.connection) -> Set[str]:
|
||||
"""Fetch all stored git hashes from the database."""
|
||||
print("Fetching known git hashes from the database...")
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT hash from git_commits;")
|
||||
return set((row[0] for row in cur.fetchall()))
|
||||
|
||||
|
||||
def update_repo(repo_dir: str) -> git.Repo:
|
||||
"""Clone or fetch local copy of the git repository."""
|
||||
if os.path.isdir(repo_dir):
|
||||
print("Fetching git repo...")
|
||||
repo = git.Repo(repo_dir)
|
||||
repo.remotes.origin.fetch(GIT_BRANCH)
|
||||
else:
|
||||
print("Cloning git repo...")
|
||||
git.Repo.clone_from(GIT_URL, repo_dir, bare=True)
|
||||
repo = git.Repo(repo_dir)
|
||||
print("repo update done.")
|
||||
return repo
|
||||
|
||||
|
||||
def parse_commits(
|
||||
conn: psycopg2.extensions.connection, repo: git.Repo, max_age: datetime.datetime
|
||||
):
|
||||
"""Parse the git repo history and upload it to the database."""
|
||||
|
||||
sql_insert_commit = """ INSERT INTO
|
||||
git_commits (hash, commit_time, phab_id, reverts_hash)
|
||||
values (%s,%s,%s,%s);
|
||||
"""
|
||||
sql_update_commit_project = (
|
||||
""" UPDATE git_commits SET mod_{} = %s where hash = %s;"""
|
||||
)
|
||||
known_hashes = get_existing_hashes(conn)
|
||||
day = None
|
||||
cur = conn.cursor()
|
||||
for commit in repo.iter_commits(GIT_BRANCH):
|
||||
# TODO: This takes a couple of minutes, maybe try using multithreading
|
||||
|
||||
# Only store new/unknown hashes
|
||||
if commit.hexsha in known_hashes:
|
||||
continue
|
||||
if commit.committed_datetime < max_age:
|
||||
break
|
||||
mycommit = MyCommit(commit)
|
||||
if mycommit.date.day != day:
|
||||
# take a snapshot commit, nice to see progress while updating the
|
||||
# database
|
||||
day = mycommit.date.day
|
||||
print(mycommit.date)
|
||||
conn.commit()
|
||||
cur.execute(
|
||||
sql_insert_commit,
|
||||
(
|
||||
mycommit.chash,
|
||||
mycommit.date,
|
||||
mycommit.phab_revision,
|
||||
mycommit.reverts_commit_hash,
|
||||
),
|
||||
)
|
||||
# Note: prasing the patches is quite slow
|
||||
for project in mycommit.modified_projects:
|
||||
# TODO find a way to make this generic for all projects, maybe user
|
||||
# "ALTER TABLE" to add columns as they appear
|
||||
# TODO: modifying the commited row is expensive, maybe find something faster
|
||||
if project in ["llvm", "libcxx", "mlir", "clang"]:
|
||||
cur.execute(
|
||||
sql_update_commit_project.format(project), (True, mycommit.chash)
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def create_csv_report(title: str, query: str, output_path: str):
|
||||
cursor = conn.cursor()
|
||||
data = cursor.execute(query)
|
||||
with open(os.path.join(output_path, title + ".csv"), "w") as csv_file:
|
||||
writer = csv.writer(csv_file)
|
||||
# write column headers
|
||||
writer.writerow([description[0] for description in cursor.description])
|
||||
for row in data:
|
||||
writer.writerow(row)
|
||||
|
||||
|
||||
def run_queries(conn: psycopg2.extensions.connection, output_path: str):
|
||||
print("running queries...")
|
||||
create_csv_report("full_db_dump", "select * from commits;", output_path)
|
||||
|
||||
query = """SELECT strftime('%Y-%m',commit_time) as month, count(hash) as num_commits, count(phab_id) as num_reviewed,
|
||||
(100.0*count(phab_id)/count(hash)) as percent_reviewed, count(reverts_hash) as num_reverted,
|
||||
(100.0*count(reverts_hash)/count(hash)) as percent_reverted
|
||||
FROM commits
|
||||
WHERE mod_{}
|
||||
GROUP BY month;
|
||||
"""
|
||||
create_csv_report("libcxx_stats", query.format("libcxx"), output_path)
|
||||
create_csv_report("mlir_stats", query.format("mlir"), output_path)
|
||||
|
||||
query = """SELECT strftime('%Y-%m',commit_time) as month, count(hash) as num_commits, count(phab_id) as num_reviewed,
|
||||
(100.0*count(phab_id)/count(hash)) as percent_reviewed, count(reverts_hash) as num_reverted,
|
||||
(100.0*count(reverts_hash)/count(hash)) as percent_reverted
|
||||
FROM commits
|
||||
GROUP BY month;
|
||||
"""
|
||||
create_csv_report("all_projects_stats", query, output_path)
|
||||
|
||||
|
||||
def update_comits():
|
||||
"""Update the git commits in the database from the git repository."""
|
||||
repo = update_repo(REPO_DIR)
|
||||
conn = connect_to_db()
|
||||
create_tables(conn)
|
||||
parse_commits(conn, repo, MAX_AGE)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
update_comits()
|
||||
# TODO: add argparse to switch between import and query mode or
|
||||
# move queries to another file
|
||||
# run_queries(conn)
|
||||
|
|
@ -1,76 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
import traceback
|
||||
import psycopg2
|
||||
from phabricator import Phabricator
|
||||
import os
|
||||
from typing import Optional
|
||||
import datetime
|
||||
import requests
|
||||
import logging
|
||||
|
||||
PHABRICATOR_URL = "https://reviews.llvm.org/api/"
|
||||
BUILDBOT_URL = "https://lab.llvm.org/buildbot/api/v2"
|
||||
|
||||
|
||||
def phab_up() -> Optional[Phabricator]:
|
||||
"""Try to connect to phabricator to see if the server is up.
|
||||
|
||||
Returns None if server is down.
|
||||
"""
|
||||
logging.info("Checking Phabricator status...")
|
||||
try:
|
||||
phab = Phabricator(token=os.getenv('CONDUIT_TOKEN'), host=PHABRICATOR_URL)
|
||||
phab.update_interfaces()
|
||||
logging.info("Phabricator is up.")
|
||||
return phab
|
||||
except Exception as ex:
|
||||
logging.error(ex)
|
||||
logging.error(traceback.format_exc())
|
||||
logging.warning("Phabricator is down.")
|
||||
return None
|
||||
|
||||
|
||||
def buildbot_up() -> bool:
|
||||
"""Check if buildbot server is up"""
|
||||
logging.info("Checking Buildbot status...")
|
||||
try:
|
||||
response = requests.get(BUILDBOT_URL)
|
||||
logging.info(f'{response.status_code} {BUILDBOT_URL}')
|
||||
logging.info(response.content)
|
||||
return response.status_code == 200
|
||||
except Exception as ex:
|
||||
logging.error(ex)
|
||||
logging.error(traceback.format_exc())
|
||||
logging.warning("Buildbot is down.")
|
||||
return False
|
||||
|
||||
|
||||
def log_server_status(phab: bool, buildbot: bool, conn: psycopg2.extensions.connection):
|
||||
"""log the phabricator status to the database."""
|
||||
logging.info("Writing Phabricator status to database...")
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"INSERT INTO server_status (timestamp, phabricator, buildbot) VALUES (%s,%s,%s);",
|
||||
(datetime.datetime.now(), phab, buildbot),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def connect_to_db() -> psycopg2.extensions.connection:
|
||||
"""Connect to the database, create tables as needed."""
|
||||
conn = psycopg2.connect(
|
||||
f"host=127.0.0.1 sslmode=disable dbname=stats user=stats password={os.getenv('DB_PASSWORD')}")
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TABLE IF NOT EXISTS server_status (timestamp timestamp, phabricator boolean, buildbot boolean);"
|
||||
)
|
||||
conn.commit()
|
||||
return conn
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logging.basicConfig(level='INFO', format='%(levelname)-7s %(message)s')
|
||||
conn = connect_to_db()
|
||||
phab = phab_up()
|
||||
buildbot = buildbot_up()
|
||||
log_server_status(phab is not None, buildbot, conn)
|
||||
|
|
@ -1,326 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import List, Optional, Tuple, Dict
|
||||
|
||||
import backoff
|
||||
from buildkite_utils import annotate, feedback_url, upload_file
|
||||
import git
|
||||
from phabricator import Phabricator
|
||||
|
||||
"""URL of upstream LLVM repository."""
|
||||
LLVM_GITHUB_URL = 'ssh://git@github.com/llvm/llvm-project'
|
||||
FORK_REMOTE_URL = 'ssh://git@github.com/llvm-premerge-tests/llvm-project'
|
||||
|
||||
|
||||
class ApplyPatch:
|
||||
"""Apply a diff from Phabricator on local working copy.
|
||||
|
||||
This script is a rewrite of `arc patch` to accommodate for dependencies
|
||||
that have already landed, but could not be identified by `arc patch`.
|
||||
|
||||
For a given diff_id, this class will get the dependencies listed on Phabricator.
|
||||
For each dependency D it will check the it's status:
|
||||
- if D is closed, skip it.
|
||||
- If D is not closed, it will download the patch for D and try to apply it locally.
|
||||
Once this class has applied all dependencies, it will apply the original diff.
|
||||
|
||||
This script must be called from the root folder of a local checkout of
|
||||
https://github.com/llvm/llvm-project or given a path to clone into.
|
||||
"""
|
||||
|
||||
def __init__(self, path: str, diff_id: int, token: str, url: str, git_hash: str,
|
||||
phid: str, push_branch: bool = False):
|
||||
self.push_branch = push_branch # type: bool
|
||||
self.conduit_token = token # type: Optional[str]
|
||||
self.host = url # type: Optional[str]
|
||||
self.diff_id = diff_id # type: int
|
||||
self.phid = phid # type: str
|
||||
if not self.host.endswith('/api/'):
|
||||
self.host += '/api/'
|
||||
self.phab = self.create_phab()
|
||||
self.base_revision = git_hash # type: str
|
||||
self.branch_base_hexsha = ''
|
||||
self.apply_diff_counter = 0
|
||||
self.build_dir = os.getcwd()
|
||||
self.revision_id = ''
|
||||
|
||||
if not os.path.isdir(path):
|
||||
logging.info(f'{path} does not exist, cloning repository...')
|
||||
self.repo = git.Repo.clone_from(FORK_REMOTE_URL, path)
|
||||
else:
|
||||
logging.info('repository exist, will reuse')
|
||||
self.repo = git.Repo(path) # type: git.Repo
|
||||
self.repo.remote('origin').set_url(FORK_REMOTE_URL)
|
||||
os.chdir(path)
|
||||
logging.info(f'working dir {os.getcwd()}')
|
||||
|
||||
@property
|
||||
def branch_name(self):
|
||||
"""Name used for the git branch."""
|
||||
return f'phab-diff-{self.diff_id}'
|
||||
|
||||
def run(self):
|
||||
"""try to apply the patch from phabricator
|
||||
"""
|
||||
try:
|
||||
diff = self.get_diff(self.diff_id)
|
||||
revision = self.get_revision(diff.revisionID)
|
||||
url = f"https://reviews.llvm.org/D{revision['id']}?id={diff['id']}"
|
||||
annotate(f"Patching changes [{url}]({url})", style='info', context='patch_diff')
|
||||
self.reset_repository()
|
||||
self.revision_id = revision['id']
|
||||
dependencies = self.get_dependencies(revision)
|
||||
dependencies.reverse() # Now revisions will be from oldest to newest.
|
||||
if len(dependencies) > 0:
|
||||
logging.info('This diff depends on: {}'.format(revision_list_to_str(dependencies)))
|
||||
plan = []
|
||||
for r in dependencies:
|
||||
if r['statusName'] == 'Closed':
|
||||
logging.info(f'skipping revision {r["id"]} - it is closed, assuming it has landed')
|
||||
continue
|
||||
d = self.get_diff(r['diffs'][0])
|
||||
plan.append((r, d))
|
||||
plan.append((revision, diff))
|
||||
logging.info('Planning to apply in order:')
|
||||
for (r, d) in plan:
|
||||
logging.info(f"https://reviews.llvm.org/D{r['id']}?id={d['id']}")
|
||||
# Pick the newest known commit as a base for patches.
|
||||
base_commit = None
|
||||
for (r, d) in plan:
|
||||
c = self.find_commit(d['sourceControlBaseRevision'])
|
||||
if c is None:
|
||||
logging.warning(f"D{r['id']}#{d['id']} commit {d['sourceControlBaseRevision']} does not exist")
|
||||
continue
|
||||
if base_commit is None:
|
||||
logging.info(f"D{r['id']}#{d['id']} commit {c.hexsha} exists")
|
||||
base_commit = c
|
||||
elif c.committed_datetime > base_commit.committed_datetime:
|
||||
logging.info(f"D{r['id']}#{d['id']} commit {c.hexsha} has a later commit date then"
|
||||
f"{base_commit.hexsha}")
|
||||
base_commit = c
|
||||
if self.base_revision != 'auto':
|
||||
logging.info(f'Base revision "{self.base_revision}" is set by command argument. Will use '
|
||||
f'instead of resolved "{base_commit}"')
|
||||
base_commit = self.find_commit(self.base_revision)
|
||||
if base_commit is None:
|
||||
base_commit = self.repo.heads['main'].commit
|
||||
annotate(f"Cannot find a base git revision. Will use current HEAD.",
|
||||
style='warning', context='patch_diff')
|
||||
self.create_branch(base_commit)
|
||||
for (r, d) in plan:
|
||||
if not self.apply_diff(d, r):
|
||||
return 1
|
||||
if self.push_branch:
|
||||
self.repo.git.push('--force', 'origin', self.branch_name)
|
||||
annotate(f"Created branch [{self.branch_name}]"
|
||||
f"(https://github.com/llvm-premerge-tests/llvm-project/tree/{self.branch_name}).\n\n"
|
||||
f"To checkout locally, run in your copy of llvm-project directory:\n\n"
|
||||
"```shell\n"
|
||||
"git remote add premerge git@github.com:llvm-premerge-tests/llvm-project.git #first time\n"
|
||||
f"git fetch premerge {self.branch_name}\n"
|
||||
f"git checkout -b {self.branch_name} --track premerge/{self.branch_name}\n"
|
||||
"```",
|
||||
style='success',
|
||||
context='patch_diff')
|
||||
logging.info('Branch {} has been pushed'.format(self.branch_name))
|
||||
return 0
|
||||
except Exception as e:
|
||||
annotate(f":bk-status-failed: Unexpected error. Consider [creating a bug]({feedback_url()}).",
|
||||
style='error', context='patch_diff')
|
||||
logging.error(f'exception: {e}')
|
||||
return 1
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def reset_repository(self):
|
||||
"""Update local git repo and origin.
|
||||
|
||||
As origin is disjoint from upstream, it needs to be updated by this script.
|
||||
"""
|
||||
# Remove index lock just in case.
|
||||
lock_file = f"{self.repo.working_tree_dir}/.git/index.lock"
|
||||
try:
|
||||
os.remove(lock_file)
|
||||
logging.info(f"removed {lock_file}")
|
||||
except FileNotFoundError:
|
||||
logging.info(f"{lock_file} does not exist")
|
||||
logging.info('Syncing local, origin and upstream...')
|
||||
if 'upstream' not in self.repo.remotes:
|
||||
self.repo.create_remote('upstream', url=LLVM_GITHUB_URL)
|
||||
self.repo.remotes.upstream.fetch()
|
||||
self.repo.git.clean('-ffxdq')
|
||||
self.repo.git.reset('--hard')
|
||||
self.repo.git.fetch('--all')
|
||||
if self.find_commit('main') is None:
|
||||
origin = self.repo.remotes.origin
|
||||
self.repo.create_head('main', origin.refs.main)
|
||||
self.repo.heads.main.set_tracking_branch(origin.refs.main)
|
||||
self.repo.heads.main.checkout()
|
||||
self.repo.git.pull('origin', 'main')
|
||||
self.repo.git.pull('upstream', 'main')
|
||||
if self.push_branch:
|
||||
self.repo.git.push('origin', 'main')
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def find_commit(self, rev):
|
||||
try:
|
||||
return self.repo.commit(rev)
|
||||
except:
|
||||
return None
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def create_branch(self, base_commit: git.Commit):
|
||||
if self.branch_name in self.repo.heads:
|
||||
self.repo.delete_head('--force', self.branch_name)
|
||||
logging.info(f'creating branch {self.branch_name} at {base_commit.hexsha}')
|
||||
new_branch = self.repo.create_head(self.branch_name, base_commit.hexsha)
|
||||
self.repo.head.reference = new_branch
|
||||
self.repo.head.reset(index=True, working_tree=True)
|
||||
self.branch_base_hexsha = self.repo.head.commit.hexsha
|
||||
logging.info('Base branch revision is {}'.format(self.repo.head.commit.hexsha))
|
||||
annotate(f"Branch {self.branch_name} base revision is `{self.branch_base_hexsha}`.",
|
||||
style='info', context='patch_diff')
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def commit(self, revision: Dict, diff: Dict):
|
||||
"""Commit the current state and annotates with the revision info."""
|
||||
self.repo.git.add('-A')
|
||||
diff.setdefault('authorName', 'unknown')
|
||||
diff.setdefault('authorEmail', 'unknown')
|
||||
author = git.Actor(name=diff['authorName'], email=diff['authorEmail'])
|
||||
message = (f"{revision['title']}\n\n"
|
||||
f"Automated commit created by applying diff {self.diff_id}\n"
|
||||
f"\n"
|
||||
f"Phabricator-ID: {self.phid}\n"
|
||||
f"Review-ID: {diff_to_str(revision['id'])}\n")
|
||||
self.repo.index.commit(message=message, author=author)
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def create_phab(self):
|
||||
phab = Phabricator(token=self.conduit_token, host=self.host)
|
||||
phab.update_interfaces()
|
||||
return phab
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def get_diff(self, diff_id: int):
|
||||
"""Get a diff from Phabricator based on its diff id."""
|
||||
return self.phab.differential.getdiff(diff_id=diff_id)
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def get_revision(self, revision_id: int):
|
||||
"""Get a revision from Phabricator based on its revision id."""
|
||||
return self.phab.differential.query(ids=[revision_id])[0]
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def get_revisions(self, *, phids: List[str] = None):
|
||||
"""Get a list of revisions from Phabricator based on their PH-IDs."""
|
||||
if phids is None:
|
||||
raise Exception('_get_revisions phids is None')
|
||||
if not phids:
|
||||
# Handle an empty query locally. Otherwise the connection
|
||||
# will time out.
|
||||
return []
|
||||
return self.phab.differential.query(phids=phids)
|
||||
|
||||
def get_dependencies(self, revision: Dict) -> List[Dict]:
|
||||
"""Recursively resolves dependencies of the given revision.
|
||||
They are listed in reverse chronological order - from most recent to least recent."""
|
||||
dependency_ids = revision['auxiliary']['phabricator:depends-on']
|
||||
revisions = self.get_revisions(phids=dependency_ids)
|
||||
result = []
|
||||
for r in revisions:
|
||||
result.append(r)
|
||||
sub = self.get_dependencies(r)
|
||||
result.extend(sub)
|
||||
return result
|
||||
|
||||
def apply_diff(self, diff: Dict, revision: Dict) -> bool:
|
||||
"""Download and apply a diff to the local working copy."""
|
||||
logging.info(f"Applying {diff['id']} for revision {revision['id']}...")
|
||||
patch = self.get_raw_diff(str(diff['id']))
|
||||
self.apply_diff_counter += 1
|
||||
patch_file = f"{self.apply_diff_counter}_{diff['id']}.patch"
|
||||
with open(os.path.join(self.build_dir, patch_file), 'wt') as f:
|
||||
f.write(patch)
|
||||
# For annotate to properly link this file it must exist before the upload.
|
||||
upload_file(self.build_dir, patch_file)
|
||||
logging.debug(f'raw patch:\n{patch}')
|
||||
proc = subprocess.run('git apply -', input=patch, shell=True, text=True,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
if proc.returncode != 0:
|
||||
logging.info(proc.stdout)
|
||||
logging.error(proc.stderr)
|
||||
message = f":bk-status-failed: Failed to apply [{patch_file}](artifact://{patch_file}).\n\n"
|
||||
if self.revision_id != revision['id']:
|
||||
message += f"**Attention! D{revision['id']} is one of the dependencies of the target " \
|
||||
f"revision D{self.revision_id}.**\n\n"
|
||||
message += (f"No testing is possible because we couldn't apply the patch.\n\n"
|
||||
f"---\n\n"
|
||||
'### Troubleshooting\n\n'
|
||||
'More information is available in the log of of *create branch* step. '
|
||||
f"All patches applied are available as *Artifacts*.\n\n"
|
||||
f":bulb: The patch may not apply if it includes only the most recent of "
|
||||
f"multiple local commits. Try to upload a patch with\n"
|
||||
f"```shell\n"
|
||||
f"arc diff `git merge-base HEAD origin` --update D{revision['id']}\n"
|
||||
f"```\n\n"
|
||||
f"to include all local changes.\n\n"
|
||||
'---\n\n'
|
||||
f"If this case could have been handled better, please [create a bug]({feedback_url()}).")
|
||||
annotate(message,
|
||||
style='error',
|
||||
context='patch_diff')
|
||||
return False
|
||||
self.commit(revision, diff)
|
||||
return True
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def get_raw_diff(self, diff_id: str) -> str:
|
||||
return self.phab.differential.getrawdiff(diffID=diff_id).response
|
||||
|
||||
|
||||
def diff_to_str(diff: int) -> str:
|
||||
"""Convert a diff id to a string with leading "D"."""
|
||||
return 'D{}'.format(diff)
|
||||
|
||||
|
||||
def revision_list_to_str(diffs: List[Dict]) -> str:
|
||||
"""Convert list of diff ids to a comma separated list, prefixed with "D"."""
|
||||
return ', '.join([diff_to_str(d['id']) for d in diffs])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description='Apply Phabricator patch to working directory.')
|
||||
parser.add_argument('diff_id', type=int)
|
||||
parser.add_argument('--path', type=str, help='repository path', default=os.getcwd())
|
||||
parser.add_argument('--token', type=str, default=None, help='Conduit API token')
|
||||
parser.add_argument('--url', type=str, default='https://reviews.llvm.org', help='Phabricator URL')
|
||||
parser.add_argument('--commit', dest='commit', type=str, default='auto',
|
||||
help='Use this commit as a base. For "auto" tool tries to pick the base commit itself')
|
||||
parser.add_argument('--push-branch', action='store_true', dest='push_branch',
|
||||
help='choose if branch shall be pushed to origin')
|
||||
parser.add_argument('--phid', type=str, default=None, help='Phabricator ID of the review this commit pertains to')
|
||||
parser.add_argument('--log-level', type=str, default='INFO')
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
|
||||
patcher = ApplyPatch(args.path, args.diff_id, args.token, args.url, args.commit, args.phid, args.push_branch)
|
||||
sys.exit(patcher.run())
|
||||
1
scripts/phab2github/.gitignore
vendored
1
scripts/phab2github/.gitignore
vendored
|
|
@ -1 +0,0 @@
|
|||
/tmp
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
This folder contains a set of scripts to mirror Phabricator *Revisions* to GitHub *Pull Requests*.
|
||||
|
||||
# Problem statement
|
||||
The LLVM project performs code reviews on [Phabricator](https://reviews.llvm.org) and the source
|
||||
code is stored on [GitHub](https://github.com/llvm/llvm-project). While there are many nice CI
|
||||
system integrations available for GitHub (e.g. buildkite, CircleCI, GitHub Actions), there are not
|
||||
that many for Phabricator. The current setup of the pre-merge tests is quite some effort to maintain
|
||||
and does not scale well with additional build agents. User do not have access to the bulid status.
|
||||
|
||||
One of the challenges is that Phabricator maintains patches, that might not be based on a git
|
||||
commit. And even if they are based on a git commit, that commit might be local to the user and
|
||||
not available in the public repository.
|
||||
|
||||
# Proposal
|
||||
Instead of integrating a CI system with Phabricator, move all the information to Github and trigger
|
||||
the builds from there.
|
||||
|
||||
1. Create a service that mirrors all Revisions form Phabricator into GitHub Pull Requests.
|
||||
1. Then use the usual CI integration for GitHub to build and test these revisions.
|
||||
1. On Phabricator, add links to the builds.
|
||||
1. At the end of the build, a script uploads the results to Phabricator (as we have it now).
|
||||
|
||||
If that works well, we can also use this to demonstrate the workflow of Github Pull Requests to the
|
||||
LLVM community.
|
||||
|
||||
# current status
|
||||
The script currently does:
|
||||
|
||||
* Get the latest 10 Revisions from Phabricator for testing.
|
||||
* For each revision:
|
||||
* Create a local git branch `phab-Dxxxxx`
|
||||
* Create a local git branch `phab-diff-yyyyy` for every Diff of the Revision using the raw patch of each Diff.
|
||||
* Layer the diffs on top of each other in the `phab-Dxxxxxx` branch.
|
||||
* Push the `phab-Dxxxxxx` branch to the [LLVM fork](https://github.com/ChristianKuehnel/llvm-project).
|
||||
* Create a [pull request](https://github.com/ChristianKuehnel/llvm-project/pulls) for each branch.
|
||||
* The GitHub/Buildkite
|
||||
[CI integration](https://buildkite.com/llvm-project/phabricator-pull-request-integration-prototype)
|
||||
then triggers the builds for the PRs.
|
||||
|
||||
As this is a prototype, it currently has some shortcommings:
|
||||
* I'm only looking at the 10 latest Revisions for testing.
|
||||
* If a patch does not apply: just ignore it
|
||||
* The Revision branches are always created from scratch, there is no incremental update.
|
||||
* I run the script manually for testing, There are no automatic pull/push updates
|
||||
|
||||
# Work items
|
||||
|
||||
This is the list (and order) of the work items for this idea.
|
||||
We will start with this on a fork of the LLVM repository running on my machine as cron job.
|
||||
Cancel the effort if some thinks would not work or the community objects.
|
||||
|
||||
* [x] For each Revision create a branch. Create a PR from that.
|
||||
* [x] Layer the diffs on top of each other in the git log.
|
||||
* [ ] Figure out what to do with chained Revisions.
|
||||
* [ ] Find a scalable way to reports results back to Phabricator.
|
||||
* [ ] Clean up old branches and PRs.
|
||||
* [ ] Create pre-merge testing service to the cloud, trigger it with every change on Phabricator.
|
||||
Also do regular full scans of Phabricator to catch everything lost in the triggers.
|
||||
* [ ] Add more build machines to buildkite and move beta testers to new infrastructure.
|
||||
* [ ] Move all pre-merge tests to new infrastructure.
|
||||
* [ ] Find way to test new scripts/builders/configuration/... without annoying the users.
|
||||
* [ ] Enable others to add more checks/builders as needed.
|
||||
|
||||
|
||||
Optional:
|
||||
* [ ] Also sync Revisions that do not trigger Harbormaster at the moment and provide feedback as
|
||||
well.
|
||||
* [ ] Find a way to copy the description and the comments from the Revision to the PR.
|
||||
* [ ] Create branches and PRs on LLVM repository, allow people to click the "Merge" button on the PR,
|
||||
reducing the manual work to land a patch.
|
||||
|
|
@ -1,216 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import os
|
||||
from typing import Optional, Union
|
||||
import git
|
||||
import logging
|
||||
from phab_wrapper import PhabWrapper, Revision
|
||||
import subprocess
|
||||
import github
|
||||
import json
|
||||
import sys
|
||||
|
||||
# TODO: move to config file
|
||||
LLVM_GITHUB_URL = 'ssh://git@github.com/llvm/llvm-project'
|
||||
MY_GITHUB_URL = 'ssh://git@github.com/christiankuehnel/llvm-project'
|
||||
MY_REPO = 'christiankuehnel/llvm-project'
|
||||
|
||||
|
||||
_LOGGER = logging.getLogger()
|
||||
|
||||
|
||||
class ApplyPatchException(Exception):
|
||||
"""A patch could not be applied."""
|
||||
pass
|
||||
|
||||
|
||||
class Phab2Github:
|
||||
|
||||
def __init__(self, workdir: str):
|
||||
self.workdir = workdir
|
||||
self.llvm_dir = os.path.join(self.workdir, 'llvm-project')
|
||||
self.repo = None # type: Optional[git.Repo]
|
||||
self.phab_wrapper = PhabWrapper()
|
||||
self.github = self._create_github()
|
||||
self.github_repo = self.github.get_repo(MY_REPO) # type: github.Repository
|
||||
|
||||
def sync(self):
|
||||
"""Sync Phabricator to Github."""
|
||||
_LOGGER.info('Starting sync...')
|
||||
self._refresh_main()
|
||||
self._delete_phab_branches()
|
||||
revisions = self.phab_wrapper.get_revisions()
|
||||
pull_requests = {p.title: p for p in self.github_repo.get_pulls(state='open')}
|
||||
for revision in revisions:
|
||||
self.create_branches_for_revision(revision)
|
||||
if self._has_branch(revision):
|
||||
if self._branches_identical(revision.branch_name, 'origin/{}'.format(revision.branch_name)):
|
||||
_LOGGER.info('Branch {} is identical to upstream. Not pushing.'.format(revision.branch_name))
|
||||
else:
|
||||
_LOGGER.info('Pushing branch {} to github...'.format(revision.branch_name))
|
||||
# TODO: do we sill need to force-push?
|
||||
self.repo.git.push('--force', 'origin', revision.branch_name)
|
||||
if revision.pr_title in pull_requests:
|
||||
_LOGGER.info('Pull request already exists: {}'.format(pull_requests[revision.pr_title].html_url))
|
||||
else:
|
||||
_LOGGER.info('Creating pull-request for branch {}...'.format(revision.branch_name))
|
||||
pr = self.github_repo.create_pull(title=revision.pr_title,
|
||||
body=revision.pr_summary,
|
||||
head=revision.branch_name,
|
||||
base='main')
|
||||
_LOGGER.info(pr.html_url)
|
||||
_LOGGER.info('Sync completed.')
|
||||
|
||||
def _refresh_main(self):
|
||||
"""Clone/update local git repo."""
|
||||
if not os.path.exists(self.workdir):
|
||||
os.mkdir(self.workdir)
|
||||
if os.path.exists(self.llvm_dir):
|
||||
# TODO: in case of errors: delete and clone
|
||||
_LOGGER.info('pulling origin and upstream...')
|
||||
self.repo = git.Repo(self.llvm_dir)
|
||||
self.repo.git.fetch('--all')
|
||||
self.repo.git.checkout('main')
|
||||
self.repo.git.pull('upstream', 'main')
|
||||
self.repo.git.push('origin', 'main')
|
||||
else:
|
||||
_LOGGER.info('cloning repository...')
|
||||
git.Repo.clone_from(MY_GITHUB_URL, self.llvm_dir)
|
||||
self.repo = git.Repo(self.llvm_dir)
|
||||
self.repo.create_remote('upstream', url=LLVM_GITHUB_URL)
|
||||
self.repo.remotes.upstream.fetch()
|
||||
_LOGGER.info('refresh of main branch completed')
|
||||
|
||||
def create_branches_for_revision(self, revision: Revision):
|
||||
"""Create branches for a Revision and it's Diffs.
|
||||
|
||||
Apply changes from Phabricator to these branches.
|
||||
"""
|
||||
name = revision.branch_name
|
||||
# TODO: only look at diffs that were not yet applied
|
||||
# TODO: can diffs be modified on Phabricator and keep their ID?
|
||||
for diff in revision.sorted_diffs:
|
||||
if self._has_branch(diff):
|
||||
continue
|
||||
self.create_branch_for_diff(diff)
|
||||
patch = self.phab_wrapper.get_raw_patch(diff)
|
||||
try:
|
||||
self.apply_patch(diff, patch)
|
||||
except ApplyPatchException as e:
|
||||
# TODO: retry on main if this fails
|
||||
_LOGGER.error('Could not apply patch for Diff {}. Deleting branch'.format(diff.id))
|
||||
_LOGGER.exception(e)
|
||||
self.repo.heads['main'].checkout()
|
||||
self.repo.delete_head(diff.branch_name)
|
||||
|
||||
diffs = [d for d in revision.sorted_diffs if self._has_branch(d)]
|
||||
if len(diffs) == 0:
|
||||
# TODO: handle error
|
||||
_LOGGER.error('Could not create branch for Revision D{}'.format(revision.id))
|
||||
return
|
||||
new_branch = self.repo.create_head(revision.branch_name, diffs[0].branch_name)
|
||||
new_branch.checkout()
|
||||
|
||||
for diff in diffs[1:]:
|
||||
_LOGGER.info('Applying Diff {} onto branch {}'.format(diff.branch_name, revision.branch_name))
|
||||
patch = self._create_patch(revision.branch_name, diff.branch_name)
|
||||
if len(patch) == 0:
|
||||
_LOGGER.warning('Diff {} is identical to last one.'.format(diff.id))
|
||||
else:
|
||||
try:
|
||||
self.apply_patch(diff, patch)
|
||||
except ApplyPatchException:
|
||||
_LOGGER.error('Applying patch failed, but should not:')
|
||||
_LOGGER.error(patch)
|
||||
raise
|
||||
|
||||
_LOGGER.info('Created branch for Revision D{}'.format(revision.id))
|
||||
|
||||
def create_branch_for_diff(self, diff: "Diff"):
|
||||
"""Create a branch for diff."""
|
||||
base_hash = diff.base_hash
|
||||
if base_hash is None:
|
||||
base_hash = 'upstream/main'
|
||||
_LOGGER.info('creating branch {} based on {}...'.format(diff.branch_name, base_hash))
|
||||
try:
|
||||
new_branch = self.repo.create_head(diff.branch_name, base_hash)
|
||||
except ValueError:
|
||||
# commit hash not found, try again with main
|
||||
_LOGGER.warning('commit hash {} not found in upstream repository. '
|
||||
'Trying main instead...'.format(diff.branch_name, base_hash))
|
||||
base_hash = 'upstream/main'
|
||||
new_branch = self.repo.create_head(diff.branch_name, base_hash)
|
||||
self.repo.head.reference = new_branch
|
||||
self.repo.head.reset(index=True, working_tree=True)
|
||||
|
||||
def apply_patch(self, diff: "Diff", raw_patch: str):
|
||||
"""Apply a patch to the working copy."""
|
||||
# TODO: impersonate the original author of the patch
|
||||
proc = subprocess.run('git apply --ignore-whitespace --whitespace=fix -', input=raw_patch, shell=True,
|
||||
text=True, cwd=self.repo.working_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
if proc.returncode != 0:
|
||||
raise ApplyPatchException('Applying patch failed:\n{}'.format(proc.stdout + proc.stderr))
|
||||
self.repo.git.add('-A')
|
||||
self.repo.index.commit(message='applying Diff {} for Revision D{}\n\n Diff: {}'.format(
|
||||
diff.id, diff.revision.id, diff.id))
|
||||
|
||||
@staticmethod
|
||||
def _create_github() -> github.Github:
|
||||
"""Create instance of Github client.
|
||||
|
||||
Reads access token from a file.
|
||||
"""
|
||||
with open(os.path.expanduser('~/.llvm-premerge-checks/github-token.json')) as json_file:
|
||||
token = json.load(json_file)['token']
|
||||
return github.Github(token)
|
||||
|
||||
def _has_branch(self, item: Union["Diff", "Revision"]) -> bool:
|
||||
"""Check if the Diff/Revision has a local branch."""
|
||||
return item.branch_name in self.repo.heads
|
||||
|
||||
def _delete_phab_branches(self):
|
||||
"""Delete all branches sarting with 'phab-'."""
|
||||
_LOGGER.info('Deleting local Phabricator-relates branches...')
|
||||
self.repo.git.checkout('main')
|
||||
for branch in [b for b in self.repo.heads if b.name.startswith('phab-')]:
|
||||
_LOGGER.info('Deleding branch {}'.format(branch))
|
||||
self.repo.git.branch('-D', branch.name)
|
||||
|
||||
def _create_patch(self, base: str, changes: str) -> str:
|
||||
"""Create patch from two commits."""
|
||||
proc = subprocess.run('git diff {} {}'.format(base, changes), shell=True, text=True,
|
||||
cwd=self.repo.working_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
if proc.returncode != 0:
|
||||
raise ApplyPatchException('Applying patch failed:\n{}'.format(proc.stdout + proc.stderr))
|
||||
return proc.stdout
|
||||
|
||||
def _branches_identical(self, left, right) -> bool:
|
||||
"""Check if two branches are identical."""
|
||||
try:
|
||||
patch = self.repo.git.diff(left, right)
|
||||
except git.GitCommandError:
|
||||
return False
|
||||
if len(patch) == 0:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
rootdir = os.path.dirname(os.path.abspath(__file__))
|
||||
tmpdir = os.path.join(rootdir, 'tmp')
|
||||
p2g = Phab2Github(tmpdir)
|
||||
p2g.sync()
|
||||
|
|
@ -1,175 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
from phabricator import Phabricator
|
||||
from typing import List, Optional, Dict
|
||||
import datetime
|
||||
|
||||
|
||||
_BASE_URL = 'https://reviews.llvm.org'
|
||||
|
||||
_LOGGER = logging.getLogger()
|
||||
|
||||
|
||||
class Revision:
|
||||
"""A Revision on Phabricator"""
|
||||
|
||||
def __init__(self, revision_dict: Dict):
|
||||
self.revision_dict = revision_dict
|
||||
self.diffs = [] # type : "Diff"
|
||||
self.phab_url = '{}/D{}'.format(_BASE_URL, self.id) # type: str
|
||||
|
||||
@property
|
||||
def id(self) -> str:
|
||||
return self.revision_dict['id']
|
||||
|
||||
@property
|
||||
def phid(self) -> str:
|
||||
return self.revision_dict['phid']
|
||||
|
||||
@property
|
||||
def status(self) -> str:
|
||||
return self.revision_dict['fields']['status']['value']
|
||||
|
||||
def __str__(self):
|
||||
return 'Revision {}: {} - ({})'.format(self.id, self.status,
|
||||
','.join([str(d.id) for d in self.diffs]))
|
||||
|
||||
@property
|
||||
def branch_name(self) -> str:
|
||||
return 'phab-D{}'.format(self.id)
|
||||
|
||||
@property
|
||||
def title(self) -> str:
|
||||
return self.revision_dict['fields']['title']
|
||||
|
||||
@property
|
||||
def pr_title(self) -> str:
|
||||
return 'D{}: {}'.format(self.id, self.title)
|
||||
|
||||
@property
|
||||
def summary(self) -> str:
|
||||
return self.revision_dict['fields']['summary']
|
||||
|
||||
@property
|
||||
def pr_summary(self) -> str:
|
||||
return '{}\n\n{}'.format(self.summary, self.phab_url)
|
||||
|
||||
@property
|
||||
def sorted_diffs(self) -> List["Diff"]:
|
||||
return sorted(self.diffs, key=lambda d: d.id)
|
||||
|
||||
|
||||
class Diff:
|
||||
"""A Phabricator diff."""
|
||||
|
||||
def __init__(self, diff_dict: Dict, revision: Revision):
|
||||
self.diff_dict = diff_dict
|
||||
self.revision = revision
|
||||
# TODO: check in git repo instead of using a flag
|
||||
|
||||
@property
|
||||
def id(self) -> str:
|
||||
return self.diff_dict['id']
|
||||
|
||||
@property
|
||||
def phid(self) -> str:
|
||||
return self.diff_dict['phid']
|
||||
|
||||
def __str__(self):
|
||||
return 'Diff {}'.format(self.id)
|
||||
|
||||
@property
|
||||
def base_hash(self) -> Optional[str]:
|
||||
for ref in self.diff_dict['fields']['refs']:
|
||||
if ref['type'] == 'base':
|
||||
return ref['identifier']
|
||||
return None
|
||||
|
||||
@property
|
||||
def branch_name(self) -> str:
|
||||
return 'phab-diff-{}'.format(self.id)
|
||||
|
||||
|
||||
class PhabWrapper:
|
||||
"""
|
||||
Wrapper around the interactions with Phabricator.
|
||||
|
||||
Conduit API documentation: https://reviews.llvm.org/conduit/
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.conduit_token = None # type: Optional[str]
|
||||
self.host = None # type: Optional[str]
|
||||
self._load_arcrc()
|
||||
self.phab = self._create_phab() # type: Phabricator
|
||||
|
||||
def _load_arcrc(self):
|
||||
"""Load arc configuration from file if not set."""
|
||||
_LOGGER.info('Loading configuration from ~/.arcrc file')
|
||||
with open(os.path.expanduser('~/.arcrc'), 'r') as arcrc_file:
|
||||
arcrc = json.load(arcrc_file)
|
||||
# use the first host configured in the file
|
||||
self.host = next(iter(arcrc['hosts']))
|
||||
self.conduit_token = arcrc['hosts'][self.host]['token']
|
||||
|
||||
def _create_phab(self) -> Phabricator:
|
||||
"""Create Phabricator API instance and update it."""
|
||||
phab = Phabricator(token=self.conduit_token, host=self.host)
|
||||
# TODO: retry on communication error
|
||||
phab.update_interfaces()
|
||||
return phab
|
||||
|
||||
def get_revisions(self) -> List[Revision]:
|
||||
"""Get relevant revisions."""
|
||||
# TODO: figure out which revisions we really need to pull in
|
||||
_LOGGER.info('Getting revisions from Phabricator')
|
||||
start_date = datetime.datetime.now() - datetime.timedelta(days=3)
|
||||
constraints = {
|
||||
'createdStart': int(start_date.timestamp())
|
||||
#'ids': [76120]
|
||||
}
|
||||
# TODO: handle > 100 responses
|
||||
revision_response = self.phab.differential.revision.search(
|
||||
constraints=constraints)
|
||||
revisions = [Revision(r) for r in revision_response.response['data']]
|
||||
# TODO: only taking the first 10 to speed things up
|
||||
revisions = revisions[0:10]
|
||||
_LOGGER.info('Got {} revisions from the server'.format(len(revisions)))
|
||||
for revision in revisions:
|
||||
# TODO: batch-query diffs for all revisions, reduce number of
|
||||
# API calls this would be much faster. But then we need to locally
|
||||
# map the diffs to the right revisions
|
||||
self._get_diffs(revision)
|
||||
return revisions
|
||||
|
||||
def _get_diffs(self, revision: Revision):
|
||||
"""Get diffs for a revision from Phabricator."""
|
||||
_LOGGER.info('Downloading diffs for Revision D{}...'.format(revision.id))
|
||||
constraints = {
|
||||
'revisionPHIDs': [revision.phid]
|
||||
}
|
||||
diff_response = self.phab.differential.diff.search(
|
||||
constraints=constraints)
|
||||
revision.diffs = [Diff(d, revision) for d in diff_response.response['data']]
|
||||
_LOGGER.info(', '.join([str(d.id) for d in revision.diffs]))
|
||||
|
||||
def get_raw_patch(self, diff: Diff) -> str:
|
||||
"""Get raw patch for diff from Phabricator."""
|
||||
_LOGGER.info('Downloading patch for Diff {}...'.format(diff.id))
|
||||
return self.phab.differential.getrawdiff(diffID=str(diff.id)).response
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
phabricator==0.7.0
|
||||
gitpython==3.0.6
|
||||
retrying==1.3.3
|
||||
PyGitHub=1.46
|
||||
2
scripts/phabtalk/.gitignore
vendored
2
scripts/phabtalk/.gitignore
vendored
|
|
@ -1,2 +0,0 @@
|
|||
venv
|
||||
test-data/
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
This folder contains Python scripts that talk to Phabricator.
|
||||
|
||||
They require a few libraries listed in `requirements.txt`.
|
||||
To install the requirements locally run `pip3 install -r requirements.txt`.
|
||||
|
|
@ -1,138 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import List, Optional
|
||||
|
||||
import backoff
|
||||
from phabricator import Phabricator
|
||||
|
||||
|
||||
class ApplyPatch:
|
||||
|
||||
def __init__(self, comment_file_path: str, git_hash: str):
|
||||
# TODO: turn os.environ parameter into command line arguments
|
||||
# this would be much clearer and easier for testing
|
||||
self.comment_file_path = comment_file_path
|
||||
self.conduit_token = os.environ.get('CONDUIT_TOKEN') # type: Optional[str]
|
||||
self.host = os.environ.get('PHABRICATOR_HOST') # type: Optional[str]
|
||||
self._load_arcrc()
|
||||
self.diff_id = os.environ['DIFF_ID'] # type: str
|
||||
self.diff_json_path = os.environ['DIFF_JSON'] # type: str
|
||||
if not self.host.endswith('/api/'):
|
||||
self.host += '/api/'
|
||||
self.phab = Phabricator(token=self.conduit_token, host=self.host)
|
||||
self.git_hash = git_hash # type: Optional[str]
|
||||
self.msg = [] # type: List[str]
|
||||
|
||||
def _load_arcrc(self):
|
||||
"""Load arc configuration from file if not set."""
|
||||
if self.conduit_token is not None or self.host is not None:
|
||||
return
|
||||
print('Loading configuration from ~/.arcrc file')
|
||||
with open(os.path.expanduser('~/.arcrc'), 'r') as arcrc_file:
|
||||
arcrc = json.load(arcrc_file)
|
||||
# use the first host configured in the file
|
||||
self.host = next(iter(arcrc['hosts']))
|
||||
self.conduit_token = arcrc['hosts'][self.host]['token']
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def update_interfaces(self):
|
||||
self.phab.update_interfaces()
|
||||
|
||||
def run(self):
|
||||
"""try to apply the patch from phabricator
|
||||
"""
|
||||
self.update_interfaces()
|
||||
|
||||
try:
|
||||
if self.git_hash is None:
|
||||
self._get_parent_hash()
|
||||
else:
|
||||
print('Use provided commit {}'.format(self.git_hash))
|
||||
self._git_checkout()
|
||||
self._apply_patch()
|
||||
finally:
|
||||
self._write_error_message()
|
||||
|
||||
def _get_parent_hash(self):
|
||||
diff = self.phab.differential.getdiff(diff_id=self.diff_id)
|
||||
# Keep a copy of the Phabricator answer for later usage in a json file
|
||||
try:
|
||||
with open(self.diff_json_path, 'w') as json_file:
|
||||
json.dump(diff.response, json_file, sort_keys=True, indent=4)
|
||||
print('Wrote diff details to "{}".'.format(self.diff_json_path))
|
||||
except Exception:
|
||||
print('WARNING: could not write build/diff.json log file')
|
||||
self.git_hash = diff['sourceControlBaseRevision']
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def _git_checkout(self):
|
||||
try:
|
||||
print('Checking out git hash {}'.format(self.git_hash))
|
||||
subprocess.check_call('git reset --hard {}'.format(self.git_hash),
|
||||
stdout=sys.stdout, stderr=sys.stderr, shell=True)
|
||||
except subprocess.CalledProcessError:
|
||||
print('WARNING: checkout of hash failed, using main branch instead.')
|
||||
self.msg += [
|
||||
'Could not check out parent git hash "{}". It was not found in '
|
||||
'the repository. Did you configure the "Parent Revision" in '
|
||||
'Phabricator properly? Trying to apply the patch to the '
|
||||
'main branch instead...'.format(self.git_hash)]
|
||||
subprocess.check_call('git checkout main', stdout=sys.stdout,
|
||||
stderr=sys.stderr, shell=True)
|
||||
subprocess.check_call('git show -s', stdout=sys.stdout,
|
||||
stderr=sys.stderr, shell=True)
|
||||
print('git checkout completed.')
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def _apply_patch(self):
|
||||
print('running arc patch...')
|
||||
cmd = 'arc patch --force --nobranch --no-ansi --diff "{}" --nocommit ' \
|
||||
'--conduit-token "{}" --conduit-uri "{}"'.format(
|
||||
self.diff_id, self.conduit_token, self.host)
|
||||
result = subprocess.run(cmd, capture_output=True, shell=True, text=True)
|
||||
print(result.stdout + result.stderr)
|
||||
if result.returncode != 0:
|
||||
msg = (
|
||||
'ERROR: arc patch failed with error code {}. '
|
||||
'Check build log for details.'.format(result.returncode))
|
||||
self.msg += [msg]
|
||||
raise Exception(msg)
|
||||
print('Patching completed.')
|
||||
|
||||
def _write_error_message(self):
|
||||
"""Write the log message to a file."""
|
||||
if self.comment_file_path is None:
|
||||
return
|
||||
|
||||
if len(self.msg) == 0:
|
||||
return
|
||||
print('writing error message to {}'.format(self.comment_file_path))
|
||||
with open(self.comment_file_path, 'a') as comment_file:
|
||||
text = '\n\n'.join(self.msg)
|
||||
comment_file.write(text)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description='Apply Phabricator patch to working directory.')
|
||||
parser.add_argument('--comment-file', type=str, dest='comment_file_path', default=None)
|
||||
parser.add_argument('--commit', type=str, dest='commit', default=None, help='use explicitly specified base commit')
|
||||
args = parser.parse_args()
|
||||
patcher = ApplyPatch(args.comment_file_path, args.commit)
|
||||
patcher.run()
|
||||
|
|
@ -1,226 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""
|
||||
Interactions with Phabricator.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional, List, Dict
|
||||
import uuid
|
||||
import argparse
|
||||
|
||||
import backoff
|
||||
from phabricator import Phabricator
|
||||
from benedict import benedict
|
||||
|
||||
|
||||
class PhabTalk:
|
||||
"""Talk to Phabricator to upload build results.
|
||||
See https://secure.phabricator.com/conduit/method/harbormaster.sendmessage/
|
||||
You might want to use it as it provides retries on most of the calls.
|
||||
"""
|
||||
|
||||
def __init__(self, token: Optional[str], host: Optional[str] = 'https://reviews.llvm.org/api/',
|
||||
dry_run_updates: bool = False):
|
||||
self._phab = None # type: Optional[Phabricator]
|
||||
self.dry_run_updates = dry_run_updates
|
||||
self._phab = Phabricator(token=token, host=host)
|
||||
self.update_interfaces()
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def update_interfaces(self):
|
||||
self._phab.update_interfaces()
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def get_revision_id(self, diff: str) -> Optional[str]:
|
||||
"""Get the revision ID for a diff from Phabricator."""
|
||||
result = self._phab.differential.querydiffs(ids=[diff])
|
||||
return 'D' + result[diff]['revisionID']
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def get_diff(self, diff_id: int):
|
||||
"""Get a diff from Phabricator based on its diff id."""
|
||||
return self._phab.differential.getdiff(diff_id=diff_id)
|
||||
|
||||
def comment_on_diff(self, diff_id: str, text: str):
|
||||
"""Add a comment to a differential based on the diff_id"""
|
||||
logging.info('Sending comment to diff {}:'.format(diff_id))
|
||||
logging.info(text)
|
||||
revision_id = self.get_revision_id(diff_id)
|
||||
if revision_id is not None:
|
||||
self._comment_on_revision(revision_id, text)
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def _comment_on_revision(self, revision: str, text: str):
|
||||
"""Add comment on a differential based on the revision id."""
|
||||
|
||||
transactions = [{
|
||||
'type': 'comment',
|
||||
'value': text
|
||||
}]
|
||||
|
||||
if self.dry_run_updates:
|
||||
logging.info('differential.revision.edit =================')
|
||||
logging.info('Transactions: {}'.format(transactions))
|
||||
return
|
||||
|
||||
# API details at
|
||||
# https://secure.phabricator.com/conduit/method/differential.revision.edit/
|
||||
self._phab.differential.revision.edit(objectIdentifier=revision,
|
||||
transactions=transactions)
|
||||
logging.info('Uploaded comment to Revision D{}:{}'.format(revision, text))
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def update_build_status(self, phid: str, working: bool, success: bool, lint: {}, unit: []):
|
||||
"""Submit collected report to Phabricator.
|
||||
"""
|
||||
|
||||
result_type = 'pass'
|
||||
if working:
|
||||
result_type = 'working'
|
||||
elif not success:
|
||||
result_type = 'fail'
|
||||
|
||||
# Group lint messages by file and line.
|
||||
lint_messages = []
|
||||
for v in lint.values():
|
||||
path = ''
|
||||
line = 0
|
||||
descriptions = []
|
||||
for e in v:
|
||||
path = e['path']
|
||||
line = e['line']
|
||||
descriptions.append('{}: {}'.format(e['name'], e['description']))
|
||||
lint_message = {
|
||||
'name': 'Pre-merge checks',
|
||||
'severity': 'warning',
|
||||
'code': 'llvm-premerge-checks',
|
||||
'path': path,
|
||||
'line': line,
|
||||
'description': '\n'.join(descriptions),
|
||||
}
|
||||
lint_messages.append(lint_message)
|
||||
|
||||
if self.dry_run_updates:
|
||||
logging.info('harbormaster.sendmessage =================')
|
||||
logging.info('type: {}'.format(result_type))
|
||||
logging.info('unit: {}'.format(unit))
|
||||
logging.info('lint: {}'.format(lint_messages))
|
||||
return
|
||||
|
||||
self._phab.harbormaster.sendmessage(
|
||||
buildTargetPHID=phid,
|
||||
type=result_type,
|
||||
unit=unit,
|
||||
lint=lint_messages)
|
||||
logging.info('Uploaded build status {}, {} test results and {} lint results'.format(
|
||||
result_type, len(unit), len(lint_messages)))
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def create_artifact(self, phid, artifact_key, artifact_type, artifact_data):
|
||||
if self.dry_run_updates:
|
||||
logging.info('harbormaster.createartifact =================')
|
||||
logging.info('artifactKey: {}'.format(artifact_key))
|
||||
logging.info('artifactType: {}'.format(artifact_type))
|
||||
logging.info('artifactData: {}'.format(artifact_data))
|
||||
return
|
||||
self._phab.harbormaster.createartifact(
|
||||
buildTargetPHID=phid,
|
||||
artifactKey=artifact_key,
|
||||
artifactType=artifact_type,
|
||||
artifactData=artifact_data)
|
||||
|
||||
def maybe_add_url_artifact(self, phid: str, url: str, name: str):
|
||||
if self.dry_run_updates:
|
||||
logging.info(f'add ULR artifact "{name}" {url}')
|
||||
return
|
||||
if phid is None:
|
||||
logging.warning('PHID is not provided, cannot create URL artifact')
|
||||
return
|
||||
self.create_artifact(phid, str(uuid.uuid4()), 'uri', {'uri': url, 'ui.external': True, 'name': name})
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def user_projects(self, user_phid: str) -> List[str]:
|
||||
"""Returns slugs of all projects user has a membership."""
|
||||
projects = benedict(self._phab.project.search(constraints={'members': [user_phid]}))
|
||||
slugs = []
|
||||
for p in projects.get('data', []):
|
||||
slug = benedict(p).get('fields.slug')
|
||||
if slug:
|
||||
slugs.append(p['fields']['slug'])
|
||||
return slugs
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def get_revision(self, revision_id: int):
|
||||
"""Get a revision from Phabricator based on its revision id."""
|
||||
return self._phab.differential.query(ids=[revision_id])[0]
|
||||
|
||||
@backoff.on_exception(backoff.expo, Exception, max_tries=5, logger='', factor=3)
|
||||
def get_diff(self, diff_id: int):
|
||||
"""Get a diff from Phabricator based on its diff id."""
|
||||
return self._phab.differential.getdiff(diff_id=diff_id)
|
||||
|
||||
|
||||
class Step:
|
||||
def __init__(self, name=''):
|
||||
self.name = name
|
||||
self.success = True
|
||||
self.duration = 0.0
|
||||
self.reproduce_commands = []
|
||||
|
||||
def set_status_from_exit_code(self, exit_code: int):
|
||||
if exit_code != 0:
|
||||
self.success = False
|
||||
|
||||
|
||||
class Report:
|
||||
def __init__(self):
|
||||
self.os = ''
|
||||
self.name = ''
|
||||
self.comments = []
|
||||
self.success = True
|
||||
self.working = False
|
||||
self.unit = [] # type: List
|
||||
self.lint = {}
|
||||
self.test_stats = {
|
||||
'pass': 0,
|
||||
'fail': 0,
|
||||
'skip': 0
|
||||
} # type: Dict[str, int]
|
||||
self.steps = [] # type: List[Step]
|
||||
self.artifacts = [] # type: List
|
||||
|
||||
def __str__(self):
|
||||
return str(self.__dict__)
|
||||
|
||||
def add_lint(self, m):
|
||||
key = '{}:{}'.format(m['path'], m['line'])
|
||||
if key not in self.lint:
|
||||
self.lint[key] = []
|
||||
self.lint[key].append(m)
|
||||
|
||||
def add_artifact(self, dir: str, file: str, name: str):
|
||||
self.artifacts.append({'dir': dir, 'file': file, 'name': name})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description='Sample interaction with Phabricator')
|
||||
parser.add_argument('--url', type=str, dest='url', default='https://reviews.llvm.org/api/')
|
||||
parser.add_argument('--token', type=str, dest='token', default=None, required=True)
|
||||
parser.add_argument('--diff_id', type=str, dest='diff_id', default=None, required=True)
|
||||
args = parser.parse_args()
|
||||
phabtalk = PhabTalk(args.token, args.url, False)
|
||||
print(phabtalk.get_revision_id(args.diff_id))
|
||||
|
||||
|
|
@ -1,101 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
import logging
|
||||
from buildkite_utils import set_metadata, BuildkiteApi
|
||||
from phabtalk.phabtalk import PhabTalk
|
||||
|
||||
if __name__ == '__main__':
|
||||
diff_id = os.getenv("ph_buildable_diff")
|
||||
revision_id = os.getenv("ph_buildable_revision", '')
|
||||
log_level = os.getenv('ph_log_level', 'INFO')
|
||||
base_commit = os.getenv('ph_base_commit', 'auto')
|
||||
run_build = os.getenv('ph_skip_build') is None
|
||||
trigger = os.getenv('ph_trigger_pipeline')
|
||||
logging.basicConfig(level=log_level, format='%(levelname)-7s %(message)s')
|
||||
|
||||
phabtalk = PhabTalk(os.getenv('CONDUIT_TOKEN'), dry_run_updates=(os.getenv('ph_dry_run_report') is not None))
|
||||
rev = phabtalk.get_revision(int(revision_id))
|
||||
user_id = rev.get('authorPHID')
|
||||
logging.debug(f'authorPHID {user_id}')
|
||||
if user_id is None:
|
||||
logging.error('cannot find author of the revision')
|
||||
sys.exit(1)
|
||||
projects = phabtalk.user_projects(user_id)
|
||||
logging.info(f'user projects: {", ".join(projects)}')
|
||||
# Cancel any existing builds.
|
||||
# Do this before setting own 'ph_buildable_revision'.
|
||||
try:
|
||||
bk = BuildkiteApi(os.getenv("BUILDKITE_API_TOKEN"), os.getenv("BUILDKITE_ORGANIZATION_SLUG"))
|
||||
for b in bk.list_running_revision_builds(os.getenv("BUILDKITE_PIPELINE_SLUG"), os.getenv('ph_buildable_revision')):
|
||||
logging.info(f'cancelling build {b.get("web_url")}')
|
||||
bk.cancel_build(b)
|
||||
except Exception as e:
|
||||
logging.error(e)
|
||||
|
||||
set_metadata('ph_buildable_diff', os.getenv("ph_buildable_diff"))
|
||||
set_metadata('ph_buildable_revision', os.getenv('ph_buildable_revision'))
|
||||
set_metadata('ph_build_id', os.getenv("ph_build_id"))
|
||||
if trigger is None:
|
||||
trigger = 'premerge-checks'
|
||||
|
||||
env = {
|
||||
'ph_scripts_refspec': '${BUILDKITE_BRANCH}',
|
||||
'ph_user_project_slugs': ",".join(projects),
|
||||
# TODO: those two are for "apply_patch.sh". Maybe just look for "ph_" env in patch_diff.py?
|
||||
'LOG_LEVEL': log_level,
|
||||
'BASE_COMMIT': base_commit,
|
||||
}
|
||||
for e in os.environ:
|
||||
if e.startswith('ph_'):
|
||||
env[e] = os.getenv(e)
|
||||
steps = [{
|
||||
'label': 'create branch',
|
||||
'key': 'create-branch',
|
||||
'commands': [
|
||||
'pip install -q -r scripts/requirements.txt',
|
||||
'scripts/apply_patch.sh'
|
||||
],
|
||||
'agents': {'queue': 'service'},
|
||||
'timeout_in_minutes': 20,
|
||||
'env': env
|
||||
}]
|
||||
if run_build:
|
||||
# trigger_build_step = {
|
||||
# 'trigger': trigger,
|
||||
# 'label': ':rocket: build and test',
|
||||
# 'async': False,
|
||||
# 'depends_on': 'create-branch',
|
||||
# 'build': {
|
||||
# 'branch': f'phab-diff-{diff_id}',
|
||||
# 'env': env,
|
||||
# },
|
||||
# }
|
||||
# steps.append(trigger_build_step)
|
||||
steps.append({
|
||||
'trigger': 'phabricator-build',
|
||||
'label': ':rocket: build and test',
|
||||
'async': False,
|
||||
'depends_on': 'create-branch',
|
||||
'build': {
|
||||
'branch': f'phab-diff-{diff_id}',
|
||||
'env': env,
|
||||
},
|
||||
})
|
||||
print(yaml.dump({'steps': steps}))
|
||||
|
|
@ -1,67 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Script runs in checked out llvm-project directory.
|
||||
|
||||
from choose_projects import ChooseProjects
|
||||
from steps import generic_linux, generic_windows, from_shell_output, extend_steps_env, bazel
|
||||
from typing import Dict
|
||||
import git
|
||||
import git_utils
|
||||
import os
|
||||
import yaml
|
||||
|
||||
steps_generators = [
|
||||
'${BUILDKITE_BUILD_CHECKOUT_PATH}/.ci/generate-buildkite-pipeline-scheduled',
|
||||
]
|
||||
|
||||
if __name__ == '__main__':
|
||||
scripts_refspec = os.getenv("ph_scripts_refspec", "main")
|
||||
no_cache = os.getenv('ph_no_cache') is not None
|
||||
log_level = os.getenv('ph_log_level', 'WARNING')
|
||||
notify_emails = list(filter(None, os.getenv('ph_notify_emails', '').split(',')))
|
||||
# Syncing LLVM fork so any pipelines started from upstream llvm-project
|
||||
# but then triggered a build on fork will observe the commit.
|
||||
repo = git_utils.initLlvmFork(os.path.join(os.getenv('BUILDKITE_BUILD_PATH', ''), 'llvm-project-fork'))
|
||||
git_utils.syncRemotes(repo, 'upstream', 'origin')
|
||||
steps = []
|
||||
|
||||
env: Dict[str, str] = {}
|
||||
for e in os.environ:
|
||||
if e.startswith('ph_'):
|
||||
env[e] = os.getenv(e, '')
|
||||
repo = git.Repo('.')
|
||||
|
||||
cp = ChooseProjects(None)
|
||||
linux_projects = cp.get_all_enabled_projects('linux')
|
||||
steps.extend(generic_linux(os.getenv('ph_projects', ';'.join(linux_projects)), check_diff=False))
|
||||
windows_projects = cp.get_all_enabled_projects('windows')
|
||||
steps.extend(generic_windows(os.getenv('ph_projects', ';'.join(windows_projects))))
|
||||
steps.extend(bazel([], force=True))
|
||||
if os.getenv('ph_skip_generated') is None:
|
||||
# BUILDKITE_COMMIT might be an alias, e.g. "HEAD". Resolve it to make the build hermetic.
|
||||
env['BUILDKITE_COMMIT'] = os.getenv('BUILDKITE_COMMIT')
|
||||
if env['BUILDKITE_COMMIT'] == "HEAD":
|
||||
env['BUILDKITE_COMMIT'] = repo.head.commit.hexsha
|
||||
env['BUILDKITE_BRANCH'] = os.getenv('BUILDKITE_BRANCH')
|
||||
env['BUILDKITE_MESSAGE'] = os.getenv('BUILDKITE_MESSAGE')
|
||||
for gen in steps_generators:
|
||||
steps.extend(from_shell_output(gen, env=env))
|
||||
|
||||
notify = []
|
||||
for e in notify_emails:
|
||||
notify.append({'email': e})
|
||||
extend_steps_env(steps, env)
|
||||
print(yaml.dump({'steps': steps, 'notify': notify}))
|
||||
|
|
@ -1,101 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Script runs in checked out llvm-project directory.
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Dict
|
||||
|
||||
from buildkite_utils import annotate, feedback_url, set_metadata
|
||||
from choose_projects import ChooseProjects
|
||||
import git
|
||||
from steps import generic_linux, generic_windows, from_shell_output, checkout_scripts, bazel, extend_steps_env
|
||||
import yaml
|
||||
|
||||
steps_generators = [
|
||||
'${BUILDKITE_BUILD_CHECKOUT_PATH}/.ci/generate-buildkite-pipeline-premerge',
|
||||
]
|
||||
|
||||
if __name__ == '__main__':
|
||||
scripts_refspec = os.getenv("ph_scripts_refspec", "main")
|
||||
diff_id = os.getenv("ph_buildable_diff", "")
|
||||
no_cache = os.getenv('ph_no_cache') is not None
|
||||
projects = os.getenv('ph_projects', 'detect')
|
||||
log_level = os.getenv('ph_log_level', 'INFO')
|
||||
logging.basicConfig(level=log_level, format='%(levelname)-7s %(message)s')
|
||||
|
||||
phid = os.getenv('ph_target_phid')
|
||||
url = f"https://reviews.llvm.org/D{os.getenv('ph_buildable_revision')}?id={diff_id}"
|
||||
annotate(f"Build for [D{os.getenv('ph_buildable_revision')}#{diff_id}]({url}). "
|
||||
f"[Harbormaster build](https://reviews.llvm.org/harbormaster/build/{os.getenv('ph_build_id')}).\n"
|
||||
f"If there is a build infrastructure issue, please [create a bug]({feedback_url()}).")
|
||||
set_metadata('ph_buildable_diff', os.getenv("ph_buildable_diff"))
|
||||
set_metadata('ph_buildable_revision', os.getenv('ph_buildable_revision'))
|
||||
set_metadata('ph_build_id', os.getenv("ph_build_id"))
|
||||
|
||||
env: Dict[str, str] = {}
|
||||
for e in os.environ:
|
||||
if e.startswith('ph_'):
|
||||
env[e] = os.getenv(e, '')
|
||||
repo = git.Repo('.')
|
||||
steps = []
|
||||
# List all affected projects.
|
||||
patch = repo.git.diff("HEAD~1")
|
||||
cp = ChooseProjects('.')
|
||||
|
||||
linux_projects = cp.choose_projects(patch = patch, os_name = "linux")
|
||||
logging.info(f'linux_projects: {linux_projects}')
|
||||
# It's now generated by .ci/generate-buildkite-pipeline-premerge
|
||||
# if len(linux_projects) > 0:
|
||||
# steps.extend(generic_linux(';'.join(linux_projects), check_diff=True))
|
||||
|
||||
windows_projects = cp.choose_projects(patch = patch, os_name = "windows")
|
||||
logging.info(f'windows_projects: {windows_projects}')
|
||||
# Generated by .ci/generate-buildkite-pipeline-premerge.
|
||||
# if len(windows_projects) > 0:
|
||||
# steps.extend(generic_windows(';'.join(windows_projects)))
|
||||
|
||||
# Add custom checks.
|
||||
if os.getenv('ph_skip_generated') is None:
|
||||
if os.getenv('BUILDKITE_COMMIT', 'HEAD') == "HEAD":
|
||||
env['BUILDKITE_COMMIT'] = repo.head.commit.hexsha
|
||||
for gen in steps_generators:
|
||||
steps.extend(from_shell_output(gen, env=env))
|
||||
modified_files = cp.get_changed_files(patch)
|
||||
steps.extend(bazel(modified_files))
|
||||
|
||||
if phid is None:
|
||||
logging.warning('ph_target_phid is not specified. Skipping "Report" step')
|
||||
else:
|
||||
steps.append({
|
||||
'wait': '~',
|
||||
'continue_on_failure': True,
|
||||
})
|
||||
|
||||
report_step = {
|
||||
'label': ':phabricator: update build status on Phabricator',
|
||||
'commands': [
|
||||
*checkout_scripts('linux', scripts_refspec),
|
||||
'$${SRC}/scripts/summary.py',
|
||||
],
|
||||
'artifact_paths': ['artifacts/**/*'],
|
||||
'agents': {'queue': 'service'},
|
||||
'timeout_in_minutes': 10,
|
||||
}
|
||||
steps.append(report_step)
|
||||
|
||||
extend_steps_env(steps, env)
|
||||
print(yaml.dump({'steps': steps}))
|
||||
|
|
@ -1,196 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the the Apache License v2.0 with LLVM Exceptions (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://llvm.org/LICENSE.txt
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Runs all check on buildkite agent.
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from typing import Callable, List, Type
|
||||
import clang_format_report
|
||||
import clang_tidy_report
|
||||
import run_cmake
|
||||
from buildkite_utils import upload_file, annotate, strip_emojis
|
||||
from exec_utils import watch_shell, if_not_matches, tee
|
||||
from phabtalk.phabtalk import Report, PhabTalk, Step
|
||||
|
||||
from choose_projects import ChooseProjects
|
||||
|
||||
|
||||
def ninja_all_report(step: Step, _: Report):
|
||||
step.reproduce_commands.append('ninja all')
|
||||
rc = watch_shell(
|
||||
sys.stdout.buffer.write,
|
||||
sys.stderr.buffer.write,
|
||||
'ninja all', cwd=build_dir)
|
||||
logging.debug(f'ninja all: returned {rc}')
|
||||
step.set_status_from_exit_code(rc)
|
||||
|
||||
|
||||
def ninja_check_all_report(step: Step, _: Report):
|
||||
print('Full log will be available in Artifacts "ninja-check-all.log"', flush=True)
|
||||
step.reproduce_commands.append('ninja check-all')
|
||||
rc = watch_shell(
|
||||
sys.stdout.buffer.write,
|
||||
sys.stderr.buffer.write,
|
||||
'ninja check-all', cwd=build_dir)
|
||||
logging.debug(f'ninja check-all: returned {rc}')
|
||||
step.set_status_from_exit_code(rc)
|
||||
|
||||
def ninja_check_projects_report(step: Step, _: Report, checks: str):
|
||||
print('Full log will be available in Artifacts "ninja-check.log"', flush=True)
|
||||
step.reproduce_commands.append(f'ninja {checks}')
|
||||
rc = watch_shell(
|
||||
sys.stdout.buffer.write,
|
||||
sys.stderr.buffer.write,
|
||||
f'ninja {checks}', cwd=build_dir)
|
||||
logging.debug(f'ninja {checks}: returned {rc}')
|
||||
step.set_status_from_exit_code(rc)
|
||||
|
||||
|
||||
def run_step(name: str, report: Report, thunk: Callable[[Step, Report], None]) -> Step:
|
||||
start = time.time()
|
||||
print(f'--- {name}', flush=True) # New section in Buildkite log.
|
||||
step = Step()
|
||||
step.name = name
|
||||
thunk(step, report)
|
||||
step.duration = time.time() - start
|
||||
# Expand section if step has failed.
|
||||
if not step.success:
|
||||
print('^^^ +++', flush=True)
|
||||
if step.success:
|
||||
annotate(f"{name}: OK")
|
||||
else:
|
||||
annotate(f"{name}: FAILED", style='error')
|
||||
report.success = False
|
||||
report.steps.append(step)
|
||||
return step
|
||||
|
||||
|
||||
def cmake_report(projects: str, step: Step, _: Report):
|
||||
global build_dir
|
||||
cmake_result, build_dir, cmake_artifacts, commands = run_cmake.run(projects, repo_path=os.getcwd())
|
||||
for file in cmake_artifacts:
|
||||
if os.path.exists(file):
|
||||
shutil.copy2(file, artifacts_dir)
|
||||
step.set_status_from_exit_code(cmake_result)
|
||||
step.reproduce_commands = commands
|
||||
|
||||
|
||||
def as_dict(obj):
|
||||
try:
|
||||
return obj.toJSON()
|
||||
except:
|
||||
return obj.__dict__
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='Runs premerge checks8')
|
||||
parser.add_argument('--log-level', type=str, default='WARNING')
|
||||
parser.add_argument('--build-and-test-all', action='store_true',
|
||||
help="Run `ninja all` and `ninja check-all`, instead of "
|
||||
"the default of running `ninja check-{project}`.")
|
||||
parser.add_argument('--check-clang-format', action='store_true')
|
||||
parser.add_argument('--check-clang-tidy', action='store_true')
|
||||
parser.add_argument('--projects', type=str, default='detect',
|
||||
help="Projects to test as a list of projects like 'clang;libc'."
|
||||
" Dependent projects are automatically added to the CMake invocation.")
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=args.log_level, format='%(levelname)-7s %(message)s')
|
||||
|
||||
ctx = strip_emojis(os.getenv('BUILDKITE_LABEL', 'default'))
|
||||
annotate(os.getenv('BUILDKITE_LABEL', 'default'), context=ctx)
|
||||
build_dir = ''
|
||||
step_key = os.getenv("BUILDKITE_STEP_KEY")
|
||||
scripts_dir = pathlib.Path(__file__).parent.absolute()
|
||||
artifacts_dir = os.path.join(os.getcwd(), 'artifacts')
|
||||
os.makedirs(artifacts_dir, exist_ok=True)
|
||||
report_path = f'{step_key}_result.json'
|
||||
report = Report()
|
||||
report.os = f'{os.getenv("BUILDKITE_AGENT_META_DATA_OS")}'
|
||||
report.name = step_key
|
||||
report.success = True
|
||||
|
||||
projects = set(args.projects.split(";"))
|
||||
cp = ChooseProjects(None)
|
||||
dependencies = cp.get_dependencies(projects)
|
||||
logging.info(f"Dependencies: {dependencies}")
|
||||
enabled_projects = ";".join(dependencies.union(projects))
|
||||
|
||||
cmake = run_step('cmake', report, lambda s, r: cmake_report(enabled_projects, s, r))
|
||||
commands_in_build = True
|
||||
if cmake.success:
|
||||
if args.build_and_test_all:
|
||||
ninja_all = run_step('ninja all', report, ninja_all_report)
|
||||
if ninja_all.success:
|
||||
run_step('ninja check-all', report, ninja_check_all_report)
|
||||
else:
|
||||
checks = " ".join(cp.get_check_targets(projects))
|
||||
logging.info(f"Running checks: {checks}")
|
||||
report_lambda: Callable[[Step, Report], None] = lambda s, r: ninja_check_projects_report(s, r, checks)
|
||||
run_step(f"ninja {checks}", report, report_lambda)
|
||||
if args.check_clang_tidy:
|
||||
if commands_in_build:
|
||||
s = Step('')
|
||||
s.reproduce_commands.append('cd ..')
|
||||
commands_in_build = False
|
||||
report.steps.append(s)
|
||||
run_step('clang-tidy', report,
|
||||
lambda s, r: clang_tidy_report.run('HEAD~1', os.path.join(scripts_dir, 'clang-tidy.ignore'), s, r))
|
||||
if args.check_clang_format:
|
||||
if commands_in_build:
|
||||
s = Step('')
|
||||
s.reproduce_commands.append('cd ..')
|
||||
commands_in_build = False
|
||||
report.steps.append(s)
|
||||
run_step('clang-format', report,
|
||||
lambda s, r: clang_format_report.run('HEAD~1', os.path.join(scripts_dir, 'clang-format.ignore'), s, r))
|
||||
logging.debug(report)
|
||||
summary = []
|
||||
summary.append('''
|
||||
<details>
|
||||
<summary>Reproduce build locally</summary>
|
||||
|
||||
```''')
|
||||
summary.append(f'git clone {os.getenv("BUILDKITE_REPO")} llvm-project')
|
||||
summary.append('cd llvm-project')
|
||||
summary.append(f'git checkout {os.getenv("BUILDKITE_COMMIT")}')
|
||||
for s in report.steps:
|
||||
if len(s.reproduce_commands) == 0:
|
||||
continue
|
||||
summary.append('\n'.join(s.reproduce_commands))
|
||||
summary.append('```\n</details>')
|
||||
annotate('\n'.join(summary), style='success')
|
||||
ph_target_phid = os.getenv('ph_target_phid')
|
||||
if ph_target_phid is not None:
|
||||
phabtalk = PhabTalk(os.getenv('CONDUIT_TOKEN'), dry_run_updates=(os.getenv('ph_dry_run_report') is not None))
|
||||
phabtalk.update_build_status(ph_target_phid, True, report.success, report.lint, [])
|
||||
for a in report.artifacts:
|
||||
url = upload_file(a['dir'], a['file'])
|
||||
if url is not None:
|
||||
phabtalk.maybe_add_url_artifact(ph_target_phid, url, f'{a["name"]} ({step_key})')
|
||||
else:
|
||||
logging.warning('ph_target_phid is not specified. Will not update the build status in Phabricator')
|
||||
with open(report_path, 'w') as f:
|
||||
json.dump(report.__dict__, f, default=as_dict)
|
||||
|
||||
if not report.success:
|
||||
print('Build completed with failures', flush=True)
|
||||
exit(1)
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue