revi-archive/llvm-premerge-checks
1
0
Fork 0
Code Activity
llvm-premerge-checks/scripts/metrics
History
Mikhail Goncharov c350101a23 Allow projects to define custom steps 
Now it's possible to allow sub-projects to define own checks and skip
"generic" ones.

To properly accomodate affected projects that might not have special
treatment we:

1. extend the set of affected projecs with dependent (e.g. add 'libc' if
'clang' was modified)

2. add custom steps for projects that define own workflow. At the moment
it's only libcxx and it has a custom trigger pipeline so it's noop.

3. add dependent projects and run generic check on them.

To illustrate: imagine that we have a dependency graph:

llvm -> clang -> openmp

and only clang was modified in a diff; also clang defines own checks.
Thus list of affected projects will be [clang, openmp].
After adding custom checks and removing their projecst: [openmp].
After adding dependencies: [llvm, clang, openmp]. Generic linux /
windows checks will be run on thouse 3 projects.
So as you can see in some scenarios projects with custom checks will
still go through generic checks.

Note that clang-format and clang-tidy checks are run only for "generic"
checks at the moment.
2020-10-02 16:19:57 +02:00
..
.gitignore script to get statistics from git repo 2020-02-21 13:19:11 +01:00
analyze_jobs.ipynb Allow projects to define custom steps 2020-10-02 16:19:57 +02:00
buildbot_status_emails.py creating tmp dir as needed 2020-05-05 16:54:08 +02:00
buildbots.py fixed initialisation 2020-02-17 14:55:10 +01:00
buildkite_master_stats.py simple report on master failues 2020-08-07 10:11:31 +02:00
jenkins.py fixed report writing 2020-05-06 17:24:26 +02:00
pull_phab_build_stats.py added more phabricator metrics 2020-08-13 13:14:54 +02:00
README.md counting failed builds on buildbots 2020-02-17 14:27:59 +01:00
repo.py code cleanup 2020-02-20 19:04:34 +01:00
repo_hist.py added raw data export 2020-03-19 09:44:41 +01:00

README.md

Metrics

To measure the impact and usefulness of the pre-merge checks, we want to collect a set of metrics. This doc will summarize the metrics and tools. All of the data shall be collected as time series, so that we can see changes over time.

  • Impact - The metrics we ultimately want to improve

    • Percentage of build-bot build on master failing. (Buildbot_percentage_failing)
    • Time to fix a broken master build: Time between start of failing builds until the build is fixed. (BuildBot_time_to_fix)
    • Percentage of Revisions on Phabricator where a broken build was fixed afterwards. This would indicate that a bug was found and fixed during the code review phase. (Premerge_fixes)
    • Number of reverts on master. This indicates that something was broken on master that slipped through the pre-merge tests or was submitted without any review. (Upstream_reverts)

  • Users and behavior - Interesting to see and useful to adapt our approach.

    • Percentage of commits to master that went through Phabricator.
    • Number of participants in pre-merge tests.
    • Percentage of Revisions with pre-merge tests executed
    • Number of 30-day active committers on master and Phabricator.

  • Builds - See how the infrastructure is doing.

    • Time between upload of diff until build results available.
    • Percentage of Revisions with successful/failed tests
    • Number of pre-merge builds/day.
    • Build queuing time.
    • Individual times for cmake, ninja all, ninja check-all per OS/architecture.
    • Result storage size.
    • Percentage of builds failing.

Requirements

  • Must:
    • Do not collect/store personal data.
  • Should:
    • Minimize the amount of additional tools/scripts we need to maintain.
    • Collect all metrics in a central location for easy evaluation (e.g. database, CSV files).
  • Nice to have:
    • As the data is from an open source project and available anyway, give public access to the metrics (numbers and charts).
    • Send out alerts/notifications.
    • Show live data in charts.

Data sources

This section will explain where we can get the data from.

  • build bot statistics

Solution

We need to find solutions for these parts:

  • Collect the data (regularly).
  • Store the time series somewhere.
  • Create & display charts.

Some ideas for this:

  • bunch of scripts:
    • Run a bunch of scripts manually to generate the metrics every now and then. Phabricator already has a database and most entries there have timestamps. So we could also reconstruct the history from that.
    • TODO: Figure out if we can collect the most important metrics this way. This requires that we can reconstruct historic values from the current logs/git/database/... entries.
  • Jenkins + CSV + Sheets:
    • collect data with jenkins
    • store numbers as CSV in this repo
    • Charts are created manually on Google Sheets
  • do it yourself:
    • Collect data with Jenkins jobs
    • Store the data on Prometheus
    • Visualize with Grafana
    • host all tools ourselves
  • Stackdriver on GCP:
    • TODO: figure out if we can get all the required data into Stackdriver
  • Jupyter notebooks:
    • TODO: figure out how that works