1
0
Fork 0
Commit graph

465 commits

Author SHA1 Message Date
Mikhail Goncharov
2e9a9563e1 enable periodic libcxx build 2020-12-15 09:24:49 +01:00
Mikhail Goncharov
268c4bb9b0 remove "pipeline_master"
it was renamed to "pipeline_main"
2020-12-10 11:03:22 +01:00
Mikhail Goncharov
99e43a8d9a rename master to main branch following LLVM 2020-12-10 09:29:24 +01:00
Mikhail Goncharov
12c4c43bb0 fix x86 debian step 2020-11-26 18:32:27 +01:00
Mikhail Goncharov
86226f3146 fix: builds were not marked as failed 2020-11-26 12:28:11 +01:00
Mikhail Goncharov
ad6c67d750 disable libcxx pipeline for periodical builds 2020-11-26 08:56:18 +01:00
Mikhail Goncharov
681fbbe2cf Process results and unit-test output of libcxx
Now "report" step combines result in a uniform way and processes unit test
results XML output. It works for sub-builds only started from the 'premerge'
pipeline, i.e. non-recursive. One downside is that now one has to wait until
all jobs have finished.

- Add instructions to setup python environment

- added option to do full report cycle but not call Phabricator

- use "annotations" to show build status. That lifts the need to filter ninja
  and other output (thus `ph_no_filter_output` param removed) and output
  everything. That is nice as script failures no longer lead to loss of logs.

- improved annotate() usability

- misc fixes
2020-11-25 15:29:50 +01:00
Mikhail Goncharov
515f086ba2 log git apply output in case of error 2020-11-04 09:34:32 +01:00
Mikhail Goncharov
90e7224ba9 improvements to patch process
- improve messaging and add instructions how to fix the patch

- fix application of patch stack. Now base commit and order of patches
  should be correct

- set origin to llvm-project fork to avoid accidental operations on origin

- use annotations to make issues visible

- create commits with original author of change

- add patches we tried to apply to artifacts

- patching / commits works locally if --push-branch=false is set

- misc renames / wordings
2020-11-03 16:15:36 +01:00
Mikhail Goncharov
43fae020f5 retry most of the requests with backoff
fixes #192

additionally:

- install python libs after scripts checkout, so we don't need to
rebuild images and restart agents only to add a new python dependency

- updated lib versions

- similar scripts checkout in steps
2020-10-12 17:08:34 +02:00
Mikhail Goncharov
322cfbda8c Add link to the build in "setup" step 2020-10-09 12:51:32 +02:00
Mikhail Goncharov
3df81c372f Do not run generic steps if project list is empty 2020-10-09 12:35:32 +02:00
Mikhail Goncharov
3925a2a844 get libcxx steps from sh output 2020-10-09 11:06:36 +02:00
Mikhail Goncharov
ab1b8d5ae2 Use buildkite API instead of passing stat via files
Previous approach had drawbacks:

- every step had to implement exporting of results in fixed format

- if step failed then failure will not be detected

Now report step will fetch results directly from Buildkite.
Agents have to be updated to have BUILDKITE_API_TOKEN env.
2020-10-08 14:40:57 +02:00
Mikhail Goncharov
563d6ba723 add backoff pip package 2020-10-08 13:48:44 +02:00
Mikhail Goncharov
1bbd71f712 Fix project exclusion rules
Previously list of projects was resolved when job run at target OS.
Now project resolution is moved to the premerge pipeline and logic
should be there.
2020-10-08 12:08:57 +02:00
Mikhail Goncharov
ea943424f3 Enable windows builds back
Now we run a pre-commit hook that tries to kill any process that locked
a file in target directory.

Updated timeout interval on windows from 120 to 90 minutes.

Fixes #243
2020-10-07 16:59:37 +02:00
Mikhail Goncharov
5cd16dde45 Don't fail if phabricator target is not specified
That is useful when testing pipelines
2020-10-05 13:33:19 +02:00
Mikhail Goncharov
c350101a23 Allow projects to define custom steps
Now it's possible to allow sub-projects to define own checks and skip
"generic" ones.

To properly accomodate affected projects that might not have special
treatment we:

1. extend the set of affected projecs with dependent (e.g. add 'libc' if
'clang' was modified)

2. add custom steps for projects that define own workflow. At the moment
it's only libcxx and it has a custom trigger pipeline so it's noop.

3. add dependent projects and run generic check on them.

To illustrate: imagine that we have a dependency graph:

llvm -> clang -> openmp

and only clang was modified in a diff; also clang defines own checks.
Thus list of affected projects will be [clang, openmp].
After adding custom checks and removing their projecst: [openmp].
After adding dependencies: [llvm, clang, openmp]. Generic linux /
windows checks will be run on thouse 3 projects.
So as you can see in some scenarios projects with custom checks will
still go through generic checks.

Note that clang-format and clang-tidy checks are run only for "generic"
checks at the moment.
2020-10-02 16:19:57 +02:00
Mikhail Goncharov
c082f1aa9b Organize scripts
Move all .py that are supposed to be run as __main__ under ./scripts
so there is no need to manipulate sys.path to import modules.

Runby cleanup
2020-10-02 14:18:22 +02:00
Mikhail Goncharov
9c383b2caf Fix refspec selection for the report step 2020-10-02 10:17:00 +02:00
Mikhail Goncharov
19974fd1cf Trigger libcxx pipeline if it's affected 2020-09-30 14:20:28 +02:00
Louis Dionne
fbdc3d9ecc Include the Phabricator review ID in commit messages
This allows jobs that trigger on these branches being created to
access this information.
2020-09-30 12:58:40 +02:00
Mikhail Goncharov
405643a089 Allow other refspecs to be used.
That gives us a way to test e.g. pull requests by specifying
ph_scripts_refspec="pull/123/head"

Also keep "ph_" prefix to remove confusion we had between
"scripts_branch" and "ph_scripts_branch" before.

Update docs and moved "testing" sections up.
2020-09-29 17:57:33 +02:00
Mikhail Goncharov
8bd8cdbdd1 disable win build on master too 2020-09-28 12:19:03 +00:00
Mikhail Goncharov
e4e63d9947 temporary disable windows step 2020-09-28 08:38:19 +00:00
Mikhail Goncharov
24336ae2e4 don't fail if branch cannot be deleted 2020-09-25 13:59:03 +00:00
Mikhail Goncharov
33c5072489 pass phid to applyPatch 2020-09-25 13:49:16 +00:00
Louis Dionne
3a0eda3819 Add the Phabricator ID to the commit messages
This allows parsing the commit message to get the Phabricator ID when
triggering jobs from unrelated pipelines.
2020-09-25 12:13:16 +00:00
apink
606f209bf2 add some platform-specific asan files to clang-tidy.ignore list
some compiler-rt files contain architecture-specific assembly code or
complex macroses which clang-tidy can't process properly.

In case of assembly code: we can get "unknown register name 'a7' in asm"
In sanitizer_common_syscalls.inc we have issuels like:
  "clang-tidy: error: unknown type name '__sanitizer_iovec'"
2020-09-13 12:11:55 +02:00
Mikhail Goncharov
8b36ff5888 remove jenkins configs and scrips
- moved linux agents to a new node pool (basically renamed old one)

- removed some out of date scripts or moved them to playlists

- removed shell scrips invoked from jenkins

- minor docs updates

Some python classes methods are not used anymore and will be cleaned up
separately.
2020-09-01 09:51:49 +02:00
Mikhail Goncharov
46c9fefb77 rename CCACHE_PATH to CCACHE_DIR
https://ccache.dev/manual/3.7.11.html#_configuration uses CCACHE_DIR to set
cache directory. CCACHE_PATH has a different meaning.

+ removed LLVM_CCACHE_DIR ccache argument to use CCACHE_DIR from env.
+ print additional info about ccache config and machine readable stats.
2020-08-26 14:22:37 +02:00
Mikhail Goncharov
dc1080fc4c
Update premerge_checks.py
use specific commit for reproducing the build
2020-08-14 15:54:11 +02:00
Mikhail Goncharov
a410c28d03
Update premerge_checks.py
add cd after clone
2020-08-14 15:17:21 +02:00
Christian Kühnel
afb8c84bde added more phabricator metrics 2020-08-13 13:14:54 +02:00
Christian Kühnel
5da900df30 added DIA support for windows image 2020-08-12 22:31:07 +02:00
Mikhail Goncharov
f3bf784fad fix no_cache flag on windows 2020-08-12 19:28:09 +02:00
Mikhail Goncharov
5e85ee2da8 typo 2020-08-12 18:58:05 +02:00
Christian Kühnel
0cb0fd670e extended metrics for phabricator 2020-08-12 13:42:14 +02:00
Christian Kühnel
e5a7ae4b36 improved stability of data download 2020-08-07 17:18:22 +02:00
Christian Kühnel
c75649476d simple report on master failues 2020-08-07 10:11:31 +02:00
Christian Sigg
36808fc0d0
No clang-tidy under mlir-{cuda,rocm}-runner. (#226)
All files include CUDA/ROCm headers, which trips up clang-tidy without --cuda-path/--rocm-path.
2020-08-03 17:15:58 +02:00
Christian Sigg
e5be8d45ca clang-tidy ignore {cuda,rocm}-runtime-wrappers.cpp
Both files include GPU header files for which clang implicitly adds include directories. However, clang-tidy does not and therefore we get compile errors (i.e. not diagnostics that we could disable with NOLINT).
2020-07-30 18:14:07 +02:00
Mikhail Goncharov
cb41bb38dd reset before checking out master 2020-07-30 09:32:50 +02:00
Mikhail Goncharov
d75dcdd2fb cleanup before checking out master 2020-07-30 09:27:05 +02:00
Mikhail Goncharov
9cfdbbb8fe fix git clenup 2020-07-30 09:20:19 +02:00
Mikhail Goncharov
e41afba72d update docs 2020-07-27 19:16:33 +02:00
Mikhail Goncharov
3c28129968 custom agent constraints
+ updated image for win machines
2020-07-24 15:59:00 +02:00
Mikhail Goncharov
fb5e91fdeb fix auto revision usage 2020-07-24 10:10:48 +02:00
Mikhail Goncharov
5b07569602 fix auto revision usage 2020-07-24 10:06:04 +02:00
Mikhail Goncharov
57702114b0 fix fallback to master branch 2020-07-24 08:58:51 +02:00
Mikhail Goncharov
2932ea50e3 handle broader range or git exceptions 2020-07-23 21:49:44 +02:00
Mikhail Goncharov
f99e98e434 use logging in apply_patch2
+ possible fix for base commit

+ ph_ options to control commit behavior
2020-07-23 20:24:02 +02:00
Mikhail Goncharov
ac3cdef23d Add clang-tidy and clang-format command to "reproduce locally" 2020-07-23 09:50:06 +02:00
Mikhail Goncharov
00a2c76d0b
Merge pull request #219 from google/ignore-tests-format
ignore **/test for clang-format
2020-07-22 19:34:34 +02:00
Mikhail Goncharov
3d1d7af150
Merge pull request #218 from google/notify
Notify
2020-07-22 19:22:52 +02:00
Mikhail Goncharov
5362bc52a3 ignore **/test for clang-format 2020-07-22 18:59:47 +02:00
Mikhail Goncharov
82d0ae033b fix script branch commit message 2020-07-22 17:04:02 +02:00
Mikhail Goncharov
3ababb3b88 notify-email 2020-07-22 16:08:52 +02:00
Mikhail Goncharov
605f18bad1 Add link to ninja log instead of XML #216
XML tooks too long to load and not that helpful

+ link tell how many tests have failed

+ link is added only if something is off
2020-07-22 15:45:53 +02:00
Mikhail Goncharov
fa11647670 retry build step if agent is gone or shut down 2020-07-21 10:42:46 +02:00
Mikhail Goncharov
d4e5126090 Update docs on ssd disk initialization
+ change number of linux agents

+ removed outputs in job analysis

+ updated windows image
2020-07-21 10:40:26 +02:00
Mikhail Goncharov
42d7237491 Add step to schedule agent restart on windows 2020-07-17 18:37:29 +02:00
Mikhail Goncharov
95cdccb68f ipython notebook to analyze build speed per agent
+ add ssd as an option to windows agent bootstrap
2020-07-16 18:04:09 +02:00
Mikhail Goncharov
304f2e88d5 Additional logging for project selection
+ option to skip win / linux build from diff checks
2020-07-14 18:26:53 +02:00
Mikhail Goncharov
89d294ef42 return exit status of ninja command
- Fix for #207. Previously grep was exiting with 1 and fail build if all
  lines from ninja are filtered. Also piping might hide exit code of the
  initial command. Now "tee" and "grep" is implemented on python level
  and we get exit code of the original command.

- Fixed default value of "scripts_branch", print scripts commit;

- Added option to filter ninja output (on by default as before).

- Copied cache usege reporting and error handling from "master" branch
  build to diff checks (useful for debugging).
2020-07-10 19:35:29 +02:00
Mikhail Goncharov
6bf3c85cc6 Print commands to reproduce build locally
- additional arguments to set enabled projects, log level and reset
cache.
2020-07-09 15:31:34 +02:00
Mikhail Goncharov
7667fc8004 Allow "building" in ninja all log
That is a temporary workaround for #207 that caused by grep exiting with
1 as whole `ninja all` output is filtered out.
2020-07-09 15:09:58 +02:00
Mikhail Goncharov
0e36e0cf7c Disable openmp build on Windows until it fixed 2020-07-09 11:00:47 +02:00
Mikhail Goncharov
3f52fb7969 Show stats on linux / windows 2020-07-08 14:32:59 +02:00
Mikhail Goncharov
f125728908 remove compiler-rt and flang temporarily 2020-07-08 12:01:17 +02:00
Mikhail Goncharov
1fa24a55ec Add option to disable cache 2020-07-08 09:45:22 +02:00
Mikhail Goncharov
1421dc1184
Merge pull request #199 from joker-eph/build_examples
Enable -DLLVM_BUILD_EXAMPLES=ON during pre-merge testing
2020-07-08 09:09:53 +02:00
Mikhail Goncharov
517f0309dd
Merge pull request #202 from fooishbar/clang-tidy-ignore-libclc
clang-tidy: Ignore all of libclc source
2020-07-07 15:48:34 +02:00
Mikhail Goncharov
efaa8b73e6 fix build command on windows 2020-07-07 15:13:43 +02:00
Mikhail Goncharov
23fecd6765 additional projects for master branch builds 2020-07-01 09:31:13 +02:00
Mikhail Goncharov
ffa1882445 Specify timeout for every step 2020-06-29 12:17:03 +02:00
Mikhail Goncharov
af7204418f error handling 2020-06-29 11:41:48 +02:00
Mikhail Goncharov
9a2a236dd0 log text updates 2020-06-29 11:14:47 +02:00
Mikhail Goncharov
6fb2c53be8 cleanup script logging and failure count fix 2020-06-29 11:07:05 +02:00
Mikhail Goncharov
ac3c287766 chmod +x cleanup_branches 2020-06-29 10:56:21 +02:00
Mikhail Goncharov
4241d6b53e Doc update 2020-06-29 10:56:21 +02:00
Mikhail Goncharov
5205b53fbf add directory 2020-06-18 11:54:28 +02:00
Mikhail Goncharov
fb830daebe Update clang version
apt.llvm.org signature has changed, we also should use debian:stable
(=buster) as a base image to correctly install packages.

Removed jdk from the base image as it was used only by Jenkins.

Log packages version for every build.

Related issues: #112
2020-06-18 10:55:15 +02:00
Mikhail Goncharov
89e3f73689 check out fork into different directory 2020-06-17 16:09:10 +02:00
Mikhail Goncharov
24da609a10 Flush stdout to fix log update
for #201
2020-06-17 15:07:19 +02:00
Mikhail Goncharov
9e3995bbe0 get queue prefix from URL params 2020-06-17 13:52:33 +02:00
Daniel Stone
d26f5ef00e clang-tidy: Ignore all of libclc source
libclc implements OpenCL C, which is not actual C/C++, and implements an
external API with different naming conventions.

As clang-tidy cannot parse the source[0], just blacklist all of libclc's
sources for now.

[0]: https://buildkite.com/llvm-project/premerge-checks/builds/695
2020-06-17 12:00:26 +01:00
Mikhail Goncharov
ffaba986c1 Deployment changes
do not restart win docker as it makes it impossible to stop agent from
buildkite UI for maintenance
2020-06-17 09:46:42 +02:00
Mikhail Goncharov
af7bd86875 disable clang format / tidy for master and release builds 2020-06-16 17:29:04 +02:00
Mikhail Goncharov
4c3cd1b6f5 Use os in queue name
That is useful for metrics collection as buildkite metrics only reports
per queue metrics
2020-06-16 10:30:24 +02:00
Mehdi Amini
655f87d98c Enable -DLLVM_BUILD_EXAMPLES=ON during pre-merge testing 2020-06-15 19:16:39 +00:00
Mikhail Goncharov
6812950908 Report build error if failed to apply patch 2020-06-05 12:32:09 +02:00
Mikhail Goncharov
de0d5afd78 Move agents to default queue
- fix windows defender exclusion path
2020-06-05 10:50:42 +02:00
Mikhail Goncharov
82daa321a2 Don't use all projects for master / release builds 2020-06-04 14:10:08 +02:00
Mikhail Goncharov
e08efbcf2a Script updates:
- option not to shut down sccache after 10 minutes
- zero sccache stats before running the build
- decrease number of jenkins agents
2020-06-04 12:36:46 +02:00
Mikhail Goncharov
3170572590 don't report back to pahabricator in master build 2020-06-04 08:47:46 +02:00
Mikhail Goncharov
770f6726bc Add new pipeline to build a master branch 2020-06-04 08:40:04 +02:00
Mikhail Goncharov
ac3e35d965 fix string format 2020-06-03 17:21:49 +02:00
Mikhail Goncharov
017ca44a75 Add windows build
- configured sccache on buildkite windows machines
- final result is sent to phabricator from "summary.py" that waits for both builds to complete
- extracted "add_url_artifact" to a runnable script
- reorganized code and fixed some of TODOs
2020-06-03 13:40:22 +02:00