1
0
Fork 0
Commit graph

408 commits

Author SHA1 Message Date
Mikhail Goncharov
82d0ae033b fix script branch commit message 2020-07-22 17:04:02 +02:00
Mikhail Goncharov
3ababb3b88 notify-email 2020-07-22 16:08:52 +02:00
Mikhail Goncharov
605f18bad1 Add link to ninja log instead of XML #216
XML tooks too long to load and not that helpful

+ link tell how many tests have failed

+ link is added only if something is off
2020-07-22 15:45:53 +02:00
Mikhail Goncharov
fa11647670 retry build step if agent is gone or shut down 2020-07-21 10:42:46 +02:00
Mikhail Goncharov
d4e5126090 Update docs on ssd disk initialization
+ change number of linux agents

+ removed outputs in job analysis

+ updated windows image
2020-07-21 10:40:26 +02:00
Mikhail Goncharov
42d7237491 Add step to schedule agent restart on windows 2020-07-17 18:37:29 +02:00
Mikhail Goncharov
95cdccb68f ipython notebook to analyze build speed per agent
+ add ssd as an option to windows agent bootstrap
2020-07-16 18:04:09 +02:00
Mikhail Goncharov
304f2e88d5 Additional logging for project selection
+ option to skip win / linux build from diff checks
2020-07-14 18:26:53 +02:00
Mikhail Goncharov
89d294ef42 return exit status of ninja command
- Fix for #207. Previously grep was exiting with 1 and fail build if all
  lines from ninja are filtered. Also piping might hide exit code of the
  initial command. Now "tee" and "grep" is implemented on python level
  and we get exit code of the original command.

- Fixed default value of "scripts_branch", print scripts commit;

- Added option to filter ninja output (on by default as before).

- Copied cache usege reporting and error handling from "master" branch
  build to diff checks (useful for debugging).
2020-07-10 19:35:29 +02:00
Mikhail Goncharov
6bf3c85cc6 Print commands to reproduce build locally
- additional arguments to set enabled projects, log level and reset
cache.
2020-07-09 15:31:34 +02:00
Mikhail Goncharov
7667fc8004 Allow "building" in ninja all log
That is a temporary workaround for #207 that caused by grep exiting with
1 as whole `ninja all` output is filtered out.
2020-07-09 15:09:58 +02:00
Mikhail Goncharov
0e36e0cf7c Disable openmp build on Windows until it fixed 2020-07-09 11:00:47 +02:00
Mikhail Goncharov
3f52fb7969 Show stats on linux / windows 2020-07-08 14:32:59 +02:00
Mikhail Goncharov
f125728908 remove compiler-rt and flang temporarily 2020-07-08 12:01:17 +02:00
Mikhail Goncharov
1fa24a55ec Add option to disable cache 2020-07-08 09:45:22 +02:00
Mikhail Goncharov
1421dc1184
Merge pull request #199 from joker-eph/build_examples
Enable -DLLVM_BUILD_EXAMPLES=ON during pre-merge testing
2020-07-08 09:09:53 +02:00
Mikhail Goncharov
517f0309dd
Merge pull request #202 from fooishbar/clang-tidy-ignore-libclc
clang-tidy: Ignore all of libclc source
2020-07-07 15:48:34 +02:00
Mikhail Goncharov
efaa8b73e6 fix build command on windows 2020-07-07 15:13:43 +02:00
Mikhail Goncharov
23fecd6765 additional projects for master branch builds 2020-07-01 09:31:13 +02:00
Mikhail Goncharov
ffa1882445 Specify timeout for every step 2020-06-29 12:17:03 +02:00
Mikhail Goncharov
af7204418f error handling 2020-06-29 11:41:48 +02:00
Mikhail Goncharov
9a2a236dd0 log text updates 2020-06-29 11:14:47 +02:00
Mikhail Goncharov
6fb2c53be8 cleanup script logging and failure count fix 2020-06-29 11:07:05 +02:00
Mikhail Goncharov
ac3c287766 chmod +x cleanup_branches 2020-06-29 10:56:21 +02:00
Mikhail Goncharov
4241d6b53e Doc update 2020-06-29 10:56:21 +02:00
Mikhail Goncharov
5205b53fbf add directory 2020-06-18 11:54:28 +02:00
Mikhail Goncharov
fb830daebe Update clang version
apt.llvm.org signature has changed, we also should use debian:stable
(=buster) as a base image to correctly install packages.

Removed jdk from the base image as it was used only by Jenkins.

Log packages version for every build.

Related issues: #112
2020-06-18 10:55:15 +02:00
Mikhail Goncharov
89e3f73689 check out fork into different directory 2020-06-17 16:09:10 +02:00
Mikhail Goncharov
24da609a10 Flush stdout to fix log update
for #201
2020-06-17 15:07:19 +02:00
Mikhail Goncharov
9e3995bbe0 get queue prefix from URL params 2020-06-17 13:52:33 +02:00
Daniel Stone
d26f5ef00e clang-tidy: Ignore all of libclc source
libclc implements OpenCL C, which is not actual C/C++, and implements an
external API with different naming conventions.

As clang-tidy cannot parse the source[0], just blacklist all of libclc's
sources for now.

[0]: https://buildkite.com/llvm-project/premerge-checks/builds/695
2020-06-17 12:00:26 +01:00
Mikhail Goncharov
ffaba986c1 Deployment changes
do not restart win docker as it makes it impossible to stop agent from
buildkite UI for maintenance
2020-06-17 09:46:42 +02:00
Mikhail Goncharov
af7bd86875 disable clang format / tidy for master and release builds 2020-06-16 17:29:04 +02:00
Mikhail Goncharov
4c3cd1b6f5 Use os in queue name
That is useful for metrics collection as buildkite metrics only reports
per queue metrics
2020-06-16 10:30:24 +02:00
Mehdi Amini
655f87d98c Enable -DLLVM_BUILD_EXAMPLES=ON during pre-merge testing 2020-06-15 19:16:39 +00:00
Mikhail Goncharov
6812950908 Report build error if failed to apply patch 2020-06-05 12:32:09 +02:00
Mikhail Goncharov
de0d5afd78 Move agents to default queue
- fix windows defender exclusion path
2020-06-05 10:50:42 +02:00
Mikhail Goncharov
82daa321a2 Don't use all projects for master / release builds 2020-06-04 14:10:08 +02:00
Mikhail Goncharov
e08efbcf2a Script updates:
- option not to shut down sccache after 10 minutes
- zero sccache stats before running the build
- decrease number of jenkins agents
2020-06-04 12:36:46 +02:00
Mikhail Goncharov
3170572590 don't report back to pahabricator in master build 2020-06-04 08:47:46 +02:00
Mikhail Goncharov
770f6726bc Add new pipeline to build a master branch 2020-06-04 08:40:04 +02:00
Mikhail Goncharov
ac3e35d965 fix string format 2020-06-03 17:21:49 +02:00
Mikhail Goncharov
017ca44a75 Add windows build
- configured sccache on buildkite windows machines
- final result is sent to phabricator from "summary.py" that waits for both builds to complete
- extracted "add_url_artifact" to a runnable script
- reorganized code and fixed some of TODOs
2020-06-03 13:40:22 +02:00
Christian Kühnel
654b6e5da8 ignoring mlir/examples/standalone in clang-tidy 2020-05-29 08:00:23 +02:00
Mikhail Goncharov
afed3a7898 Update docs and script for installing win machines
I have tried to install new windows machine for buildkite and updated
some scrips and docs on along the way.

- Updated base image for k8s agent installation as it gave a warning that
previous version was old.

- Now buildkite secret is configured in a powershell script along with
tags (and possible other parameters).

- Split "windows_agent_start.ps1" to "..jenkins" and "..buildkite" as
  some parameters are different.

- Created a "windows-development" machine in GCP stat can be resumed
  to build docker images / expriment.
2020-05-27 15:41:49 +02:00
Mikhail Goncharov
d796cfec6d fix typo 2020-05-26 12:54:31 +02:00
Mikhail Goncharov
72156ea180
Merge pull request #188 from google/bk-build-2
building with buildkite
2020-05-26 10:42:54 +02:00
Mikhail Goncharov
8eee5355ea building with buildkite 2020-05-25 16:42:40 +02:00
Christian Kühnel
856ce4b148 added missing dependencies to libc 2020-05-20 06:10:28 +02:00
Mikhail Goncharov
1505cfb635 build 2020-05-14 12:57:38 +02:00
Mikhail Goncharov
a152d97e3c Simple service to integrate harbormaster w/ buildkite
Harbormaster can send a request in url encoded form while buildkite
expects a POST request with json. Instead of modifying harbormastar or
buildkite this adds a simple proxy that accepts url encoded form and
creates a request that buildkite expects. To avoid potential abuse,
nginx asks for simple http auth credentials stored in harbormaster.

All build parameters passed by proxy are put into build metadata and as
ph_* env variables available during the build.

Secrets involved:
- harbormastert knows http-auth to proxy (stored in privatly and in k8
  buildkite/http-auth as auth file);
- proxy knows buildkite api token (mine atm);
- build agent knows conduit API token (mine atm),
  and SSH key of llvm-premerge-tests-bot (in k8 buildkite/github-ssh).

Sample build: https://reviews.llvm.org/harbormaster/build/64828/8/
https://buildkite.com/llvm-project/premerge/builds/48
2020-05-13 13:13:33 +02:00
Christian Kühnel
169744c617 extended log scan preiod to 90 days
fixes #181
2020-05-12 08:46:24 +02:00
Christian Kühnel
514a87aa4f added missing dependency 2020-05-11 16:55:20 +02:00
Christian Kühnel
2264ff6c41 fix issue as 'all' can't be built 2020-05-11 16:09:27 +02:00
Christian Kühnel
07404db9a2 disabling lldb again 2020-05-11 09:18:57 +02:00
Christian Kühnel
da722d3e40 continuing after failures 2020-05-08 16:24:16 +02:00
Christian Kühnel
510ed904f8 adding flush after every print
so it appears in the Jenkins logs
2020-05-08 15:36:13 +02:00
Christian Kühnel
8ea2a3b846 using different way for deleting remote branches 2020-05-08 15:33:45 +02:00
Christian Kühnel
d48ff97d0d only push if we deleted something 2020-05-08 15:27:09 +02:00
Christian Kühnel
45b28dd0df first version of repo cleanup script 2020-05-08 15:23:10 +02:00
Christian Kühnel
4c48379707 improved windows agent creation 2020-05-08 13:19:04 +02:00
Christian Kühnel
f9a9fb2be0 lldb tests were fixed, re-enabling it 2020-05-07 17:27:07 +02:00
Christian Kühnel
795d8270f7 fixed report writing 2020-05-06 17:24:26 +02:00
Christian Kühnel
10454dfba8 improved error handling for agent identification 2020-05-06 13:11:28 +02:00
Christian Kühnel
0e12d53def added agent detection 2020-05-06 12:43:35 +02:00
Christian Kühnel
971d4b8b74 fixes #175 2020-05-06 12:41:58 +02:00
Christian Kühnel
d55096bb7c getting workflow stages 2020-05-06 12:06:40 +02:00
Christian Kühnel
27da151340 creating tmp dir as needed 2020-05-05 16:54:08 +02:00
Christian Kühnel
9fabda9800 wiping sccache on timeouts 2020-05-04 14:22:49 +00:00
Christian Kühnel
b819a22205 added --dryrun 2020-05-04 14:16:50 +00:00
Christian Kühnel
5fd70b614d disabled lldb because of failing tests
see https://results.llvm-merge-guard.org/master_debian_clang8-2152/console-log.txt
2020-04-29 12:17:52 +02:00
Christian Kühnel
9841a13bbf re-enabled lldb
after failing test was fixed
2020-04-29 11:46:26 +02:00
Mikhail Goncharov
39667898a7 use env variables 2020-04-24 15:43:09 +02:00
Mikhail Goncharov
1a3952afc4
Merge pull request #171 from google/run-bootstrap-from-scripts
Checkout scripts and define pipeline in python
2020-04-24 14:48:09 +02:00
Mikhail Goncharov
e860c29a47 Checkout scripts and define pipeline in python 2020-04-24 14:47:00 +02:00
Christian Kühnel
9db1fd9654 disabled compiler-rt 2020-04-24 13:56:28 +02:00
Christian Kühnel
cd7f06c58d removed lldb 2020-04-24 11:59:37 +02:00
Christian Kühnel
6e2d4d612a disabled lldb
test keeps failing, problem with lack of IPv6 networking on Kubernetes:
lldb-unit :: Host/./HostTests/ConnectionFileDescriptorTest.TCPGetURIv6
2020-04-24 09:29:56 +02:00
Christian Kühnel
184d216b24 Flush stdout
So that the log output appears in the Jenkins logs before the cmake output.
2020-04-23 15:07:54 +02:00
Christian Kühnel
58ce45a053 Revert "added -DCMAKE_BUILD_TYPE=RelWithDebInfo for Windows builds"
This reverts commit 963a5abc7f.
2020-04-23 11:45:52 +02:00
Christian Kühnel
d1a1896933 excluding flang on Windows 2020-04-23 11:40:40 +02:00
Christian Kühnel
963a5abc7f added -DCMAKE_BUILD_TYPE=RelWithDebInfo for Windows builds 2020-04-23 11:31:38 +02:00
Christian Kühnel
71144d1922 By default: building all enabled projects.
This is the fallback solution in case the project detection fails for
whatever reason. This is also used for building the master branches.
2020-04-23 11:22:31 +02:00
ChristianKuehnel
a47751ebe8
Upgrade Windows agents to Visual Studio 2019 and sccache (#162)
* added cmake parameters for sccache

* added sccache

* removed stray debug output

* starting sccache with VS environment

* added container for Visual Studio 2019

* fixed comments

* considering WIndows version

* updated to vs2019

* using MS install method

* snapshot of VS2019 experiments

* using --installRecommended

* cleanup of dockerfile

* updated path

* fixed dockerfile

* dumped version number

* exclude for virus scan

* added testing option

this does not start the agent

* write results on failures

* added timeouts for pipelines

* moving master pipelines to python scripts

* added flang to automatic project selection

based on #159, this will enable flang for beta testers

* added persistent workspace for testing containers

* added secure delete function

* added better log message

* deleting read-only files

* checking existence before setting flags

* using unlink

* deleting recursively

* using pathlib for chmod

* using custom workspace

* fixed drive

* separate handling of single files

* simplified read-only handling again

* removed vsdevcmd calls

as it's already set in the docker Entrypoint

* renamed container folders

* windows version not needed any more

* bumped version number

* bumped version number

* script cleanup

* added cmake parameters for sccache

* added sccache

* starting sccache with VS environment

* added container for Visual Studio 2019

* fixed comments

* considering WIndows version

* updated to vs2019

* using MS install method

* snapshot of VS2019 experiments

* using --installRecommended

* cleanup of dockerfile

* updated path

* fixed dockerfile

* dumped version number

* removed vsdevcmd calls

as it's already set in the docker Entrypoint

* renamed container folders

* windows version not needed any more

* bumped version number

* bumped version number

* script cleanup

* removed sccache from vs2017

* making windows image configurable

* added versioning

* created windows BETA pipeline

* added Jenkins label for vs2019
2020-04-21 11:46:41 +02:00
Christian Kühnel
ba3e9c29da added lldb 2020-04-21 09:27:59 +02:00
Christian Kühnel
f4f6f5881c removed redundant backslashes 2020-04-17 16:33:51 +02:00
Christian Kühnel
7b90806189 removing --ignore-whitespace --whitespace=fix from git apply
Now patches are applies as they are with whatever white space changes
they contain.

fixes #147
2020-04-17 16:33:26 +02:00
Christian Kühnel
140d9f6f59 changed Error to Info
fixes #155
2020-04-17 16:25:27 +02:00
Mehdi Amini
aea569d4c8
Add flang to the list of project to test (#159)
Co-authored-by: ChristianKuehnel <kuhnel@google.com>
2020-04-17 14:30:18 +02:00
Christian Kühnel
bf0fb3fc68 simplified read-only handling again 2020-04-16 13:54:50 +02:00
Christian Kühnel
620945da3c separate handling of single files 2020-04-16 13:52:08 +02:00
Christian Kühnel
8e45e65375 using pathlib for chmod 2020-04-16 10:25:24 +00:00
Christian Kühnel
e3b384dc7f deleting recursively 2020-04-16 11:23:02 +02:00
Christian Kühnel
2850e18a47 using unlink 2020-04-16 11:15:40 +02:00
Christian Kühnel
66444f4952 checking existence before setting flags 2020-04-16 11:06:26 +02:00
Christian Kühnel
7521aeeda1 deleting read-only files 2020-04-16 10:30:39 +02:00
Christian Kühnel
b2c68fa9be added better log message 2020-04-16 10:12:57 +02:00
Christian Kühnel
eaa931da8c added secure delete function 2020-04-15 18:16:41 +02:00
Christian Kühnel
b8dcbcb273 added flang to automatic project selection
based on #159, this will enable flang for beta testers
2020-04-15 17:59:02 +02:00
Christian Kühnel
e1a8db6108 removed stray print statement 2020-04-15 17:54:18 +02:00