1
0
Fork 0
Commit graph

191 commits

Author SHA1 Message Date
Christian Kühnel
6394ae39b4 Adding an emptyDir to kubernetes deployment
This should resolve the out-of-disk problems reported in #379
2022-01-19 17:44:42 +01:00
Christian Kühnel
8d10520ca3 Revert "Updated configuration for BOLT"
This reverts commit 823df07a07.
2021-12-16 15:52:05 +00:00
Christian Kühnel
823df07a07 Updated configuration for BOLT 2021-12-16 09:29:41 +01:00
dependabot[bot]
431d15b9ca Bump lxml from 4.6.3 to 4.6.5 in /containers/stats
Bumps [lxml](https://github.com/lxml/lxml) from 4.6.3 to 4.6.5.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-4.6.3...lxml-4.6.5)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-12-14 11:39:43 +01:00
Mikhail Goncharov
cec188edc4 Add openmp deps and update toolchain to 13 2021-12-14 11:39:17 +01:00
Martin Storsjö
972d727767 Fix building the agent-windows-vs2019 image
The current version of Python fails to build some dependencies, but
pinning the previous version works.

ActivePerl is no longer installable via Chocolatey (neither current
nor older versions) as ActiveState don't provide the downloads
without registration any longer (and even if registered, only
provide the latest, continuously updated version for free users, which
can't match the package manager checksums), see http://disq.us/p/2ipditb
for further discussion.

Replace ActivePerl with StrawberryPerl. StrawberryPerl has other
drawbacks, as it installs a whole C/C++ toolchain as part of it,
and copies of pkg-config tools that can cause package misdetections.
Remove these after installing it, see
https://github.com/StrawberryPerl/Perl-Dist-Strawberry/issues/11
for discussion on the inconveniences of installing it.
2021-10-20 10:21:22 +02:00
Martin Storsjö
26153acfee Install llvm-mingw in the windows agent docker image
This bundles clang/lld and associated tools, configured for a mingw
target, with corresponding mingw sysroots.

As we already install the 'llvm' chocolatey package, which also
provides clang (defaulting to a MSVC target), add this one last in the
path. This lets 'clang' refer to the default MSVC-targeting tool, while
one can use a GCC style triple prefixed driver name, like
x86_64-w64-mingw32-clang, to use this tool.

This allows setting up a much-needed mingw configuration for the libcxx
CI.
2021-10-20 10:21:08 +02:00
Mikhail Goncharov
3b7d720763 add post checkout to run git gc 2021-09-30 12:46:36 +02:00
dependabot[bot]
8745fb51d3 Bump urllib3 from 1.26.4 to 1.26.5 in /containers/stats
Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.4 to 1.26.5.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.4...1.26.5)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-21 13:13:24 +02:00
Mikhail Goncharov
6c2410440e Run bazel builds in premerge
if user is a member of "bazel_build" https://reviews.llvm.org/project/view/107/
or modified /utils/bazel/*

For #328
2021-07-28 16:01:44 +02:00
Mikhail Goncharov
f7058aaa51 add post-checkout to run git prune
fix #321
2021-07-27 18:16:40 +02:00
Mikhail Goncharov
8960ff7dc9 Pre-checkout for windows
Kill all running processes in BUILDKITE_BUILD_PATH and force unlock git
folder.
2021-06-01 15:26:23 +02:00
Mikhail Goncharov
a44473098c cron jobs for buildbot and phab monitoring
+ fix phabricator / buildbot uptime monitoring

+ data is stored in separate databases
2021-05-20 21:44:19 +02:00
Mikhail Goncharov
6b2003a664 Cron job to load BK data to DB 2021-05-20 17:30:43 +02:00
Mikhail Goncharov
f0d1cc6ab3 update linux node pool (machines w/ no external ssd)
updated llvm version to 12 with relevant fixes in scrips
2021-05-17 10:06:54 +02:00
Mikhail Goncharov
1b603e884a migrage linux-agents to a new pool 2021-05-15 10:58:23 +02:00
Mikhail Goncharov
e703a856cb Add graceful termination period for linux agents
Now builds will not be dropped on cluster upgrades. That requires
container updates.
2021-05-14 14:56:18 +02:00
Mikhail Goncharov
aeba5401d1 remove gnuwin and cmake
relanding #290
also apparently vs cmake was always used, removing seaprate installation
2021-04-27 16:58:37 +00:00
Mikhail Goncharov
ded88cee7f Revert "Remove GnuWin, add Git Bash to the path"
This reverts commit 18ff65644e.

Image with that change fails to build with some obscure errors
with file access.
2021-04-24 19:01:01 +02:00
Mikhail Goncharov
f6814312b5 remove version argument 2021-04-24 16:58:03 +00:00
Mikhail Goncharov
4f4bf4629e Remove "version" from windows versions.
Current mechanism does not work well due to need to upload version.
Rebuilding a container from an alder revision also leads to overriding
existing container version.
2021-04-24 16:56:18 +02:00
Mikhail Goncharov
d29b27ff10 published new images for win 2021-04-23 13:55:50 +00:00
Martin Storsjö
ffd904922d Install an older version of ActivePerl
The current default version in Chocolatey, 5.28, fails to install.
This was previously worked around by pinning a newer upcoming
version, 5.28.0.20210106. However this version never got approved
into Chocolatey (for unknown reasons), so it's no longer available.
Therefore instead request the previous older version, 5.24.3.2404001,
which installs correctly.
2021-04-23 14:49:31 +02:00
Martin Storsjö
18ff65644e Remove GnuWin, add Git Bash to the path
Both the main and the release/12.x branches of llvm-project should
work when built in a Windows Container with Git Bash in path now.
2021-04-23 14:48:24 +02:00
Mikhail Goncharov
609f614040 put back gnuwin 2021-03-19 14:35:02 +00:00
Mikhail Goncharov
feb44f59ad update build_run for win 2021-03-19 10:20:17 +00:00
Martin Storsjö
59993105bd windows: Stop installing GnuWin
These tools aren't actually needed; the same tools from Git
from Windows fill the same need.
2021-03-19 10:17:43 +01:00
Martin Storsjö
515acd1736 windows: Install a specific version of ActivePerl
The current default version in Chocolatey (5.28) fails to install,
with the following error:

    ERROR: The response content cannot be parsed because the Internet
    Explorer engine is not available, or Internet Explorer's first-launch
    configuration is not complete. Specify the UseBasicParsing parameter
    and try again.

This is fixed in a newer version of packaging of ActivePerl, thus request
this particular version.
2021-03-19 09:57:18 +01:00
Martin Storsjö
e7cc8d1de8 windows: Add Git Bash to the path
The libcxx tests currently require bash to be available (although
that might change), and the bash provided with Git is enough for
fulfilling this need. (Bash can also be useful for running other
scripts in the libcxx CI chain outside of the tests themselves.)

For the main LLVM testing, this works once
https://reviews.llvm.org/D98858 is merged.

This also allows getting rid of GnuWin altogether.
2021-03-19 09:56:58 +01:00
Martin Storsjö
b11b61d04f windows: Don't manually add c:\Program Files (x86)\GnuWin32\bin to the path
The GnuWin32 tools installed by Chocolatey reside in C:\GnuWin, and
are added automatically to the path.

The CMake tools aren't added automatically to the path though, so
that entry still is useful.

(However, the Visual Studio installation also brings along
a different version of CMake that ends up found if the one we
installed via Chocolatey isn't available.)
2021-03-19 09:56:58 +01:00
Martin Storsjö
dee182ad1c windows: Install LLVM/Clang in the docker image
The main LLVM build runs fine with MSVC itself, but building libcxx
isn't supported with cl.exe, only with clang-cl.

This shouldn't matter for the main LLVM build, as it still is
configured to use CXX=cl (and even then, CMake defaults to cl
if nothing is specified).
2021-03-19 09:56:44 +01:00
Mikhail Goncharov
99e43a8d9a rename master to main branch following LLVM 2020-12-10 09:29:24 +01:00
Mikhail Goncharov
ea943424f3 Enable windows builds back
Now we run a pre-commit hook that tries to kill any process that locked
a file in target directory.

Updated timeout interval on windows from 120 to 90 minutes.

Fixes #243
2020-10-07 16:59:37 +02:00
Mikhail Goncharov
8b36ff5888 remove jenkins configs and scrips
- moved linux agents to a new node pool (basically renamed old one)

- removed some out of date scripts or moved them to playlists

- removed shell scrips invoked from jenkins

- minor docs updates

Some python classes methods are not used anymore and will be cleaned up
separately.
2020-09-01 09:51:49 +02:00
Mikhail Goncharov
5380ea63b4 docs about machine config and ingress 2020-08-28 13:08:40 +02:00
Mikhail Goncharov
46c9fefb77 rename CCACHE_PATH to CCACHE_DIR
https://ccache.dev/manual/3.7.11.html#_configuration uses CCACHE_DIR to set
cache directory. CCACHE_PATH has a different meaning.

+ removed LLVM_CCACHE_DIR ccache argument to use CCACHE_DIR from env.
+ print additional info about ccache config and machine readable stats.
2020-08-26 14:22:37 +02:00
Mikhail Goncharov
06db27454c add ATL component, build windows image without cache 2020-08-12 22:31:07 +02:00
Christian Kühnel
5da900df30 added DIA support for windows image 2020-08-12 22:31:07 +02:00
Mikhail Goncharov
5e9a7be295 add perl to windows image 2020-07-08 10:25:05 +02:00
Christian Kühnel
f84641775a removed buildbot container 2020-06-26 10:36:52 +02:00
Mikhail Goncharov
77b08754c6 use unstable debian release 2020-06-18 16:16:07 +02:00
Mikhail Goncharov
fb830daebe Update clang version
apt.llvm.org signature has changed, we also should use debian:stable
(=buster) as a base image to correctly install packages.

Removed jdk from the base image as it was used only by Jenkins.

Log packages version for every build.

Related issues: #112
2020-06-18 10:55:15 +02:00
Mikhail Goncharov
dd4e6210bb pre-checkout hook
That will fix #112 by checking out into 'llvm-project' directory
(derived from repository name).
2020-06-17 14:41:00 +02:00
Mikhail Goncharov
e08efbcf2a Script updates:
- option not to shut down sccache after 10 minutes
- zero sccache stats before running the build
- decrease number of jenkins agents
2020-06-04 12:36:46 +02:00
Mikhail Goncharov
017ca44a75 Add windows build
- configured sccache on buildkite windows machines
- final result is sent to phabricator from "summary.py" that waits for both builds to complete
- extracted "add_url_artifact" to a runnable script
- reorganized code and fixed some of TODOs
2020-06-03 13:40:22 +02:00
Mikhail Goncharov
afed3a7898 Update docs and script for installing win machines
I have tried to install new windows machine for buildkite and updated
some scrips and docs on along the way.

- Updated base image for k8s agent installation as it gave a warning that
previous version was old.

- Now buildkite secret is configured in a powershell script along with
tags (and possible other parameters).

- Split "windows_agent_start.ps1" to "..jenkins" and "..buildkite" as
  some parameters are different.

- Created a "windows-development" machine in GCP stat can be resumed
  to build docker images / expriment.
2020-05-27 15:41:49 +02:00
Mikhail Goncharov
1505cfb635 build 2020-05-14 12:57:38 +02:00
Mikhail Goncharov
a152d97e3c Simple service to integrate harbormaster w/ buildkite
Harbormaster can send a request in url encoded form while buildkite
expects a POST request with json. Instead of modifying harbormastar or
buildkite this adds a simple proxy that accepts url encoded form and
creates a request that buildkite expects. To avoid potential abuse,
nginx asks for simple http auth credentials stored in harbormaster.

All build parameters passed by proxy are put into build metadata and as
ph_* env variables available during the build.

Secrets involved:
- harbormastert knows http-auth to proxy (stored in privatly and in k8
  buildkite/http-auth as auth file);
- proxy knows buildkite api token (mine atm);
- build agent knows conduit API token (mine atm),
  and SSH key of llvm-premerge-tests-bot (in k8 buildkite/github-ssh).

Sample build: https://reviews.llvm.org/harbormaster/build/64828/8/
https://buildkite.com/llvm-project/premerge/builds/48
2020-05-13 13:13:33 +02:00
Christian Kühnel
d6e9b1f6c4 manual commands for vs config 2020-05-04 14:23:02 +00:00
Mikhail Goncharov
39667898a7 use env variables 2020-04-24 15:43:09 +02:00
Mikhail Goncharov
e860c29a47 Checkout scripts and define pipeline in python 2020-04-24 14:47:00 +02:00
Mikhail Goncharov
8f2cc6f5b4
Merge pull request #166 from google/update-image-pipeline-bootstrap
Forked docker image for premerge
2020-04-24 12:30:36 +02:00
Mikhail Goncharov
27ab5abd0a Forked docker image for premerge
- added simple script to define pipeline, it will check out from this
  repo later

- refactored docker container, e.g. merged most of the RUN steps into
  one and moved steps that are likely to change closer to the end. That
  should improve rebuild / upload speed.

Sample run: https://buildkite.com/llvm-project/premerge/builds/14
2020-04-24 12:29:44 +02:00
Christian Kühnel
8039b8a6f7 changed the way sccache is started 2020-04-24 09:54:36 +00:00
Christian Kühnel
7a87f7d453 fixed version file path 2020-04-24 09:54:36 +00:00
Christian Kühnel
d8a19b36e4 fixed vs code target architecture 2020-04-24 09:54:36 +00:00
Christian Kühnel
b9f8ce9f20 bumped version numbers 2020-04-23 09:47:47 +00:00
Mikhail Goncharov
eca78eb366 Add new deployment for agent that will do premerge checks
maybe we will later use same agents for everything but for now I don't
want to modify working build much. But I have updated deployment of
release agent to use correct docker image (there were renamed a while
ago) and pass tags and agent token via env variables.

Have not deployed new deployment for release yet.
2020-04-22 12:11:23 +02:00
ChristianKuehnel
a47751ebe8
Upgrade Windows agents to Visual Studio 2019 and sccache (#162)
* added cmake parameters for sccache

* added sccache

* removed stray debug output

* starting sccache with VS environment

* added container for Visual Studio 2019

* fixed comments

* considering WIndows version

* updated to vs2019

* using MS install method

* snapshot of VS2019 experiments

* using --installRecommended

* cleanup of dockerfile

* updated path

* fixed dockerfile

* dumped version number

* exclude for virus scan

* added testing option

this does not start the agent

* write results on failures

* added timeouts for pipelines

* moving master pipelines to python scripts

* added flang to automatic project selection

based on #159, this will enable flang for beta testers

* added persistent workspace for testing containers

* added secure delete function

* added better log message

* deleting read-only files

* checking existence before setting flags

* using unlink

* deleting recursively

* using pathlib for chmod

* using custom workspace

* fixed drive

* separate handling of single files

* simplified read-only handling again

* removed vsdevcmd calls

as it's already set in the docker Entrypoint

* renamed container folders

* windows version not needed any more

* bumped version number

* bumped version number

* script cleanup

* added cmake parameters for sccache

* added sccache

* starting sccache with VS environment

* added container for Visual Studio 2019

* fixed comments

* considering WIndows version

* updated to vs2019

* using MS install method

* snapshot of VS2019 experiments

* using --installRecommended

* cleanup of dockerfile

* updated path

* fixed dockerfile

* dumped version number

* removed vsdevcmd calls

as it's already set in the docker Entrypoint

* renamed container folders

* windows version not needed any more

* bumped version number

* bumped version number

* script cleanup

* removed sccache from vs2017

* making windows image configurable

* added versioning

* created windows BETA pipeline

* added Jenkins label for vs2019
2020-04-21 11:46:41 +02:00
Christian Kühnel
b36a7733de fixed jeninkins version
To avoid upgrade to later version with mandatory CSRF enabled.
This did break the Phabricator integration in #163.
2020-04-20 21:52:07 +02:00
Christian Kühnel
651bf69a89 added persistent workspace for testing containers 2020-04-15 18:00:53 +02:00
Christian Kühnel
2899b93c16 added gdb
fixed python bdist_wheel warning
2020-03-27 13:51:54 +01:00
Christian Kühnel
24e751f6ec updated container and deployment to store ssh keys for github uploads 2020-03-23 15:34:23 +01:00
Christian Kühnel
9b2e225461 addd buildbot installation 2020-03-19 14:53:19 +01:00
Christian Kühnel
f182e84f1a Docker build toolchain working 2020-03-19 14:21:32 +01:00
Christian Kühnel
e18bd6fca7 incremented version number 2020-02-24 15:28:59 +00:00
Christian Kühnel
d00cd053e1 tagging and pushing "latest" 2020-02-20 16:45:04 +00:00
Christian Kühnel
548e07d5d4 added versioning to windows docker containers 2020-02-20 15:18:26 +00:00
Christian Kühnel
a0f676dfba set fixed windows version number 2020-02-20 09:15:11 +00:00
Christian Kühnel
dd375f71db using upstream jenkins image 2020-02-17 10:50:37 +01:00
Christian Kühnel
a4e320ac04 Merge branch 'master' of ssh://github.com/google/llvm-premerge-checks 2020-02-06 14:08:26 +00:00
Christian Kühnel
66a653578b changed way of authenticating gsutil 2020-02-06 14:07:51 +00:00
Christian Kühnel
5e0b5873a9 changing authentication for gsutil 2020-02-06 14:26:46 +01:00
Christian Kühnel
39319ab89d named the buildkite windows agents 2020-02-06 14:05:46 +01:00
Christian Kühnel
9e431d4ab9 added comment on host names 2020-02-06 13:32:26 +01:00
Christian Kühnel
e8a9d7bf17 removing perl
messes up the entire build toolchain
2020-02-04 15:46:47 +00:00
Christian Kühnel
1f7eff3c41 Merge branch 'master' of ssh://github.com/google/llvm-premerge-checks 2020-02-04 10:14:36 +00:00
Christian Kühnel
7cceab06d9 using push/pop-location 2020-02-04 11:14:22 +01:00
Christian Kühnel
dc132819d6 removed perl from the path 2020-02-04 08:58:35 +00:00
Christian Kühnel
69ebbaa3f7 moved temp to SSD 2020-02-03 10:46:29 +00:00
Christian Kühnel
87c591d6b2 added perl for OpenMP 2020-01-31 13:19:07 +00:00
Christian Kühnel
64e18e7f85 changed path to requirements.txt 2020-01-28 16:40:08 +01:00
Mikhail Goncharov
ed0fb93e67 updated buildkite to llvm-10 2020-01-24 10:06:42 +01:00
Mikhail Goncharov
caef6c8238 fix build 2020-01-23 17:55:56 +01:00
Mikhail Goncharov
e13822f0f5 Use llvm-10 on phabricator agent
- removed 'clang8' from the image name;

- added --no-cache to ./build_deploy.sh to make sure that image will use
  most recent packages.

Tested locally
2020-01-23 16:17:19 +01:00
Mikhail Goncharov
403973a356 Line endings, docs 2020-01-22 19:04:13 +01:00
Christian Kühnel
da4237c666 buildkite not running as root anymore 2020-01-20 10:07:54 +01:00
Christian Kühnel
06826a8310 using command line arguments 2020-01-17 15:50:52 +01:00
Christian Kühnel
f2170f4955 updated buildkite windows 2020-01-17 11:52:31 +00:00
Christian Kühnel
ae519e1887 set of flag 2020-01-17 12:51:51 +01:00
Christian Kühnel
8d6af88016 moved build path to SSD 2020-01-17 11:42:27 +01:00
Christian Kühnel
c350bec730 config files to run buildkite for Linux 2020-01-17 10:33:08 +01:00
Christian Kühnel
a62a99f0d3 added pip to windows image 2020-01-16 14:19:47 +00:00
Christian Kühnel
4f4bcc7f65 installed gsutils in container 2019-12-20 11:07:09 +00:00
Christian Kühnel
67f6d1d48d migrated to python3 2019-12-20 08:47:39 +00:00
Christian Kühnel
4d6362b377 cleanup of powershell invocations 2019-12-20 08:21:46 +00:00
Christian Kühnel
451671cd88 tweaks for docker on windows 2019-12-18 17:29:21 +00:00
Christian Kühnel
ffb70a2922 cleanup 2019-12-16 16:38:04 +00:00
Christian Kühnel
f05a4240f2 fixed error handling 2019-12-16 10:02:28 +00:00
Christian Kühnel
250e527cf1 added access to result storage 2019-12-13 15:08:41 +00:00