1
0
Fork 0
Commit graph

72 commits

Author SHA1 Message Date
Mikhail Goncharov
00e8188f47 deployment definition 2020-07-21 15:21:06 +02:00
Mikhail Goncharov
d4e5126090 Update docs on ssd disk initialization
+ change number of linux agents

+ removed outputs in job analysis

+ updated windows image
2020-07-21 10:40:26 +02:00
Mikhail Goncharov
ffaba986c1 Deployment changes
do not restart win docker as it makes it impossible to stop agent from
buildkite UI for maintenance
2020-06-17 09:46:42 +02:00
Mikhail Goncharov
af7bd86875 disable clang format / tidy for master and release builds 2020-06-16 17:29:04 +02:00
Mikhail Goncharov
4c3cd1b6f5 Use os in queue name
That is useful for metrics collection as buildkite metrics only reports
per queue metrics
2020-06-16 10:30:24 +02:00
Mikhail Goncharov
397f7f38e9 removed 'release' buildkite queue 2020-06-15 17:07:36 +02:00
Mikhail Goncharov
de0d5afd78 Move agents to default queue
- fix windows defender exclusion path
2020-06-05 10:50:42 +02:00
Mikhail Goncharov
e08efbcf2a Script updates:
- option not to shut down sccache after 10 minutes
- zero sccache stats before running the build
- decrease number of jenkins agents
2020-06-04 12:36:46 +02:00
Mikhail Goncharov
017ca44a75 Add windows build
- configured sccache on buildkite windows machines
- final result is sent to phabricator from "summary.py" that waits for both builds to complete
- extracted "add_url_artifact" to a runnable script
- reorganized code and fixed some of TODOs
2020-06-03 13:40:22 +02:00
Christian Kühnel
a4d7420a2c increased resources for jenkins master 2020-05-28 17:19:20 +02:00
Mikhail Goncharov
afed3a7898 Update docs and script for installing win machines
I have tried to install new windows machine for buildkite and updated
some scrips and docs on along the way.

- Updated base image for k8s agent installation as it gave a warning that
previous version was old.

- Now buildkite secret is configured in a powershell script along with
tags (and possible other parameters).

- Split "windows_agent_start.ps1" to "..jenkins" and "..buildkite" as
  some parameters are different.

- Created a "windows-development" machine in GCP stat can be resumed
  to build docker images / expriment.
2020-05-27 15:41:49 +02:00
Mikhail Goncharov
1505cfb635 build 2020-05-14 12:57:38 +02:00
Mikhail Goncharov
a152d97e3c Simple service to integrate harbormaster w/ buildkite
Harbormaster can send a request in url encoded form while buildkite
expects a POST request with json. Instead of modifying harbormastar or
buildkite this adds a simple proxy that accepts url encoded form and
creates a request that buildkite expects. To avoid potential abuse,
nginx asks for simple http auth credentials stored in harbormaster.

All build parameters passed by proxy are put into build metadata and as
ph_* env variables available during the build.

Secrets involved:
- harbormastert knows http-auth to proxy (stored in privatly and in k8
  buildkite/http-auth as auth file);
- proxy knows buildkite api token (mine atm);
- build agent knows conduit API token (mine atm),
  and SSH key of llvm-premerge-tests-bot (in k8 buildkite/github-ssh).

Sample build: https://reviews.llvm.org/harbormaster/build/64828/8/
https://buildkite.com/llvm-project/premerge/builds/48
2020-05-13 13:13:33 +02:00
Christian Kühnel
cbb2820ef0 changed machine name to argument 2020-05-08 11:31:48 +02:00
Mikhail Goncharov
d09a194b1c Update specs to match cert-manager 0.15
Certificates are created automatically and not needed
2020-05-07 14:38:48 +02:00
Mikhail Goncharov
8f2cc6f5b4
Merge pull request #166 from google/update-image-pipeline-bootstrap
Forked docker image for premerge
2020-04-24 12:30:36 +02:00
Mikhail Goncharov
27ab5abd0a Forked docker image for premerge
- added simple script to define pipeline, it will check out from this
  repo later

- refactored docker container, e.g. merged most of the RUN steps into
  one and moved steps that are likely to change closer to the end. That
  should improve rebuild / upload speed.

Sample run: https://buildkite.com/llvm-project/premerge/builds/14
2020-04-24 12:29:44 +02:00
Christian Kühnel
230fb5b48b changed to 16 cores 2020-04-24 11:59:37 +02:00
Mikhail Goncharov
51734a7175 Remove port mapping and rename buildkite release deployment 2020-04-22 12:26:52 +02:00
Mikhail Goncharov
eca78eb366 Add new deployment for agent that will do premerge checks
maybe we will later use same agents for everything but for now I don't
want to modify working build much. But I have updated deployment of
release agent to use correct docker image (there were renamed a while
ago) and pass tags and agent token via env variables.

Have not deployed new deployment for release yet.
2020-04-22 12:11:23 +02:00
Christian Kühnel
9b393a2f53 upgraded to 4 Linux agents 2020-04-20 21:56:03 +02:00
Christian Kühnel
b36a7733de fixed jeninkins version
To avoid upgrade to later version with mandatory CSRF enabled.
This did break the Phabricator integration in #163.
2020-04-20 21:52:07 +02:00
Christian Kühnel
24e751f6ec updated container and deployment to store ssh keys for github uploads 2020-03-23 15:34:23 +01:00
Christian Kühnel
7f59673bb4 created script to generate/upload ssh keys 2020-03-23 12:14:19 +01:00
Christian Kühnel
dd375f71db using upstream jenkins image 2020-02-17 10:50:37 +01:00
Christian Kühnel
1804caf1ed using standard again,
builds started running out of RAM.
2020-02-03 15:00:20 +01:00
Christian Kühnel
5fe1299160 added windows cluster 2020-02-03 12:41:54 +01:00
Christian Kühnel
cc5b821973 removed windows agents 2020-02-03 12:41:54 +01:00
Mikhail Goncharov
e13822f0f5 Use llvm-10 on phabricator agent
- removed 'clang8' from the image name;

- added --no-cache to ./build_deploy.sh to make sure that image will use
  most recent packages.

Tested locally
2020-01-23 16:17:19 +01:00
Christian Kühnel
5752b6939c fixed namespaces 2020-01-17 11:42:16 +01:00
Christian Kühnel
53dfed10c0 added deployment for buildkite 2020-01-17 10:53:21 +01:00
Christian Kühnel
c350bec730 config files to run buildkite for Linux 2020-01-17 10:33:08 +01:00
Christian Kühnel
d7996865de added scripts to set up cloud storage 2019-12-18 18:30:43 +01:00
Christian Kühnel
ccef5871df converted line endings 2019-12-17 11:09:22 +01:00
Christian Kühnel
adf51c6551 update of scripts for native windows agent 2019-12-16 18:56:02 +00:00
Christian Kühnel
1127782f09 started with scripts to bootstrap windows agents 2019-12-16 19:01:09 +01:00
Christian Kühnel
d4702d1c3f draft for agent installer 2019-12-16 16:45:06 +00:00
Christian Kühnel
1c20e83bf8 moved results storage to larger disk 2019-12-16 12:01:36 +01:00
Christian Kühnel
250e527cf1 added access to result storage 2019-12-13 15:08:41 +00:00
Christian Kühnel
3c0c4547ff added -it so container can be killed 2019-12-02 14:17:30 +01:00
Christian Kühnel
68b1b78080 added docker pull to get latest image 2019-12-02 14:15:40 +01:00
Christian Kühnel
56b62412f7 fixed docker command 2019-12-02 13:42:28 +01:00
Christian Kühnel
9fcc035a2e detecting ID of SSH 2019-12-02 12:47:54 +01:00
Christian Kühnel
438acecb69 made boot disk larger to fit docker image 2019-12-02 12:47:28 +01:00
Christian Kühnel
b9dfc9ba9a added scripts to create windows agents 2019-12-02 12:14:39 +01:00
ChristianKuehnel
de34fb9572
Create docker container for Windows build (#52)
* first steps towards windows builds #25

* added build scripts for windows

* moved from powershell to CMD

* script cleanup

* set up jenkins swarm plugin

* added deployment script for windows

* first draft of jenkins file for windows

* first shot at windows pods

* first draft for buildkite agent

* added call to run_cmake.bat

* added call to run_ninja.bat

* added new buildkite script

* added Dockerfile for buildkite agent

* fixed userprofile path

* added /IncludeOptional for VS

I hope that includes all required libraries

* moved work folder to W:

* added git config for long file names

* added more CMake flags

* setting x64 bit arch in a different way

* and yet another way to set 64bit

* merged buildkite changes to jenkins scripts

* hiding license in batch script runs

* running release build

* added test xml log

* compilation is working, but not all tests

* cmake is in patch now

* updated the dependencies from buildkite config

* dockerfile cleanup

* change how git handles line endings

* updated comment

* performance improvment for volume

* changed workdir folder

* using gnuwin package

split installation into separate layers to speed up incremental changes
2019-11-26 17:10:30 +01:00
Christian Kühnel
efa707699a moved auth file out of the source tree 2019-11-08 16:48:21 +01:00
Christian Kühnel
ee083ffe76 fixed urls after moving to new project 2019-10-29 09:43:00 +01:00
ChristianKuehnel
40e8c91295
configuration for new GCP project (#31)
* configuration for new GCP project

* fixed typo

* fixed image names

* set temporary urls during migration phase

* updated config after benchmarks
2019-10-28 09:35:49 -07:00
Christian Kühnel
f433a82f1a updated config after benchmarks 2019-10-24 09:17:37 -07:00