1
0
Fork 0
Commit graph

80 commits

Author SHA1 Message Date
Mikhail Goncharov
1ce0e1ff35 remove bucket as we don't use it anymore 2020-10-15 13:19:36 +02:00
Mikhail Goncharov
ab1b8d5ae2 Use buildkite API instead of passing stat via files
Previous approach had drawbacks:

- every step had to implement exporting of results in fixed format

- if step failed then failure will not be detected

Now report step will fetch results directly from Buildkite.
Agents have to be updated to have BUILDKITE_API_TOKEN env.
2020-10-08 14:40:57 +02:00
Mikhail Goncharov
381b1bf1a6 Update nginx configuration to capture all traffic to the static IP
Right now we have only one service we need to serve.

I kept cert manager config for now in the case we want to get a hostname
in the near future.

Updated doc + minor doc / setup fixes.

fixes #242
2020-09-29 11:44:12 +02:00
Mikhail Goncharov
8b36ff5888 remove jenkins configs and scrips
- moved linux agents to a new node pool (basically renamed old one)

- removed some out of date scripts or moved them to playlists

- removed shell scrips invoked from jenkins

- minor docs updates

Some python classes methods are not used anymore and will be cleaned up
separately.
2020-09-01 09:51:49 +02:00
Mikhail Goncharov
cb41bb38dd reset before checking out master 2020-07-30 09:32:50 +02:00
Mikhail Goncharov
3c28129968 custom agent constraints
+ updated image for win machines
2020-07-24 15:59:00 +02:00
Mikhail Goncharov
c384553e0b Increase linux agent memory #215 2020-07-21 18:40:36 +02:00
Mikhail Goncharov
79537bfc38 updated deployment 2020-07-21 17:42:02 +02:00
Mikhail Goncharov
00e8188f47 deployment definition 2020-07-21 15:21:06 +02:00
Mikhail Goncharov
d4e5126090 Update docs on ssd disk initialization
+ change number of linux agents

+ removed outputs in job analysis

+ updated windows image
2020-07-21 10:40:26 +02:00
Mikhail Goncharov
ffaba986c1 Deployment changes
do not restart win docker as it makes it impossible to stop agent from
buildkite UI for maintenance
2020-06-17 09:46:42 +02:00
Mikhail Goncharov
af7bd86875 disable clang format / tidy for master and release builds 2020-06-16 17:29:04 +02:00
Mikhail Goncharov
4c3cd1b6f5 Use os in queue name
That is useful for metrics collection as buildkite metrics only reports
per queue metrics
2020-06-16 10:30:24 +02:00
Mikhail Goncharov
397f7f38e9 removed 'release' buildkite queue 2020-06-15 17:07:36 +02:00
Mikhail Goncharov
de0d5afd78 Move agents to default queue
- fix windows defender exclusion path
2020-06-05 10:50:42 +02:00
Mikhail Goncharov
e08efbcf2a Script updates:
- option not to shut down sccache after 10 minutes
- zero sccache stats before running the build
- decrease number of jenkins agents
2020-06-04 12:36:46 +02:00
Mikhail Goncharov
017ca44a75 Add windows build
- configured sccache on buildkite windows machines
- final result is sent to phabricator from "summary.py" that waits for both builds to complete
- extracted "add_url_artifact" to a runnable script
- reorganized code and fixed some of TODOs
2020-06-03 13:40:22 +02:00
Christian Kühnel
a4d7420a2c increased resources for jenkins master 2020-05-28 17:19:20 +02:00
Mikhail Goncharov
afed3a7898 Update docs and script for installing win machines
I have tried to install new windows machine for buildkite and updated
some scrips and docs on along the way.

- Updated base image for k8s agent installation as it gave a warning that
previous version was old.

- Now buildkite secret is configured in a powershell script along with
tags (and possible other parameters).

- Split "windows_agent_start.ps1" to "..jenkins" and "..buildkite" as
  some parameters are different.

- Created a "windows-development" machine in GCP stat can be resumed
  to build docker images / expriment.
2020-05-27 15:41:49 +02:00
Mikhail Goncharov
1505cfb635 build 2020-05-14 12:57:38 +02:00
Mikhail Goncharov
a152d97e3c Simple service to integrate harbormaster w/ buildkite
Harbormaster can send a request in url encoded form while buildkite
expects a POST request with json. Instead of modifying harbormastar or
buildkite this adds a simple proxy that accepts url encoded form and
creates a request that buildkite expects. To avoid potential abuse,
nginx asks for simple http auth credentials stored in harbormaster.

All build parameters passed by proxy are put into build metadata and as
ph_* env variables available during the build.

Secrets involved:
- harbormastert knows http-auth to proxy (stored in privatly and in k8
  buildkite/http-auth as auth file);
- proxy knows buildkite api token (mine atm);
- build agent knows conduit API token (mine atm),
  and SSH key of llvm-premerge-tests-bot (in k8 buildkite/github-ssh).

Sample build: https://reviews.llvm.org/harbormaster/build/64828/8/
https://buildkite.com/llvm-project/premerge/builds/48
2020-05-13 13:13:33 +02:00
Christian Kühnel
cbb2820ef0 changed machine name to argument 2020-05-08 11:31:48 +02:00
Mikhail Goncharov
d09a194b1c Update specs to match cert-manager 0.15
Certificates are created automatically and not needed
2020-05-07 14:38:48 +02:00
Mikhail Goncharov
8f2cc6f5b4
Merge pull request #166 from google/update-image-pipeline-bootstrap
Forked docker image for premerge
2020-04-24 12:30:36 +02:00
Mikhail Goncharov
27ab5abd0a Forked docker image for premerge
- added simple script to define pipeline, it will check out from this
  repo later

- refactored docker container, e.g. merged most of the RUN steps into
  one and moved steps that are likely to change closer to the end. That
  should improve rebuild / upload speed.

Sample run: https://buildkite.com/llvm-project/premerge/builds/14
2020-04-24 12:29:44 +02:00
Christian Kühnel
230fb5b48b changed to 16 cores 2020-04-24 11:59:37 +02:00
Mikhail Goncharov
51734a7175 Remove port mapping and rename buildkite release deployment 2020-04-22 12:26:52 +02:00
Mikhail Goncharov
eca78eb366 Add new deployment for agent that will do premerge checks
maybe we will later use same agents for everything but for now I don't
want to modify working build much. But I have updated deployment of
release agent to use correct docker image (there were renamed a while
ago) and pass tags and agent token via env variables.

Have not deployed new deployment for release yet.
2020-04-22 12:11:23 +02:00
Christian Kühnel
9b393a2f53 upgraded to 4 Linux agents 2020-04-20 21:56:03 +02:00
Christian Kühnel
b36a7733de fixed jeninkins version
To avoid upgrade to later version with mandatory CSRF enabled.
This did break the Phabricator integration in #163.
2020-04-20 21:52:07 +02:00
Christian Kühnel
24e751f6ec updated container and deployment to store ssh keys for github uploads 2020-03-23 15:34:23 +01:00
Christian Kühnel
7f59673bb4 created script to generate/upload ssh keys 2020-03-23 12:14:19 +01:00
Christian Kühnel
dd375f71db using upstream jenkins image 2020-02-17 10:50:37 +01:00
Christian Kühnel
1804caf1ed using standard again,
builds started running out of RAM.
2020-02-03 15:00:20 +01:00
Christian Kühnel
5fe1299160 added windows cluster 2020-02-03 12:41:54 +01:00
Christian Kühnel
cc5b821973 removed windows agents 2020-02-03 12:41:54 +01:00
Mikhail Goncharov
e13822f0f5 Use llvm-10 on phabricator agent
- removed 'clang8' from the image name;

- added --no-cache to ./build_deploy.sh to make sure that image will use
  most recent packages.

Tested locally
2020-01-23 16:17:19 +01:00
Christian Kühnel
5752b6939c fixed namespaces 2020-01-17 11:42:16 +01:00
Christian Kühnel
53dfed10c0 added deployment for buildkite 2020-01-17 10:53:21 +01:00
Christian Kühnel
c350bec730 config files to run buildkite for Linux 2020-01-17 10:33:08 +01:00
Christian Kühnel
d7996865de added scripts to set up cloud storage 2019-12-18 18:30:43 +01:00
Christian Kühnel
ccef5871df converted line endings 2019-12-17 11:09:22 +01:00
Christian Kühnel
adf51c6551 update of scripts for native windows agent 2019-12-16 18:56:02 +00:00
Christian Kühnel
1127782f09 started with scripts to bootstrap windows agents 2019-12-16 19:01:09 +01:00
Christian Kühnel
d4702d1c3f draft for agent installer 2019-12-16 16:45:06 +00:00
Christian Kühnel
1c20e83bf8 moved results storage to larger disk 2019-12-16 12:01:36 +01:00
Christian Kühnel
250e527cf1 added access to result storage 2019-12-13 15:08:41 +00:00
Christian Kühnel
3c0c4547ff added -it so container can be killed 2019-12-02 14:17:30 +01:00
Christian Kühnel
68b1b78080 added docker pull to get latest image 2019-12-02 14:15:40 +01:00
Christian Kühnel
56b62412f7 fixed docker command 2019-12-02 13:42:28 +01:00