1
0
Fork 0
Commit graph

111 commits

Author SHA1 Message Date
Mikhail Goncharov
1505cfb635 build 2020-05-14 12:57:38 +02:00
Mikhail Goncharov
a152d97e3c Simple service to integrate harbormaster w/ buildkite
Harbormaster can send a request in url encoded form while buildkite
expects a POST request with json. Instead of modifying harbormastar or
buildkite this adds a simple proxy that accepts url encoded form and
creates a request that buildkite expects. To avoid potential abuse,
nginx asks for simple http auth credentials stored in harbormaster.

All build parameters passed by proxy are put into build metadata and as
ph_* env variables available during the build.

Secrets involved:
- harbormastert knows http-auth to proxy (stored in privatly and in k8
  buildkite/http-auth as auth file);
- proxy knows buildkite api token (mine atm);
- build agent knows conduit API token (mine atm),
  and SSH key of llvm-premerge-tests-bot (in k8 buildkite/github-ssh).

Sample build: https://reviews.llvm.org/harbormaster/build/64828/8/
https://buildkite.com/llvm-project/premerge/builds/48
2020-05-13 13:13:33 +02:00
Christian Kühnel
cbb2820ef0 changed machine name to argument 2020-05-08 11:31:48 +02:00
Mikhail Goncharov
d09a194b1c Update specs to match cert-manager 0.15
Certificates are created automatically and not needed
2020-05-07 14:38:48 +02:00
Mikhail Goncharov
8f2cc6f5b4
Merge pull request #166 from google/update-image-pipeline-bootstrap
Forked docker image for premerge
2020-04-24 12:30:36 +02:00
Mikhail Goncharov
27ab5abd0a Forked docker image for premerge
- added simple script to define pipeline, it will check out from this
  repo later

- refactored docker container, e.g. merged most of the RUN steps into
  one and moved steps that are likely to change closer to the end. That
  should improve rebuild / upload speed.

Sample run: https://buildkite.com/llvm-project/premerge/builds/14
2020-04-24 12:29:44 +02:00
Christian Kühnel
230fb5b48b changed to 16 cores 2020-04-24 11:59:37 +02:00
Mikhail Goncharov
51734a7175 Remove port mapping and rename buildkite release deployment 2020-04-22 12:26:52 +02:00
Mikhail Goncharov
eca78eb366 Add new deployment for agent that will do premerge checks
maybe we will later use same agents for everything but for now I don't
want to modify working build much. But I have updated deployment of
release agent to use correct docker image (there were renamed a while
ago) and pass tags and agent token via env variables.

Have not deployed new deployment for release yet.
2020-04-22 12:11:23 +02:00
Christian Kühnel
9b393a2f53 upgraded to 4 Linux agents 2020-04-20 21:56:03 +02:00
Christian Kühnel
b36a7733de fixed jeninkins version
To avoid upgrade to later version with mandatory CSRF enabled.
This did break the Phabricator integration in #163.
2020-04-20 21:52:07 +02:00
Christian Kühnel
24e751f6ec updated container and deployment to store ssh keys for github uploads 2020-03-23 15:34:23 +01:00
Christian Kühnel
7f59673bb4 created script to generate/upload ssh keys 2020-03-23 12:14:19 +01:00
Christian Kühnel
dd375f71db using upstream jenkins image 2020-02-17 10:50:37 +01:00
Christian Kühnel
1804caf1ed using standard again,
builds started running out of RAM.
2020-02-03 15:00:20 +01:00
Christian Kühnel
5fe1299160 added windows cluster 2020-02-03 12:41:54 +01:00
Christian Kühnel
cc5b821973 removed windows agents 2020-02-03 12:41:54 +01:00
Mikhail Goncharov
e13822f0f5 Use llvm-10 on phabricator agent
- removed 'clang8' from the image name;

- added --no-cache to ./build_deploy.sh to make sure that image will use
  most recent packages.

Tested locally
2020-01-23 16:17:19 +01:00
Christian Kühnel
5752b6939c fixed namespaces 2020-01-17 11:42:16 +01:00
Christian Kühnel
53dfed10c0 added deployment for buildkite 2020-01-17 10:53:21 +01:00
Christian Kühnel
c350bec730 config files to run buildkite for Linux 2020-01-17 10:33:08 +01:00
Christian Kühnel
d7996865de added scripts to set up cloud storage 2019-12-18 18:30:43 +01:00
Christian Kühnel
ccef5871df converted line endings 2019-12-17 11:09:22 +01:00
Christian Kühnel
adf51c6551 update of scripts for native windows agent 2019-12-16 18:56:02 +00:00
Christian Kühnel
1127782f09 started with scripts to bootstrap windows agents 2019-12-16 19:01:09 +01:00
Christian Kühnel
d4702d1c3f draft for agent installer 2019-12-16 16:45:06 +00:00
Christian Kühnel
1c20e83bf8 moved results storage to larger disk 2019-12-16 12:01:36 +01:00
Christian Kühnel
250e527cf1 added access to result storage 2019-12-13 15:08:41 +00:00
Christian Kühnel
3c0c4547ff added -it so container can be killed 2019-12-02 14:17:30 +01:00
Christian Kühnel
68b1b78080 added docker pull to get latest image 2019-12-02 14:15:40 +01:00
Christian Kühnel
56b62412f7 fixed docker command 2019-12-02 13:42:28 +01:00
Christian Kühnel
9fcc035a2e detecting ID of SSH 2019-12-02 12:47:54 +01:00
Christian Kühnel
438acecb69 made boot disk larger to fit docker image 2019-12-02 12:47:28 +01:00
Christian Kühnel
b9dfc9ba9a added scripts to create windows agents 2019-12-02 12:14:39 +01:00
ChristianKuehnel
de34fb9572
Create docker container for Windows build (#52)
* first steps towards windows builds #25

* added build scripts for windows

* moved from powershell to CMD

* script cleanup

* set up jenkins swarm plugin

* added deployment script for windows

* first draft of jenkins file for windows

* first shot at windows pods

* first draft for buildkite agent

* added call to run_cmake.bat

* added call to run_ninja.bat

* added new buildkite script

* added Dockerfile for buildkite agent

* fixed userprofile path

* added /IncludeOptional for VS

I hope that includes all required libraries

* moved work folder to W:

* added git config for long file names

* added more CMake flags

* setting x64 bit arch in a different way

* and yet another way to set 64bit

* merged buildkite changes to jenkins scripts

* hiding license in batch script runs

* running release build

* added test xml log

* compilation is working, but not all tests

* cmake is in patch now

* updated the dependencies from buildkite config

* dockerfile cleanup

* change how git handles line endings

* updated comment

* performance improvment for volume

* changed workdir folder

* using gnuwin package

split installation into separate layers to speed up incremental changes
2019-11-26 17:10:30 +01:00
Christian Kühnel
efa707699a moved auth file out of the source tree 2019-11-08 16:48:21 +01:00
Christian Kühnel
ee083ffe76 fixed urls after moving to new project 2019-10-29 09:43:00 +01:00
ChristianKuehnel
40e8c91295
configuration for new GCP project (#31)
* configuration for new GCP project

* fixed typo

* fixed image names

* set temporary urls during migration phase

* updated config after benchmarks
2019-10-28 09:35:49 -07:00
Christian Kühnel
f433a82f1a updated config after benchmarks 2019-10-24 09:17:37 -07:00
Christian Kühnel
9fe5764047 removed unused ssh service 2019-10-14 18:26:47 +02:00
Christian Kühnel
a88e8f2ce0 agents now connect through swarm plugin 2019-10-14 18:03:21 +02:00
Mikhail Goncharov
3078056d6f lower memory requirements for agent 2019-10-14 13:27:31 +02:00
Mikhail Goncharov
5370c990f0 Add 2 more jenkins nodes.
+ update local setup script
2019-10-14 12:22:25 +02:00
Christian Kühnel
4cb862a908 backup for github launch 2019-10-14 09:07:34 +02:00
Christian Kühnel
ce9e33f696 enabled basic auth on jenkins prod 2019-10-09 18:44:40 +02:00
Christian Kühnel
a9b250519c forwarding credentials working on jenkins-staging 2019-10-09 18:29:44 +02:00
Christian Kühnel
a65d82bb43 removed comments 2019-10-09 14:03:22 +02:00
Christian Kühnel
2d13b1688e TLS certs working 2019-10-09 13:59:34 +02:00
Christian Kühnel
7f68d2d664 snapshot of http working via nginx-ingress 2019-10-09 13:45:46 +02:00
Christian Kühnel
5d2387492d added skeleton for static ip
but it did not work
2019-10-09 12:48:37 +02:00
Christian Kühnel
fb85a8b83a added license statements 2019-10-09 10:18:07 +02:00
Christian Kühnel
a026d83261 First version of TLS configuration 2019-10-09 10:16:31 +02:00
Christian Kühnel
39724eb627 moved results to jenkins namespace 2019-10-09 09:54:32 +02:00
Christian Kühnel
d5d538f9bf Add now machines with integrated SSD
This might speed up the builds significantly
2019-10-07 14:08:34 +02:00
Christian Kühnel
8b96eaa7b0 build results are published to nfs 2019-10-05 11:33:39 +02:00
Christian Kühnel
cc1b61309a added NFS server to share files 2019-10-05 10:35:32 +02:00
Christian Kühnel
8008eaed06 Added another node to the cluster for misc services 2019-10-05 09:08:51 +02:00
Christian Kühnel
639ce6447a empty deployment of nginx 2019-10-05 08:48:31 +02:00
Christian Kühnel
6c99e22efc repo cleanup 2019-10-04 14:35:05 +02:00
Christian Kühnel
63d7054b20 added license 2019-10-04 14:31:57 +02:00
Christian Kühnel
dd96ebb61d initial import of configuration files 2019-10-04 10:26:20 +02:00