KubeCon + CloudNativeCon North America 2018

Migrating Jenkins to Kubernetes Broke Our Brains

1. How Migrating Jenkins to Kubernetes Broke our Brains github.com/solarhess/jenkins_kube_brains INTERNAL Jonathan Hess & Loren Mountain Trout, SAP December, 2018 © 2018 SAP SE or an SAP affiliate company. All rights reserved. 1
2. Welcome © 2018 SAP SE or an SAP affiliate company. All rights reserved. 2
3. Agenda Bad old Jenkins Jenkins on K8s in just 3 easy steps What happens when you let the developers drive You can do this too Questions © 2018 SAP SE or an SAP affiliate company. All rights reserved. 3
4. The story of our bad old Jenkins. © 2018 SAP SE or an SAP affiliate company. All rights reserved. 4
5. Requirements for new Jenkins Simple enough that a developer can manage it On the corporate network (not the cloud) Free Efficient and Repeatable Fast © 2018 SAP SE or an SAP affiliate company. All rights reserved. 5
6. 3 easy steps to a new Jenkins Prepare VMs © 2018 SAP SE or an SAP affiliate company. All rights reserved. Prepare Kubernetes Configure Jenkins 6
7. “Kubernetes is open source” © 2018 SAP SE or an SAP affiliate company. All rights reserved. 7
8. github.com/solarhess/jenkins_kube_brains © 2018 SAP SE or an SAP affiliate company. All rights reserved. 8
9. What we started with © 2018 SAP SE or an SAP affiliate company. All rights reserved. 9
10. What we ended up with © 2018 SAP SE or an SAP affiliate company. All rights reserved. 10
11. Kubernetes broke our brains Prepare VMs Why is it slow and hanging? © 2018 SAP SE or an SAP affiliate company. All rights reserved. Prepare Kubernetes What’s up with the network? Prepare Jenkins Storage for Jenkins Home? 11
12. Prepare VMs: Why is it slow and hanging? © 2018 SAP SE or an SAP affiliate company. All rights reserved. 12
13. VMs with attached volumes Where are the big, fast volumes? © 2018 SAP SE or an SAP affiliate company. All rights reserved. 13
14. Mount and move Don’t forget to update /etc/fstab © 2018 SAP SE or an SAP affiliate company. All rights reserved. 14
15. Prepare Kubernetes: What’s up with the network not connecting? © 2018 SAP SE or an SAP affiliate company. All rights reserved. 15
16. IP address ranges © 2018 SAP SE or an SAP affiliate company. All rights reserved. 16
17. © 2018 SAP SE or an SAP affiliate company. All rights reserved. 17
18. kubeadm init © 2018 SAP SE or an SAP affiliate company. All rights reserved. 18
19. Prepare Jenkins: Where do we put Jenkins Home? © 2018 SAP SE or an SAP affiliate company. All rights reserved. 19
20. Persistent Storage on a self-hosted cluster © 2018 SAP SE or an SAP affiliate company. All rights reserved. 20
21. Persistent Storage on a self-hosted cluster © 2018 SAP SE or an SAP affiliate company. All rights reserved. 21
22. Persistent Storage NFS hosted inside K8s © 2018 SAP SE or an SAP affiliate company. All rights reserved. 22
23. Jenkins broke our brains Configure Jenkins to be Efficient and Repeatable © 2018 SAP SE or an SAP affiliate company. All rights reserved. Benchmarking your build VM’s IO 23
24. Efficient and repeatable Jenkins configuration © 2018 SAP SE or an SAP affiliate company. All rights reserved. 24
25. Traditional Style Jenkins Configuration Hand-configured Jenkins with the Jenkins user interface © 2018 SAP SE or an SAP affiliate company. All rights reserved. 25
26. GitOps Style Jenkins Configuration Jenkins configuration in Jenkinsfile or Groovy DSL © 2018 SAP SE or an SAP affiliate company. All rights reserved. 26
27. Don’t choose, use both! Traditional Style for Global Config © 2018 SAP SE or an SAP affiliate company. All rights reserved. GitOps Style for Project Builds 27
28. Performance Benchmarking A little bit of math can save you a lot of time © 2018 SAP SE or an SAP affiliate company. All rights reserved. 28
29. © 2018 SAP SE or an SAP affiliate company. All rights reserved. 29
30. “And then we let developers take it for a drive.” © 2018 SAP SE or an SAP affiliate company. All rights reserved. 30
31. “Barely working” © 2018 SAP SE or an SAP affiliate company. All rights reserved. 31
32. Operating Jenkins broke our brains Builds fail intermittently OOM © 2018 SAP SE or an SAP affiliate company. All rights reserved. VMs die suddenly Dangling Docker Berries Builds back up endlessly Pod Tetris 32
33. Builds fail intermittently “What in heck is OOM?” © 2018 SAP SE or an SAP affiliate company. All rights reserved. 33
34. Java doesn’t play nice with containers © 2018 SAP SE or an SAP affiliate company. All rights reserved. 34
35. VMs die suddenly “What in heck are Dangling DockerBerries?” © 2018 SAP SE or an SAP affiliate company. All rights reserved. 35
36. Dangling DockerBerries NODES DISKS © 2018 SAP SE or an SAP affiliate company. All rights reserved. 36
37. Prune the DockerBerries hourly © 2018 SAP SE or an SAP affiliate company. All rights reserved. 37
38. Builds back up endlessly “What in heck is Pod Tetris?” © 2018 SAP SE or an SAP affiliate company. All rights reserved. 38
39. Pod Tetris How to lose The entire 24 node K8s cluster v Kubernetes Pod count: 23 Cluster used: 100% Master (Large pod) 3.5 CPU 12 GB Limit Memory © 2018 SAP SE or an SAP affiliate company. All rights reserved. 39
40. Pod Tetris How to WIN The entire 24 node K8s cluster v Kubernetes Pod count: 23 Cluster used: 30% Master (Large pod) 3 CPU 10 GB Memory (Small pod) 2 CPU 6 GB Memory (Tiny pod) 1 CPU 2 GB Memory © 2018 SAP SE or an SAP affiliate company. All rights reserved. 40
41. 10-4 good buddy © 2018 SAP SE or an SAP affiliate company. All rights reserved. 41
42. “Pods are NOT VMs.” Source: Trout © 2018 SAP SE or an SAP affiliate company. All rights reserved. 42
43. If you want to do this too Benchmark IO performance Jenkinsfiles for builds (don’t bother scripting global config) Play Pod Tetris to Size your jenkins Don’t try to build Kubernetes yourself (consider Gardener) © 2018 SAP SE or an SAP affiliate company. All rights reserved. 43
44. Gardener Open source Kubernetes management from SAP © 2018 SAP SE or an SAP affiliate company. All rights reserved. 44
45. Seed Cluster Garden Cluster Shoot Cluster Administrator kubify End-User kubectl Kubernetes Dashboard gardenctl Garden Cluster Garden Cluster API LB Garden Cluster Control Plane Ingress LB Storage [K8s] DS, RS, SS, J, ... [CRD] Shoot, Seed, ... Worker Gardener Dashboard Seed Cluster Seed Cluster API LB etcd Main D Gardener API Server D G Controller Manager D Shoot Cluster Shoot Cluster VPN LB Storage Seed Cluster Control Plane Kubelet + Container Runtime ... Shoot Cluster API LB [K8s] DS, RS, SS, J, ... [CRD] Machine Deployment, ... W + Container Runtime Kubelet Gardener Dashboard BckUp SS etcd Events API Server SS VPN Worker Worker Main Kubelet + Container Runtime PV Events PV D Scheduler D Controller Manager D Addon Manager D Machine Controller D Terraformer J Monitoring ... ... IaaS VPN Logging Target IaaS VPN D Calico DS Kube Proxy DS Kube DNS D Optional Addons Actual Workload ... Shoot Cluster Control Plane © 2018 SAP SE or an SAP affiliate company. All rights reserved. Kubernetes Dashboard kubectl Account/Network Prep ... Target IaaS/Account 45
46. SAP has a booth Learn more about our projects on Kubernetes from SAP Gardener Manage Kubernetes clusters as a service at scale Connect and extend products using serverless computing and microservices architecture Monitor repair, patch or upgrade your clusters in an automated way Customize your solution using the technology you want Central dashboard that simplifies administrative tasks across cloud infrastructures gardener.cloud/ Build an end-to-end customer experience using your existing technology kyma-project.io/ © 2018 SAP SE or an SAP affiliate company. All rights reserved. 46
47. Thank you SAP Open source foundations and noteworthy projects by SAP Service Manager © 2018 SAP SE or an SAP affiliate company. All rights reserved. 47
48. Questions & Answers Loren Trout Devops Engineer, SAP Loren.trout@sap.com @enzotrout Jonathan Hess Software Engineer, SAP jonathan.hess@sap.com @solarhess github.com/solarhess/jenkins_kube_brains © 2018 SAP SE or an SAP affiliate company. All rights reserved. 48
49. Images attributions “Fury Truck” by pxhere is licensed under CC BY 2.0 “Jenkins” by Jenkis.io is licensed under CC BY 4.0 “Runaway Truck Ramp - Monarch Pass” by Larry Lamsa is licensed under CC BY 2.0 “JunkYard” by Tom Fisk is licensed under pexels licence “Truck into Mountains” by 500photos is licensed under pexels licence “Berries” by PhotoMIX is licensed under pexels licence © 2018 SAP SE or an SAP affiliate company. All rights reserved. 49