Kubernetes and The GitOps Face Off

1. Kubernetes and The GitOps Face-Off Javeria Khan & Ricardo Aravena
2. Ricardo Aravena (rico) Sr .Data Ops Engineer @raravena80, <raravena@branch.io> Javeria Khan Sr. Systems Engineer @javeriak_, <jkhan@branch.io>
3. K8s and GitOps @ Branch 5 Tools Flux/Bazel/homegrown build & deploy 4 3 K8s in prod for 3 years API services - Lots of different languages Big Data CI/CD Spark/Flink CI/CD Circle+Jenkins w/Github 2 1 Where? Almost Everywhere https://branch.io/
4. Agenda Background ● Our Scale & Usage ● GitOps History Tools ● ● Why use Skaffold ● ● Popular tools Pros & Cons Side by Side (Build) tools ● ● Why use Popular Tools ● Side by Side Choosing ● ● Production Ready Ease of use ● ● Community Support Stability Future ● Enhancements
5. 8 B requests a day (+70% y/y) 3B user sessions per day 100K requests per second 10 TB of data per day 200+ microservices ~300 builds per day 10 Kubernetes clusters 10,000s containers per cluster
6. GitOps History 2003 2011 2013 2015 2017 buildbot Jenkins/Travis Cloud CI/CD Kubernetes GitOps First release First releases CircleCI, AppVeyor, etc 1.0 Release Deploy to K8s https://en.wikipedia.org/wiki/Comparison_of_continuous_integration_software
7. GitOps
8. GitOps Tools
9. Version Control Use a tool Both Infra & Code Compare current state to desired state Why GitOps? Automate Deployments Gain reversibility, have an audit trail and transparency
10. Kubernetes @ Branch In Production since 2016 10 kubernetes clusters Run builds every ~5 min 10,000s containers per cluster
11. GitOps Infra Kubernetes Clusters CI dev d pro Code Repos Config Repos config sync tool Prod PRs dev sync tool pre-prod dev & testing port-forward remote apps stream remote logs
12. Desired Features ✓ Dependencies Deployment tools: Flux, Argo - Learning curve ✓ Handles source code - Iterative development Skaffold, Metaparticle App Development tools: Draft, GitKube ✓ Local development/test, remote cluster ✓ Logging ✓ Stability / Support (alpha, beta) ✓ Production use Templating tools: Helm, Ksonnet
13. Ksonnet “YAMLs is for computers. Ksonnet is for people” https://github.com/ksonnet/ksonnet
14. Ksonnet Who? ● Heptio/Bitnami What? ● Defining Apps How? ● Jsonnet manifests ● No (uses existing images) Docker daemon required ● No Local and remote support ● Yes Dockerfile requirement https://ksonnet.io
15. Ksonnet Pros ● Modularity: Dependency Cons ● package management Dependencies: knowledge of jsonnet (github, filesys, helm repos) ● Stability: early 0.13 release ● Support: active community ● Source Mgmt: Does not ● Supports deploying multiple versions to multiple clusters handle source code updates ● No integrated image builder https://ksonnet.io
16. Draft “Streamlined Kubernetes Development” https://github.com/Azure/draft
17. Draft Who? ● MS Open Source What? ● App development How? ● Draft packs/cli Dockerfile requirement ● No (can use) Local and Remote ● Yes Docker daemon required ● Yes, local https://github.com/Azure/draft
18. Draft Languages https://github.com/Azure/draft
19. Draft Pros ● Dependencies: No Cons ● Dockerfile requirement, No K8s manifests needed ● Support: active community ● Supports many languages ● Integrated docker image builder Dependencies: Helm + Tiller, local docker daemon ● Stability: 0.16.x experimental release ● Source Mgmt: No automated updates to remote https://github.com/Azure/draft
20. GitKube “Build & Deploy using git push” https://github.com/hasura/gitkube
21. GitKube Who ● Hasura What? ● App Deployment How? ● Git push Dockerfile requirement ● Yes Docker daemon required ● Yes (in cluster) Local and Remote ● Yes. https://github.com/hasura/gitkube
22. GitKube https://github.com/hasura/gitkube
23. GitKube Pros ● ● ● ● ● ● Dependencies:Uses existing common tools (git, kubectl) Easy setup Supports RBAC Supports any language Source Mgmt: Handles source code No cli, runs on cluster Cons ● ● Dependencies: Needs remote Docker, Dockerfile, k8s manifests in repo, CRDs Stability: Early release 0.2.1, no active community https://github.com/hasura/gitkube
24. Flux “Achieve Continuous Delivery and Integration” https://github.com/weaveworks/flux
25. Flux Who ● Weaveworks What? ● App Deployment How? ● ● git push fluxctl cmd Dockerfile requirement ● No Local and remote ● ● git for local fluxctl for remote Docker Daemon required ● No https://github.com/weaveworks/flux
26. Flux Pros ● ● ● ● Dependencies: Easy to deploy controller and CRD, supports Helm charts Source Mgmt: Automatic sync with docker registry and repo Stability: Mature 1.8.x release Support: active community Cons ● ● ● ● ● No local git management No integrated docker image builder Does it scale? Rollbacks possible but hard Canaries, blue-green hard https://github.com/weaveworks/flux
27. Flux Credit: https://github.com/weaveworks/flux
28. Skaffold “Easy and Repeatable Kubernetes Development” https://github.com/GoogleContainerTools/skaffold
29. Skaffold Who? ● Google Cloud ● App Development & Deployment ● ● Watches git repo Git push Dockerfile requirement ● Yes Local and remote ● Yes Docker daemon required ● Yes. Local and remote What? How? https://github.com/GoogleContainerTools/skaffold
30. Skaffold Pros Cons ● Supports local and remote ● Has controller Docker, Dockerfile, Helm, K8s ● Supports any language/tool manifests in repo ● Support: Active community ● Source Mgmt: Automated ● ● Dependencies: Needs local Stability: Early release 0.16.x updates from source code ● Integrated docker image builder https://github.com/GoogleContainerTools/skaffold
31. Skaffold https://github.com/GoogleContainerTools/skaffold
32. Argo “Open source Kubernetes native workflows, events, CI & CD” https://github.com/argoproj/argo
33. Argo Who? ● Applatix What? ● CI/CD ● K8s Controller and CRDs Dockerfile requirement ● No Local and Remote ● No local docker build Docker daemon required ● No How? https://github.com/argoproj/argo
34. Argo Pros ● Declarative YAML for Cons ● ● No integrated docker image builder Complicated setup More of a workflow tool ● Cli is a wrapper for kubectl ● Pipelines ● Nice UI ● Kubernetes integrated ● Stability: Mature 2.x release ● Support: Active development and community https://github.com/argoproj/argo
35. Metaparticle “Cloud Native standard library for Containers & Kubernetes” https://github.com/metaparticle-io
36. Metaparticle Who? ● Brendan Burns ● Templating + Deployments How? ● Using actual code Dockerfile requirement ● No Local and Remote ● Yes Docker daemon required ● No What? https://github.com/metaparticle-io
37. Metaparticle Pros ● ● ● ● Dependencies: No Dockerfile, YAML or config files Code based deployment Infra as real code More language support coming ○ Go ○ Rust ○ Ruby Cons ● Dependencies: Needs local ● Docker Limited language support ● Stability: Very alpha, no ● community Idiomatic https://github.com/metaparticle-io
38. GitOps Tools Side by Side Dockerfile requirement Docker Daemon req Function Method Local / Remote Helm integration Draft No Yes, in cluster Deploy to K8s Draft packs Yes Yes Flux Yes No Full lifecycle git push fluxctl Yes Yes GitKube Yes Yes, in cluster Deploy to K8s git push Yes No Skaffold Yes Yes, local & remote Deploy to K8s K8s YAML/JSON Yes No Argo No, but can use No CI/CD K8s YAML/JSON Remote only No Ksonnet No, but can use No Deploy to K8s Jsonnet Yes Yes Metaparticle No, but can use Yes, local Deploy to K8s Code libs Yes No Tool
39. Container Build tools
40. Why use something other than docker build?
41. Kaniko https://github.com/GoogleContainerTools/kaniko
42. Img https://github.com/genuinetools/img
43. Orca Build/Umoci https://github.com/cyphar/orca-build
44. Buildah https://github.com/containers/buildah
45. FTL - Faster Than Light https://github.com/GoogleCloudPlatform/runtimes-common/tree/master/ftl
46. Bazel Docker/OCI https://github.com/containers/buildah
47. Knative https://github.com/knative/build-templates
48. Image Build Tools Side by Side Tool Who? What? How? Docker daemon In K8s Cluster Kaniko Google Cloud Build in K8s Image builder, un-nested No Yes Img Jess Frazelle Build Unprivileged Dockerless, RawProc No Yes, nested SUSE Just Build Unprivileged, needs runC No Yes, nested. Red Hat Just Build Requires privilege escalation No Yes, w/Knative Google Cloud Just Build Layers/Dockerless No Yes, with Kaniko Google Just Build Bazel definition No Yes, w/Kaniko-Knative Google Cloud Build templ. In K8s Using templates No Yes, requires istio Umoci/Orca-build Buildah FTL Bazel Knative
49. GitOps with Build Tools GitOps Tool Build Tools Flux ✓ Ksonnet ✓ Metaparticle ✓ Argo ✓ Draft X GitKube X Skaffold X
50. What to use in prod?
51. What to use in prod?
52. What to use in prod?
53. What to use in prod?
54. Future GitOps ● ● ● 1.0 releases New tools CRD integration and development Build Tools ● ● ● 1.0 releases More tools to build with Kubernetes Direct integration with K8s Helm ● ● ● Helm v3 Lua plugins/hooks Event driven architecture
55. Resources Scaffold https://github.com/GoogleContainerTools/skaffold Draft https://github.com/Azure/draft Flux https://github.com/weaveworks/flux GitKube https://github.com/hasura/gitkube Argo https://github.com/argoproj/argo Ksonnet https://github.com/argoproj/argo Kaniko https://github.com/GoogleContainerTools/kaniko Img https://github.com/genuinetools/img Orca build/Umoci https://github.com/openSUSE/umoci Buildah https://github.com/projectatomic/buildah FTL https://github.com/GoogleCloudPlatform/runtimes-common/tree/master/ftl Bazel Docker/OCI https://github.com/bazelbuild/rules_docker
56. Thanks! Branch Engineering is hiring! https://branch.io/careers/