KubeCon NA 2018 Cross cloud CI Intro slides 12112018

1. CNCF Cross-cloud CI Project Intro Taylor Carpenter & Denver Williams, Vulk.coop
2. Project Intro CNCF Cross-Cloud CI tests Kubernetes (K8s) and projects running on K8s across multiple cloud providers.
3. Project Intro - 35 minutes ● ● ● ● ● ● ● History of the Project Why and What Project Overview Technology Overview Lessons Learned What’s Next Q&A
4. Cross-cloud CI Intro: History History and why the project exists: The project started from as an idea in early 2016 to test the compatibility of Kubernetes versions on cloud providers along with applications running on those clusters. This was before the K8s Conformance Group, Cluster API project and well before a GA version of kubeadm existed. At that time there were very few cloud providers offering K8s as a service, such as Google Kubernetes Engine (GKE).
5. Cross-cloud CI Intro: What is it? What? The Cross-cloud CI project consists of a composable base CI system, a status repository server and a dashboard. The underlying CI testing system has 3 components (build pipeline, multi-cloud provisioner cross-cloud, cross-project) that continually validate the interoperability of each CNCF project for any commit on stable and head across all supported cloud providers. The testing system can reuse artifacts from a project’s existing CI system or generate new build artifacts. The status repository server collects the test results and the dashboard displays them.
6. Goal: to target CNCF projects Incubating Graduated Orchestration Distributed Tracing API Logging Remote Procedure Call Storage Service Discovery Container Runtime Container Runtime Networking API Distributed Tracing Software Update Spec Monitoring Security Messaging Service Mesh Package Management Storage Registry Distributed K/V Service Mesh SANDBOX Identity Spec Identity Policy Serverless Tooling Metrics Spec Distributed K/V Monitoring Packaging Spec Container Security Image Distribution Nodeless
7. Cross-cloud CI Intro: cncf.ci
8. Goal: to target non-CNCF projects Implemented 8
9. Goal: to target public, bare metal & private clouds + 9
10. CNCF CI Platform started (Feb 28, 2017) • Feb 1st, 2017: CNCF CI Goals posted to mailing list • Feb 28, 2017: CI Platform started CI Platform started Feb 28 10
11. CI Dashboard Greenlight (Sept 22nd, 2017) •Greenlight to start implementation of CI Dashboard •Granted during Open Source Summit in Los Angeles CI Platform started 1st demo of CI Platform CI Platform v.1.0.0 Release Greenlight for Dashboard Feb 28 June 27 Aug 13 Sept 22 11
12. Cncf.ci v1.0.0 Release (Jan 26th, 2018) •Cncf.ci v1.0.0 released to production •Included the Dashboard, Status API server, 4 active projects and 5 active clouds •Projects: K8s, Prometheus, CoreDNS, Linkerd •Clouds: AWS, Azure, GCE, GKE, Packet CI Platform started 1st demo of CI Platform CI Platform v.1.0.0 Release Greenlight for Dashboard Dashboard v1.0.0 Release Feb 28 June 27 Aug 13 Sept 22 Jan 26 12
13. Cncf.ci v1.1+ Releases (March, 2018) •Cncf.ci v1.1.0 released • Added Fluentd and IBM Cloud •Cncf.ci v1.2.0 released • Added ONAP SO project •Cncf.ci v1.3.0 released • Added OpenStack CI Platform started 1st demo of CI Platform CI Platform v.1.0.0 Release Greenlight for Dashboard Dashboard v1.0.0 Release Dashboard v1.1.+ Releases Feb 28 June 27 Aug 13 Sept 22 Jan 26 Mar 20 13
14. Cncf.ci v1.5.0 Release (Sept 7, 2018) •Cncf.ci v1.5.0 released to production •Added Envoy to projects CI Platform started 1st demo of CI Platform CI Platform v.1.0.0 Release Greenlight for Dashboard Dashboard v1.0.0 Release Dashboard v1.1.+ Releases Dashboard v1.5.0 Release Feb 28 June 27 Aug 13 Sept 22 Jan 26 Mar 20 Sep 7 14
15. Cncf.ci v1.6.0 Release (Oct 31, 2018) • Cncf.ci v1.6.0 released to production • Added OCI to deployments CI Platform started 1st demo of CI Platform CI Platform v.1.0.0 Release Greenlight for Dashboard Dashboard v1.0.0 Release Dashboard v1.1.+ Releases Dashboard v1.5.0 Release Feb 28 June 27 Aug 13 Sept 22 Jan 26 Mar 20 Sep 7 Dashboard v1.6.0 Release Oct 31 15
16. CI Status Dashboard Overview CI Dashboard Overview: ● Overview of cncf.ci ○ Shows status of 3 pipeline stages: Build, Provision and App Deployments ○ Refreshes at 3:00am Eastern Time every day ○ Supports dynamically adding/removing active clouds and projects ○ Clicking on Build status badge opens CI system build job URL ○ Clicking on Release Name opens project’s GitHub commit URL ○ Clicking on Deployment status badge opens “provisioning/app-deploy” job URL 16
17. 17
18. 1. Build projects from source 18
19. 1. Build projects from source 19
20. 2. Provision w/ cross-cloud 20
21. 3. Deploy w/ cross-project 21
22. CI Status: Sometimes things fail ● See dashboard snapshot ○ From Monday, December 10, 2018 22
23. 23
24. Project Configuration 24
25. Cloud Configuration 25
26. Refresh triggered 26
27. Testing System Overview ● Build Pipeline per project (optional, can use project’s build artifacts) ● Cloud Provisioning Pipeline (cross-cloud) ● App Deployment Pipeline (cross-project) 27
28. Build Pipeline Stage 1. Build: Compile binaries and e2e tests 2. Package: Create containers, create artifact pinning config and push to registry 3. Update-Dashboard: Update build status badges 28
29. Cloud Provisioning Pipeline Stage 1. Build: Prepare provisioning software from the cross-cloud project 2. Artifacts: Collect K8s artifact pinnings from the previous K8s builds 3. Cross-Cloud: Deploy K8s onto each cloud using cross-cloud provisioner 4. Update-Dashboard: Update deployment badges 29
30. App Deployment Pipeline Stage 1. Artifacts: Collect project artifact pinnings from the previous build stages 2. Cross-Project: Use Helm charts to deploy each project 3. End-to-End: Run e2e tests for each project 4. Dashboard-Update: Update deployment status badges 30
31. Technology Overview 31
32. CI System Technology Overview ● Unified CI/CD platform: GitLab ● Cross-cloud provisioning: Terraform, Cloud-init, and per cloud K8s configuration ● App deployments: K8s manifest management with Helm ● E2e tests: Custom containers + Helm ● Automated builds and deployments: Git + per project yaml configuration 32
33. Dashboard Technology Overview ● Frontend: Vue.js ● Status repository: Elixir and Erlang ● Automated builds and deployments: Git + per project yaml configuration 33
34. Lessons Learned 34
35. Lessons Learned Starting with the end-user interface is advised ● Serve one audience as your focus ● Simplify what you are communicating ● Graceful failures in the Web UI 35
36. 36
37. Lessons Learned Don’t start tightly coupled to an existing CI platform ● API, pipeline, event hooks ● Mixing CI for the software components with implementation ● Supporting existing project CI systems 37
38. Lessons Learned Design for projects and cloud providers to help from the start ● The entire system should be reproducible by others ● Support composable portions from the start (eg. kubeadm) 38
39. What’s Next? 39
40. What’s next for Cross-cloud CI? Cross-Group Collaboration: ● OpenCI community ● OPNfv collaboration ● Network Service Mesh collaboration ● Gathering feedback from End Users, CNCF Projects and K8s Community on next iteration of cncf.ci Status Dashboard 40
41. What’s next for Cross-cloud CI? Adding More CNCF Projects: ● Jaeger ● Notary ● Vitess ● etc 41
42. What’s next for Cross-cloud CI? Adding New Features: ● Automate project release updates ● API for history of builds, deployments and end-to-end tests ● Rollback to previous working release ● Integrate with other test platforms ● New screens TBD: Per Project, Per Deployment, etc 42
43. Upcoming Events CNCF Cross-cloud CI Deep Dive at KubeCon North America 2018 ○ Day: Wednesday, December 12 ○ Time: 10:50am - 11:25am Pacific Time ○ Location: Tahoma 5 @ TCC The Conference Center (TCC) CI WG Public Meetings ○ Day: 4th Tuesday of month ○ Time: 11:00am Pacific Time (US and Canada) ○ Location: https://zoom.us 43
44. How to Collaborate ● Attend CI WG meetings: ○ https://github.com/cncf/wg-ci ● Subscribe to the CNCF CI public mailing list: ○ https://lists.cncf.io/g/cncf-ci-public ● Create issues on GitHub: ○ https://github.com/crosscloudci/crosscloudci/issues 44
45. Connect with the team @crosscloudci @crosscloudci #cncf-ci slack channel crosscloudci@vulk.coop 45
46. Q&A 46
47. Thank you for your feedback! ● W. Watson @wavell ● Lucina Stricko @lixuna ● Denver Williams @denverwilliams ● Taylor Carpenter @taylor