KubeCon NA 2018 Cross cloud CI Deep Dive Slides 12122018

1. CNCF Cross-cloud CI Deep Dive: Using Cross-cloud with Cloud-native Network Functions (CNFs) Taylor Carpenter & Denver Williams, Vulk.coop
2. CNCF Cross-cloud CI + CNF The Cross-Cloud CI project tests Kubernetes (K8s) and projects running on K8s across multiple cloud providers. The CNF project provides reference code and test comparisons of Cloud-native Network Functions.
3. Agenda - 35 Minutes ● Intro to CNCF Cross-cloud CI Project ○ Cross-cloud Kubernetes provisioner ● Intro to CNCF Cloud-native Network Functions (CNFs) Project ○ Reproducible CI for NFV testing ○ Using cross-cloud with CNFs ● Lessons learned / Challenges ● What’s next ● How to collaborate ● Q/A 3
4. Quick Intro to CNCF Cross-cloud CI https://github.com/crosscloudci/crosscloudci 4
5. What is CNCF Cross-cloud CI? What? The CNCF Cross-cloud CI project consists of a composable base CI system, a status repository server and a dashboard. The underlying CI testing system has 3 stages (build pipeline per project, cloud provisioning with cross-cloud, app deployments with cross-project) that continually validate the interoperability of each CNCF project for any commit on stable and head across all supported cloud providers. The testing system can reuse artifacts from a project’s existing CI system or generate new build artifacts. The status repository server collects the test results and the dashboard displays them.
6. Goal: to Target CNCF Projects Incubating Graduated Orchestration Distributed Tracing API Logging Remote Procedure Call Container Runtime Container Runtime Networking API Service Mesh Service Mesh Package Management Distributed Tracing Monitoring Software Update Spec Storage Security Service Discovery Messaging Storage Registry Service Mesh SANDBOX Identity Spec Identity Policy Serverless Tooling Metrics Spec Distributed K/V Monitoring Packaging Spec Container Security Image Distribution Nodeless 6
7. Goal: to Target Non-CNCF Projects 7
8. Goal: to Target Public, Bare Metal & Private Clouds + 8
9. CNCF CI Platform Timeline • Feb 28, 2017: CI Platform started • Jan 26, 2018: v.1.0.0 Dashboard released • Mar 20, 2018: v1.2.0 included ONAP • Sept 7, 2018: v1.5.0 included Envoy • Oct 31, 2018: v1.6.0 included OCI CI Platform started 1st demo of CI Platform CI Platform v.1.0.0 Release Greenlight for Dashboard Dashboard v1.0.0 Release Dashboard v1.1.+ Releases Dashboard v1.5.0 Release Feb 28 June 27 Aug 13 Sept 22 Jan 26 Mar 20 Sep 7 Dashboard v1.6.0 Release Oct 31 9
10. CI Status Dashboard Overview CI Dashboard Overview: ● Overview of cncf.ci ○ Shows status of 3 pipeline stages: Build, Provision and App Deployments ○ Refreshes at 3:00am Eastern Time every day ○ Supports dynamically adding/removing active clouds and projects ○ Clicking on Build status badge opens CI system build job URL ○ Clicking on Release Name opens project’s GitHub commit URL ○ Clicking on Deployment status badge opens “provisioning/app-deploy” job URL 10
11. CI Dashboard at cncf.ci 11
12. Testing System Overview ● Build pipeline per project (optional, can use project’s build artifacts) ● Kubernetes provisioning pipeline (cross-cloud) ● App deployment pipeline (cross-project) 12
13. Testing System Overview ● Build pipeline per project (optional, can use project’s build artifacts) ● Kubernetes provisioning pipeline (cross-cloud) ● App deployment pipeline (cross-project) 13
14. K8s Provisioning Pipeline Stage 1. Build: Prepare provisioning software from the cross-cloud project 2. Artifacts: Collect K8s artifact pinnings from the previous K8s builds 3. Cross-Cloud: Deploy K8s onto each cloud using cross-cloud provisioner 4. Update-Dashboard: Update deployment badges 14
15. 1. Testing K8s stable and head 15
16. 2. Provision w/ cross-cloud 16
17. 3. Deploy w/ cross-project 17
18. CI System Technology Overview ● Unified CI/CD platform: GitLab ● App deployments and e2e tests: K8s manifest management with Helm ● Cross-cloud provisioning: Terraform, Cloud-init and per cloud K8s configuration ● Automated builds and deployments: Git + per project yaml configuration 18
19. Quick Intro to CNCF Cloud-native Network Functions (CNFs) https://github.com/cncf/cnfs 19
20. CNF Project Intro CNCF is ushering the evolution of Virtualized Network Functions (VNFs) to Cloud-native Network Functions (CNFs) running on Kubernetes in public, private, or hybrid clouds. The transition to CNFs will provide 3 major benefits to service providers: 1. 2. 3. Cost savings (capex/opex) Improved resiliency Higher development velocity 20
21. CNF Project Intro The CNF project facilitates open collaboration on the development and use of Cloud-native Network Functions for real world use cases. The project provides reference code and test comparisons of CNFs. https://github.com/cncf/cnfs 21
22. CNF Project Goals The ideal outcome of the CNF project is that a third party developer can run the provided CNF reference code/benchmarking tests with an API key and a couple of CLI commands. Note: the CNF project is still in the prototype stage. Additional reference code and benchmarking tests will be added incrementally. 22
23. Reproducible CI for NFV testing 23
24. Creating a Neutral Test Environment Hardware Public: ● Packet Cloud ● FD.io CSIT lab Software Open: ● 100% open source ● Vanilla Kubernetes ● Helm Community Collaboration: ● Cross-group ● Multi-vendor 24
25. Reproducible Infrastructure Machines Networking 25
26. Provisioning Physical Machines The machines running the network functions use commodity hardware and all configuration is documented. Specs at a glance: ● ● ● ● ● CPU: Dual socket Xeon Gold 5120 (2.2Ghz) Cores: 24 per CPU (48 total) Memory: 384 GB of DDR4 ECC Storage: 3.2 TB of NVMe Flash NIC: Quad port Intel x710 The system hardware configuration is based on the Packet m2.xlarge.x86. Using either the default dual port Mellanox ConnectX-4 NIC or a quad port Intel x710 NIC. 26 The NIC ports are connected to 10GbE ports on the top-of-rack switches.
27. Vanilla K8s Clusters Kubernetes test environment: ● Deployment of Kubernetes with cross-cloud ● Kubernetes services running on bare metal Kubernetes Node CNF CNF CNF CNF 27
28. Layer-2 Network Wiring Worker Node Port #1 Port #2 Port #3 Port #4 Packet switch VLAN #1,2 Port #1 Port #2 Port #3 Port #4 Traffic Generator 28
29. Layer-2 Host Configuration NF NF NF Bridge Domain #1 NF NF NF VPP vSwitch Eth-Sub VLAN #1 Port #2 Bridge Domain #2 Eth-Sub VLAN #2 Worker Node Port #3 29
30. Layer 2 CNF Connections Kubernetes Node CNF CNF CNF CNF Data Plane (vSwitch) Kubernetes Node CNF CNF CNF Data Plane (vSwitch) CNF
31. Deploying K8s w/Layer-2 Support! Kubernetes test clusters: ● Deployment of Kubernetes with cross-cloud ○ + Ansible + cpu management policies ● Kubernetes services running on bare metal ● High-performance Layer-2 networking w/VPP for CNFs and host vSwitch ● Connecting CNFs over memif sockets 31
32. Using Cross-cloud with CNF Project 32
33. Cross-cloud Enhanced for CNFs New features: ● Added support for Ubuntu 18.04 as a host OS ● Support reserved Packet instances ● Enable support for cpu-management-policies ● Support worker node reboots for kernel config (eg. grub) updates 33
34. Lessons Learned / Challenges 34
35. Lessons Learned / Challenges Creating neutral and easily reproducible test comparisons has its obstacles, including: ● General challenges transitioning from VNFs to CNFs ● Reproducible infrastructure provisioning ● Reproducible clusters with high-performance data planes 35
36. Transitioning from VNFs to CNFs ● Moving from network functionality from physical hardware to encapsulating the software in a virtual machine (P2V) is generally easier than containerizing the software (P2C or V2C) ● Many network function virtualization VMs rely on kernel hacks or otherwise do not restrict themselves to just the stable Linux kernel userspace ABI ○ ● They also often need to use DPDK or SR-IOV to achieve sufficient performance Containers provide nearly direct access to the hardware with little or no virtualization overhead ○ But they expect containerized applications to use the stable userspace Linux kernel ABI, not to bypass it 36
37. Reproducible Infrastructure Provisioning ● Just because it’s in the API does not mean it works ● Access to the switch configuration does not mean you can set it up as expected ● Limits in provider facilities and between customer projects 37
38. Reproducible Clusters ● OpenStack ● Host OS - builds, packages, defaults ● Standard plugins vs source builds 38
39. What’s Next? 39
40. What’s Next for Cross-cloud? Cross-Group Collaboration: ● Network Service Mesh (NSM) collaboration ● Merge enhancements from forks/pull requests (eg. VMWare) ● Gathering feedback from End Users, CNCF Projects and K8s Community ● Planning next iteration of project 40
41. What’s Next for Cross-cloud? Features: ● Supporting Network Service Mesh ● Layer-2 for different providers ● Supporting kubeadm ● Supporting offline services like DNS 41
42. What’s Next for CNCF CNFs? Events and presentations: ● KubeCon CNFs BoF on Wed, Dec 12 at 2:35pm PT ○ ● Mobile World Congress, Barcelona, February 25-28, 2019 ○ ● https://sched.co/JCLS https://www.mwcbarcelona.com Open Networking Summit, San Jose, California, April 3 - 5, 2019 ○ https://events.linuxfoundation.org/events/open-networking-summitnorth-america-2019/ 42
43. What’s Next for CNCF CNFs? Enhancements: ● Comparisons with OpenStack, Firecracker, Singularity ● Supporting more environments (eg. Amazon bare metal) ● Adding more use cases 43
44. How to collaborate 44
45. How to Collaborate with Cross-cloud ● Attend CI WG meetings: ○ https://github.com/cncf/wg-ci ● Subscribe to the CNCF CI public mailing list: ○ https://lists.cncf.io/g/cncf-ci-public ● Create issues on GitHub: ○ https://github.com/crosscloudci/cross-cloud/issues ● Review KubeCon Cross-cloud CI Intro Slides: ○ https://kccna18.sched.com/event/Grci 45
46. Connect with Cross-cloud CI @crosscloudci @crosscloudci #cncf-ci slack channel crosscloudci@vulk.coop 46
47. Connect with CNF Project @cncf/cnfs @vulkcoop #cnf slack channel cncfcnfs@vulk.coop 47
48. Q&A 48
49. Thank you for your participation! ● W. Watson @wavell ● Lucina Stricko @lixuna ● Denver Williams @denverwilliams ● Taylor Carpenter @taylor
50. Thank you! Today’s Demo Prepared by: taylor@vulk.coop lucina@vulk.coop watson@vulk.coop denver@debian.nz 50