KubeCon + CloudNativeCon North America 2018

Connecting Multiple Kubernetes Clusters Across Cloud Providers

1. Connecting Kubernetes Clusters Across Cloud Providers Thomas Graf, Co-Founder & CTO, Isovalent @tgraf__ 1
2. About the Speaker Thomas Graf ● ● ● Linux kernel developer for many years at Red Hat Working on networking, security and BPF Founder of the Cilium project 2
3. Goal of this Session: Run Services Across Cloud providers Cluster Frontend Backend 3
4. Goal of this Session: Run Services Across Cloud providers Cluster Cluster Frontend Frontend Backend Backend 4
5. Goal of this Session: Run Services Across Cloud providers Cluster Cluster Frontend Frontend Backend Backend 5
6. Goal of this Session: Run Services Across Cloud providers Cluster Cluster Frontend Backend Encrypted Frontend Backend 6
7. Goal of this Session: Run Services Across Cloud providers Cluster Cluster Frontend Backend Encrypted Frontend ← failover Backend 7
8. What Tools do we need? Kubernetes ● ● ● Open Source (Apache) Managed or self-managed Kubernetes services Cilium ● ● ● ● Infrastructure APIs ● ● VPC concept with routing IPSec compatible VPN Gateway with IKEv1 support ● ... ● Open Source (Apache) Based on new BPF technology Networking (CNI) Kubernetes services ○ Replacing kube-proxy ○ Multi-cluster capability (1.4) Network security ○ Identity-based, DNS aware, API aware, data protocol aware ○ Transparent encryption (1.4) Envoy/Istio Integration ○ Sidecar Acceleration ○ Transparent SSL visibility (kTLS) 8
9. What is BPF? Highly efficient sandboxed virtual machine in the Linux kernel. Making the Linux kernel programmable at native execution speed. Jointly maintained by Cilium and Facebook engineers with collaborations from Google, Red Hat, Netflix, and many others. $ clang -target bpf -emit-llvm -S \ 32-bit-example.c $ llc -march=bpf 32-bit-example.ll $ cat 32-bit-example.s cal: r1 = *(u32 *)(r1 + 0) r2 = *(u32 *)(r2 + 0) r2 += r1 *(u32 *)(r3 + 0) = r2 exit 9
10. Who uses BPF? Every packet toward facebook.com has been processed by BPF/XDP enabled application since May, 2017 Nikita V. Shirokov, Facebook Traffic team Linux Networking Summit 2018 Source: http://vger.kernel.org/lpc_net2018_talks/LPC_XDP_Shirokov_v2.pdf 10
11. Who uses BPF? ● ● ● ● ● ● ● L3-L4 Load balancing Network security Traffic optimization Profiling QoS & Traffic optimization Network Security Profiling ● ● ● ● Working upstream to replacing iptables with BPF Profiling & Tracing Performance Troubleshooting & Monitoring Check out bpftrace and Brendan Gregg’s blog posts 11
12. Who contributes to BPF? 380 Daniel Borkmann (Cilium, Maintainer) 161 Alexei Starovoitov (Facebook, Maintainer) 160 Jakub Kicinski Netronome 110 John Fastabend (Cilium) 96 Yonghong Song (Facebook) 95 Martin KaFai Lau (Facebook) Top contributors of the total 186 contributors to BPF from January 2016 to November 2018. 94 Jesper Dangaard Brouer (Red Hat) 74 Quentin Monnet (Netronome) 45 Roman Gushchin (Facebook) 45 Andrey Ignatov (Facebook) 12
13. Connecting Clusters with & . 13
14. Deployments & Pods Pod View Deployment View Frontend frontend-1 replicas=3 Backend frontend-2 frontend-3 Scale backend-1 backend-2 replicas=4 $ kubectl get deployment backend NAME DESIRED CURRENT UP-TO-DATE backend 4 4 4 backend-3 backend-4 Scale AVAILABLE 4 AGE 1d 14
15. What is a Service? 10.0.0.1 10.0.0.2 10.0.0.3 frontend-1 frontend-2 frontend-3 backend-1 backend-2 backend-3 backend-4 10.0.0.4 10.0.0.5 10.0.0.6 10.0.0.7 15
16. What is a Service? 10.0.0.1 10.0.0.2 10.0.0.3 frontend-1 frontend-2 frontend-3 Backend Service 10.100.0.1 backend-1 backend-2 backend-3 backend-4 10.0.0.4 10.0.0.5 10.0.0.6 10.0.0.7 16
17. Health Checks [...] livenessProbe: Backend Service 10.100.0.1 exec: command: - check-status failureThreshold: 3 periodSeconds: 2 backend-1 backend-2 backend-3 backend-4 10.0.0.4 10.0.0.5 10.0.0.6 10.0.0.7 Are you alive? Are you alive? Are you alive? Are you alive? 17
18. What are Endpoints? $ kubectl get svc backend NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE Backend <none> 80/TCP 1d ClusterIP 10.39.245.245 $ kubectl get endpoints backend NAME ENDPOINTS AGE Backend 1d ● ● 10.36.1.219:80,10.36.2.249:80 Kubernetes creates a shadow Endpoints object for every Service. The Endpoints object lists all pod IPs and port mappings of healthy pods based on the liveness health-check. Advanced: ● You can maintain the Endpoints object as a user as well. 18
19. Global Services Cluster Cluster frontend-1 frontend-2 frontend-1 Backend Service backend-1 frontend-2 frontend-3 Backend Service backend-2 backend-1 backend-2 metadata: annotations: io.cilium/global-service: "true" 19
20. Demo 20
21. Design Principles Simple ● ● Simple to use ○ Standard Kubernetes Services) ○ Avoid need for networking degree Simple to troubleshoot & debug Resilient ● ● ● Secure ● ● ● Encryption Security policies spanning clusters with identity-based enforcement Mutual TLS compatibility Preserve and respect availability zones and failure domains. Failures in one cluster should not impact other clusters. Avoid requirement of Kubernetes clusters to be aware of each other Efficient ● ● Native networking speeds Direct pod to pod connections without intermediate termination (proxies). 21
22. Supported Service Annotations io.cilium/global-service: {true false} Whether to include endpoints of other clusters. io.cilium/shared-service: {true false} Whether to share local with other clusters. Defaults to true if global-service. Coming Soon: io.cilium/service-affinity: {local-cluster local-node remote none} Whether to prefer local or remote endpoints. 22
23. ClusterMesh Control Plane Cluster Cluster Cilium etcd Pod Pod agent Pod agent Pod Pod Pod agent apiserver Pod Pod agent Sync Services Endpoints TLS operator LB Watching (read-only access) ● ● Control plane access to other clusters is always read-only (notification stream). Not all clusters must be aware of each other apiVersion: v1 kind: Service metadata: name: cilium-etcd-external annotations: cloud.google.com/load-balancer-type: "Internal" 23
24. Step-by-step Setup VPC1 10.1.0.0/16 ● ● VPC2 192.168.0.0/16 Create VPCs in each cloud provider with non-overlapping CIDR ranges. Support for overlapping CIDR ranges is possible but only complicates the setup. 24
25. Step-by-step Setup VPC1 10.1.0.0/16 ● ● ● VPC2 192.168.0.0/16 Create a VPN gateway and redundant VPN tunnels using IPSec. (Instructions in references) IPSec is the standard. All cloud providers support it. Setup routing to route from VPC1 to VPC2 via VPN and vice versa. 25
26. Step-by-step Setup Cluster VPC1 10.1.0.0/16 ● Cluster VPC2 192.168.0.0/16 Setup Kubernetes clusters using the created VPCs. 26
27. Step-by-step Setup Cluster Cluster Pod Pod Pod agent agent Cilium etcd TLS Pod agent Pod Pod agent VPC1 ● ● ● Cilium etcd TLS VPC2 Deploy Cilium Setup internal LoadBalancer to expose control plane on VPC Deploy Kubernetes secrets (clustermesh-secrets) to establish connections 27
28. Use Cases 28
29. High Availability Fail over to another cluster Cluster Cluster frontend-1 frontend-2 frontend-1 frontend-2 Backend Service frontend-3 Backend Service ← failover backend-1 backend-2 backend-1 backend-2 29
30. Shared Services Not all services need to be run in every cluster Cluster Cluster pod-1 pod-2 pod-1 Vault Service pod-2 pod-3 Vault Service Shared Services Vault Service vault-1 vault-2 30
31. Split Stateless and Stateful Keep your clusters dependency free Stateless Stateless ingress frontend-1 frontend-2 frontend-1 Datastore Service frontend-2 frontend-3 Datastore Service Shared Services data-1 ingress Datastore Service data-2 data-3 data-n 31
32. CVE-2018-1002105 CVE-2018-1002105: proxy request handling in kube-apiserver can leave vulnerable TCP connections Time to Abandon Cluster 32
33. Evacuation Move the stateless pieces 1.9 Frontend Ingress frontend-1 frontend-2 frontend-3 Datastore Service backend-1 backend-2 33
34. Evacuation Move the stateless pieces 1.9 1.13 Frontend Ingress frontend-1 frontend-2 frontend-3 Datastore Service backend-1 backend-2 34
35. Evacuation Move the stateless pieces 1.9 1.13 Evacuate stateless services Keep stateful datastores in old cluster Datastore Service backend-1 frontend-1 Frontend Ingress frontend-2 frontend-3 Datastore Service backend-2 35
36. Evacuation Move the stateless pieces Restrict Access 1.9 1.13 Evacuate stateless services Keep stateful datastores in old cluster Datastore Service backend-1 frontend-1 Frontend Ingress frontend-2 frontend-3 Datastore Service backend-2 36
37. Istio Integration Dataplane Control Plane ● ● ● ● + Service Management Identity Provider Integration Telemetry Collection ... Knows what to do ● ● ● + ● ● ● ● Handling network packets and L7 requests Routing & load balancing Security policy enforcement ... Knows how to do it Both Istio and Kubernetes currently provide service definitions We chose to implement Kubernetes services first for simplicity Istio services can be supported with the same simplicity and performance 37
38. Summary ● ● ● ● ● Based on new BPF technology Networking (CNI) Kubernetes services ○ Replacing kube-proxy ○ Multi-cluster capability (1.4) Network security ○ Identity-based, DNS aware, API aware, data protocol aware ○ Transparent encryption (1.4) Envoy/Istio Integration ○ Sidecar Acceleration ○ Transparent SSL visibility (kTLS) 38
39. Thank You! More Information: Slack: https://cilium.io/slack GitHub: https://github.com/cilium/cilium Docs: https://docs.cilium.io/ Twitter: @ciliumproject Want to hear more about Cilium at KubeCon? Implementing Least Privilege Security and Networking with BPF on Kubernetes 1:45pm - 2:20pm, Ballroom 6C 39

相关幻灯片