1. Protect Your Kubernetes Data Friends Don’t Let Friends Leave Their Kubernetes Data Unprotected Rita Zhang Principal Software Engineer @ Microsoft @ritazzhang
2. Rita Zhang • Software engineer @ Microsoft, San Francisco • Kubernetes upstream features, Azure Kubernetes Service • Maintainer for K8s KMS plugin for Azure Key Vault, Keyvaultflexvolume, aad-pod-identity @ritazzhang
3. @ritazzhang
4. @ritazzhang
5. An attacker who can successfully access your cluster database can compromise your entire cluster and have access to your cloud resources. @ritazzhang
6. Kubernetes Database ž Uses etcd as its persistent storage for API objects ž Stores secrets as base64 encoded plaintext @ritazzhang https://kubernetes.io/docs/concepts/overview/components/#etcd
7. “Authentication was added in etcd 2.1. … etcd before 2.1 was a completely open system; anyone with access to the API could change keys. In order to preserve backward compatibility and upgradability, this feature is off by default.” Read more from coreos etcd doc @ritazzhang https://elweb.co/the-security-footgun-in-etcd/
8. I did a simple search on shodan and came up with 2,284 etcd servers on the open internet. CREDENTIALS, a lot of CREDENTIALS. Credentials for things like cms_admin, mysql_root, postgres, etc. Passwords for databases of all kinds, AWS secret keys, and API keys and secrets for a bunch of services. GET http://<ip address>:2379/v2/keys/?recursive=true @ritazzhang
9. @ritazzhang
11. So…How do I secure my cluster? • There are many things you can do • Control access to the Kubernetes APIs • Control access to the Kubelet • Control privileges containers run with • Restrict network access • Restrict resource access • Restrict access to etcd • Encrypt etcd data at rest • Store application secrets outside of Kubernetes • Restrict access to resources with pod identity @ritazzhang https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/
12. Secrets kubectl create secret generic secret1 Secret API Server etcd Kubernetes Master Kubelet Node
13. Pod using Secret kubectl create –f pod-using-secret.yaml Pod API Server etcd Kubernetes Master Kubelet Pod Node Mount path: /etc/foo Pod secret
14. Encryption at Rest Secret API Server etcd Kubernetes Master @ritazzhang • Kubernetes v1.7+ • etcd v3 required • encryption using keys in config file on master • Plain text, encoded with base64 https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
15. @ritazzhang
16. Key Management Service (KMS) Provider for Encryption at Rest KMS KMS provider Secret API Server etcd Kubernetes Master @ritazzhang • Kubernetes v1.10, v1.13 stable • etcd v3 required • Separate key management from K8s cluster management • Supports encryption using keys stored in external trusted Key Management Service (KMS), e.g. Azure Key Vault, Google Cloud KMS • HSM-protected keys https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/
17. Before V1.13 @ritazzhang
18. V1.13 @ritazzhang
19. Demo: K8s cluster with Azure Key Vault data encryption @ritazzhang https://github.com/Azure/kubernetes-kms
20. High Level Design @ritazzhang
21. With the KMS provider plugin, we can encrypt Kubernetes data stored in etcd at rest with a KMS managed key. @ritazzhang
22. What if instead of storing my secrets in etcd, I want to store and manage access to application secrets outside of Kubernetes? @ritazzhang
23. Pod using Key Vault Flexvolume kubectl create -f pod-using-keyvaultflexvolume.yaml Pod API Server Kubernetes Master Kubelet Pod Node KMS Mount path: /etc/foo Pod Keyvault flexvolume
24. Kubernetes Key Vault FlexVolume ž Flexvolume enables users to mount vendor volumes into kubernetes. It expects vendor drivers to be installed in the volume plugin path on every kubelet node. ž Key Vault flexvol driver makes a request to Key Management Service, (e.g. Azure Key Vault) and mounts secret and secret value as a volume to pods ž Separation of concern/role based management access to secrets @ritazzhang https://github.com/Azure/kubernetes-keyvault-flexvol
25. Demo: K8s cluster with Azure Key Vault flexvolume @ritazzhang https://github.com/Azure/kubernetes-keyvault-flexvol
26. With the Kubernetes Key Vault FlexVolume driver, we can store and retrieve secrets from an Azure Key Vault instance and mount the values as a volume to containers. @ritazzhang
27. What if I want to restrict access to my cloud resources to specific pods? @ritazzhang
28. AAD Pod Identity ž Restrict/enable pods to access individual resources that depend on Azure AD for access with its own identity (e.g. Azure SQL server or your own custom API that uses AAD) ž Kubernetes Custom Resource Definition objects that map pods to Azure AD identities ž When pods request access to a resource that uses Azure AD for access, a matching Azure identity is assigned @ritazzhang https://github.com/Azure/aad-pod-identity
29. AAD Pod Identity @ritazzhang https://docs.microsoft.com/en-us/azure/aks/operator-best-practices-identity - use-pod-identities
30. Demo: K8s cluster with AAD Pod Identity @ritazzhang https://github.com/Azure/aad-pod-identity
31. With AAD Pod Identity, we can restrict and enable specific pods access to resources that need Azure AD for access based on its identity. @ritazzhang
32. Demo: K8s cluster with Azure Key Vault Flexvolume and AAD Pod Identity @ritazzhang https://github.com/Azure/kubernetes-keyvault-flexvol - option-2---pod-identity
33. Recap Azure Key Vault KMS plugin • Use a key in Key Vault for etcd encryption • Secrets/keys/certs are stored in etcd, managed as part of Kubernetes • Restrict access using K8s concepts: RBAC, Service Accounts, namespaces • Bring your own keys, import or generate HSM-protected (Hardware Security Modules) keys • Available on AKS-engine Flexvol + AAD Pod Identity • Mounts secrets/keys/certs to the pod using a flexvolume • Secrets/keys/certs are stored in Azure Key Vault • Restrict access to secrets/keys/certs with specific pod identities • Manage secrets in Key Vault via Azure API’s/CLI/Portal • Separation of concerns/role-based management access to secrets • Import or generate HSM-protected (Hardware Security Modules) keys • Industry compliance • More granular RBAC at the pod level @ritazzhang
34. Resources https://ritazh.com/using-azure-key-vault-for-kubernetes-data-encryptiond5eac8daee71 https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/ https://github.com/Azure/aksengine/blob/master/docs/kubernetes/features.md#azure-key-vault-data-encryption https://github.com/kubernetes/kubernetes/pull/55684 https://github.com/Azure/kubernetes-kms https://github.com/GoogleCloudPlatform/k8s-cloudkms-plugin/ https://github.com/Azure/kubernetes-keyvaultflexvol @ritazzhang https://github.com/Azure/aad-pod-identity