Kubernetes 日志平台建设最佳实践

QCon大会

2019/05/14 发布于 技术 分类

QCon  QCon2019 

文字内容
1. -
3. p KK PW • KKn • + • A e I sr s D 4 O n 1 2s Bs / T b ) o (02s (02 PW
4. • • • •
5. • • • •
6. 3 1 G 9 2 2 0 D 2
7. B AP LP /P O T I
8. Kubernetes &
9. Kubernetes • • • • • • • • • • • • Kubernetes Stdout Journal Event • • • IoT • • • 100W * 1M = 1TB 100W * 1% = 10000 / • • • SAAS LogReduce
10. • • • • : 16PB+ : 40 + Logtail : 200W+ CPU Kubernetes 10 + 5PB+ 1.6% 8W+ 43MB
11. • • • •
12. 1 DaemonSet or Sidecar DaemonSet Sidecar • • DaemonSet PASS Sidecar
13. 2 • • 5W 40 40GB SSD 2PB SSD /app/data/logs Sidecar
14. M 3 b kP iLogtail A J C D f Ak B h a f b B C k C k Gvn uh uh Fk S vn b vn Fk b ts C o B lp lp lp m e r ce o o
15. PB grep pgm awk sed egrep pssh A and B or (C and D) order.value > 10 and status : ok 1GB 1TB 1PB
16. • • • • Session TraceId
18. SQL92 / Join IP
20. (1/2) • • • • • A Ingress • 90% A/B Test • • • • >10S Service Service In A Ratio ID=1002 95 68% ID=1002 Method=Get 80 74% ID=1002 Method=Get IP=120343 40 37% ID=1003 20 19% Service B B A In A Ratio In B Ratio ID=1002 95 68% 100 3% ID=1002 Method=Get 80 74% 100 3% ID=1002 Method=Get IP=120343 40 37% 250 7.5% ID=1003 20 19% 500 15%
21. (2/2) +
22. Big Data Analytics Mobile & Web Mobile Web Massive Structured, Unstructured & Semistructured Data OSS PAI Batch Processing Text & Logs Logs MaxCompute Pig Hybrid Storage Array SQL NoSQL Services & Languages HIVE EMR Hadoop Interactive Analytics Visualization QuickBI DataV DLA Log Service / LogShipper Real-time Data Stream IoT & Devices Camera IoT Log Service / LogHub Log Service / Analytics Stream Processing SparkStreaming Flink Storm Function Compute Log Service / Dashboard
23. Kubernetes Audit Ingress Mesh Event HPA DaemonSet Stdout Sidecar Journal Mesh Event Appender … Ø Ø Ø Ø Ø Ø 1-3 / >
24. • • • •
25. 1- • • I PAAS Mesh/Trace c Ingress R • I • • DCS da + ServiceMesh 0e Ingress B + +1 e A
26. 2Trace • • API SLS • API Trace
27. • • • •
28. Automation • FUSE • • • • All in DaemonSet • • • ITOps