滴滴 梁李印 实时计算在滴滴出行的实践之路

CodeWarrior

2019/07/08 发布于 编程 分类

GIAC2019 

文字内容
3. H • H • • 20 1 6 A4
4. 4 2 & & 3104 4
5. StreamSQL &CEP 1. 2. 3. 2018 2017 2016 20+ 1. 2. Spark Flink 3. I 1. 2. 3. 2019 StreamSQL II CEP StreamSQL 1. Flink 2.StreamSQL 3.Meta
6. StreamSQL IDE YARN Router/ Flink Flink Flink YARN YARN YARN HDFS HDFS HDFS DC1 DC2 DC3
7. 2 BU2 1 BU1 GroupA GroupB GroupC GroupD GroupE NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager LabelA LabelB NoLabel LabelE
8. • APP1 APP2 APP4 APP5 APP6 APP7 APP3 APP8 Queue1 • Queue2 • Queue3 C ( • NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager LabelA LabelB NoLabel )
9. • % • ResourceManager % • heartbeat update vcore heartbeat • update vcore NodeManager update monitor Container monitor % NodeManager update monitor Container update Container update monitor Container 5 0 3 C C
10. R • N • Y A • • A
11. 4 2 & & 3104 4
12. POI • • • • • • -->ES/HDFS • binlog->Druid -->HBase/GeoMesa • • • • • • •
13. Join/ Flink Druid
14. Flink KV Store
15. xx /Join CEP xx
17. • •
18. 4 2 & & 3104 4
19. DDL UDX Join SQL IDE
20. HBase DDMQ StreamSQL Kafka GeoMesa • ElasticSearch • Druid • KV Store HDFS
21. UDF JSON UDX UDTF UDX UDAF MAP UDX Hive UDX UDF UDTF UDAF
22. source1 Join Operator Sink source2 TTLState1 TTLState2 • • • Join Operator source Sink TTLState cache
23. il • D • H • kS F J L 7 e m 7XQU 7 %% 0 Tm 7a
24. ,0 Sa • 1 6 _ • B 1 11 2 && = = =B 3( B =3 3 ) 3 5 = . 2 = = =B 3( M • && • P T Fb c E
25. 4 2 & & 3104 4
26. SQL IDE
27. Killed/Failed NodeManager JobManager TaskManager Flume Druid Kafka
28. DB Kafka Flink Kafka Druid Dashboard
31. DB Kafka table1 topic1 table1-stream table2 topic2 table2-stream table3 topic3 table3-stream StreamSQL
32. Flink StreamSQL IDE StreamSQL Service MetaStore SQL Flink Spark MetaStore Hive Presto
33. Q • E • P • • S L C
34. Thank You