腾讯游戏 周东祥 大数据分析系统在游戏领域的迭代与实践

CodeWarrior

2019/07/08 发布于 编程 分类

GIAC2019 

文字内容
1. xk I
3. u 2 E Te 2 c F c - 2 D 2 c
4. M t“
5. + , j l , o
6. 2
7. M t“
8. s ü ü ü o ü ü G ü ü ü N2G G 3SLNS ü o –” a–” ü ü t –” ü o ü t ü t ü t d
9. i p –” t o 3 l w - s o 1
10. t
11. o o o t 1 o o G d –” t t - o 3 d –” t
12. c d 52V NJ Kafka Storm TGDruid –” o d MQ t –” MySQL d o HDFS RPC&HTTP o SQL RDS CloudDB 59GVW 54GI
13. M c d x Ø 59GVW Ø 54GI Ø 52V NJ t“
14. o 1 c d j HDFS RDS h CloudDB o au b –” o r –”
15. o 1 c d j 2019.03 2019.04 LSLJ JNW NSI NS MY T IG LY OLYL H L IL LLU a% , b HU a% LVT U ] NS MG NSL RG Q Q HU W R UG]R S 2019.05 Fliter r ‘201903’ - ‘201905’ Merge max(level) Fliter max(level)>30 sum(payment) sum(payment)>100 group by uin , (b Distinct distinct(uin)
16. o 1 c d j 2019.03 2019.04 LSLJ W R F W V UG]R S F W V JG MY T IG YHJRGH SLM Q PU U IG LY U F VGIPFG NS. F W V NS OLYL IG LY H L IL LLU a% , bHU a% , (b LVT U ] NS JG 2019.05 Walk through A Fliter a% a% a% , b , b , (b Merge Merge group by uin group by uin sum(payment) sum(payment) a% a% a% , b , b , (b
17. o 1 i t“ m == Filter 1 2 3 Fliter Distinct 20 10 4 Merge group by uin max(level)>30 ‘201903’ - ‘201905’ DataNode RPC max(level) sum(payment)>100 sum(payment) r 1 2 10 20 n g n 2G G TJ 1001001 1001002 1001003 1001004 ----------- 2G G TJ % 2001001 2001002 2001003 2001004 ----------- 2G G TJ 1101001 2101002 3101003 4101004 ----------- 2G G TJ % 1201001 2201002 3201003 4201004 ----------- g n n n n 9<< distinct(uin)
18. o 1 t“ %i + s x n <1 n , s n n n v ÷
19. o 1 t“ %i n g n h n 0 && (o ),+ 000010000100 0000100001000010…0101 . I MM 1001 n -+ g
20. o 1 59GVW c ((& Ø d t“ c l d f(+.C %%W Ø ( W Ø [ ([ Ø Ø u % d e 30 7 4 15 5 7 3 10 2 0 iData 2000 4000 TDW Spark 6000 TDW MR 8000
21. o 1 t“ 59GVW Y JL Y 7Y _ WBJOL SLY i v9<< h n IBJOL ( SLY = n n ( CH R5 LJ 1WW Y n %i 1WW 4H H= = n n L i L H= L UP Y LLH= LH= H= LLL h Dt Eh ÷h h n n m m
22. M c d x Ø 59GVW Ø 54GI Ø 52V NJ t“
23. yw l j m t 1 2-10 11-20 l e m e t 1 2-10 11-20 e
24. yw r g+&&& f l j q ~ +&&& *&&& &# + cl +&&& *&&& +&&& *&&& )&&& +&&& *&&& )&&& vk d )&&& (&& & )& &&& % +&&& ,&& (&& *&&& )&&& (&& &&&
25. - t“ v 54GI DataNode-1 1 Storage Scheduler Data Stats Gather 9<< Block 1 9> n … n DataNode-2 n Block 1 … SQL Parser Query Optimizer DataNode-n Executor-n Block Execution Plan Bitcode Emitter Execute Scheduler 1 … Data Mapper Col1 Col2 Col n Bitmap Index Generator Bitmap Filter Builder Execute Engine Dynamic Bitmap Index Cache Aggregate Merger
26. i i c t“ d 1 DataNode A B Executor DataNode C Executor Block1 ,&& ) … [_ B A =8 % h [ % 2G G TJ C [ % [
27. - t“ %i )W Executor-n B j Data Mapper Col1 Col2 Col n Bitmap Index Generator Bitmap Filter Builder Execute Engine B5 53C IP THWGPU L 7A D 2F J U 6 A &&( (& ,& ( Dynamic Bitmap Index Cache E85A5 L SJ V . a b 0 2 GL 13 33 GL gender d e d e &d e d e + 0 2 %) 0 2 GV G . a a d(&e 0 2 &d e d((e d /e . pb b0 2 + %) & & GV G d me 0 2 &d te &d e &di e Aggregate Merger
28. yw j A1 B1 C1 0-10 D2 C2 11-20 B2 A1∩B1∩ j C1∩D1 D1 D1 f B1 B2 j ( j ) j * , j - j * j + j j . j / j & j j * j + j ( D1 A2 o j ) j
29. i 9> 76 SELECT mean(BKey_1 / 8) / 6 FROM #1412923.20151019 WHERE BKey_2 = 1 AND BKey_1 BETWEEN 10 AND 100 =8 Ø B B LSLJ Ø 9C 5TP LY_ A n d 9Ce y LSLJ AL TLYNL SUM S SUM SUM BKey_1 / 8 SUM COUNT COUNT COUNT COUNT MEAN mean(BKey_1 / 8) / 6
30. 54GI t“ m l l 30 Ø ) CB+f(* f,*7 Ø u • , j • , • &o h (g 20 10 %(W % W 1.058 0.919 0 2 1.292 1.489 10 15 5 W iData spark h 9> n 35 30 25 20 15 10 5 0 0.855 0.879 1.364 1.243 1.182 . - =8 , 76 + *& o & )( o (. o n (* o (& o n n 30.902 28.854 28.23 26.54 25.385 (g c n 27.758 24.283 24.129 23.911 iData spark
31. M c d x Ø 59GVW Ø 54GI Ø 52V NJ t“
32. t o 3
33. t o u 3 ~ 3,174,023 2,661,884
34. t x o c 3 d v2V NJ n n <00 AB n n n 42 2V NJ TJ OW n n v<VTUM
35. M t“
36. 2
37. M c t“ d Ø 59GVW Ø 54GI Ø 52V NJ g