【刘光聪】TensorFlow内核剖析

饶森丽

2017/12/31 发布于 技术 分类

2017年,droidcon 第2次来到中国,并将于2017年11月在北京盛大开幕。参会人群包括业界领袖、技术大咖、技术开发者、大众创业者及领域从业者。大会将邀请来自Google、微软、Facebook、Ebay、Intel、Telenav、阿里巴巴、腾讯、小米、乐视、联想等国内外安卓技术与应用领域的大咖,沿袭历年国际大会特色,聚焦行业最前沿技术,碰撞切磋技术火花。

文字内容
1. Architecture Overview Programming Model Execution Model Model Training Bibliography Introduction to TensorFlow Internals Guangcong Liu Software Architect@ZTE 2017-11-16 Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 1 / 44
2. Architecture Overview Contents Programming Model Execution Model Model Training Bibliography 1 Architecture Overview 2 Programming Model 3 Execution Model 4 Model Training 5 Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 2 / 44
3. Architecture Overview Programming Model Execution Model Model Training Bibliography Architecture Overview 1 System Architecture 2 Design Principles 3 Graph Transformation Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 3 / 44
4. Architecture Overview Programming Model System Architecture System Architecture Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 4 / 44
5. Architecture Overview Programming Model Design Principles Design Principles Execution Model Model Training Bibliography Deferred ExecutionğThe construction and execution of graph are separated, and the graph execution is delayed. Primitive OPğOP is the basic computation unit. Abstract AcceleratorğSupport CPU, GPU, and ASIC. Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 5 / 44
6. Architecture Overview Programming Model Graph Transformation Graph Construction Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 6 / 44
7. Architecture Overview Programming Model Graph Transformation Graph Exection Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 7 / 44
8. Architecture Overview Graph Transformation Split Graph Programming Model Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 8 / 44
9. Architecture Overview Programming Model Graph Transformation Register Graph Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 9 / 44
10. Architecture Overview Graph Transformation Run Graph Programming Model Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 10 / 44
11. Architecture Overview Programming Model Execution Model Model Training Bibliography Programming Model 1 Dataflow Graph 2 Variable 3 Session 4 Graph Construction & Exection Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 11 / 44
12. Architecture Overview Programming Model Execution Model Dataflow Graph Graph = Set{OP} + Set{Tensor} Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 12 / 44
13. Architecture Overview Programming Model Execution Model Dataflow Graph OP: Abstract Computation Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 13 / 44
14. Architecture Overview Programming Model Dataflow Graph Tensor: Dataflow Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 14 / 44
15. Architecture Overview Programming Model Variable Initialization Model Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 15 / 44
16. Architecture Overview Programming Model Execution Model Variable Initialization Dependency Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 16 / 44
17. Architecture Overview Programming Model Session Life Cycle: Python Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 17 / 44
18. Architecture Overview Programming Model Session Life Cycle: C++ Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 18 / 44
19. Architecture Overview Programming Model Execution Model Graph Construction & Exection Graph Construction & Serialization Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 19 / 44
20. Architecture Overview Programming Model Graph Construction & Exection Example: OP Constructor Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 20 / 44
21. Architecture Overview Programming Model Graph Construction & Exection Example: Create OP Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 21 / 44
22. Architecture Overview Programming Model Execution Model Graph Construction & Exection Example: Graph Construction Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 22 / 44
23. Architecture Overview Programming Model Execution Model Model Training Bibliography Execution Model 1 Execution Model 2 Distributed Example Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 23 / 44
24. Architecture Overview Programming Model Execution Model Local Runtime Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 24 / 44
25. Architecture Overview Programming Model Execution Model Distributed Runtime Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 25 / 44
26. Architecture Overview Programming Model Client Session Create ClientSession Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 26 / 44
27. Architecture Overview Programming Model Client Session Polymorphism Creation Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 27 / 44
28. Architecture Overview Programming Model Client Session Create MasterSession Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 28 / 44
29. Architecture Overview Programming Model Client Session MasterSession Model Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 29 / 44
30. Architecture Overview Programming Model Run Step Split Graph by Task) Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 30 / 44
31. Architecture Overview Programming Model Run Step Split Graph by Device Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 31 / 44
32. Architecture Overview Example Split Graph Programming Model Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 32 / 44
33. Architecture Overview Programming Model Example Receive Tensor Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 33 / 44
34. Architecture Overview Programming Model Execution Model Model Training Bibliography Training Model 1 Compute Gradients 2 Apply Gradients 3 Training Workflow Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 34 / 44
35. Architecture Overview Programming Model Execution Model Compute Gradients Optimizer: Compute Gradients Model Training Bibliography § class Optimizer(object): def minimize(self, loss, var_list=None, global_step=None): grads_and_vars = self.compute_gradients( loss, var_list=var_list) return self.apply_gradients( grads_and_vars, global_step=global_step) def compute_gradients(loss, var_list): grads = gradients(loss, var_list, grad) return list(zip(grads, var_list)) def gradients(loss, var_list, grads=1): ops_and_grads = {} for op in reversed_graph(loss).topological_sort(): grad = op.grad_fn(grad) ops_and_grads[op] = grad ¦ return [ops_and_grads.get(var) for var in var_list] Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 ¥ . .. . . .. 35 / 44
36. Architecture Overview Programming Model Compute Gradients Gradient Function Execution Model Model Training Bibliography § @ops.RegisterGradient("op_name") def grad_func(op, grad): """construct gradient subgraph for an op type. Returns: A list of gradients, one per each input of op. """ ¦ return cons_grad_subgraph(op, grad) (y1, y2, (∂L/∂x1, ..., ym ∂L/∂x2, ) = f (x1, x2, ..., ∂L/∂xn) = .g..(,xx1n,)x2 , ..., xn ; ∂L/∂ y1, ∂L/∂ y2 , ..., ∂ L/∂yn ) ¥ Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 36 / 44
37. Architecture Overview Programming Model Compute Gradients ExampleğSquare Execution Model § @ops.RegisterGradient("Square") def SquareGrad(op, grad): x = op.inputs[0] with ops.control_dependencies([grad.op]): ¦ return grad * (2.0 * x) Model Training Bibliography ¥ Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 37 / 44
38. Architecture Overview Programming Model Apply Gradients Apply Gradients Execution Model Model Training § def apply_gradients(grads_and_vars, learning_rate): for (grad, var) in grads_and_vars: ¦ apply_gradient_descent(learning_rate, grad, var) Bibliography ¥ Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 38 / 44
39. Architecture Overview Programming Model Training Workflow Critical Path: RunStep Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 39 / 44
40. Architecture Overview Programming Model Training Workflow Distributed Initialization Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 40 / 44
41. Architecture Overview Programming Model Execution Model Model Training Bibliography Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 41 / 44
42. Architecture Overview Bibliography Papers Programming Model Execution Model Model Training Bibliography TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems, Google Inc. TensorFlow: A System for Large-Scale Machine Learning, Google Inc. Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 42 / 44
43. Architecture Overview Acknowledgments Q&A Programming Model Execution Model Model Training Bibliography Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 43 / 44
44. Architecture Overview Acknowledgments Programming Model Execution Model Model Training Bibliography Thanks Guangcong Liu (ZTE) TensorFlow Internals . . . .... .... .... . . . . .... .... .... . . 2017.11 . .. . . .. 44 / 44