ebay中国技术主管茹炳晟 - 测试基础架构最佳实践

公羊鹤梦

2017/11/14 发布于 技术 分类

全球化的测试基础架构平台设计与实践 测试平台框架设计 测试数据准备的最佳实践 测试与DevOps的整合实践 全球化测试的基础架构实践

文字内容
1. 测试基础架构 最佳实践 茹炳晟(Robin Ru) ebay中国研发中心
2. 茹炳晟 (Robin Ru) 产地:上海 主要工作经历: § ebay 中国研发中心 -- 测试基础架构技术主管 § Hewlett-Packard 惠普软件(中国)研发中心 -- 测试架构师、资深测试专家 § Alcatel-Lucent阿尔卡特朗讯(上海)研发中心 -- 测试技术主管 § Cisco 思科(中国)研发中心 -- 资深测试开发工程师 兴趣爱好: § 户外运动爱好者 § 全球自由行爱好者 § 高级开放水域潜水员 + 高氧空气潜水员 联系方式: § biru@ebay.com or 微信
3. 引言 q3小时 à 讲什么 (What) q各种测试架构和测试框架 à 为什么(Why) q知其然知其所以然 à 怎么讲 (How) q站在前人的肩膀上 à 收获 (Best Practice)
4. 大型网站分布式架构 &'16HUYHUV 5HYHUVH 3UR[\6HUYHUV /RDG %DODQFHU Application B Server n Application B Server 2 Application B Server 1 0HVVDJH4XHXH 6HUYHUV Application A Server n Application A Server 2 Application A Server 1 Distributed Service i Server n Distributed Service i Server 2 Distributed Service i Server 1 &RPPRQ6HUYLFH /RFDO&DFKH 'DWD$FFHVV 0RGXOH 6HDUFK(QJLQH 6HUYHUV 1R64/ 6HUYHUV Distributed Cache Server n Distributed Cache Server 2 Distributed Cache Server 1 Remote Distributed Cache Distributed File Server n Distributed File Server 2 Distributed File Server 1 Files Distributed DB Server n Distributed DB Server 2 Distributed DB Server 1 Database
5. GUI Automation Test 架构 Business Requirement Feature Requirement Test Requirement Unified Flow Framework Base on UF Reusable script snippet Unified Business Flow Abstract Adopt Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Page Encapsulation Code Generator Test Case Design 'DWDGULYHQ 7HVW'DWD Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW Test Date Provider Test Date Preparation Tool Encapsulated utiliites which call API or directly DB access WRSUHSDUH WHVWGDWD 6\VWHP8QGHU7HVW 687 Use OOB/GDS test data 2XWRIER[7HVW'DWD *ROGHQ'DWD6HW
6. API Automation Test 架构 API Design Specification API Parameters’ Type API Test Case Design API Test Case Data-driven Test Data Parallel Execution Controller Automatic Test Data Generator 5HTXHVW5HVSRQVH +LVWRU\'DWDEDVH Load Generator Cluster API Test Executor 0$8,%UHH]H5(67$VVXUHG ReTXHVW Response 6\VWHP8QGHU7HVW 687 Test Data Service
7. Test Data Platform 架构 Test Date Adapter Web Service 5(67IXO UI Interface for management Test Data Core Service Offering data Meta data management Recipe data management Data quantity management Data quality management Test Data Meta DB Recipe Definition Meta data for business objects Out-of-box test data Meta data for data quantity Meta data for data quality Unified Test Data Controller Test Data Preparation Tool Test Data Validation Tool DEV Test Data Preparation Tool 6\VWHP8QGHU7HVW  687
8. Test Execution Environment 架构 Business Requirement Feature Requirement Test Requirement Test Case Design 7HVW&DVH 5HSRVLWRU\ Pull Build Jenkins Slave n Jenkins Slave 2 Test Requests Jenkins Slave 1 Test Runner Test Execution System Test Requests dispatch Jenkins Master Dynamicallty add/delete -HQNLQV VODYHE\XVLQJ'RFNHU Jenkins auto scale service Push Test Case Locally develop/debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Run Run Run Run 6HOHQLXP 1RGH 6HOHQLXP 1RGH  6HOHQLXP 1RGHQ Dispatch and auto scale service dispatch dispatch dispatch 6HOHQLXP+XE dispatch 666HHHOOEHEHOEHQ\Q\Q\LLX'X'LX'PPRRPRFFF1N1N1NHRHRHRUGUGUGHHHVVV Dynamicallty add/delete selenium node in Grid accoriding to test load
9. Test Report Platform 架构 6HOHQLXP -HQNLQV 6SOXQN Requirement Management System (JIRA/DOORS/ALM) Test Report Service 7HVW([HFXWLRQ 5HSRUW0HWD'DWD Defect close then send root cause back to Analysis Service, it will be used as machine learning sample set Full Trace Test Report Failed Test Cases Test Report Trace Analysis Service Defect Automatic Classificate and Submit Defect Management System (JIRA$/0) GUI Video-based Test Report GUI Story Board Test Report 'HIHFW$QDO\VLV 0DFKLQH/HDUQLQJ .QRZOHGJH'DWDEDVH Multi-Site Story Board Comparison Report Generator Service LQA submit translation and layer out defect Multi-Site Story Board Comparison Report Test Summary Report Test Report Statistics Service Test Trend Report
10. Agenda q 测试基础架构的范畴 q 从大型网站技术架构的演变谈起 q 测试基础架构的演变 Ø GUI Automation Test Framework的演变 Ø Test Data Platform的演变 Ø API Automation Test Framework的演变 Ø Test Execution Environment的演变 Ø Test Execution and Management Platform的演变 Ø Test Report Platform的演变
11. Agenda q 代码级单元测试的最佳实践 Ø 测试架构师在代码级单元测试扮演什么样的角色 Ø 代码的基本特征 Ø 代码缺陷产生的原因 Ø 代码错误的分类 Ø 代码级测试方法的分类 Ø 完备测试用例的设计 Ø 代码测试覆盖率的衡量
12. Agenda q 全球化Site自动化测试的最佳实践 Ø What‘s Global Site Test ? Ø Global Site Test Challenge Ø Global Test Capability Brief Introduction and Key Value Ø Global Test Capability Design Philosophy Ø Go deeper into Global Test Capability v Global site test base utilities v Global site test data utilities v Global Configuration Repository v Unified Flow Framework v Multi-Site Story Board Test Report v Global Site Test Overall Architecture
13. 测试基础架构的范畴 q Automation Test Framework Ø Code Level Unit Test Ø Code Level Integration Test Ø GUI Level Ø API Level q Test Execution Environment Ø Jenkins Ø Selenium Gird Ø Other q Test Case Execution and Management Platform q Test Data Platform q DevOps Integration Interface
14. 从大型网站技术架构的演变谈起 网站架构Hello World Application Server Files $SSOLFDWLRQ Database
15. 从大型网站技术架构的演变谈起 应用服务和数据服务分离 Application Server $SSOLFDWLRQ File Server Files DB Server Database
16. 从大型网站技术架构的演变谈起 使用Cache改善网站性能 Application Server $SSOLFDWLRQ /RFDO&DFKH Distributed Cache Server nn Distributed Cache Server 2 Distributed Cache Server 1 Remote Distributed Cache File Server Files DB Server Database
17. 从大型网站技术架构的演变谈起 使用Application Server Cluster改善网站并发处理能力 /RDG%DODQFHU Application Server n Application Server 2 Application Server 1 $SSOLFDWLRQ /RFDO&DFKH Distributed Cache Server nn Distributed Cache Server 2 Distributed Cache Server 1 Remote Distributed Cache File Server Files DB Server Database
18. 从大型网站技术架构的演变谈起 数据库读写分离 /RDG%DODQFHU Application Server n Application Server 2 Application Server 1 $SSOLFDWLRQ /RFDO&DFKH 'DWD$FFHVV 0RGXOH Read DB Server (Slave) Database Distributed Cache Server nn Distributed Cache Server 2 Distributed Cache Server 1 Remote Distributed Cache File Server Files Write DB Server 0DVWHU Database Master/Slave Sync up
19. 从大型网站技术架构的演变谈起 使用Reverse Proxy和CDN加速网站响应 &'16HUYHUV 5HYHUVH 3UR[\6HUYHUV /RDG %DODQFHU Application Server n Application Server 2 Application Server 1 $SSOLFDWLRQ /RFDO&DFKH 'DWD$FFHVV 0RGXOH Read DB Server (Slave) Database Distributed Cache Server nn Distributed Cache Server 2 Distributed Cache Server 1 Remote Distributed Cache File Server Files Write DB Server 0DVWHU Database Master/Slave Sync up
20. 从大型网站技术架构的演变谈起 使用Distributed File Server和Distributed DB Server &'16HUYHUV 5HYHUVH 3UR[\6HUYHUV /RDG %DODQFHU Application Server n Application Server 2 Application Server 1 $SSOLFDWLRQ /RFDO&DFKH 'DWD$FFHVV 0RGXOH Distributed Cache Server n Distributed Cache Server 2 Distributed Cache Server 1 Remote Distributed Cache Distributed File Server n Distributed File Server 2 Distributed File Server 1 Files Distributed DB Server n Distributed DB Server 2 Distributed DB Server 1 Database
21. 从大型网站技术架构的演变谈起 使用NoSQL和Search Engine &'16HUYHUV 5HYHUVH 3UR[\6HUYHUV /RDG %DODQFHU Application Server n Application Server 2 Application Server 1 $SSOLFDWLRQ /RFDO&DFKH 'DWD$FFHVV 0RGXOH 6HDUFK(QJLQH 6HUYHUV 1R64/ 6HUYHUV Distributed Cache Server n Distributed Cache Server 2 Distributed Cache Server 1 Remote Distributed Cache Distributed File Server n Distributed File Server 2 Distributed File Server 1 Files Distributed DB Server n Distributed DB Server 2 Distributed DB Server 1 Database
22. 从大型网站技术架构的演变谈起 业务拆分 Application B Server n Application B Server 2 Application B Server 1 &'16HUYHUV 5HYHUVH 3UR[\6HUYHUV /RDG %DODQFHU 0HVVDJH4XHXH6HUYHUV Application A Server n Application A Server 2 Application A Server 1 $SSOLFDWLRQ /RFDO&DFKH 'DWD$FFHVV 0RGXOH 6HDUFK(QJLQH 6HUYHUV 1R64/ 6HUYHUV Distributed Cache Server n Distributed Cache Server 2 Distributed Cache Server 1 Remote Distributed Cache Distributed File Server n Distributed File Server 2 Distributed File Server 1 Files Distributed DB Server n Distributed DB Server 2 Distributed DB Server 1 Database
23. 从大型网站技术架构的演变谈起 基于分布式服务架构 &'16HUYHUV 5HYHUVH 3UR[\6HUYHUV /RDG %DODQFHU Application B Server n Application B Server 2 Application B Server 1 0HVVDJH4XHXH 6HUYHUV Application A Server n Application A Server 2 Application A Server 1 Distributed Service i Server n Distributed Service i Server 2 Distributed Service i Server 1 &RPPRQ6HUYLFH /RFDO&DFKH 'DWD$FFHVV 0RGXOH 6HDUFK(QJLQH 6HUYHUV 1R64/ 6HUYHUV Distributed Cache Server n Distributed Cache Server 2 Distributed Cache Server 1 Remote Distributed Cache Distributed File Server n Distributed File Server 2 Distributed File Server 1 Files Distributed DB Server n Distributed DB Server 2 Distributed DB Server 1 Database
24. 测试架构的演变 q GUI Automation Test Framework的演变 q Test Data Platform的演变 q API Automation Test Framework的演变 q Test Execution Environment的演变 q Test Execution and Management Platform的演变 q Test Report Platform的演变 q DevOps Integration Interface的演变
25. 测试架构的演变 - GUI Automation Test Framework 最原始的GUI测试 Business Requirement Feature Requirement Test Requirement Test Case Design manually WHVW execute according to test case design /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687
26. 测试架构的演变 - GUI Automation Test Framework 基于录制回放的GUI自动化测试 Business Requirement Feature Requirement Test Requirement Test Case Design UI Automation Test Case Create test case by UI recording Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687
27. 测试架构的演变 - GUI Automation Test Framework 基于可重用测试代码片段构成GUI测试用例 Business Requirement Feature Requirement Test Requirement Test Case Design Operation-based reusable script snippet UI recording Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687
28. 测试架构的演变 - GUI Automation Test Framework 基于Component Abstract构成GUI测试用例 Business Requirement Feature Requirement Test Requirement Test Case Design Reusable script snippet Component Operation Abstract Component Abstract Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687
29. 测试架构的演变 - GUI Automation Test Framework 基于Page Abstract构成GUI测试用例 Business Requirement Feature Requirement Test Requirement Test Case Design Reusable script snippet Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687
30. 测试架构的演变 - GUI Automation Test Framework 基于Business Flow Abstract构成GUI测试用例 Business Requirement Feature Requirement Test Requirement Reusable script snippet Business Flow Abstract Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Test Case Design Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687
31. 测试架构的演变 - GUI Automation Test Framework 测试数据 – 数据驱动的测试 + 测试数据准备 Business Requirement Feature Requirement Test Requirement Reusable script snippet Business Flow Abstract Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Test Case Design Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 'DWDGULYHQ 7HVW'DWD Test Date Provider Test Date Preparation Tool Encapsulated utiliites which call API or directly DB access WRSUHSDUH WHVWGDWD
32. 测试架构的演变 - GUI Automation Test Framework 测试数据 – 使用Out-of-box Test Data / Golden Data Set Business Requirement Feature Requirement Test Requirement Reusable script snippet Business Flow Abstract Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Test Case Design 'DWDGULYHQ 7HVW'DWD Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW Test Date Provider Test Date Preparation Tool Encapsulated utiliites which call API or directly DB access WRSUHSDUH WHVWGDWD 6\VWHP8QGHU7HVW 687 Use OOB/GDS test data 2XWRIER[7HVW'DWD *ROGHQ'DWD6HW
33. 测试架构的演变 - GUI Automation Test Framework 基于Unified Flow Framework实现Flow Branch控制 Business Requirement Unified Flow Framework Feature Requirement Test Requirement Base on UF Reusable script snippet Unified Business Flow Abstract Adopt Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Test Case Design 'DWDGULYHQ 7HVW'DWD Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW Test Date Provider Test Date Preparation Tool Encapsulated utiliites which call API or directly DB access WRSUHSDUH WHVWGDWD 6\VWHP8QGHU7HVW 687 Use OOB/GDS test data 2XWRIER[7HVW'DWD *ROGHQ'DWD6HW
34. 测试架构的演变 - GUI Automation Test Framework 基于Page Encapsulation Code Generator提高Page Abstract的效率 Business Requirement Unified Flow Framework Feature Requirement Test Requirement Base on UF Reusable script snippet Unified Business Flow Abstract Adopt Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Page Encapsulation Code Generator Test Case Design 'DWDGULYHQ 7HVW'DWD Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW Test Date Provider Test Date Preparation Tool Encapsulated utiliites which call API or directly DB access WRSUHSDUH WHVWGDWD 6\VWHP8QGHU7HVW 687 Use OOB/GDS test data 2XWRIER[7HVW'DWD *ROGHQ'DWD6HW
35. 测试架构的演变 – Test Data Platform 引入Test Data Service,提供统一的测试数据准备服务 Business Requirement Feature Requirement Test Requirement Test Case Design Unified Flow Framework Base on UF Reusable script snippet Unified Business Flow Abstract Test Case Adopt Page Operation Abstract Page Abstract Replay test case to execute test Component Operation Abstract Component Abstract /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Page Encapsulation Code Generator 'DWDGULYHQ 7HVW'DWD Test Date Provider Test Date Adapter Call TDS to prepare(create/ search/update) test data on SUT Test Date Service
36. 测试架构的演变 – Test Data Platform Test Data Service的雏形 Business Requirement Unified Flow Framework Feature Requirement Test Requirement Base on UF Reusable script snippet Unified Business Flow Abstract Adopt Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Page Encapsulation Code Generator Test Case Design Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 'DWDGULYHQ 7HVW'DWD Test Date Provider Test Date Adapter Call TDS to prepare(create/ search/update) test data on SUT Test Date Service Test Date Adapter Web Service 5(67IXO Test Data Preparation Tool Encapsulated utiliites which call API or directly DB access WR SUHSDUHWHVWGDWD Use OOB/GDS test data 2XWRIER[7HVW'DWD *ROGHQ'DWD6HW 6\VWHP8QGHU7HVW 687
37. 测试架构的演变 – Test Data Platform 引入Test Data Core Service和Recipe Business Requirement Unified Flow Framework Feature Requirement Test Requirement Base on UF Reusable script snippet Unified Business Flow Abstract Adopt Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Page Encapsulation Code Generator Test Case Design Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 'DWDGULYHQ 7HVW'DWD Test Date Provider Test Date Adapter Call TDS to prepare(create/ search/update) test data on SUT Test Date Service Test Date Adapter Web Service 5(67IXO Test Data Core Service Offering data Meta data management Recipe data management Test Data Preparation Tool Encapsulated utiliites which call API or directly DB access WRSUHSDUH WHVWGDWD 6\VWHP8QGHU7HVW  687 Test Data Meta DB Recipe Definition Meta data for business objects Out-of-box test data
38. 测试架构的演变 – Test Data Platform 引入Data Quantity / Quality管理 Business Requirement Unified Flow Framework Feature Requirement Test Requirement Base on UF Reusable script snippet Unified Business Flow Abstract Adopt Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Page Encapsulation Code Generator Test Case Design Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 'DWDGULYHQ 7HVW'DWD Test Date Provider Test Date Adapter Call TDS to prepare(create/ search/update) test data on SUT Test Date Service Test Date Adapter Web Service 5(67IXO UI Interface for management Test Data Core Service Offering data Meta data management Recipe data management Data quantity management Data quality management Unified Test Data Controller Test Data Preparation Tool Encapsulated utiliites which call API or directly DB access WRSUHSDUH WHVWGDWD 6\VWHP8QGHU7HVW  687 Test Data Validation Tool Test Data Meta DB Recipe Definition Meta data for business objects Out-of-box test data Meta data for data quantity Meta data for data quality
39. 测试架构的演变 – Test Data Platform 引入Unified Controller接入不同Test Data Tools Test Date Adapter Business Requirement Unified Flow Framework Feature Requirement Test Requirement Base on UF Reusable script snippet Unified Business Flow Abstract Adopt Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Page Encapsulation Code Generator Test Case Design Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 'DWDGULYHQ 7HVW'DWD Test Date Provider Test Date Adapter Call TDS to prepare(create/ search/update) test data on SUT Test Date Service Web Service 5(67IXO UI Interface for management Test Data Core Service Offering data Meta data management Recipe data management Data quantity management Data quality management Test Data Meta DB Recipe Definition Meta data for business objects Out-of-box test data Meta data for data quantity Meta data for data quality Unified Test Data Controller Test Data Preparation Tool Test Data Validation Tool DEV Test Data Preparation Tool Performance Test Data Preparation Tool 6\VWHP8QGHU7HVW  687
40. 测试架构的演变 – API Automation Test Framework 最原始的API测试 API Design Specification API Test Case Design API Test Tool (POSTMAN / SOAPUI) ReTXHVW Response 6\VWHP8QGHU7HVW 687 Collections Team Library
41. 测试架构的演变 – API Automation Test Framework 引入API Test Executor实现Code-based API自动化测试 API Design Specification API Test Case Design API Test Case with Test Data API Test Executor 0$8,%UHH]H5(67$VVXUHG ReTXHVW Response 6\VWHP8QGHU7HVW 687
42. 测试架构的演变 – API Automation Test Framework Test Case和Test Data分离实现Data-Driven Test API Design Specification API Test Case Design API Test Case Data-driven Test Data API Test Executor 0$8,%UHH]H5(67$VVXUHG ReTXHVW Response 6\VWHP8QGHU7HVW 687
43. 测试架构的演变 – API Automation Test Framework 引入Data-Driven Test Data的自动生成 API Parameters’ Type API Design Specification API Test Case Design API Test Case Data-driven Test Data Automatic Test Data Generator API Test Executor 0$8,%UHH]H5(67$VVXUHG ReTXHVW Response 6\VWHP8QGHU7HVW 687
44. 测试架构的演变 – API Automation Test Framework 引入Test Data Service生成测试数据 API Parameters’ Type API Design Specification API Test Case Design API Test Case Data-driven Test Data Automatic Test Data Generator API Test Executor 0$8,%UHH]H5(67$VVXUHG ReTXHVW Response 6\VWHP8QGHU7HVW 687 Test Data Service
45. 测试架构的演变 – API Automation Test Framework 引入Parallel Execution Controller实现API并发测试 API Parameters’ Type API Design Specification API Test Case Design API Test Case Data-driven Test Data Automatic Test Data Generator Parallel Execution Controller API Test Executor 0$8,%UHH]H5(67$VVXUHG ReTXHVW Response 6\VWHP8QGHU7HVW 687 Test Data Service
46. 测试架构的演变 – API Automation Test Framework 引入Load Generator Cluster实现API高并发和压力测试 API Parameters’ Type API Design Specification API Test Case Design API Test Case Data-driven Test Data Automatic Test Data Generator Parallel Execution Controller Load Generator Cluster API Test Executor 0$8,%UHH]H5(67$VVXUHG ReTXHVW Response 6\VWHP8QGHU7HVW 687 Test Data Service
47. 测试架构的演变 – API Automation Test Framework 引入2R History Database实现API Diff Identification API Parameters’ Type API Design Specification API Test Case Design API Test Case Data-driven Test Data Automatic Test Data Generator Parallel Execution Controller 5HTXHVW5HVSRQVH +LVWRU\'DWDEDVH Load Generator Cluster API Test Executor 0$8,%UHH]H5(67$VVXUHG ReTXHVW Response 6\VWHP8QGHU7HVW 687 Test Data Service
48. 测试架构的演变 – API Automation Test Framework 微服务架构下的API测试挑战 q API的种类数量多 q API测试高覆盖率的代价大 q 微服务架构之间的耦合 q 第三方服务的耦合 6HUYLFH$ 6HUYLFH% ReTXHVW Response ReTXHVW Response 6HUYLFH7  687
49. 测试架构的演变 – API Automation Test Framework 基于消费者契约的API测试 – 基于契约的测试与验证 6HUYLFH$ &RQVXPHU$ 6HUYLFH% FRQVXPHU% Consumer A Request/Response Suite Record Proxy 6HUYLFH7  6873URYLGHU Test Execution Test Validation Service T Contract Suite Consumer B Request/Response Suite
50. 测试架构的演变 – API Automation Test Framework 基于消费者契约的API测试 – 基于契约的Mock Service实现API依赖解耦 Service T API Test 6HUYLFH7  687 6HUYLFH; 6HUYLFH<
51. 测试架构的演变 – API Automation Test Framework 基于消费者契约的API测试 – 基于契约的Mock Service实现API依赖解耦 Service T API Test 6HUYLFH7  687 Mock Service X 6HUYLFH; 6HUYLFH< Mock Service Y Service X Contract Suite Service Y Contract Suite
52. 测试架构的演变 - Test Execution Environment 基于Jenkins触发测试执行 Business Requirement Feature Requirement Test Requirement Test Case Design 7HVW&DVH 5HSRVLWRU\ Pull Build Push Test Case Locally develop/debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Jenkins Run 5HPRWH7HVW([HFXWLRQ (QYLURQPHQW 7HVW%HG
53. 测试架构的演变 - Test Execution Environment 基于Test Runner / Test Execution System Business Requirement Feature Requirement Test Requirement Test Case Design 7HVW&DVH 5HSRVLWRU\ Pull Build Push Test Case Locally develop/debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Test Runner Test Execution System Test Requests Jenkins Run 5HPRWH7HVW([HFXWLRQ (QYLURQPHQW 7HVW%HG
54. 测试架构的演变 - Test Execution Environment 基于Selenium Grid提高测试并行执行能力 Business Requirement Feature Requirement Test Requirement Test Case Design Test Runner Test Execution System Test Requests 7HVW&DVH 5HSRVLWRU\ Pull Build Jenkins Push Test Case Locally develop/debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Run Run Run 6HOHQLXP 1RGH 6HOHQLXP 1RGH  6HOHQLXP 1RGHQ dispatch Test Requests dispatch 6HOHQLXP+XE dispatch
55. 测试架构的演变 - Test Execution Environment 基于Jenkins Cluster提高测试并行执行能力 Business Requirement Feature Requirement Test Requirement Test Case Design 7HVW&DVH 5HSRVLWRU\ Pull Build Test Runner Test Execution System Jenkins Slave n Jenkins Slave 2 Jenkins Slave 1 Test Requests dispatch Jenkins Master Push Test Case Locally develop/debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Run Run Run 6HOHQLXP 1RGH 6HOHQLXP 1RGH  6HOHQLXP 1RGHQ dispatch Test Requests dispatch 6HOHQLXP+XE dispatch
56. 测试架构的演变 - Test Execution Environment 基于测试负载,用Docker实现Selenium Grid的动态扩展与收缩 Business Requirement Feature Requirement Test Requirement Test Case Design 7HVW&DVH 5HSRVLWRU\ Pull Build Jenkins Slave n Jenkins Slave 2 Test Requests Jenkins Slave 1 Test Runner Test Execution System Test Requests dispatch Jenkins Master Push Test Case Locally develop/debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Run Run Run Run 6HOHQLXP 1RGH 6HOHQLXP 1RGH  6HOHQLXP 1RGHQ Dispatch and auto scale service dispatch dispatch dispatch 6HOHQLXP+XE dispatch 666HHHOOEHEHOEHQ\Q\Q\LLX'X'LX'PPRRPRFFF1N1N1NHRHRHRUGUGUGHHHVVV Dynamicallty add/delete selenium node in Grid accoriding to test load
57. 测试架构的演变 - Test Execution Environment 基友Docker实现Jenkins Cluster的动态扩展与收缩 Business Requirement Feature Requirement Test Requirement Test Case Design 7HVW&DVH 5HSRVLWRU\ Pull Build Jenkins Slave n Jenkins Slave 2 Test Requests Jenkins Slave 1 Test Runner Test Execution System Test Requests dispatch Jenkins Master Dynamicallty add/delete -HQNLQV VODYHE\XVLQJ'RFNHU Jenkins auto scale service Push Test Case Locally develop/debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Run Run Run Run 6HOHQLXP 1RGH 6HOHQLXP 1RGH  6HOHQLXP 1RGHQ Dispatch and auto scale service dispatch dispatch dispatch 6HOHQLXP+XE dispatch 666HHHOOEHEHOEHQ\Q\Q\LLX'X'LX'PPRRPRFFF1N1N1NHRHRHRUGUGUGHHHVVV Dynamicallty add/delete selenium node in Grid accoriding to test load
58. 测试架构的演变 - Test Execution and Management Platform 引入Test Report Service生成各种测试报告 Business Requirement Feature Requirement Test Requirement Test Case Design Business Coverage Calculation Feature Coverage Calculation Test Coverage Calculation Test Report Service Test Execution Report Meta Data Story Board Test Report for GUI Full Trace Test Report Test Summary Report 7HVW&DVH 5HSRVLWRU\ Pull Build Push Test Case Locally develop/ debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Test Runner Test Execution Service Test Requests Auto Scale Jenkins Cluster Dispatch 'RFNHUEDVHG 6HOHQLXP*LUG
59. 测试架构的演变 - Test Execution and Management Platform Test Report Service生成的各种测试报告 q Full Trace Test Report for DEV/QE q GUI Video-based Test Report for PM/PO q Story Board Test Report for PM/PO q Multi-Site Story Board Comparison Test Report for LQA q Test Summary Report for management team q Test Trend Report for management team
60. 测试架构的演变 - Test Report Platform 典型的Test Report Platform架构 6HOHQLXP Requirement Management System (JIRA/DOORS/ALM) -HQNLQV Test Report Service 6SOXQN 7HVW([HFXWLRQ 5HSRUW0HWD'DWD Business Requirement Feature Requirement Test Requirement Test Case Design Business Coverage Calculation Feature Coverage Calculation Test Coverage Calculation Test Report Service Test Execution Report Meta Data Story Board Test Report for GUI Full Trace Test Report Test Summary Report 7HVW&DVH 5HSRVLWRU\ Pull Build Push Test Case Locally develop/ debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Test Runner Test Execution Service Test Requests Auto Scale Jenkins Cluster Dispatch 'RFNHUEDVHG 6HOHQLXP*LUG Full Trace Test Report GUI Video-based Test Report GUI Story Board Test Report Test Summary Report
61. 测试架构的演变 - Test Report Platform 引入Test Analysis Service提高Defect分类效率 Business Requirement Feature Requirement Test Requirement Test Case Design Business Coverage Calculation Feature Coverage Calculation Test Coverage Calculation Test Report Service Test Execution Report Meta Data Story Board Test Report for GUI Full Trace Test Report Test Summary Report 7HVW&DVH 5HSRVLWRU\ Pull Build Push Test Case Locally develop/ debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Test Runner Test Execution Service Test Requests Auto Scale Jenkins Cluster Dispatch 'RFNHUEDVHG 6HOHQLXP*LUG 6HOHQLXP -HQNLQV 6SOXQN Requirement Management System (JIRA/DOORS/ALM) Test Report Service 7HVW([HFXWLRQ 5HSRUW0HWD'DWD Defect close then send root cause back to Analysis Service, it will be used as machine learning sample set Full Trace Test Report Failed Test Cases Test Report Trace Analysis Service Defect Automatic Classificate and Submit Defect Management System (JIRA$/0) GUI Video-based Test Report GUI Story Board Test Report 'HIHFW$QDO\VLV 0DFKLQH/HDUQLQJ .QRZOHGJH'DWDEDVH Test Summary Report
62. 测试架构的演变 - Test Report Platform 引入Multi-Site Comparison Report提高LQA 测试效率 Business Requirement Feature Requirement Test Requirement Test Case Design Business Coverage Calculation Feature Coverage Calculation Test Coverage Calculation Test Report Service Test Execution Report Meta Data Story Board Test Report for GUI Full Trace Test Report Test Summary Report 7HVW&DVH 5HSRVLWRU\ Pull Build Push Test Case Locally develop/ debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Test Runner Test Execution Service Test Requests Auto Scale Jenkins Cluster Dispatch 'RFNHUEDVHG 6HOHQLXP*LUG Defect close then send root cause back to Analysis Service, it will be used as machine learning sample set 6HOHQLXP -HQNLQV 6SOXQN Requirement Management System (JIRA/DOORS/ALM) Test Report Service 7HVW([HFXWLRQ 5HSRUW0HWD'DWD Full Trace Test Report Failed Test Cases Test Report Trace Analysis Service Defect Automatic Classificate and Submit Defect Management System (JIRA$/0) GUI Video-based Test Report GUI Story Board Test Report 'HIHFW$QDO\VLV 0DFKLQH/HDUQLQJ .QRZOHGJH'DWDEDVH Multi-Site Story Board Comparison Report Generator Service LQA submit translation and layer out defect Multi-Site Story Board Comparison Report Test Summary Report
63. 测试架构的演变 - Test Report Platform 引入Test Statistics Service Business Requirement Feature Requirement Test Requirement Test Case Design Business Coverage Calculation Feature Coverage Calculation Test Coverage Calculation Test Report Service Test Execution Report Meta Data Story Board Test Report for GUI Full Trace Test Report Test Summary Report 7HVW&DVH 5HSRVLWRU\ Pull Build Push Test Case Locally develop/ debug/run /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Test Runner Test Execution Service Test Requests Auto Scale Jenkins Cluster Dispatch 'RFNHUEDVHG 6HOHQLXP*LUG Defect close then send root cause back to Analysis Service, it will be used as machine learning sample set 6HOHQLXP -HQNLQV 6SOXQN Requirement Management System (JIRA/DOORS/ALM) Test Report Service 7HVW([HFXWLRQ 5HSRUW0HWD'DWD Full Trace Test Report Failed Test Cases Test Report Trace Analysis Service Defect Automatic Classificate and Submit Defect Management System (JIRA$/0) GUI Video-based Test Report GUI Story Board Test Report 'HIHFW$QDO\VLV 0DFKLQH/HDUQLQJ .QRZOHGJH'DWDEDVH Multi-Site Story Board Comparison Report Generator Service LQA submit translation and layer out defect Multi-Site Story Board Comparison Report Test Summary Report Test Report Statistics Service Test Trend Report
64. 代码级单元测试的最佳实践 测试架构师在代码级单元测试扮演什么样的角色? • 单元测试框架选型(JUnit,TestNG,Spock,etc) • Mock / Stub 框架选型 • 静态扫描的规则定义与优化 • 单元测试的CI/CD集成 • 系统性方法排除开发的思维惯性 • 基于代码级错误分析,帮助提高开发人员代码质量意识
65. 代码级单元测试的最佳实践 代码的基本特征 对数据进行分类处理 每一次条件判定,就是一次分类处理 嵌套的条件和循环,也是分类处理 对子函数的调用获得内部输入
66. 代码级单元测试的最佳实践 代码缺陷产生的原因 分类遗漏 分类错误 处理错误
67. 代码级单元测试的最佳实践 代码错误的分类 功能层面 有特征 处理错误 性能层面 无特征 时间性能 空间性能 语法特征 边界行为特征 经验特征 算法错误 部分算法错误
68. 代码级单元测试的最佳实践 代码级测试方法的分类 静态方法 测试方法分类 动态方法 静态分析代码 动态执行代码 人工 自动 人工 自动
69. 代码级单元测试的最佳实践 代码级测试方法的分类 人工静态方法 自动静态方法 人工动态方法 自动动态方法 人工静态方法: 通过人工阅读代码来查找代码中潜在的各 种错误,通常采用开发人员代码走查、结对编程、同行评 审等手段。显然效率低下,效果完全取决于个人的能力。 自动静态方法:不实际执行代码,通过扫描代码来发现语 法错误,潜在语义错误和部分动态错误。这类可被发现的 错误通常具有某些特征。目前广泛采用,并用于代码风格 检查。 人工动态方法:设计代码的输入和预期的正确输出的集合 ,执行代码,并判断实际输出是否符合预期。传统意义上 的单元测试和代码级集成测试。 自动动态方法:基于代码自动生成边界测试用例并执行来 捕捉潜在的异常、崩溃和超时。只能发现行为特征错误, 因为工具不可能自动了解代码所要实现的功能。
70. 代码级单元测试的最佳实践 代码级测试方法示例 人工静态方法 可能发现 自动静态方法 没有语法错误和潜在动态错误,无法发现问题 人工动态方法 通过设定输入,执行程序并判断输出与预期的差 异,可以发现问题。但是或许只是部分问题 自动动态方法 可能发现,自动采用边界测试用例来执行代码, 捕捉潜在的异常、崩溃和超时
71. 代码级单元测试的最佳实践 完备测试用例的设计 什么是好的测试用例? •发现了缺陷的用例就是好的用例 用例发现了缺陷是好用例,缺陷修复后,同样的用例难道就不是好用例了吗? • 发现缺陷可能性大的用例就是好用例 如何评估用例发现缺陷的可能性? • 发现至今未被发现的缺陷的用例就是好用例 如何评估是否还存在未被发现的缺陷?
72. 代码级单元测试的最佳实践 完备测试用例的设计 好的测试用例 一定是一个完备的集合,能够 覆盖所有等价类以及各种边界值 渔网捕鱼的例子 – 如果把被测试代码视为一个池塘,Defect视为池塘中的鱼,建立用例集的过程 就像是在编织渔网 – 好的用例集就是一张能够覆盖整个池塘的渔网,只要池塘里有鱼,网就一定能 把鱼捞上来。 – 如果网是完整的且合格的,那么捞不到鱼,就能证明池塘中没有鱼。 – 渔网的好坏与是池塘中否有鱼无关
73. 代码级单元测试的最佳实践 完备测试用例的设计 “好”的测试用例集合 等价类划分的准确性 对于每个等价类都能保 证只要其中一个输入测 试通过,其他输入也一 定测试通过 等价类集合的完备性 所有等价类都已经正确 识别并且测试通过 边界值/条件处理的正确性 所有可能的边界值和边 界条件都已经正确识别 并且测试通过 做到了以上三点,就可以肯定测试是充分且完备的,既做到了完整的输入覆盖
74. 代码级单元测试的最佳实践 代码测试覆盖率的衡量 通常采用白盒测试覆盖率来度量代码测试的充分性和完整性 覆盖率=(至少被执行了一次的条目数)/条目总数 * 100% 其中条目可以是语句,条件,判定,路径等,分别对应不同种类的代码覆盖率 • 评估代码测试的充分性和完整性 • 单纯统计覆盖率意义不大,关键是帮助找 出遗漏用例,有针对性的补充用例 • 识别不可达的“僵尸”代码 • 基于已存在的代码,在惯性思维作用下, 与开发人员自行测试的局限性一致 • 测试成本随着覆盖率的提高而增加,越往 后越难
75. 代码级单元测试的最佳实践 探讨覆盖率的局限性 有了很高的白色覆盖率,是否就高枕无忧了呐? 比如:某项目的代码达到了MC/DC覆盖率100% 极端的例子:一个被测函数,里面只有一行代码,衡量代码质量的所有覆 盖率指标都是100% Ø白盒覆盖是基于现有代码的,不能发现“未考虑某些输入”形成的缺陷。 Ø满足白盒覆盖率测试的要求,并不意味整个测试已完成,而只能说明测 试对象已不需要基于此技术再进行额外的测试 Ø“未考虑得得某些输入”通常是边界输入,会导致有特征的错误(崩溃, 异常等),自动动态善于捕捉此类错误 Ø不要一味追求高覆盖率,覆盖率仅仅是手段,透过现象看本质
76. 全球化Site自动化测试的挑战 • What‘s Global Site Test ? • Global Site Test Challenge • Global Test Capability Key Value • Global Test Capability Brief Introduction • Global Test Capability Design Philosophy
77. 全球化Site自动化测试的挑战 • Go deeper into Global Test Capability Ø Global site test base utilities Ø Global site test data utilities Ø Global Configuration Repository Ø Unified Flow Framework Ø Multi-Site Story Board Test Report • Global Site Test Overall Architecture
78. What‘s Global Site Test? www.XXX.com www.XXX.mx www.XXX.uk www.XXX.de www.XXX.fr www.XXX.cn
79. Global Site Test Challenge • Hardcode tech debt (site, feature toggle) in test cases • Not support different site switch during CBT test in framework level • Not support global test data preparation • Test report not design for LQA, need manually effort to do screenshot • Not support validation on Currency, Date, Number and Price in different site with IBR rules • The limitation of test execution capability
80. What’s our solution? Global Test Platform
81. Global Test Platform Key Value • Scalability for test case • Highly reusable test components (Page, Flow) to combine E2E test • Easy to achieve high test coverage • Support CBT in test framework level • Flexible test validation to support global feature • Everything is configurable instead of hardcode • Unified Flow Framework to handle flow branch • Designed test report for global test
82. Global Test Platform Brief Introduction Desktop Website support different browser (Chrome, Firefox, Safari, IE, etc.) Base on Mobile Website support different mobile browser with same code as desktop QE-Infrastructure and inherit all core test capability Mobile Native Application Support both iOS and Android Web Service support both RESTFul and SOAP styles
83. Global Test Platform Brief Introduction Support both GUI and API level test case development Support both domain and E2E level test case development Core automation framework and library to provide global test capability Support global test data preparation for DEV/QE/PE Support CBT test capability
84. Global Test Capability Design Philosophy • Scalable for support new site launch test • Configurable instead of hardcode • Design for reusable of global test artifacts • Easy understanding test report for linguist QA
85. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report q Format Util – DateUtil, CurrencyUtil, PriceUtil, NumberUtil q Environment Util – Global ENV Support Util
86. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report q Global Base Util – SiteUtil, CountryUtil, StoreUtil q CBT Util – Change site in same test case
87. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report q Site X Support Util – Data-driven for new site launch
88. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report q Using Global Test Data Utilities to support test data preparation for DEV/QE/PE
89. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report q Build Test Data Service base on Global Test Data Utilities to support test data preparation Test Date Adapter Web Service 5(67IXO UI Interface for management Test Data Core Service Offering data Meta data management Recipe data management Data quantity management Data quality management Test Data Meta DB Recipe Definition Meta data for business objects Out-of-box test data Meta data for data quantity Meta data for data quality Unified Test Data Controller Global Test Data Utilties Test Data Validation Tool DEV Test Data Preparation Utilties Performance Test Data Preparation Utilties 6\VWHP8QGHU7HVW  687
90. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report Global Registry is used to manage user signal based properties and store global/local configuration, which groups certain features together to allow users to easily turn on and off features in a particular dimension q Decouple configure (feature toggle) from test code q Multi site test with same test code
91. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report Before Global Configuration Repository After
92. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report q One flow to combine all flow branches q Reusable Page-Flow module between domain and E2E q Support multi entry/exit page for same flow q Support the switch of validation ON/OFF in flow level q Support lambda expression and chain code style q Support customized operations before and after each Page method
93. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report UnifiedSellFlow ViewListingFlow UnifiedCheckoutFlow ViewOrderFlow ViewSalesFlow
94. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report
95. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report Design Purpose q Provide easy understanding test report for all stakeholders q Improve efficiency for Linguist QA Major Feature q Easy to go though the business flow which is made up with a list of screenshots with progress bar q Highlight the element the current action focus on q Show flow name, action name and description if you use Unified Flow
96. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report
97. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report
98. Go deeper into Global Test Capability Global Test Base Utilities Global Test Data Utilities Global Configuration Repository Unified Flow Framework Multi-Site Story Board Test Report
99. Test Architecture - GUI Automation Business Requirement Feature Requirement Test Requirement Unified Flow Framework Global Configuration Repository Base on UF Reusable script snippet Unified Business Flow Abstract Adopt Page Operation Abstract Page Abstract Component Operation Abstract Component Abstract Page Encapsulation Code Generator Test Case Design 'DWDGULYHQ 7HVW'DWD Test Case Replay test case to execute test /RFDO7HVW([HFXWLRQ (QYLURQPHQW 6\VWHP8QGHU7HVW 687 Test Date Provider Test Date Adapter Call TDS to prepare(create/ search/update) test data on SUT Test Date Service
100. Q & A