Booking,com

  • 23 浏览
文字内容
1. 8 Booking.com MySQL 1 0 2 数据库构架 C C T D 吴鑫 2018-05-10
2. ● ● 2002-2006 芬兰 2006-2015 瑞典 Database Developer ● Database Administrator ● Database Consultant ● ● ● Database Engineer & Team Lead Email: xin.wu@booking.com Linkedin: linkedin.com/in/xinwu1 C C T D 8 1 0 2 2015-2018 荷兰
3. Agenda. • Booking.com 概况 • MySQL数据构架 • C C • 数据库HA高可用性解决方案 • 测试环境搭建 T D 8 1 0 2 自动化案例分析
4. Booking.com. ● Booking.com于1996年成立于荷兰阿姆斯特丹,现 在是全球最大的在线酒店预订网站。目前隶属于 Booking Holdings Inc(Booking.com, Agoda, priceline.com, Kayak, RentalCars, OpenTable) ● 拥有超过1.5万名员工,就职于全世界70个国家及 地区的198家办公室 ● Booking.com的网站和移动端App支持40余种语言, 网站注册房源总数超过28,469,348, 遍布全球229 个国家及地区的 131,481个目的地 ● 日均客房预订间夜数超过1,550,000 ● Booking.com使用MySQL做为主要的线上数据库解决 方案,目前是欧洲最大的MySQL用户 T D C C 0 2 8 1
5. MySQL Thousands of MySQL baremetal servers 0 2 8 1 Hundreds of Replication chains(master - slave) C C T D Several data centers for redundancy MySQL 5.6, 5.7, 8.0 & MariaDB
6. Standard Replication Chain. Master Intermediate Master 1 Slave1 Slave2 IM3 C C T D Slave3 Slave4 8 1 0 2 IM2 Slave5 Slave6 IMn Slave7 Slaven
7. RW Master Master CNAME slave slave slave zooRoster slave RO zooAnimal Application 0 2 C C T D frontend -pool slave 8 1 slave zooRoster cronjob -pool backup clone 24h delay
8. Automation. ● Auto upgrade MySQL/MariaDB 数据库版本升级 ● Auto upgrade OS 自动操作系统升级 ● Auto execute tasks 指定任务执行 0 2 8 1 ● Auto grow/shrink pool 自动扩展和收缩服务器池 C C ● Auto schema changes 自动执行DDL 表结构改变语句 ● Backup/Restore 自动数据备份和恢复 T D
9. How to scale read. DB1 DB1 DB1 0 2 C C T D DB1 8 1 Automatically grow the pool DB1 DB1
10. Needs-Setup  OS  MySQL  Configuration (puppet) Badmin (booking admin) Detect under capacity Clone Data • Clone source rsync • Backup – restore • Torrenting Provision API Provisioning New server C C T D 8 1 0 2 Installed  MySQL replicating  Configuration Live  Enable in the pool  Serving traffic Warmup • Tcpdump from servers in the same pool
11. How to scale write. Downtime 5-10s DB2-CNAME DB1-CNAME DB1-RW DB2-RW DB1 T D DB1 DB1-RO 0 2 C C tableA tableB tableC DB1 8 1 Replication Filter DB1 DB2 DB2 tableB tableC DB2 DB2 DB2-RO
12. HA solution Orchestrator. ● ● ● What is orchestrator What failure does it handle Demo T D C C 0 2 8 1
13. What is orchestrator Orchestrator is a MySQL replication topology management and visualization tool, allowing for: ● ● ● ● ● ● ● ● 8 1 Detection and investigation of replication clusters Safe topology refactoring: moving replicas around the topology Sleek topology visualization Replication problems visualization Topology changes via intuitive drag & drop Maintenance mode declaration and enforcement Auditing of operations More… T D C C Developed by Shlomi Noach. 0 2 https://github.com/github/orchestrator
14. Orchestrator GUI T D C C 0 2 8 1
15. Orchestrator GUI T D C C 0 2 8 1
16. What failures does it handle ● Master failures ○ 8 1 Operational hooks to external systems which need to be aware of these faulure ● Intermediate master failures ● Does NOT care about leaf slaves or applications ● Works with Oracle or MariaDB GTID ● Works without GTID: Can add pseudo-GTID (events injected on the master are T D C C 0 2 used to find a match) so no need to migrate ● Handles multi-level topologies
17. Test Environment. ● ● ● ● Database testing environment is shared with everyone Database instances are running on VM All production data Data refresh everyday T D C C 0 2 8 1
18. Test Production snapbox M slave slave 0 2 C C T D 8 1 rdbprod Master CNAME master iscsi slave Slave CNAME
19. Master CNAME Test2 Test1 master snapbox snapbox 24H refresh iscs i slave rdbprod Active 0 2 C C T D Slave CNAME 8 1 master iscs i slave rdbprod Passive
20. We are hiring. 0 2 C C T D 8 1 Check https://workingatbooking.com/ xin.wu@booking.com
21. T D C C 0 2 8 1