Facebook大数据模块快速部署和实时更新 冯冀 (2)

Razor

2019/10/19 发布于 技术 分类

文字内容
1. Service Deployment with Real Time and Large Volume Data Yi Feng 冯翼 Performance and Capacity Engineer, Facebook Inc. Facebook Inc. © All Rights Reserved
3. Who am I? Before industry PhD, Computer Science, City University of New York Approximation Algorithm Prove theorems and disprove conjectures MSExchange Diagostics Production Debugging, Diagnostics, and bug fix Microsoft, LinkedIn Programmer – Monitoring and Diagnostic Agent Performance and Scalability Engineer – Infrastructure, Instagram Infra Distribution Service Cluster and Interface Optimization Facebook Inc. © All Rights Reserved
4. Takeaways Make Synergetic Application in your practice Learn the principles and keys of similar practice Entertained and Delight Facebook Inc. © All Rights Reserved
5. Challenge in Operations • Reliability • Agility • Efficiency • Procedure – engineering practice • Coordination – engineer practice • Engineering Effort – Operation Cost Facebook Inc. © All Rights Reserved
6. About Facebook • Social network • Communication • Content Sharing Millions of Servers Tens of EB storage (1EB = 1000PB = 1M TB) • 2.42B Monthly Active Users (SEC Filing 10Q, Q2 2019) 10+ Geographical Regions Globally Facebook Inc. © All Rights Reserved
7. Dedicated Cluster Infra Overview Co-allocated Infra Overview Load balance Desktop FE service Mobile Partner EdgeTransport Feed retrieval and ranking ADs FE 前端 Content Photo / Video Warm storage Replication Table storage (文件 表存 储) OS App Blob storage (二进制 存储块) Search Store Migration BE 后端 Facebook Inc. © All Rights Reserved
8. Deployment Facebook Inc. © All Rights Reserved
9. Deployment Challenge Amount of Servers – tens of thousands Data Reatime-ness Amount of Data in Memory Mega Tera Bytes Completion Time Availability During Deployment Incident Recovery time Facebook Inc. © All Rights Reserved
10. Deployment Specification • Scope: Service clusters => geographical region, servers • Meta spec: current version, target version, steps: 2%, 10%, 25%, 50%, 99%, 100% • Availability: concurrent down percentage • SLA:'>SLA: starting time, duration, hours to avoid Example: • Instagram ranking services, 6 regions: Ireland, Sweden, Carolina, Altoona, Oregon, and Texas. Total servers involved ~27K. • Target version: Release Candidate Stable [time stamp[ • Increment: 25% => 100% • Concurrent down percentage: <10%. • SLA:'>SLA: starting 7am PT, target complete 1pm PT (before EU peak hour starts). Facebook Inc. © All Rights Reserved
11. Case 1: ranking service – Data online loading 350-500QPS, P99 latency 1350ms • Each server contains ~40GB in memory cached data + stacked decision trees Inbound request Ranking server (cache) DB1 DB2 DB15 DB9 DB7 A request kicks in • Queries and update ~5 DB servers (local cache miss), ~150 entries. • Update local cache, and respond • Inbound request comes at 350-500 QPS, P99 latency at 1350ms • 320s transfer time =>( 0+320) / 2 * 400 * 150 = 320K entries updated • Takes another 320s to back to service. Facebook Inc. © All Rights Reserved
12. Case 1: Infrastructure – Conventional deployment New Version of Service Sweden 1200 servers Ireland 1300 servers US East 2500 severs US central 2000 Servers US South 2500 Servers 120 servers 120 servers US West 3000 Servers Milliseconds 120 120 servers Seconds servers • • • • • 120 servers Effective Bandwidth 0.5-1Gbps (same building) Concurrent down:10% Total Data Transfer Per server: 40GBytes Ideal Single server transfer time: 40GB / 0.125GBps = 320s Ideal Total Transfer Time: 40GBytes * 2000 Destination Servers / 120 Source Server / 0.125GBps = ~2 hours 120 servers Facebook Inc. © All Rights Reserved
13. • Each server takes 11 minutes to be online • 10% of services down each time • Service stop, update binary takes 3 min 20s • Practical total deployment time 5-9 hours Cache hit rate Case 1: Real Production Deployment — Production Time of a day Facebook Inc. © All Rights Reserved
14. Case 1: Use Service Cluster Neighbors as Source Inbound request ranking service Ash0109 ash33212 ash12234 ash0228 DB SVC • Bring service up with empty cache and decision tree parameters • Break down inbound request • Query neighbors to fulfill and update cache • Mark fully only when cache miss rate back to normal (99.5%) • Entire Server back time: 3m 20s, stop, update binary, start • Takes 1 minutes for a server full back online (350-500 QPS) Facebook Inc. © All Rights Reserved
15. • Bandwidth bottleneck on DB side - check • DB overload by update write • Server data load time < 1 minute • Entire deployment – seamless • Total deployment time: 30-40 minutes (compared 4-9 hours) Cache hit rate Case 1: Problem solved – 搞定 Noise* Time of a day Facebook Inc. © All Rights Reserved
16. Principles of Enigneering Simplicity Frugality Efficiency Engineering Effort Resilience Operational Effort Agility Infra maintain Effort Facebook Inc. © All Rights Reserved
17. Case 2: Problem – deploy a large indexing services Facebook Market Place (跳蚤市场) 400M SKU (SEC 10K, 2018) 400 TB in Memory Indexing Shard – on copy per geographical location Index on product categories, info tag, geographical location Amazon (100M), 10K, 2017 Deploy a new version Facebook Inc. © All Rights Reserved
18. Case 2: Large in Memory Indices - Challenge 4-5PB of inmemory data – storage / bandwidth Service availability during deployment Consistent across Planet Eartch Facebook Inc. © All Rights Reserved
19. Case 2: Solution – Theory NE SC Atlantic Ocean VA OR IRE SWE DEN • No storage needed • Risk: all regions down is almost zero NM • Minimized cross Atlantic traffic TX North America • Index copied from neighbor region Europe Facebook Inc. © All Rights Reserved
20. Case 2: Implementation • Index Partitioned by Servers VA OR NE NM SC TX • Each region has a disjoint partition updated concurrently • Failed request request (up to 12%, 90s) goes to a neighbor region Facebook Inc. © All Rights Reserved
21. Case 2 Solution No storage needed Optimized bandwidth Service Availability constant
22. Recap Takeaways 贼不走空 Make Synergetic Application in your practice 学以致用 Learn the principles and keys of similar practice 去粗取精 Entertained and Delight 听个乐呵 Facebook Inc. © All Rights Reserved
24. Questions? Facebook Inc. © All Rights Reserved