1. Apache Pulsar 分层分片架构 翟佳 Streamlio
3. • Pulsar 简介 • Pulsar的分层分片架构 • Benchmark • Pulsar在智联的实践
4. Apache Pulsar是什么
5. Apache Pulsar是什么 “Flexible Pub/Sub messaging backed by durable log/stream storage”
6. 项目状态 • 2012在Yahoo内部启动,经历了了⽆无数的迭代 • 2016年年九⽉月Yahoo将Pulsar开源 • 2017年年六⽉月Yahoo将Pulsar捐献给了了Apache软件基⾦金金会 • 2018年年九⽉月Pulsar毕业成为顶级项⽬目 • 2400+ commits - 22 Yahoo releases - 9 Apache releases • 24 committers from 8 companies, 78 contributors • 30+ companies on production
7. Apache Pulsar 特性 Durability Data replicated and synced to disk Ordering Guaranteed ordering Delivery Guarantees At least once, at most once and effectively once High throughput Can reach 1.8 M messages/s in a single partition Multi-tenancy A single cluster can support many tenants and use cases Low Latency Low publish latency of 5ms at 99pct Geo-replication Out of box support for geographically distributed applications Unified messaging model Support both Streaming and Queuing Highly scalable & available Can support millions of topics HA
8. • Pulsar 简介 • Pulsar的分层分片架构 • Benchmark • Pulsar在智联的实践
9. 存储和计算分离 分层架构: Brokers & Bookies • 独⽴立扩展 • 灵活容错 • 快速扩容
10. 服务层 — brokers
11. 存储层 — BookKeeper BookKeeper分布式⽇日志/流存储 • 低延时、⾼高吞吐、持久化 • 强⼀一致 (repeatable read consistency) • ⾼高可⽤用 • 单节点可以存储很多⽇日志 • I/O隔离
12. 读写流程 (1) (1) ⽣生产者发送消息给Broker
13. 读写流程 (2) (2) Broker并发写N个副本
14. 读写流程 (3) (3) Broker等待来⾃自于Bookies的Quorum Acks
15. 读写流程 (4) (4) Broker返回确认给⽣生产者,并投递给消费者
16. 分片存储 • 分区(Partition)是逻辑上的⼀一个概念 • 分区按照时间或者⼤大⼩小被切成分⽚片 (Segment) • 分⽚片被打散存放到集群中的所有节点
17. Broker容错
18. Bookie容错
19. 存储扩容
20. Broker 分区 (主副本) 物理分区 vs 逻辑分区 物理理分区 Segment 1 逻辑分区 Segment 2 Segment 3 Segment n Broker Broker Broker Broker Broker 分区 (拷⻉贝) 分布 (从拷⻉贝) Segment 1 ..
Segment 2 Segment n Segment 2 ..
Segment 3 Segment n Segment 3 ..
Segment 1 Segment n Segment 1 ..
Segment 2 Segment n 分区架构 ! 物理理分区 ! 存储和计算紧耦合 ! 容错恢复需要拷⻉贝物理理分区 ! 扩容需要迁移物理理分区来达到负载均衡 分⽚片架构 ! 逻辑分区,“物理理”分⽚片 ! 存储和计算分离 ! 失效处理理相互分离,快速、⽆无痛点 ! 弹性扩容
21. 物理分区 vs 逻辑分区 https://jack-vanlightly.com/sketches/2018/10/2/kafka-vs-pulsar-rebalancing-sketch
22. 物理分区 vs 逻辑分区 Write
23. 物理分区 vs 逻辑分区 Tailing Read
24. 物理分区 vs 逻辑分区 Catchup Read
25. 物理分区 vs 逻辑分区 All together
26. • Pulsar 简介 • Pulsar的分层分片架构 • Benchmark • Pulsar在智联的实践
27. Benchmark • 测试⽬目标 • Throughput & latency • Min 2 guaranteed copies • Kafka Settings • Topic settings replicationFactor=3 min.insync.replicas=2 • Running on 3 EC2 VMs with local SSDs log.flush.interval.ms= • Pulsar Settings • Use ensemble=3 write=3 ack=2 # Using default: means no fsyncs • Kafka producer config acks=all • Data synced on disk before ack linger.ms=1 batch.size=131072 https://github.com/openmessaging/openmessaging-benchmark
28. Throughput Max Throughput 1 Topic 1 Partition 1KB payload
29. Latency at fixed throughput 50K msg/s 1 Topic 1 Partition 1KB payload Latency
30. Latency at fixed throughput — 99pct 50K msg/s 1 Topic 1 Partition 1KB payload Latency
31. • Pulsar 简介 • Pulsar的分层分片架构 • Benchmark • Pulsar在智联的实践
32. Pulsar统一模型 队列列 + 流
33. 智联: 队列 + 流
34. 智联 Pulsar
35. 智联 Pulsar
36. Cassandra Pulsar 生态 Pulsar IO Kinesis MySQL MongoDB Messaging Pulsar Brokers Event Processing Pulsar Functions Complex Stream Processing Stream Storage BookKeeper Analytics Pulsar SQL AWS S3 Tiered Storage Google Cloud Storage Azure Blob Storage HDFS