StreamNative 翟佳 2019 GIAC 6 23 下一代分布式消息系统pulsar

CodeWarrior

2019/07/08 发布于 编程 分类

GIAC2019 

文字内容
1. Apache Pulsar StreamNative
3. • • Apache Pulsar, Apache BookKeeper PMC • EMC -> StreamNative • StreamNative Core Engineer • ->
4. • Apache Pulsar • • Pulsar • Pulsar
5. 2003 2012 2010 2006 2011
6. Apache Pulsar Flexible Pub/Sub Messaging backed by durable log Storage
7. “ ” • - • Topics - - - • (Cloud Native) • • Topics • • • IO Backlog - rebalance
8. Apache Pulsar Durability Data replicated and synced to disk Ordering Guaranteed ordering Delivery Guarantees At least/At most /Exactly once High throughput 1.8 M messages/ s in a single partition Low Latency Low publish latency of 5ms at 99pct Unified messaging model Both Streaming and Queuing Multi-tenancy A single cluster can support many tenants and use cases Geo-replication Out of box support for geographically distributed applications Highly scalable & available Millions of topics HA
9. • Pulsar Pulsar • Pulsar • Pulsar •
10. • • Pulsar Pulsar • • •
11. Pulsar +
12. Pulsar — Broker Producer Subscription1 Topic … Subscription2 Write Ack Consumer readEntries ManagedLedger Cursor + 1 2 3 4 5 6 7 1 2 3 4 5 6 7 BookKeeper 1 2 3 4 5 6 7 Ledger stores Data ABCD E Ledger stores Cursor changes Ack/N-Ack -/+
13. Pulsar — Key-Shared
14. • • Pulsar Pulsar • • •
16. Geo-replication Beijing Shanghai Producer (P1) Topic (T1) Topic (T1) Producer (P2) Consumer (C1) SubscripEon (S1) SubscripEon (S1) Consumer (C2) Producer (P3) Topic (T1) Guangzhou Broker Pub/Sub
17. • • Pulsar Pulsar • • •
18. : • • • •
19. +
20. • Pulsar Pulsar • Pulsar • Pulsar •
21. Apache BookKeeper • • Parallel Quorum Replication Ensemble, Write Quorum, Ack Quorum)
22. Entry, Ledger & Log Log / Stream / Partition
23. +
24. • Pulsar Pulsar • Pulsar • + • • IO
25. Broker catchup
26. Bookie
27. re-balance
29. • Pulsar Pulsar • Pulsar • + • • IO
30. IO — Write Write
31. IO — Tailing Read Tailing Read
32. IO — Catchup Read Catchup Read
33. IO — All
34. — Segmented Stream Segment Readers Producer Segmented Stream Segment0 Segment1 Segment2 Segment3 Segment4 Segment 5 Consumer Time
35.
36. • Pulsar Pulsar • Pulsar • Pulsar •
37. Pulsar Pulsar IO Cassandra Kafka MySQL Messaging Pulsar Brokers Debezium MongoDB Hdfs Event Processing Complex Stream Pulsar Functions Processing Stream Storage BookKeeper Flume Analytics Pulsar SQL Hive Tiered Storage AWS S3 Google Cloud Storage Azure Blob Storage HDFS
38.Twitter: @apache_pulsar Wechat Subscription: ApachePulsar Mailing Lists dev@pulsar.apache.org, users@pulsar.apache.org Slack https://apache-pulsar.slack.com (#china) register: https://apache-pulsar.herokuapp.com/ Localization https://crowdin.com/project/apache-pulsar Github https://github.com/apache/pulsar https://github.com/apache/bookkeeper
39. • 2.4.0 Release
40. Thanks
41. msup