姜凤波:全用户态服务开发套件F-Stack

箕斯雅

2017/11/14 发布于 技术 分类

腾讯云在业务在快速发展中面临大量的L4/L7接入服务,业务成本急剧上升,亟需提升接入层性能以节省服务器资源降低成本。而F-Stack是腾讯云开发的基于DPDK+FreeBSD协议栈的全用户态网络服务开发套件,可以方便的接入原有应用并提升性能,已经在腾讯云DNS/CDN/COS等业务中使用,获得了非常好的效果。本次分享介绍当前业务接入层在网络上遇到的问题和腾讯云的解决方案 F-Stack 的具体架构和实现,以及在实际产品中的应用实践。

文字内容
1. 全用户态演服S讲A务者C开:C姜2发0凤1套波7 件F-Stack
2. Explosive growth of data services  Challenges  service traffic:CDN,live streaming(RTMP),etc.  DDoS, CC 2017 Solution C NIC: 25GbE40GbE100GbE SAC CPU: 24Cores48Cores56Cores  L2/L3: Kernel bapass(DPDK)  L4/L7: ? 2
3. Thread5 Thread4 Thread3 Thread2 Thread1 Thread0 The problem - Kernel SAC内核C拥堵2017 User space Kernel space CPU cores(NUMA) NIC queues
4. SACC2017 User space Process3 Process2 Process1 Process0 Kernel space Data plane Control plane Kernel Bypass
5. User space TCP/IP stack  Seastar's native stack  problems in WAN  incompatible with existing programs 2017 mTCP's user-level stack  tcp only. SACC too simple to be a production ready stack VLAN/VxLAN/Tunnel/Bonding/Network Tools are not supported
6. What does F-Stack do? APP (nginx, redis, memcache) SDK (corouting) Posix API SACC2017 FreeBSD TCP/IP stack DPDK
7. F-Stack …… …… …… …… …… APP (Nginx, Redis) SDK (Corouting) 1、Full user-space ……  No content switch ……  Zero copy SACC2017Posix-like API FreeBSD TCP/IP stack DPDK ……  No hirqs and sirqs …… 2、Shared-nothing architectur ……  Linear scalability  No schedule  No lock  No cache locality miss
8. Why FreeBSD?  Why full TCP/IP stack?  Advantageous functional, production ready stack, IGMP/IPv6/VLAN/VxLAN/Tunnel/Bonding/Net Tools  Stable. CC2017 Why not Linux? SA complicated logic.  GPL
9. User space FreeBSD TCP/IP stack phymem uma_page_slab_has h uma kmem_malloc malloc hook mmap/malloc (rte_mempool/rte_ malloc) timecounter ticks hz timer kernel,irq threading sched wakeup sleep SACC2017replace remove & replace rte_timer ticks timecounter poll memory clock schedule mtx rw rm sx cond remove empty macro locks
10. Posix-like API ff_init(argc, argv) ff_socket ff_read ff_write …… SACC2017ff_run(loop, arg); ff_kqueue ff_kevent ff_epoll_ctl ff_epoll_wait API F-Stack Lib
11. Coroutine Synchronous programming app event 2017Add IO Event CAsynchronous SACexecution Microthread Scheduler Add Event MMicMircoMritcoMhirtcorhrietcorharetordhaetdrEhaevdrEaeevdEnaevdEtnevtEnevtnetnt Execute Event Get Event
12. Tools  sysctl  ifconfig  route  netstat  top  etc.. SACC2017
13. Nginx_F-Stack Bandwidth(G) RPS cache body RPS test(No Keep Alive) 7.00 6.00 5.15 5.00 4.00 20173.00 2.00 C1.00 SAC0.00 0.340.40 1 0.77 0.39 2 2.92 3.94 1.44 0.42 0.44 0.48 4 8 12 CPU Cores 0.47 16 5.82 1200000 1000000 800000 600000 400000 0.47 20 200000 0 BW(Nginx) BW(Nginx_FStack) RPS(Nginx) RPS(Nginx_FStack)
14. F-Stack in Tencent Cloud  2013 UDP DNS server based on DPDK  2014 TCP(user-space TCP/IP stack) DNS server  2015 L4/L7 gateway application 2017 2016 Used on HttpDNS,CDN,COS(Cloud Object Storage) SACC 2017 Port FreeBSD network stack to F-Stack and open source
15. Roadmap Watchdog/tools APP (nws,nginx,redis,……) SDK (corouting,php,python,……) https://github.com/f-stack SACC2017Posix API(LD_RRELOAD) FreeBSD TCP/IP (TCPA,SHIELD,Hooks) DPDK(SPDK) http://www.f-stack.org F-Stack WeChat subscription
16. SACC2017