構(gòu)建高性能IP負(fù)載均衡技術(shù)

1、構(gòu)建高性能IP負(fù)載均衡技術(shù)byjgli搜狐研究院 基礎(chǔ)架構(gòu)部1硬件體系架構(gòu)linux內(nèi)核虛擬服務(wù)器DPDK：intel數(shù)據(jù)平面解決方案Missouri：IP負(fù)載均衡解決方案2硬件體系架構(gòu)Xeon E5-2600 Product Family82599ES Ethernet ControllerPCI Express Interface3PCI Express Interface4PCI Express Interface版本原始傳輸率單向單通道帶寬雙向x1通道帶寬雙向x2通道帶寬雙向x4通道帶寬雙向x8通道帶寬雙向x16通道帶寬PCIE 1.02.5GT/s250MB/s500MB/s1GB/

2、s2GB/s4GB/s8GB/sPCIE 2.05.0GT/s500MB/s1GB/s2GB/s4GB/s8GB/s16GB/sPCIE 3.08.0GT/s1GB/s2GB/s4GB/s8GB/s16GB/s32GB/sPCIE 4.016.0GT/s2GB/s4GB/s8GB/s16GB/s32GB/s64GB/s5PCI Express Interface# dmidecode t slot6x86體系架構(gòu)變遷7Xeon E5-2600 Product Family8Xeon E5-2600 Product FamilyDirect Media Interface 2 (DMI2) co

3、nnects the processor to the Platform Controller Hub (PCH). DMI2 is similar to a four-lane PCI Express* supporting a speed of 5 GT/s per lane. 4GB/sIntel Xeon Processor E5-2620QPI Speed 7.2 GT/s3.6 GHz 2 bits/Hz （雙倍數(shù)據(jù)傳送速率） 20 （QPI連接寬度） (64/80) （數(shù)據(jù)比特?cái)?shù)/數(shù)據(jù)數(shù)據(jù)包比特?cái)?shù)） 2 （各向發(fā)送和接收作業(yè)同步進(jìn)行） 8 （每字節(jié)比特?cái)?shù)）= 28.8 GB/s1

4、 lik: 28.8GB/s2 lik: 57.6GB/sIntel Xeon Processor E5-2620DDR3-800/1066/1333DDR3-800 Bandwidth= 4*0.8Gb/s*64b8 = 25.6GB/sDDR3-1066 Bandwidth = 4*1.066Gb/s*64b8 = 34.1GB/sDDR3-1333 Bandwidth= 4*1.333Gb/s*64b8 = 42.6GB/s1 chan:10.6GB/s4 chan:42.6GB/s The Physical layer consists of the actual wires carr

5、ying the signals, as well as circuitry and logic to support ancillary features required in the transmission and receipt of the 1s and 0s. The unit of transfer at the Physical layer is 20-bits, which is called a Phit (for Physical unit). The Link layer is responsible for reliable transmission and flo

6、w control. The Link layers unit of transfer is 80-bits, which is called a Flit (for Flow control unit). The Routing layer provides the framework for directing packets through the fabric. The Transport layer is an architecturally defined layer (not implemented in the initial products) providing advan

7、ced routing capability for reliable end-to-end transmission. The Protocol layer is the high-level set of rules for exchanging packets of data between devices. A packet is comprised of an integral number of FlitsPCIE e3 x8: 16GB/s 9Xeon E5-2600 Product Familygoogle： What every programmer should know

8、about memory1082599ES Ethernet Controller1182599ES Ethernet Controller1282599ES Ethernet Controller1382599ES Ethernet Controller1482599ES Ethernet Controller15linux內(nèi)核虛擬服務(wù)器LVS ：Linux Virtual Serverkernel tunninglinux kernel network data flow16 Linux Networking LayersNetwork ApplicationsBSD SocketsINE

9、T SocketsTCPUDPIPPPPSLIPEthernetARPUserKernelSocket InterfaceProtocol LayersNetwork Devices17linux kernel network data flow18linux kernel network data flowlinux kernel network data flow19LVS ：Linux Virtual Server20LVS ：Linux Virtual Server工作模式：NAT/FULLNAT/DR/TUNNEL傳輸協(xié)議：TCP/UDP調(diào)度算法：RR/WRR/LCS/WLCS/.2

10、1LVS ：FULLNAT22LVS ：FULLNATreal servere間可以跨vlan通訊，in/out流都經(jīng)過lvs，降低運(yùn)維成本集群方式部署，擴(kuò)展容易synflood及其他TCP FLAGS DDOS攻擊防御，抗攻擊能力強(qiáng)23kernel tunning多隊(duì)列網(wǎng)卡，中斷綁定軟中斷均衡，RPS/RFSlvs代碼：session hash table，session hash bucket lock 等其他24DPDK：intel數(shù)據(jù)平面解決方案whowhywhat25國內(nèi)的應(yīng)用百度：數(shù)據(jù)中心阿里：協(xié)議棧，linux虛擬服務(wù)器騰訊：流量清洗其他：26DPDK產(chǎn)生背景27DPDK如何解決

11、問題28其他編程優(yōu)化技術(shù)Software prefetchBranch Predictioninline, complier option, $ gcc -O3 test.c -o normal$ gcc -O3 test.c -o branch$ objdump -d branch branch.asm$ objdump -d normal normal.asm 29我們?yōu)槭裁匆芯緿PDK30Missouri：IP負(fù)載均衡解決方案performance testroadmapframeworkabout31What is the MissouriAlibaba LVS v3DPDK mod

12、eFULLNAT，TCP/UDPpercpu data32Missouri：network flow33performance testIEEE 802.3x雙工模式(Full Duplex)中提供流量控管的機(jī)制(Flow Control)；如網(wǎng)絡(luò)交換器產(chǎn)生擁塞時(shí)就會(huì)利用PAUSE frame 通知傳送者暫緩送出封包，以減緩擁塞情況。34test environmentclient: Dell PowerEdge R720Xeon CPU E5-2650 v2 2.60GHz82599ES 10-Gigabit SFI/SFP+server: Dell PowerEdge R720 xdXeo

13、n CPU E5-2620 2.00GHz10-Gigabit X540-AT2Memory 16G4 cpus4 pktgen threads25500 IP35Missouri vs LVS36Missouri vs LVS37Missouri vs LVSserver: Dell PowerEdge R720 xdXeon CPU E5-2620 2.00GHz10-Gigabit X540-AT2client: Dell PowerEdge R720Xeon CPU E5-2650 v2 2.60GHz82599ES 10-Gigabit SFI/SFP+64字節(jié)20 倍38Missouri：roadmapARP/RARP/ICMP/