North-South & East-West.
Data Center traffic is characterized in to North-South(NS) and East-West(EW). ES is between Machine-Machine that is localized to a DC. NS is between Machine-User & Machine-Machine (Inter-DC, hybrid clouds are few examples) that traverses WAN edge either to internet or over to VPN (using L2/L3/MPLS VPN or OTV).
Pic Courtesy: Nango's Facebook Talk
According to Facebook's Nanog Talk in 2013
- EW constitutes about 95% of DC traffic. NS takes remaining 5%.
- EW traffic is growing at an exponential rate while NS traffic has stagnated pretty much.
- For every 2 bytes of data generated by NS there is corresponding 98 bytes of data generated by EW.
These number's clearly indicate that EW traffic have to be well engineered for better performance.
East-West Traffic Types
EW traffic can be classified in to
- Tenant Traffic
- Infrastructure Traffic.
Tenant traffic are between the VM's. Infrastructure traffic consists of Management, Storage & VMotion. Typically, Storage & VMotion are high in bandwidth and storage is latency sensitive. Management is low bandwidth but required for the management of Compute, Network and Storage nodes.
Tenant traffic are virtualized using protocols like VxLAN, NvGRE, MPLSoGRE(Contrail/Nuage), etc. Infrastructure traffic are not virtualized as they are between the hypervisors and does not make sense to do so.
North-South Traffic Types
NS traffic are due to
- Inter-DC case.
- To & from Internet
Inter-DC is a case where VM's that communicate are located in different DC's or one of them is located in a public cloud. Usually, there is VPN connection (L2VPN, L3VPN, MPLS-VPN or OTV) between the DC's.
User talk to a webserver thro' internet. These traffic go thro' a firewall and get NAT'ed. Few hybrid cloud also use the model.
VLAN Provisioning
VLANs are provisioned at vSwitch. Each traffic type are placed in different VLANs and are allocated different bandwidth based on requirement. If the fabric is L3, all VLANs gets terminated at ToR with exception of Edge VLAN which gets plumbed from vSwitch to WAN edge including all intermediate networking nodes.
All NS traffic are place in Edge VLAN. Of-course, Edge VLAN is not required when MPLSoGRE based solution is used. Contrail and Nuage are MPLSoGRE based.
Each of EW traffic are placed in different VLAN on vSwitch. As VLAN get's terminated at ToR, EW traffic are routed then on. Yes. VMotion, Storage, Management & Tenant traffic will go over routed network. Even though tenant traffic are part of single VLAN's, the virtualized nature provides tenant isolation.
Summary
A server running VMWs vSphere as hypervisor with single vSwitch having 4 NIC's attached will have IP with VLAN created using VMKernel option for each of
- Management
- Storage
- VMotion
- Tenant (NVO'ed)
- Edge VLAN.
If the fabric is L3, default GW would be first hop ToR. All traffic would be routed from then on. Edge VLAN is for NS traffic and would have VLANs plumbed from vSwitch to WAN edge.
According to NSX design guide, Fabric design should be approved with VMW for VMotion to be supported over L3.
Data Center traffic is characterized in to North-South(NS) and East-West(EW). ES is between Machine-Machine that is localized to a DC. NS is between Machine-User & Machine-Machine (Inter-DC, hybrid clouds are few examples) that traverses WAN edge either to internet or over to VPN (using L2/L3/MPLS VPN or OTV).
Pic Courtesy: Nango's Facebook Talk
According to Facebook's Nanog Talk in 2013
- EW constitutes about 95% of DC traffic. NS takes remaining 5%.
- EW traffic is growing at an exponential rate while NS traffic has stagnated pretty much.
- For every 2 bytes of data generated by NS there is corresponding 98 bytes of data generated by EW.
These number's clearly indicate that EW traffic have to be well engineered for better performance.
East-West Traffic Types
EW traffic can be classified in to
- Tenant Traffic
- Infrastructure Traffic.
Tenant traffic are between the VM's. Infrastructure traffic consists of Management, Storage & VMotion. Typically, Storage & VMotion are high in bandwidth and storage is latency sensitive. Management is low bandwidth but required for the management of Compute, Network and Storage nodes.
Tenant traffic are virtualized using protocols like VxLAN, NvGRE, MPLSoGRE(Contrail/Nuage), etc. Infrastructure traffic are not virtualized as they are between the hypervisors and does not make sense to do so.
North-South Traffic Types
NS traffic are due to
- Inter-DC case.
- To & from Internet
Inter-DC is a case where VM's that communicate are located in different DC's or one of them is located in a public cloud. Usually, there is VPN connection (L2VPN, L3VPN, MPLS-VPN or OTV) between the DC's.
User talk to a webserver thro' internet. These traffic go thro' a firewall and get NAT'ed. Few hybrid cloud also use the model.
VLAN Provisioning
VLANs are provisioned at vSwitch. Each traffic type are placed in different VLANs and are allocated different bandwidth based on requirement. If the fabric is L3, all VLANs gets terminated at ToR with exception of Edge VLAN which gets plumbed from vSwitch to WAN edge including all intermediate networking nodes.
All NS traffic are place in Edge VLAN. Of-course, Edge VLAN is not required when MPLSoGRE based solution is used. Contrail and Nuage are MPLSoGRE based.
Each of EW traffic are placed in different VLAN on vSwitch. As VLAN get's terminated at ToR, EW traffic are routed then on. Yes. VMotion, Storage, Management & Tenant traffic will go over routed network. Even though tenant traffic are part of single VLAN's, the virtualized nature provides tenant isolation.
Summary
A server running VMWs vSphere as hypervisor with single vSwitch having 4 NIC's attached will have IP with VLAN created using VMKernel option for each of
- Management
- Storage
- VMotion
- Tenant (NVO'ed)
- Edge VLAN.
If the fabric is L3, default GW would be first hop ToR. All traffic would be routed from then on. Edge VLAN is for NS traffic and would have VLANs plumbed from vSwitch to WAN edge.
According to NSX design guide, Fabric design should be approved with VMW for VMotion to be supported over L3.
