Geneve: A Network Virtualization Encapsulation With A Difference

Before Geneve, each vendor had their virtualization encaps like VxLAN, STT, NvGRE, MPLSoGRE, DOVE etc. Though, there were few pros & cons associated with each of these encapsulation(encap), it is believed that encap does not matter as it is used to transport VM's packet from A to B. Geneve seems to move away from this premise and also tries to bring all vendors towards common encap.

To summarize, Geneve addresses

    1) A common encap that consumes all virtualization tunneling protocol.

    2) Encap does matter and it does more than just transporting the packet.

From the pictures above, it is quite obvious that Geneve has added few more fields to VxLAN header. Fields those are important to this discussion are

    1) O & C bit

    2) Protocol Type

    3) Variable Length Options. 

Common Encapsulation:

Geneve attempts to bring all vendors under one umbrella WRT encap. This helps in lot of ways

    - DC operators would have single encap in a heterogeneous hypervisor environment.

    - Vendors like NVO GWs, NIC vendors, Firewall system, IDS etc have to provide support for only one encap.

    - Life becomes much easy from Monitoring and Debugging perspective.

Protocol_Type indicates the ether_type that appears after Geneve header. Ether_Type could be any as defined by IEEE. This provides an ability to encapsulate any protocol with an Ethertype. With VxLAN, it is possible only to encap an Ethernet frame.

Encapsulation Matters:

Traditional protocols carry meta-data only on control plane not on data plane where-as Geneve carries meta-data on data plane. This is where Geneve thinks differently. It views DC as a single large distributed router with each hypervisors as a line card. It is typical of such distributed system to carry meta-data on data plane between different line-cards. Geneve rightly does that for network virtualization too.

Meta-data is immensely powerful for things like service chaining, storing context for firewall, etc. Meta-data can do something like what communities and attributes did for BGP.

Interestingly, Geneve does not define these meta-data but leaves it to vendor to define their own meta-data using Variable length options field. 

Backward Compatibility:

Geneve is not backward compatible (from data plane perspective) with any of NVO encap including VxLAN. Perhaps, VxLAN can be made forward compatible with Geneve.

So, what is the future of VxLAN and its support by various HW, SW & NIC vendors. Bruce explains about it here

Remaining Headers:

O is OAM bit, when set is consumed by tunnel endpoint rather than forwarding to the tenant. This is more on the lines of VxLAN Router-Alert. One could possibly simulate BFD using O bit and Options provided.

C is set when the packets are consumed first by tunnel end point and then forwarded to the tenant. This seems more like piggy-backing options to tunnel data plane on the dataplane.

Final Thoughts:

At this point it is not clear as how much flexibility HW GWs can provide for Options field. These would pan-out as HW vendors starting to support Geneve. 

I still believe MPLSoGRE that Nuage and Contrail uses works well for Inter-DC cases given that ASICs already supports MPLSoGRE encap & decap. Would have been good had Geneve considered Inter-DC as well. Possibly it does not matter if operators start using vRouters as WAN edge.