VMware NSX Replication Modes
VMware NSX has three replication modes: unicast, hybrid, and multicast. The replication mode defines how VMware NSX VTEP VXLAN handles traffic when a virtual machine sends a frame to a MAC address or sends layer 2 BUM traffic. VTEP VXLAN uses an existing layer 2 mechanism for flooding and learning MAC addresses. When a VTEP needs to send a packet to multiple hosts, this will be considered as broadcast, unknown unicast, and multicast (also known as BUM) traffic:
- Broadcast: This refers to sending data from one source device to all devices in a local network. The sender sends a broadcast frame with a destination MAC address of FF:FF:FF:FF:FF:FF and the packet will be received by all hosts on that local layer 2 network. Broadcast is used during ARP resolutions, DHCP requests, and route advertisements on some routing protocols.
- Unknown unicast: This refers to sending data from one source device to one destination device. If the MAC address of the destination device is unknown, the frame will be flooded to all devices in that layer 2 local network if it is Broadcast traffic. Once the MAC address is learned, future communication will be forwarded normally as unicast.
- Multicast: This refers to sending data from one device to multiple devices in a group but not all. The data is sent to a multicast group and only members of that group receive the multicast traffic.
Referring to RFC 7348, VXLAN uses multicast to carry layer 2 BUM (broadcast, unknown, multicast). In VMware NSX, the NSX controller can handle this BUM traffic when using the unicast replication mode, therefore eliminating the need for layer 2 or layer 3 multicasts in the physical network. You would need to consider using hybrid or multicast when you have large VTEP subnets, expecting large BUM traffic, and would like to offload BUM traffic to your physical network switches.
There is a very good explanation of how the three replication modes work behind the scene at https://telecomoccasionally.wordpress.com/2015/01/11/nsx-for-vsphere-vxlan-control-plane-modes-explained/. In summary, you typically require the following information to decide which replication mode to choose:
- Application traffic patterns
- Expected BUM traffic on logical switches
- Number of VMs on logical switches
- Number of ESXi hosts or VTEPs
- VTEP subnets design
- Multicast familiarity, experiences, and day-2 operations considerations