The device under test for this project was the Cisco Nexus 9516 data center core switch/router, a 16-slot chassis equipped with 1,024 50-gigabit Ethernet interfaces and two supervisor modules. Cisco equipped the switch with its N9K-X9732C-EX line cards, each of which offers 32, 64, or 128 ports of 100-, 50-, and 25-gigabit Ethernet capacity.
The traffic generator/analyzer was Spirent TestCenter equipped with its 10/25/40/50/100G MX3 modules. The Spirent instrument has a measurement precision of +/- 2.5 nanoseconds.
We assessed the Cisco devices with three performance tests and one test of power consumption. The performance tests measured throughput, latency, and jitter with three kinds of traffic – IPv4 unicast, IPv6 unicast, and IPv4 multicast – and a large amount of routing state in each case.
In the IPv4 unicast tests, we enabled BGP routing, with default timers, on all 1,024 ports of the Cisco device. The device also ran bidirectional forwarding detection (BFD), which rapidly finds link faults and is commonly used in production networks. We also configured the Spirent test tool to emulate a different external BGP (eBGP) peering session on each of the 1,024 ports. After establishing the BGP sessions, the Spirent test tool then advertised 1,024 unique, noncontiguous routes on each port, for 1,048,576 routes total. We used noncontiguous routes to prevent route aggregation, thus forcing the Cisco device to fully populate its routing tables.
After verifying the Cisco and Spirent devices had exchanged all BGP routing information, the Spirent tool then offered traffic to all 1 million routes in a fully meshed pattern, meaning all ports exchanged traffic with all other ports. This pattern is the most stressful on switch fabrics. We offered each of eight frame sizes for a duration of 300 seconds each, and measured throughput, latency, and jitter. The frame sizes included the seven standard sizes given in RFC 2544 (64, 128, 256, 512, 1,024, 1,280, and 1,518 bytes) and 9,216-byte jumbo frames.
The offered load for all tests was equivalent to 99.99 percent of line rate. This is Ethernet’s nominal line rate minus 100 parts per million (ppm). We used the 99.99 offered load instead of 100.00 percent to avoid any clocking variations between the test instrument and the switch.
In the IPv6 unicast tests, we enabled multiprotocol BGP (MP-BGP) and repeated essentially the same procedures as with IPv4 unicast. The only difference was the use of IPv6 test traffic, and the use of a 78-byte minimal-size frame instead of a 64-byte minimum. This is to accommodate the larger IPv6 header (40 bytes, instead of 20 bytes as with IPv4) and the “signature field” in test traffic to measure latency, jitter, and sequencing.
In the IPv4 multicast tests, one Spirent test port acted as transmitter and all others acted as receivers, forcing the Cisco device to replicate each multicast frame 1,023 times. We enabled protocol independent multicast-sparse mode (PIM-SM) routing and Internet group multicast protocol version 3 (IGMPv3) on the Cisco device. We also configured the Spirent test instrument to emulate subscribers to 10,000 unique multicast group addresses on each of 1,023 receiver ports.
The Spirent tool began the test by joining all 10,000 multicast groups on all 1,023 receiver ports. This created more than 10 million unique multicast routes on the Cisco device. After verifying all groups had been joined on all ports, we configured the Spirent tool to offer test traffic on one transmitter port. We used the same frame sizes, offered load and test duration as in the IPv4 unicast tests, and again measured throughput, latency, and jitter.
To measure power consumption, we attached a Voltech Power Analyzer to the lab’s three-phase power. The Cisco device used 10 power supplies, each attached to either phase 1 or phase 2. After verifying via the Cisco CLI that the device shared power evenly across power supplies, we measured volts, amperage, and watts for two power supplies and then multiplied results by 5 to derive power consumption for the entire system.
We repeated the test nine times: With the switch idle and with the nine frame sizes used in throughput testing, again offered at 99.99 percent of line rate from the Spirent instrument. These power tests used the same BGP control plane (with 1 million unique routes) and traffic pattern as in the IPv4 unicast test case.