NVIDIA Networking Core Topics

This study guide was originally created as a study guide for the NCP-AIN, but it also serves as a good summary/review of NVIDIA Networking topics.

For each topic: What is it? / Why does it exist? / What problem does it solve? / What monitors it? / What breaks if misconfigured?

• What is it? A protocol that carries RDMA (Remote Direct Memory Access) traffic over Ethernet. RoCEv1 is Layer 2 only (not routable); RoCEv2 encapsulates RDMA in UDP/IP, making it routable across L3 networks.
• Why does it exist? To get InfiniBand-like RDMA performance (kernel bypass, zero-copy, low CPU overhead) while running on cheaper, more ubiquitous Ethernet infrastructure.
• What problem does it solve? Traditional TCP/IP networking has high CPU overhead from kernel involvement and data copies. RoCE lets NICs move data directly between application memory on different hosts, cutting latency and freeing CPU cycles - critical for GPU-to-GPU AI training traffic.
• What tool monitors it? NetQ, WJH, NIC counters (ConnectX/ethtool), UFM (if IB is involved), switch RoCE/ECN/PFC counters.
• What breaks if misconfigured? RDMA is loss-intolerant - without correctly tuned PFC/ECN/DCQCN, any packet loss causes massive retransmissions, congestion spreading, throughput collapse, and stalled GPU collectives (e.g., NCCL AllReduce hangs).

• What is it? A per-priority (per-traffic-class) Ethernet flow control mechanism. When a queue for a given priority nears full, the receiver sends a PAUSE frame telling the sender to stop - only for that priority, not the whole link.
• Why does it exist? Standard Ethernet is “lossy” by design (drop on congestion). RDMA/RoCE requires a lossless fabric since RDMA doesn't tolerate packet loss well. PFC provides that losslessness per-class.
• What problem does it solve? Prevents buffer overflow and drops on RDMA traffic classes without pausing unrelated (e.g., normal TCP) traffic sharing the same link.
• What tool monitors it? NetQ, WJH (specifically flags PFC storms/deadlocks), switch pause-frame counters.
• What breaks if misconfigured? PFC storms (cascading pause frames), head-of-line blocking, congestion spreading across the fabric, and even PFC deadlocks (circular pause dependencies that freeze traffic entirely). If PFC is missing entirely, RoCE loses its lossless guarantee and suffers drop-induced retransmit storms.

• What is it? An IP-header marking scheme. When a switch queue occupancy crosses a configured threshold, it marks the packet's CE (Congestion Experienced) bit instead of dropping it.
• Why does it exist? To give proactive, early congestion signaling before buffers actually overflow - enabling senders to slow down before loss or PFC pauses are needed.
• What problem does it solve? Reduces reliance on blunt, reactive mechanisms (drops or PFC pauses) by letting endpoints react to marked packets and self-throttle earlier - this is the “E” half of DCQCN.
• What tool monitors it? NetQ, switch telemetry/counters, WJH congestion reporting.
• What breaks if misconfigured? Thresholds set too low over-mark and hurt throughput; set too high and ECN never fires before PFC kicks in, meaning you effectively lose ECN's benefit and fall back on lossy PFC pausing - leading to the storms/deadlocks described above.

• What is it? An end-to-end congestion control algorithm for RoCEv2 that combines ECN (network-side marking) with rate-limiting on the NIC. When ECN-marked packets arrive, the receiver sends Congestion Notification Packets (CNPs) back to the sender, which throttles its transmit rate.
• Why does it exist? RDMA/RoCE has no native TCP-like congestion control. DCQCN was designed specifically to provide congestion control for RDMA traffic at the NIC level.
• What problem does it solve? Prevents incast congestion (many senders to one receiver, common in AI collective operations) and reduces dependence on PFC as the only congestion defense, since PFC alone can cause storms/deadlocks.
• What tool monitors it? NetQ, NIC counters (ConnectX CNP/ECN counters via ethtool), WJH.
• What breaks if misconfigured? Poorly tuned rate-increase/decrease parameters cause unfair bandwidth sharing between flows, sluggish recovery from congestion, underutilized links, or - if too weak - congestion collapse that falls back onto PFC pausing.

• What is it? A dynamic path-selection mechanism (available in InfiniBand switches and NVIDIA Spectrum-X Ethernet) that routes flows/packets based on real-time link congestion and utilization, rather than static hashing (like ECMP).
• Why does it exist? Static hash-based routing can send multiple “elephant flows” down the same path while other equal-cost paths sit idle, causing hotspots even when aggregate capacity exists.
• What problem does it solve? Load imbalance and congestion hotspots in fabrics carrying bursty, many-to-many AI/HPC traffic patterns (e.g., all-to-all GPU collectives).
• What tool monitors it? UFM (InfiniBand), NetQ (Ethernet/Spectrum-X), switch telemetry counters.
• What breaks if misconfigured? Disabled or misconfigured adaptive routing leads to persistent hotspots and poor fabric utilization; if enabled without proper reordering support, per-packet adaptive routing can cause out-of-order delivery issues for protocols sensitive to ordering.

• What is it? 16-bit values assigned to IB ports/nodes that define logical partitions within a subnet - conceptually similar to VLANs but for InfiniBand.
• Why does it exist? A shared physical IB fabric often serves multiple tenants or workloads that must not be able to communicate with each other.
• What problem does it solve? Multi-tenancy isolation and security - nodes without a shared PKey simply cannot exchange traffic at the fabric level.
• What tool monitors it? UFM, the Subnet Manager (SM), ibdiagnet.
• What breaks if misconfigured? Nodes that should communicate but lack a shared PKey will fail to connect at all (hard-to-diagnose “unreachable” errors). Conversely, overly broad PKey membership creates a security hole, allowing unintended cross-tenant communication.

• What is it? NVIDIA's network operations and telemetry platform. It continuously collects state/telemetry from switches (Cumulus Linux/Spectrum) and provides real-time visibility, historical trending, and automated validation of the fabric.
• Why does it exist? Operators need fleet-wide, correlated visibility into fabric health - you can't troubleshoot a distributed AI fabric one switch at a time via CLI.
• What problem does it solve? The difficulty of detecting configuration drift, link flaps, congestion, and RoCE-specific issues (PFC/ECN events) across a large fabric without manual, per-device inspection.
• What tool monitors it? NetQ itself is the monitoring layer - it's typically surfaced via its own UI/CLI, and can feed data into Grafana/SNMP dashboards. Its own health is checked via NetQ agent status.
• What breaks if misconfigured? If NetQ agents aren't deployed/configured correctly on switches, you lose fabric-wide visibility - congestion, drops, and misconfigurations go undetected, drastically increasing mean-time-to-resolution (MTTR).

• What is it? A hardware-based telemetry feature (built into the Spectrum ASIC / Cumulus Linux) that reports, in real time, why packets were dropped - with specific reason codes (buffer overflow, ACL deny, RoCE ECN/PFC events, routing issues, etc.).
• Why does it exist? Traditional switches are largely black boxes when it comes to explaining drops; packet captures are slow and impractical at line rate for this purpose.
• What problem does it solve? The “why did my packet get dropped” black-box problem - WJH gives root-cause visibility at line rate without needing packet captures.
• What tool monitors it? WJH is itself the diagnostic layer, accessed via CLI (e.g., nv show system wjh on Cumulus) or surfaced through NetQ dashboards.
• What breaks if misconfigured? If WJH is disabled or not integrated, you lose real-time root-cause insight into drops/congestion, forcing much slower manual troubleshooting (e.g., blind packet captures) and longer outages.

• What is it? A Kubernetes CNI meta-plugin that lets a pod attach to multiple network interfaces, instead of the single default pod network interface.
• Why does it exist? Default Kubernetes networking gives each pod exactly one interface. AI/HPC workloads need dedicated, high-performance secondary interfaces (RDMA, SR-IOV VFs) in addition to the standard cluster network.
• What problem does it solve? Enables pods to get a high-performance secondary NIC (e.g., an SR-IOV VF for RDMA) while retaining normal pod networking for control-plane/service traffic.
• What tool monitors it? kubectl describe on pods/NetworkAttachmentDefinitions, NVIDIA Network Operator status, Kubernetes events, Prometheus.
• What breaks if misconfigured? Pods get stuck in ContainerCreating, fail to attach the secondary interface, or silently fall back to the slow default network path - causing AI training/RDMA jobs to fail to start or run at degraded performance.

• What is it? A PCIe hardware feature that lets one physical NIC present multiple lightweight Virtual Functions (VFs), each of which can be passed directly into a VM, container, or pod as if it were its own NIC.
• Why does it exist? To give virtualized/containerized workloads near bare-metal network performance by bypassing the hypervisor/software network stack.
• What problem does it solve? Virtualization networking overhead - SR-IOV lets containers/pods use RDMA directly with near-native throughput and latency, essential for GPU training in Kubernetes.
• What tool monitors it? NetQ, NVIDIA Network Operator status, ethtool/ip link on host and VF, ibdev2netdev, Kubernetes SR-IOV device plugin status.
• What breaks if misconfigured? Insufficient VFs provisioned, driver mismatches, or wrong PF/VF binding causes pods to fail scheduling (no allocatable resource), RDMA to silently not work, or performance degradation if VFs are oversubscribed on the same physical NIC.

• What is it? A Kubernetes operator that automates deployment and lifecycle management of NVIDIA's networking stack in a cluster - NIC drivers, RDMA shared device plugin, SR-IOV device plugin, IPoIB, and Multus configuration.
• Why does it exist? Manually installing/matching drivers and device plugins across every node in a large K8s cluster is complex and error-prone. (Analogous to the NVIDIA GPU Operator, but for networking.)
• What problem does it solve? Simplifies and standardizes deployment of high-performance networking (RDMA/SR-IOV) across many nodes declaratively, ensuring consistent driver/plugin versions cluster-wide.
• What tool monitors it? kubectl get pods/status in the operator's namespace, its CR status conditions, Prometheus metrics.
• What breaks if misconfigured? A misconfigured Network Operator custom resource causes driver load failures, missing device plugins, pods unable to request RDMA/SR-IOV resources - effectively blocking any AI workload that needs high-performance networking from scheduling at all.

• PFC = pause per priority (lossless). ECN = mark before full (proactive). DCQCN = the NIC-side algorithm that uses ECN marks (via CNPs) to actually throttle senders, with PFC as the backstop.
• NetQ = fleet-wide visibility/validation. WJH = “why was THIS packet dropped, right now.”
• PKeys = InfiniBand's VLAN equivalent (isolation).
• Multus = extra NICs for a pod. SR-IOV = the hardware mechanism that often provides those NICs. Network Operator = the automation that wires Multus + SR-IOV + drivers together in K8s.

AI Knowledge