The TCP/IP protocol has been the backbone of Internet communication for many years, but for AI networks, TCP/IP has fatal shortcomings in some ways. The TCP/IP protocol has a high latency, usually around tens of microseconds, and also causes a severe load on the CPU. RDMA can access memory data directly through the network interface without the intervention of the operating system kernel. This allows for high-throughput, low-latency network communication, especially when used in massively parallel computer clusters.
There are four implementations of RDMA technology: InfiniBand, RoCEv1, RoCEv2, and iWARP. Among them, RoCEv1 has been deprecated, and iWARP is less commonly used. The most commonly used networking solutions in the industry today are InfiniBand and RoCEv2.
- InfiniBand: Designed with RDMA in mind, the physical link layer, network layer, and transport layer are redesigned to ensure reliable transmission at the hardware level, providing higher bandwidth and lower latency. However, the cost is high, and it needs to support IB NICs and switches.
- iWARP: TCP-based RDMA network, which uses TCP to achieve reliable transmission. Compared with RoCE, in the case of large-scale networking, a large number of TCP connections in iWARP will occupy a large amount of memory resources and require higher system specifications. You can use a common Ethernet switch, but you need an iWARP-capable network card.
- RoCE: RoCE implements RDMA over Ethernet, which can bypass TCP/IP and use hardware offload, thereby reducing CPU utilization. There are two main versions of RoCE: RoCEv1 and RoCEv2. RoCEv1 is an RDMA protocol implemented at the Ethernet link layer. The switch needs to support flow control technologies such as PFC to ensure reliable transmission at the physical layer. RoCEv2 is implemented at the UDP layer of the Ethernet TCP/IP protocol, which was introduced to solve the scalability problem.
RoCEv2 supports the transmission of RDMA routes over Layer 3 Ethernet networks. RoCEv2 replaces the InfiniBand network layer with IP and UDP headers on the Ethernet link layer, which makes it possible to route RoCE between traditional IP-based routers.
So, InfiniBand or RoCE, who is better suited for AI data center networking?
First of all, let’s take a look at the basic concepts and requirements of the computing center:
- Parallel computing: An AI workload is a unified infrastructure of multiple machines running the same application/compute task.
- Scale: HPC/AI tasks can scale up to thousands of compute engines (such as GPUs, CPUs, and FPGAs).
- Job type: Different tasks differ in size, run duration, size and number of datasets to consider, types of answers that need to be generated, and the different languages used to code the application and the type of hardware it runs, all of which lead to constant changes in the in-network traffic patterns built to run HPC/AI workloads.
- Lossless: In traditional data centers, lost messages are retransmitted, while in AI workloads, lost messages mean that the entire calculation is either wrong or stuck. As a result, AI data centers need a lossless network.
- Bandwidth: High-bandwidth traffic needs to run between servers in order for the application to be able to fetch data. In modern deployments, each compute engine for AI or other high-performance computing capabilities has interface speeds of up to 400Gbps.
These complexities pose significant challenges to AI networks, which need to be characterized by high bandwidth, low latency, no jitter, no packet loss, and long-term stability.
Let’s first take a look at the basic definitions of the two networking methods:
IB networking
IB Network is a high-performance computing network designed and developed based on the InfiniBand technical specification to simplify server-to-server connectivity and its associated peripherals, reduce latency, and enhance interoperability. It has the characteristics of high bandwidth and low latency, and is widely used in the field of high-performance computing (HPC). IB networks can be isolated through partition keys, and IB networks of different tenants can be isolated through different partition keys, similar to Ethernet VLANs. This network isolation mechanism helps improve the security and manageability of the network.
ROCE networking
The RDMA over Converged Ethernet (RoCE) protocol is a cluster network communication protocol that enables RDMA (Remote Memory Direct Access) over Ethernet, which greatly reduces the latency of Ethernet communication and improves bandwidth utilization.
RDMA (Remote Direct Memory Access) technology stands for Remote Direct Data Access, which is created to solve the delay of server-side data processing in network transmission. RDMA transmits data directly to a computer’s storage over a network, quickly moving data from one system to remote system memory without any impact on the operating system, eliminating the need for much computer processing. It eliminates the overhead of external memory replication and context switching, freeing up memory bandwidth and CPU cycles for improved application performance.
IB vs. RoCE
Compared to InfiniBand, RoCE has greater versatility and a relatively lower cost. It can be used not only to build high-performance RDMA networks, but also to traditional Ethernet. However, configuring parameters such as Headroom, PFC (Priority-Based Flow Control), and ECN (Explicit Congestion Notification) on the switch can be complex. In large-scale deployments, the overall throughput performance of a RoCE network may be slightly lower than that of an InfiniBand network.
From a technology perspective, InfiniBand employs a variety of technologies to improve network forwarding performance, reduce failure recovery time, improve scalability, and reduce operational complexity.
In terms of service performance, InfiniBand has lower end-to-end latency compared to RoCEv2, so networks built on InfiniBand have advantages in application-level business performance.
When it comes to bandwidth and latency, factors such as congestion and routing can impact high-performance network interconnects.
In general, ROCE and IB networking have their own advantages and disadvantages, and should be selected according to specific application scenarios and requirements. ROCE networking has certain advantages in terms of cost and scalability, while IB networking may be superior in terms of performance and reliability.
