Virtualized NFSoRDMA-based Disaggregated Storage Solution for AI Workloads

Disaggregated storage with NFSoRDMA offers a high-performance solution for AI workloads. It combines the NFS protocol with RDMA technology to achieve low latency and high throughput. The key objectives of disaggregated storage solutions for AI include high throughput, maximum IOPS, simplicity in configuration, and deployment flexibility. Performance requirements for AI tasks include fast data loading, quick checkpoint writing, and parallel access to data from multiple clients. The proposed solution integrates a high-performance storage engine with file system services, focusing on software-defined RAID, tuned file systems, and RDMA-based interfaces. Virtualization of NFSoRDMA and xiRAID Opus overcomes limitations in the Linux kernel space and enhances performance. The solution achieves high throughput, saturating a 2x200 Gbit interface, and reduces CPU load compared to mdraid. Testing results demonstrate the superiority of xiRAID Opus over mdraid in both sequential and random operations. The implementation of disaggregated storage based on NFSoRDMA shows significant advancements in performance and efficiency for AI workloads.