Description:

In this document, we walk through the steps to enable InfiniBand SR-IOV on a dual-port Mellanox ConnectX-5 VPI adapter card in vSphere 7.x. We build on the most current VMware and Mellanox documentation and cover the steps from BIOS, ESXi, and vCenter to the functionality test on the VM guest operating system using ibverbs, Intel cluster checker, and the Ohio State University (OSU) microbenchmark suite. We also introduce how to use the vHPC toolkit, an open-source tool developed by VMware, to speed up the deployment of an HPC cluster in vSphere. Finally, we present a performance study of five HPC applications across multiple vertical domains, all concerned with dynamic systems (including manufacturing, weather forecasting, and the life sciences). We conclude that virtual HPC clusters with VMware vSphere perform nearly as well as bare-metal HPC clusters while offering all the advantages of virtualization with vSphere like increased IT agility, flexibility, scalability, and significant cost savings of hardware. --Authored by Yuankun Fu