The 520 node Infiniband cluster at Fermilab.

Research and development work on commodity clusters has been carried out under the Lattice QCD SciDAC grant at FNAL and JLab. The objective has been to provide computing platforms to test the QCD API, and to determine optimal configurations for the terascale clusters planned for FY 2006 and beyond. In addition, the clusters that have been constructed are being used to carry out important research in QCD.

The 6n cluster at JLab.

A wide range of processors and communications systems has been evaluated, and both switched and mesh communications systems have been studied. Myrinet and InfiniBand fabrics have been tested for switched clusters, and gigabit ethernet has been used for the mesh ones. A total of eight clusters of various sizes have been built with a total throughput of approximately 1.7 teraflop/s on production code. The most recent gigabit ethernet cluster was built at JLab, shown at left. It has 384 2.8 GHz Xeon processors, and sustains approximately 500 gigaflop/s. The latest switched architecture cluster, which was built at FNAL, has 520 3.2 GHz P4 nodes, and an InfiniBand network which sustains approximately 850 gigaflop/s. These clusters are shown at the right. The experience gained with these prototype clusters will enable the U.S. lattice QCD community to build highly cost effective terascale production clusters in the coming year.