The Jpsi cluster at Fermilab.

Research and development work on commodity clusters has been carried out under the Lattice QCD SciDAC grant at FNAL and JLab. This has led to the terascale resources for lattice QCD that have been deployed at the two labs.

The 7n cluster at JLab.

A wide range of processors and communications systems has been evaluated, and both switched and mesh communications systems have been studied. Myrinet and InfiniBand fabrics have been tested for switched clusters, and gigabit ethernet has been used for the mesh ones. A total of eleven clusters of various sizes have been built, which will provide a total throughput of approximately 16.1 Teraflop/s on lattice QCD production code when the 2009 installation is complete. (This is the equivalent of 70-75 TFlops in terms of the Linpack benchmarks.) The most recent cluster built at JLab is shown at left. It consists of 396 nodes of AMD Opteron (quad-core) CPUs connected via DDR InifiniBand switched networks, with a throughput of 2.99 TFops on lattice QCD code. Jpsi, the latest cluster built at FNAL, consists of 586 nodes with 2.1 GHz dual CPU quad core Opterons with a double data rate Infiniband switch, sustaining 5.75 TFlops on lattice QCD code. This cluster is shown at the right. It will be expanded in 2009 with another 270 nodes, bring its total performance to 8.4 TF.