USQCD Collaboration, May 6–7, 2011
Brookhaven LQCD Facility     Fermilab LQCD Facility     JLab LQCD Facility

All Hands' Home

JLab USQCD 2011 Site


Call for Proposals


2010 Proposals

2011 Machine performance


Argonne ALCF

Oak Ridge NCCS



Collaboration links

2010 Meeting

2009 Meeting

2008 Meeting

2007 Meeting

2006 Meeting

2005 Meeting

Call for Proposals

From: Frithjof Karsch 
Subject: [sdac] 2011/12 USQCD Call for Proposals
Date: February 8, 2011 10:15:31 AM CST

Dear Colleagues,

This message is a Call for Proposals for awards of time on the USQCD
computer resources dedicated to lattice QCD and other lattice field
theories. These are the clusters at Fermilab and JLab, the GPU-clusters 
at Fermilab and JLab and awards to USQCD from the INCITE program.

In this call for proposals we expect to distribute about 
                 262.3 M Jpsi-core hours on clusters at JLAB and FNAL 
                   4.2 M GPU-hours on GPU clusters at JLAB and FNAL 
                  27.0 M Jpsi-core hours on BG/P at ALCF (*)
                  15.0 M Jpsi-core hours on XT5 at Oak Ridge (*)

In addition we expect that USQCD can utilize about
                  45.0 M Jpsi-core hours of zero priority time on the
                                   BG/P at ALCF (**)

Moreover, USQCD distributes about 5 M Jpsi-equivalent core-hours which we
expect to charge for disc and tape usage.

 (*) available in CY2012
(**) estimate based on CY2010; allocation starts already on April 1st. 
     Further remarks on the nature of the INCITE award and additional 
     requirements for projects that apply for resources on leadership
     class computers are given in section (iv).

All members of the USQCD Collaboration are eligible to submit proposals.
Those interested in joining the Collaboration should contact Paul Mackenzie 

Let us begin with some important dates:

        February  8: this Call for Proposals
        March    12: proposals due
        April    15: reports to proponents sent out
        May     6/7: All Hands' Meeting at JLab
        May      20: allocations announced
                     NOTE: INCITE allocations will already be announced
                           and will start on April 1.(see section (iv))
        July      1: new allocations start

Proponents are invited to make an oral presentation of their proposals
at the All Hands' Meeting.  These presentations are not mandatory, but
are recommended for large projects and in those cases where the report
raises serious issues.

The web site for the All Hands' Meeting is 


The requests can be of three types:

    A) requests for large amounts of supercomputer time
          ---more than 2.5 M Jpsi-equivalent core-hours on clusters 
             or leadership class computers or more than 100K GPU hours ---
       to support calculations of benefit for the whole USQCD Collaboration;

    B) requests for medium amounts of supercomputer time 
          --- 2.5 M Jpsi-equivalent core-hours or less on clusters
              or 100 K GPU hours or less on GPU clusters  ---
       to support calculations that are scientifically sound but not 
       necessarily of such broad benefit;

    C) requests for exploratory calculations, such as those needed to
       develop and/or benchmark code, acquire expertise on the use of 
       the machines, or to perform investigations of limited scope.
       The amount of time used by such projects should not exceed
       100 K Jpsi core-hours on clusters or 5 K GPU-hours on the 

Requests of Type A and B must be made in writing to the Scientific
Program Committee and are subject to the policies spelled out below.
These proposals must also specify the amount of disk and tape storage 
needed.  Projects will be charged for new disks and tapes.  How this
will be implemented is discussed in section (iii). 

Requests of Type C should be made in an e-mail message to

    Paul Mackenzie ( for clusters at FNAL,

    Chip Watson ( for clusters at JLAB.

Type C requests will be honored up to a total not exceeding 5% of 
the available time on USQCD hardware. If the demand exceeds such limits, 
the Scientific Program Committee will reconsider the procedures for access.

Collaboration members who wish to perform calculations on USQCD hardware 
or on resources awarded to USQCD through the INCITE program can present 
requests according to procedures specified below. The Scientific Program 
Committee would like to handle requests and awards on leadership class
computers and cluster in the equivalent core-hours for the FNAL "Jpsi" 
cluster. Requests on the GPU clusters will be handled in GPU hours.   
Conversion factors for clusters and leadership class computers are given 

As projects usually are not flexible enough to switch between running on 
GPUs and clusters, we refrain at present from introducing conversion factors 
for this purpose.

In determining the size of the resources requested for a particular project 
and in assessing the overall balance of the USQCD program, the Scientific 
Program Committee will, however, combine the total requested cluster and
leadership facility time measured in J/Psi core hours and the GPU hours 
converted to J/Psi core hours by the formula 1 GPU hour = 15 J/Psi core hour 
(a ratio based on current procurement costs).

                         - o -

The rest of this message deals with requests of Types A and B.  It is
organized as follows:

    i)   policy directives regarding the usage of awarded resources;

    ii)  guidelines for the format of the proposals and deadline for

    iii) procedures that will be followed to reach a consensus on the
         research programs and the allocations;

    iv)  policies for handling awards on leadership-class machines

    v)   description of USQCD resources at Fermilab and JLAB

i) Policy directives.

1) This Call for Proposals is for calculations that will further the
physics goals of the USQCD Collaboration, as stated in the proposals for
funding submitted to the DOE (see, and have the
potential of benefiting additional research projects by members of the

2) Proposals of Type A are for investigations of very large scale,
which will require a substantial fraction of the available resources.
Proposals of Type B are for investigations of medium to large scale,
which will require a smaller amount of resources.  Proposals requesting
more than 2.5 M Jpsi-equivalent core-hours on clusters or more than
100 K hours on GPU-clusters will be considered as Type A, smaller ones as 
Type B.  

Proposals that request time on the leadership-class computers at Argonne
and Oak Ridge should be of Type A and should demonstrate that they
(i) can efficiently make use of large partitions of leadership class 
computers, and (ii) will run more efficiently on leadership class computers 
than on clusters.

It is hoped that on USQCD hardware about 80% of the available resources 
will be allocated to proposals of Type A and about 15% to proposals of
Type B, with the rest being reserved for small allocations and
contingencies.  Because our process is proposal-driven, however, we
cannot guarantee the 80-15-5 split.

3) Proposals of Type A are for investigations that benefit the whole
USQCD Collaboration.  Thus it is expected that the calculations will
either produce data, such as lattice gauge fields or quark propagators,
that can be used by the entire Collaboration, or that the calculations
produce physics results listed among the Collaboration's strategic goals.

Accordingly, proponents planning to generate multi-purpose data must
describe in their proposal what data will be made available to the whole
Collaboration, and how soon, and specify clearly what physics analyses
they would like to perform in an "exclusive manner" on these data (see
below), and the expected time to complete them.

Similarly, proponents planning important physics analyses should explain
how the proposed work meets our strategic goals and how its results
would interest the broader physics community.

Projects generating multi-purpose data are clear candidates to use
USQCD's award(s) on leadership-class computers.  Therefore, these
proposals must provide additional information on several fronts:
they should

	demonstrate the potential to be of broad benefit, for example
	by providing a list of other projects that would use the shared

	present a roadmap for future planning, presenting, for example,
	criteria for deciding when to stop with one ensemble and start
	with another;

	discuss how they would cope with a substantial increase in
	allocated resources, from the portability of the code and
	storage needed to the availability of competent personnel
	to carry out the running;

Some projects carrying out strategic analyses are candidates for running
on the leadership-class machines. They should provide the same information 
as above.  

4) Proposals of Type B are not required to share data or to work towards
stated Collaboration goals, although if they do that it is a plus.
Type B proposals may also be scientifically valuable even if not closely
aligned with USQCD goals.  In that case the proposal should contain a
clear discussion of the physics motivations.  If appropriate, Type B
proposals may discuss data-sharing and strategic importance as in the
case of Type A proposals.

5) The data that will be made available to the whole Collaboration will
have to be released promptly.  "Promptly" should be interpreted with
common sense.  Lattice gauge fields and propagators do not have to be
released as they are produced, especially if the group is still testing
the production environment.  On the other hand, it is not considered
reasonable to delay release of, say, 444 files, just because the last 56
will not be available for a few months.

After a period during which such data will remain for the exclusive use
of the members of the USQCD Collaboration, and possibly of members of
other collaborations under reciprocal agreements, the data will be made
available worldwide as decided by the Executive Committee.

6) The USQCD Collaboration recognizes that the production of shared data
will generally entail a substantial amount of work by the investigators
generating the data.  They should therefore be given priority in
analyzing the data, particularly for their principal physics interests.
Thus, proponents are encouraged to outline a set of physics analyses that
they would like to carry out with these data in an exclusive manner and
the amount of time that they would like to reserve to themselves to
complete such calculations.

When using the shared data, all other members of the USQCD collaboration
agree to respect such exclusivity.  Thus, they shall refrain from using
the data to reproduce the reserved or closely similar analyses.  In its
evaluation of the proposals the Scientific Program Committee will in
particular examine the requests for exclusive use of the data and will
ask the proposers to revise it in case the request was found too broad or
excessive in any other form.  Once an accepted proposal has been posted
on the Collaboration website, it should be deemed by all parties that the
request for exclusive use has been accepted by the Scientific Program
Committee.  Any dispute that may arise in regards to the usage of such
data will have to be directed to the Scientific Program Committee for
resolution and all members of the Collaboration should abide by the
decisions of this Committee.

7)  Usage of the USQCD software, developed under our SciDAC grants, is
recommended, but not required.  USQCD software is designed to be
efficient and portable, and its development leverages efforts throughout
the Collaboration.  If you use this software, the SPC can be confident
that your project can use USQCD resources efficiently.  Software
developed outside the collaboration must be documented to show that it
performs efficiently on its target platform(s).  Information on
portability is welcome, but not mandatory.

8) The investigators whose proposals have been selected by the Scientific
Program Committee for a possible award of USQCD resources shall agree to
have their proposals posted on a password protected website, available
only to our Collaboration, for consideration during the All Hands'

9) The investigators receiving an allocation of time following this Call
for Proposals must maintain a public web page that reasonably documents
their plans, progress, and the availability of data.  These pages should
contain information that funding agencies and review panels can use to
determine whether USQCD is a well-run organization.  The public web page
need not contain unpublished scientific results, or other sensitive

The SPC will not accept new proposals from old projects that still have
no web page.  Please communicate the URL to

ii) Format of the proposals and deadline for submission.

The proposals should contain a title page with title, abstract and the
listing of all participating investigators.  The body, including
bibliography and embedded figures, should not exceed 12 pages in length
for requests of Type A, and 10 pages in length for requests of Type B,
with font size of 11pt or larger.  If necessary, further figures, with
captions but without text, can be appended, for a maximum of 8 additional
pages.  CVs, publication lists and similar personal information are not
requested and should not be submitted.  Title page, proposal body and
optional appended figures should be submitted as a single pdf file, in an
attachment to an e-mail message sent to

The deadline for receipt of the proposals is Friday, March 11, 2011.

The last sentence of the abstract must state the total amount of computer
time in Jpsi-equivalent core-hours and for GPU-clusters in
GPU-hours (see below).  Proposals lacking this information will be 
returned without review (but will be reviewed if the corrected proposal 
is returned quickly and without other changes).

The body of the proposal should contain the following information,
if possible in the order below:

1) The physics goals of the calculation.

2) The computational strategy, including such details as gauge and
fermionic actions, parameters, computational methods.

3) The software used, including a description of the main algorithms
and the code base employed.  If you use USQCD software, it is not
necessary to document performance in the proposal.  If you use your own
code base, then the proposal should provide enough information to show
that it performs efficiently on its target platform(s).  Information on
portability is welcome, but not mandatory.  As feedback for the software
development team, proposals may include an explanation of deficiencies
of the USQCD software for carrying out the proposed work.

4) The amount of resources requested in Jpsi-equivalent core-hours or
GPU hours. Here one should also state which machine is most desirable and 
why, and whether it is feasible or desirable to run some parts of the 
proposed work on one machine, and other parts on another.

The Scientific Program Committee will use the following table to convert:

        1 J/psi core-hour = 1      Jpsi core-hour
        1 Ds    core-hour = 1.33   Jpsi core-hour
	1 7n    core-hour = 0.77   Jpsi core-hour
        1 9q    core-hour = 2.2    Jpsi core-hour
        1 10q   core-hour = 2.3    Jpsi core-hour
        1 9g/FNAL(GPU) hour      = 1 GPU-hour 
        1 BG/P  core-hour = 0.54   Jpsi core-hour
        1 XT5   core-hour = 0.50   Jpsi core-hour  

The above numbers are based on the average of asqtad and DWF fermion
inverters. In the case of XT5 we used the average of asqtad (HISQ)
and clover inverters. See for details.

The total request(s) on clusters and GPUs should also be specified 
in the last sentence of the proposal's abstract (see above). 

Proposals of Type A should indicate longer-term computing needs here.

In addition to CPU, proposals must specify how much mass storage is
needed.  The resources section of the proposal should state how much
existing storage is in use, and how much new storage is needed, for disk 
and tape, in Tbytes.  In addition, please also restate the storage request
in Jpsi-equivalent core-hours, using the following conversion factor, which
reflect the current replacement costs for disk storage and tapes:

        1 Tbyte disk = 26,940 Jpsi-equivalent core-hour
        1 Tbyte tape =  2,694 Jpsi-equivalent core-hour

Projects using disk storage will be charged 12% of these costs every 
three months. Projects will be charged for tape usage when a file is written 
at the full cost of tape storage; when tape files are deleted, they will
receive a 40% refund of the charge.

Proposals should discuss whether these files will be used by one, a few,
or several project(s).  The cost for files (e.g., gauge configurations)
that are used by several projects will borne by USQCD and not a specific
physics project.  The charge for files used by a single project will be
deducted from the computing allocation: projects are thus encouraged to
figure out whether it is more cost-effective to store or re-compute a
file.  If a few (2-3) projects share a file, they will share the

5) What data will be made available to the entire Collaboration, and
the schedule for sharing it.

6) What calculations the investigators would like to perform in an
"exclusive manner" (see above in the section on policy directives),
and for how long they would like to reserve to themselves this
exclusive right.

iii) Procedure for the awards.

The Scientific Program Committee will receive proposals until the deadline 
of Friday, March 11, 2011.  

Proposals not stating the total request in the last sentence of the 
abstract will be returned without review (but will be reviewed if the 
corrected proposal is returned quickly and without other changes).

Proposals that are considered meritorious and conforming to the goals of
the Collaboration will be posted on the web at,
in the Collaboration's password-protected area.  Proposals recommended
for awards in previous years can be found there too.

The Scientific Program Committee (SPC) will make a preliminary
assessment of the proposals.  On  April 15, 2011, the SPC will send 
a report to the proponents raising any concerns about the proposal.

The proposals will be presented and discussed at the All Hands' Meeting,
May 6-7, 2011, at JLab; see however,

  ---- section (iv) for special treatment of INCITE proposals---

Following the All Hands' Meeting the SPC will determine a set of
recommendations on the awards.  The quality of the initial proposal, the
proponents' response to concerns raised in the written report, and the
views of the Collaboration expressed at the All Hands' Meeting will all
influence the outcome.  The SPC will send its recommendations to the
Executive Committee shortly after the All Hands' Meeting, and inform the
proponents once the recommendations have been accepted by the Executive
Committee.  The successful proposals and the size of their awards will be
posted on the web.

The new USQCD allocations will commence July 1, 2011.

Scientific publications describing calculations carried out with these
awards should acknowledge the use of USQCD resources, by including the
following sentence in the Acknowledgments:

"Computations for this work were carried out in part on facilities of
the USQCD Collaboration, which are funded by the Office of Science of
the U.S. Department of Energy."

Projects whose sole source of computing is USQCD should omit the phrase
"in part".

iv) INCITE award CY2011/2012 and zero priority time at Argonne

Since 2007, USQCD policy has been to apply as a Collaboration for time
on the "leadership-class" computers, installed at Argonne and Oak Ridge
National Laboratories, and allocated through the DOE's INCITE Program
(see The first successful three-year
INCITE grant period ended 12/2010. 

A new grant proposal has been successful and received first funding 
for CY2011. 

For CY2011 USQCD was awarded 50 M core-hours on the BG/P at Argonne 
and 30 M core-hours on the Cray XT5 at Oak Ridge.  Following closely
the project proposal laid down in the INCITE project proposal, this 
allocation has already been awarded by the SPC to USQCD projects for 
the period 01/11-12/11. 

We expect to receive a similar allocation in CY2012. In accordance
with the usage pattern we have seen during the last years we will
distribute half of the expected CY2012 INCITE allocation at Oak Ridge
and the entire allocation on the BG/P at Argonne through this Call for 
Proposals for the period 01/2012-06/2012.

In addition we expect to receive in CY2011 also "zero priority time" 
on the BG/P at Argonne. We want to handle this similar to the CY2010
zero priority time.  Zero priority time will become available once 
the regular INCITE allocation is used up. This will presumably be the 
case already in April 2011. For this reason INCITE allocations should
be made already before the All Hands Meeting in May.

With this Call for Proposals USQCD will grant access to ''zero priority
time'' at Argonne in CY2011 as well as to the regular INCITE allocation
which we expect to receive in CY2012. In order to make efficient use of the 
expected ''zero priority time'' the time schedule for INCITE allocations 
has been fixed as follows: 

A) New INCITE allocations will already be announced on April 1st and can 
   be used from then on. Adjustments to these allocations may be made
   after the All Hands Meeting.

B) Starting January 2012 the regular INCITE allocation for CY2012 will
   be used.

C) As soon as USQCD's official INCITE allocation is used up in CY2012, 
   projects may be allowed to use again zero priority time until the end 
   of the USQCD allocation period (06/30/2012).

The usage of the INCITE allocations should be monitored by all PIs of 
INCITE projects on the USQCD WEB-page: 

v) USQCD computing resources.

The Scientific Program Committee will allocate 7200 hours/year to
Type A and Type B proposals.  Of the 8766 hours in an average year the
facilities are supposed to provide 8000 hours of uptime.  We then reserve
400 hours (i.e., 5%) for each host laboratory's own use, and another 400
hours for Type C proposals and contingencies.


856 node cluster ("J/Psi")
   856 quad-core, dual-socket 2.1 GHz Opteron nodes
   (6848 total cpu cores available)
   8 GB memory/node
   Infiniband network
   88 GB local scratch disk/node
   total: 7200*6848*1 = 49.3 M Jpsi-equivalent core-hours
      1 J/Psi node-hour = 8 Jpsi-equivalent core-hour

421 node cluster ("Ds")
    32 cores per node
    64 GB memory/node
    1 Ds core-hour =  1.33 Jpsi-equivalent core-hour
    1 Ds node-hour = 43.56 Jpsi-equivalent core-hours
    total: 7200*421*43.56=  132 M Jpsi-equivalent core-hours

The new GPU cluster at FNAL (after Sept. 1,2011):

65 node cluster equipped with 2GPUs NVIDIA C2050 (Fermi Tesla)

    10 / 12 * 7200 * 130 = 780K GPU hours

These clusters will share about 135 TBytes of disk space in a combination
of dCache and Lustre file systems, plus another approximately 20 TBytes in
conventional NFS-mounted storage.  We will add another 67 TBytes of Lustre
(or dCache) space by the end of the allocation year.  The cluster will have
access to ~ 750 TByte of tape storage. The current maximum size of files
on tape is 400 GBytes.

For further information see


320 node Infiniband cluster ("9q")
  320 quad-core, dual-processor  2.4 GHz Intel Nehalem
  24 GB memory/node, QDR IB fabric in partitions of up to 128 nodes
  total: 7200*320*8*2.2 =   40.5 M  Jpsi-equivalent core-hours
        1 9q    core-hour =  2.2   Jpsi-equivalent core-hours

192 node Infiniband cluster ("10q")
  192 quad-core, dual-processor  2.53 GHz Intel Westmere
  24 GB memory/node, QDR IB fabric in partitions of 32 nodes
  total: 7200*192*8*2.3 =   25.4 M  Jpsi-equivalent core-hours
        1 10q    core-hour =  2.3   Jpsi-equivalent core-hours

340 node Infiniband cluster ("7n")
  340 quad-core, dual-processor 1.9 GHz Opteron (Barcelona)
  8 GB memory/node, DDR IB fabric, 50 GB local scratch disk/node
  total: 7200*340*8*0.77  = 15.1 M Jpsi-equivalent core-hours
        1  7n   core-hour = 0.77   Jpsi-equivalent core-hours

480 node GPU cluster at JLab:

32 node cluster equipped with 4GPUs NVIDIA C2050 (Fermi Tesla)
46 node cluster equipped with 4GPUs GTX-480 (Fermi gaming card) 
32 node cluster equipped with 4GPUs GTX-285 (last year's model)
 2 node cluster equipped with 4GPUs GT200b Tesla (last year's model)
32 node cluster equipped with 1GPU  GTX285 
      total: 7200*480 =  3,456K GPU hours

Further details and comments on the JLab GPU clusters:

1) 32 nodes of quad NVIDIA C2050 (Fermi Tesla) GPUs
   GPU memory (ECC on) 2.6 GB / GPU
   dual Intel Xeon 2.53 GHz quad core (Westmere)
   48 GB memory
   QDR Infiniband in half bandwidth slot

   These are the most flexible nodes, in that the GPUs
   have ECC memory, so are suitable for everything that
   can be run on a GPU, even small scale configuration generation.
   All 32 are on a single switch, so a job can exploit 
   128 GPUs.

2) 46 nodes of quad GTX-480 (Fermi gaming card)
   1.5 GB memory / GPU
   20 are 2.53 GHz Xeon, 28 are 2.4 GHz Xeon
   48 GB memory
   SDR Infiniband in half bandwidth slot, up to 20 nodes per switch

   These are the fastest nodes, 35% faster than Tesla w/ ECC on
   They exhibit a small but non-zero rate of memory errors, 
   are only suitable for inverters, and the application
   must test the residual and discard results if too high.

3) 32 nodes of quad GTX-285 (last year's model)
   2 GB memory / GPU
   dual quad core 2.4 GHz Xeon (Nehalem)
   48 GB memory
   SDR Infiniband in half bandwidth slot
   These are also gaming cards, but show only extremely
   rare memory errors (4+ orders of magnitude lower
   than the 480s).  Some people use these for more than
   inverters.  So far no observed memory errors.

4) 2 nodes of quad GT200b Tesla (last year's model)
   4 GB memory / GPU
   dual quad core 2.4 GHz Xeon (Nehalem)
   48 GB memory
   SDR Infiniband

   While technically "Tesla" professional cards, these
   do not have ECC memory, and are essentially the
   same as the 285s with twice the memory but ~20%
   lower performance.

5) 32 nodes of single GTX-285
   2 GB memory / GPU
   dual quad core 2.4 GHz Xeon (Nehalem)
   24 GB memory
   QDR Infiniband (full bandwidth)

   These nodes are actually a part of the 10q Infiniband
   cluster, and sometimes run non-GPU jobs.  More than 32
     nodes exist with GPUs, so sometimes more than 32 are
     used for GPUs, with an expected average of 32.

   These nodes are suitable for small jobs, or for
   multi-GPU jobs in which more CPU cores are
   needed since there are 8 Intel cores per GPU. 
   Up to 32 nodes may be in a single job
   (one rack, full bandwidth, non oversubscribed QDR)

Measured anisotropic Clover inverter performance, on 
a 24^3 x 128 lattice, multi-GPU running, per GPU:

   ------------   -----------
   C2050          189
   GTX480         256
   GTX285         154
   old Tesla      120

For further information see also

Back to Top

Andreas Kronfeld Legal Notices