USQCD Collaboration, April29–30, 2016
Brookhaven LQCD Facility     Fermilab LQCD Facility     JLab LQCD Facility

All Hands' Home



Call for Proposals


USQCD Science

2016 Machine performance


Argonne ALCF

Oak Ridge OLCF

Social Dinner


2015 Meeting

2014 Meeting

2013 Meeting

2012 Meeting

2011 Meeting

2010 Meeting

2009 Meeting

2008 Meeting

2007 Meeting

2006 Meeting

2005 Meeting

Call for Proposals



Dear Colleagues,


This message is a Call for Proposals for awards of time on the USQCD computer resources dedicated to lattice QCD and other lattice field theories. These are the clusters at Fermilab and JLab, the GPU-clusters at Fermilab and JLab, the BG/Q at BNL.


The awards will be for calculations that further the scientific goals of the collaboration, as laid out in the recent white papers and USQCD proposals that can be found at

/collaboration, noting that an important reason for funding is relevance to the DOE experimental program.


In this allocation year, we expect to distribute about


           71 M BG/Q core-hours at BNL 

           263 M Jpsi-core hours on clusters at FNAL and JLAB

           4.6 M GPU-hours on GPU clusters at FNAL and JLAB

           approximately 250M JPSI=core-hours at JLAB on new resources


           32 M Jpsi-equivalent core-hours which we expect to 

                charge for disc and tape usage. 


USQCD also has community resources through the DoE INCITE program at Argonne and Oak Ridge and on the NSF supercomputer Blue Waters at NCSA which are not being allocated this year.  We have just submitted a new Blue Waters proposal and we do not yet know how we fared.  In order to better present our diverse national program, we split our request into four separate proposals in each of our four main subject areas.  For these Blue Waters proposals we were instructed to apply for only two years, and our understanding is that the Blue Waters program will end at the conclusion of that period, so that these were our last Blue Waters proposals.  We will also apply for new INCITE time for the period 2017-9 this year.  We are seriously considering restructuring our traditional unified INCITE proposal into several subject-based proposals, as we did for Blue Waters. 



USQCD is adopting a new policy this year to encourage more even use of allocations throughout the year.  At the beginning of the current allocation year last July, a large fraction of the projects at Fermilab and JLab were not ready to run.  After alerting the allocated projects of this fact, the site managers properly began letting unallocated projects run in opportunistic mode in order that cycles not go completely to waste.  This was not an optimal use of our resources which had been so painstakingly allocated by the SPC.  Further, it undermines the case of our friends in Germantown who are arguing for more resources for us if we are not ready to use the resources they have already given us.  In order to discourage this situation, we are adopting a policy already in use at supercomputer centers such as NERSC and requiring projects to use some of their allocation in each calendar quarter.  Projects that fail to do this will forfeit some of their allocation for the quarter, which will be added to the allocations of projects that are ready to run.  (See for a detailed statement of the NERSC rules that we will use.) 

UPDATE 10-17-2016: NERSC has changed the page which we reference above. A new USQCD Policy page has been posted at //reductions.html


All members of the USQCD Collaboration are eligible to submit proposals.  Those interested in joining the Collaboration should contact Paul Mackenzie (



Let us begin with some important dates:



      February   14: this Call for Proposals

      March     11: proposals due for Type A proposals

      April     18: reports to proponents sent out

      April      29-30: All Hands' Meeting at BNL (ending ~5pm)

      May       31: allocations announced

      July       1: new allocations start


The Scientific Program Committee (SPC) will request some number of presentations by the proponents of proposals at the All Hands' Meeting.  Proponents may in general request to make an oral presentation of their proposals; however, the logistical constraints of the meeting may preclude some number of talks.


The web site for the All Hands' Meeting is




Requests can be of three types:


  A) requests for potentially large amounts of time on USQCD  

     dedicated resources and/or leadership class computers, to

     support calculations of benefit for the whole USQCD

     Collaboration and/or addressing critical scientific needs.

     There is no minimum size to the request. However, small

     requests will be not considered suitable for leadership

     resources. Allocations are for one year on USQCD resources.


  B) requests for medium amounts of time on USQCD dedicated

     resources intended to support calculations in an early

     stage of development which address, or have the potential

     to address the scientific needs of the collaboration;

        --- No maximum, but encouraged to be below 2.5 M

            Jpsi-equivalent core-hours or less on clusters, 

            or 100 K GPU hours or less on GPU clusters. 

            No suggested size for BNL BG/Q requests ---

     Allocations are for up to 6 months.


  C) requests for exploratory calculations, such as those

     needed to develop and/or benchmark code, acquire expertise

     on the use of the machines, or to perform investigations of

     limited scope. The amount of time used by such projects

     should not exceed 100 K Jpsi core-hours on clusters or 10 K

     GPU-hours on the GPU-clusters. Requests for BG/Q at BNL

     should be handled on a case basis.





 Requests of Type A and B must be made in writing to the Scientific Program     

Committee and are subject to the policies spelled out below.

These proposals must also specify the amount of disk and tape storage they will carry forward and the amount of each that will be created in the coming year.  Projects will be charged for new disks and tapes as well as existing disk usage.  How this will be implemented is discussed in section (iii).


Requests of Type B can be made anytime of the year, and will start in the nearest month. Requests should be sent in an e-mail message to

Anna Hasenfratz (


Requests of Type C should be made in an e-mail message to

  Paul Mackenzie ( for clusters at FNAL,

  Robert Mawhinney ( for the BG/Q at BNL,

  Chip Watson ( for clusters at JLAB.


Type B requests will be considered up to a total not exceeding 15% of the available time on USQCD hardware.  Type C requests will be considered up to a total not exceeding 5% of the available time on USQCD hardware.  If the demand exceeds such limits, the Scientific Program Committee will reconsider the procedures for access.


Collaboration members who wish to perform calculations on USQCD hardware or on resources awarded to USQCD through the INCITE program can present requests according to procedures specified below. The Scientific Program Committee would like to handle requests and awards on leadership class computers and cluster in their respective units, namely Blue Gene core hours or Cray core hours. Requests for the BG/Q will be handled in BG/Q core hours, and requests on the GPU clusters will be handled in GPU hours.  Conversion factors for clusters, GPUs, and leadership class computers are given below. As projects usually are not flexible enough to switch between running on GPUs, BG/Q, and clusters, we choose to allocate in their respective units. In addition, since the various GPU clusters have quite different properties, it may be useful if proposals asking for GPU time included a preference, if any, for particular USQCD GPU.

 However, as nominal conversion factors are available, we describe at the end of the document the total resources available to USQCD in TFlop-years.




The rest of this message deals with requests of Types A and B.  It is organized as follows:


  i)   policy directives regarding the usage of awarded resources;


  ii)  guidelines for the format of the proposals and deadline for submission;


  iii) procedures that will be followed to reach a consensus on the research programs and the allocations;


  iv)  policies for handling awards on leadership-class machines


  v)   description of USQCD resources at Fermilab and JLAB



i) Policy directives.


1) This Call for Proposals is for calculations that will further the physics goals of the USQCD Collaboration, as stated in the proposals for funding submitted to the DOE (see /), and have the potential of benefiting additional research projects by members of the Collaboration. In particular, the scientific goals are described in the science sections of the recent SciDAC proposals and in the recent white papers, which are placed on the same web-site.  It is important to our success in continued funding that we demonstrate continued importance in helping DoE experiments to succeed. 


2) Proposals of Type A are for investigations of very large scale,which may require a substantial fraction of the available resources. Proposals of Type B are for investigations in an early stage of development, and are medium to large scale which will require a smaller amount of resources. There is no strict lower limit for requests within Type A proposals, and there is no upper limit on Type B Proposals. However, Type B requests for significantly more than 2.5 M Jpsi-equivalent core-hours on clusters or more than 100 K hours on GPU-clusters, will receive significant scrutiny.

 Proposals that request time on the leadership-class computers at Argonne and Oak Ridge should be of Type A and should demonstrate that they (i) can efficiently make use of large partitions of leadership class computers, and (ii) will run more efficiently on leadership class computers than on clusters.


3) All Type A and B proposals are expected to address the scientific needs of the USQCD Collaboration.  Proposals of Type A are for investigations that benefit the whole USQCD Collaboration.  Thus it is expected that the calculations will either produce data, such as lattice gauge fields or quark propagators, that can be used by the entire Collaboration, or that the calculations produce physics results listed among the Collaboration's strategic goals.

 Accordingly, proponents planning to generate multi-purpose data must describe in their proposal what data will be made available to the whole Collaboration, and how soon, and specify clearly what physics analyses they would like to perform in an "exclusive manner" on these data (see below), and the expected time to complete them.

 Similarly, proponents planning important physics analyses should explain how the proposed work meets our strategic goals and how its results would interest the broader physics community.

 Projects generating multi-purpose data are clear candidates to use USQCD's award(s) on leadership-class computers.  Therefore, these proposals must provide additional information on several fronts: they should

   - demonstrate the potential to be of broad benefit, for example by providing a list of other projects that would use the shared data, or how the strategic scientific needs of USQCD are addressed;

   - present a roadmap for future planning, presenting, for example, criteria for deciding when to stop with one ensemble and start with another;

   - discuss how they would cope with a substantial increase in allocated resources, from the portability of the code and storage needed to the availability of competent personnel to carry out the running;


Some projects carrying out strategic analyses are candidates for running on the leadership-class machines. They should provide the same information as above.


4) Proposals of Type B are not required to share data, although if they do so it is a plus.  Type B proposals may also be scientifically valuable even if not closely aligned with USQCD goals.  In that case the proposal should contain a clear discussion of the physics motivations.  If appropriate, Type B proposals may discuss data-sharing and strategic importance as in the case of Type A proposals.


5) The data that will be made available to the whole Collaboration will have to be released promptly.  "Promptly" should be interpreted with common sense.  Lattice gauge fields and propagators do not have to be released as they are produced, especially if the group is still testing the production environment.  On the other hand, it is not considered reasonable to delay release of, say, 444 files, just because the last 56 will not be available for a few months.

 After a period during which such data will remain for the exclusive use of the members of the USQCD Collaboration, and possibly of members of other collaborations under reciprocal agreements, the data will be made available worldwide as decided by the Executive Committee.


6) The USQCD Collaboration recognizes that the production of shared data will generally entail a substantial amount of work by the investigators generating the data.  They should therefore be given priority in analyzing the data, particularly for their principal physics interests.  Thus, proponents are encouraged to outline a set of physics analyses that they would like to carry out with these data in an exclusive manner and the amount of time that they would like to reserve to themselves to complete such calculations.

 When using the shared data, all other members of the USQCD collaboration agree to respect such exclusivity.  Thus, they shall refrain from using the data to reproduce the reserved or closely similar analyses.  In its evaluation of the proposals the Scientific Program Committee will in particular examine the requests for exclusive use of the data and will ask the proposers to revise it in case the request was found too broad or excessive in any other form. Once an accepted proposal has been posted on the Collaboration website, it should be deemed by all parties that the request for exclusive use has been accepted by the Scientific Program Committee. Any dispute that may arise in regards to the usage of such data will have to be directed to the Scientific Program Committee for resolution and all members of the Collaboration should abide by the decisions of this Committee.


7) Usage of the USQCD software, developed under our SciDAC grants, is recommended, but not required.  USQCD software is designed to be efficient and portable, and its development leverages efforts throughout the Collaboration.  If you use this software, the SPC can be confident that your project can use USQCD resources efficiently. Software developed outside the collaboration must be documented to show that it performs efficiently on its target platform(s). Information on portability is welcome, but not mandatory.


8) The investigators whose proposals have been selected by the Scientific Program Committee for a possible award of USQCD resources shall agree to have their proposals posted on a password protected website, available only to our Collaboration, for consideration during the All Hands' Meeting.


9) The investigators receiving a Type A allocation of time following this Call for Proposals must maintain a public web page that reasonably documents their plans, progress, and the availability of data.  These pages should contain information that funding agencies and review panels can use to determine whether USQCD is a well-run organization.  The public web page need not contain unpublished scientific results, or other sensitive information.


The SPC will not accept new proposals from old projects that still have no web page.  Please communicate the URL to


ii) Format of the proposals and deadline for submission.


The proposals should contain a title page with title, abstract and the listing of all participating investigators.  The body, including bibliography and embedded figures, should not exceed 12 pages in length for requests of Type A, and 10 pages in length for requests of Type B, with font size of 11pt or larger.  If necessary, further figures, with captions but without text, can be appended, for a maximum of 8 additional pages.  CVs, publication lists and similar personal information are not requested and should not be submitted. Title page, proposal body and optional appended figures should be submitted as a single pdf file, in an attachment to an e-mail message sent to


The deadline for receipt of Type A proposals is Friday, March 11, 2016.


The last sentence of the abstract must state the total amount of computer time in Jpsi-equivalent core-hours for clusters, GPU-clusters in GPU-hours, and in BG/Q core hours for those machines. Proposals lacking this information will be returned without review (but will be reviewed if the corrected proposal is returned quickly and without other changes).


The body of the proposal should contain the following information, if possible in the order below:


1) The physics goals of the calculation.

2) The computational strategy, including such details as gauge and fermionic actions, parameters, computational methods.

3) The software used, including a description of the main algorithms and the code base employed.  If you use USQCD software, it is not necessary to document performance in the proposal.  If you use your own code base, then the proposal should provide enough information to show that it performs efficiently on its target platform(s).

Information on portability is welcome, but not mandatory.  As feedback for the software development team, proposals may include an explanation of deficiencies of the USQCD software for carrying out the proposed work.


4) The amount and type of resources requested. Here one should also state which machine is most desirable and why, and whether it is feasible or desirable to run some parts of the proposed work on one machine, and other parts on another.  If relevant, proposals of Type A should indicate longer-term computing needs here.


The Scientific Program Committee will use the following table to convert:


      1 J/psi    core-hour = 1      Jpsi core-hour

      1 Ds       core-hour = 1.33   Jpsi core-hour

      1 9q       core-hour = 2.2    Jpsi core-hour

      1 10q      core-hour = 2.3    Jpsi core-hour

      1 12s      core-hour = 2.3    Jpsi core-hour

      1 XK7      core-hour = 1.0    Jpsi core-hour

      1 BG/Q     core-hour = 1.64   Jpsi core-hour

      1 C2050    GPU hour  = 82     Jpsi equivalent core-hour

      1 K20      GPU hour  = 172    Jpsi equivalent core-hour

      1 K40      GPU hour  = 224    Jpsi equivalent core-hour

      1 Phi      MIC hour  = 164    Jpsi equivalent core-hour

      1 Jpsi     core-hour = 1.22   GFlop/sec-hour


The above numbers are based on appropriate averages of asqtad, DWF fermion inverters, and Clover inverters. In the case of XK7 performance is based on a Clover inverter run on the GPUs at leadership scale. The conversion of GPU to Jpsi is based on the average of application performance on user jobs across all GPU systems at FNAL and JLab (including gamer as well as non-gamer cards).  See

performance.html for details.


In addition to CPU, proposals must specify how much mass storage is needed.  The resources section of the proposal should state how much tape and disk storage is already in use, and how much new storage is needed, for disk and tape, in Tbytes.  In addition, please also restate the storage request in Jpsi-equivalent core-hours, using the following conversion factor, which reflect the current replacement costs for disk storage and tapes:


      1 Tbyte disk =   40 K Jpsi-equivalent core-hour

      1 Tbyte tape =    6 K Jpsi-equivalent core-hour


Projects using disk storage will be charged 25% of these costs every three months. Projects will be charged for tape usage when a file is written at the full cost of tape storage; when tape files are deleted, they will receive a 40% refund of the charge.

 Proposals should discuss whether these files will be used by one, a few, or several project(s).  The cost for files (e.g., gauge configurations) that are used by several projects will borne by USQCD and not a specific physics project.  The charge for files used by a single project will be deducted from the computing allocation: projects are thus encouraged to figure out whether it is more cost-effective to store or re-compute a file.  If a few (2-3) projects share a file, they will share the charge.


Projects that expect to have large I/O requirements, such as those that use eigenvalue and deflation methods, are requested to note that in their proposal and to work with the site managers to handle these needs as painlessly as possible.


5) If relevant, what data will be made available to the entire Collaboration, and the schedule for sharing it.


6) What calculations the investigators would like to perform in an "exclusive manner" (see above in the section on policy directives), and for how long they would like to reserve to themselves this exclusive right.


iii) Procedure for the awards.


The Scientific Program Committee will receive proposals until the deadline of Friday, March 11, 2016.  Proposals not stating the total request in the last sentence of the abstract will be returned without review (but will be reviewed if the corrected proposal is returned quickly and without other changes).


Proposals that are considered meritorious and conforming to the goals of the Collaboration will be posted on the web at /, in the Collaboration's password-protected area.

Proposals recommended for awards in previous years can be found there too.


The Scientific Program Committee (SPC) will make a preliminary assessment of the proposals.  On April 18, 2016, the SPC will send a report to the proponents raising any concerns about the proposal.


A few proposals will be presented and discussed at the All Hands' Meeting, April 29-30, 2016, at BNL.


Following the All Hands' Meeting the SPC will determine a set of recommendations on the awards. The quality of the initial proposal, the proponents' response to concerns raised in the written report, and the views of the Collaboration expressed at the All Hands' Meeting will all influence the outcome.  The SPC will send its recommendations to the Executive Committee after the All Hands' Meeting, and inform the proponents once the recommendations have been accepted by the Executive Committee.  The successful proposals and the size of their awards will be posted on the web.


The new USQCD allocations will commence July 1, 2016.


Scientific publications describing calculations carried out with these awards should acknowledge the use of USQCD resources, by including the following sentence in the Acknowledgments:


"Computations for this work were carried out in part on facilities of the USQCD Collaboration, which are funded by the Office of Science of the U.S. Department of Energy."


Projects whose sole source of computing is USQCD should omit the phrase "in part".




We expect to receive in CY2016 zero-priority time on the BG/Q at Argonne.  Based on previous usage and availability, we will distribute zero-priority time starting in 2016 as soon as our INCITE allocation has been consumed and zero-priority time becomes available, according to the allocations made during last yearŐs, 2015 allocation process.  This is expected to begin in April 2016.  These 2015 allocations for 2016 zero-priority time will complete June 30, 2016.   As part of the current, 2016 allocations process, driven by this Call for Proposals, we will allocate the remainder of CY2016 ANL zero-priority time as well as that which becomes available to USQCD in the first half of 2017.   As the total amount of time cannot be reliably estimated, we will allocate percentages of zero-priority usage. The SPC may readjust these percentage allocations based upon observed usage. The Oak Ridge facility does not provide a zero-priority queue.


The usage of the INCITE allocations should be monitored by all PIs of INCITE projects on the USQCD WEB-page:


v) USQCD computing resources.


The Scientific Program Committee will allocate 7200 hours/year to Type A and Type B proposals.  Of the 8766 hours in an average year the facilities are supposed to provide 8000 hours of uptime.  We then reserve 400 hours (i.e., 5%) for each host laboratory's own use, and another 400 hours for Type C proposals and contingencies.





60% of a 1024 node BG/Q rack

  16 cores/node, up to 4 threads per core

  16 GB memory/node

  10% of a BNL rack with time donated to USQCD.

  50% of a rack owned by USQCD.

     total: 7200*1024*16*0.60 = 70.8 M BG/Q core-hours = 116 M Jpsi-equivalent core-hours


There is no tape storage at BNL for USQCD activities.  We have 100+ TBytes of disk space, which should be ample for users to stage their calculation to/from BNL,  but long term storage on tape will continue to be done at FNAL and JLAB.






224 node cluster ("Bc")

 Eight-core, quad-socket 2.8 GHz AMD Opteron (Abu Dhabi) nodes

 32 cores per node

 64 GB memory/node

 1 core-hour = 1.48 JPsi-equivalent core-hours

     total: 7200*224*32*1.48 = 76.4 M Jpsi-equivalent core-hours


314 node cluster ("Pi0")

 Eight-core, dual-socket 2.6 GHz Intel Xeon (Ivy Bridge) nodes

 16 cores per node

 128 GB memory/node

 1 core-hour = 3.14 JPsi-equivalent core-hours

     total: 7200*314*16*3.14 = 113.6 M Jpsi-equivalent core-hours


32 node cluster ("Pi0g")

 Eight-core, dual-socket 2.6 GHz Intel Xeon (Ivy Bridge) nodes

 128 GB memory/node

 4 GPUs NVIDIA K40m (Kepler Tesla) per node, GPU rating 2.6

 (128 total GPUs available)

 GPU memory (ECC on) 11.5 GB/GPU

 Each K40 gpu-hr is equivalent to 2.6 Fermi-gpu-hr

     total = 7200*128*2.6 = 2396 K GPU-hours)


These clusters will share about 1000 TBytes of disk space in Lustre file systems. Tape access is also available.


For further information see /fnal/






276 node cluster ("12s")

  Eight-core, dual processor Intel Sandy Bridge nodes

  16 cores per node

  32 GB memory/node

  QDR network card, with full bi-sectional bandwidth network fabric

  1 12s core-hour = 2.3 Jpsi cores

      total: 7200*276*16*2.3 =  73.1 M  Jpsi-equivalent core hours 



42 node GPU cluster at JLab  ("12k")

    Dual 8 core Intel Sandy Bridge host nodes

    4 NVIDIA K20m (2012 Kepler Tesla) GPUs/node

    128 GB memory/node

    FDR network fabric with full bi-sectional bandwidth

    1 K20m = 2 C2050

    168 total GPUs available: in C2050 units -> 336 GPUs total

    total: 7200*42*4*2 = 2.4 M GPU hours

    (equivalent to ~ 200 M Jpsi core hours from the listed conversion factors)


There is a projected procurement of 49 TFlops in FY16. This might come in the form of 220M as GPU (equivalent to ~4.4 M GPU hours) plus 80M Jpsi conventional, or it might come as a pure KNL resource of that same capacity (300 M Jpsi core hours for this allocation year). At present we are allocating 250 M Jpsi equivalent core hours for this projected resource


For further information see also . Machine descriptions can be found at


At JLAB, the systems will have access to about 1 PBytes of disk space. Tape access is also available.



Resource estimates


Based upon the performance conversions used above, the total resources available in this call are shown below.


BG/Q (BNL):                 116 M Jpsi ->  16 TF-yr

Clusters (FNAL):           190 M Jpsi ->  26 TF-yr

Clusters (JLab):               73 M Jpsi ->  10 TF-yr

New resources(JLAB);   250 M Jpsi ->  34 TF-yr  (for 2nd half of allocation period)

Tesla GPUs (FNAL):      283 M Jpsi ->  39 TF-yr

Total (USQCD):              962 M Jpsi -> 132 TF-yr




Back to Top

Andreas Kronfeld Legal Notices