|
|
| |
|
|
|
| |
|
All Hands' Home USQCD Science |
Call for ProposalsDear Colleagues, This message is a Call for Proposals for
awards of time on the USQCD computer resources dedicated to lattice QCD and other lattice field theories. These are the clusters at Fermilab and JLab, the GPU-clusters at Fermilab and JLab, the BG/Q at
BNL. The awards will be for calculations that
further the scientific goals of the collaboration, as laid out in the
recent white papers and USQCD proposals that can be found at /collaboration, noting that an important reason for funding is relevance to the DOE
experimental program. In this allocation year, we expect to
distribute about 71 M BG/Q core-hours at BNL 263 M
Jpsi-core hours on clusters at FNAL and JLAB 4.6 M
GPU-hours on GPU clusters at FNAL and JLAB
approximately 250M JPSI=core-hours at JLAB on new resources 32 M Jpsi-equivalent core-hours which we expect to
charge for disc and tape usage.
USQCD also has community
resources through the DoE INCITE program at Argonne and Oak Ridge and on the
NSF supercomputer Blue Waters at NCSA which are not being allocated this year. We have just submitted a new Blue Waters
proposal and we do not yet know how we fared. In order to better present our diverse
national program, we split our request into four separate proposals in each of
our four main subject areas. For
these Blue Waters proposals we were instructed to apply for only two years, and
our understanding is that the Blue Waters program will end at the conclusion of
that period, so that these were our last Blue Waters proposals. We will also apply for new INCITE time
for the period 2017-9 this year. We
are seriously considering restructuring our traditional unified INCITE proposal
into several subject-based proposals, as we did for Blue Waters. USQCD is adopting a new policy this year to
encourage more even use of allocations throughout the year. At the beginning of the current
allocation year last July, a large fraction of the projects at Fermilab and JLab were not ready to run. After alerting the allocated projects of
this fact, the site managers properly began letting unallocated projects run in
opportunistic mode in order that cycles not go completely to waste. This was not an optimal use of our
resources which had been so painstakingly allocated by the SPC. Further, it undermines the case of our
friends in Germantown who are arguing for more resources for us if we are not
ready to use the resources they have already given us. In order to discourage this situation,
we are adopting a policy already in use at supercomputer centers such as NERSC
and requiring projects to use some of their allocation in each calendar
quarter. Projects that fail to do
this will forfeit some of their allocation for the quarter, which will be added
to the allocations of projects that are ready to run. (See UPDATE 10-17-2016: NERSC has changed the page which we reference above. A new USQCD Policy page has been posted at //reductions.html All members of the USQCD Collaboration are
eligible to submit proposals. Those
interested in joining the Collaboration should contact Paul Mackenzie (mackenzie@fnal.gov). Let us begin with some important dates: ======================================= February 14: this
Call for Proposals March 11:
proposals due for Type A proposals April 18:
reports to proponents sent out April 29-30: All Hands' Meeting at BNL (ending
~5pm) May
31: allocations announced July
1: new allocations start The Scientific Program Committee (SPC) will
request some number of presentations by the
proponents of proposals at the All Hands' Meeting. Proponents may in general request to make an
oral presentation of their proposals; however, the logistical constraints of the meeting may preclude some number of talks. The web site for the All Hands' Meeting is Requests can be of three types: A) requests for potentially large
amounts of time on USQCD dedicated resources and/or
leadership class computers, to support calculations of
benefit for the whole USQCD Collaboration and/or
addressing critical scientific needs. There is no minimum size
to the request. However, small requests will be not
considered suitable for leadership resources. Allocations are
for one year on USQCD resources. B) requests for medium amounts of time
on USQCD dedicated resources intended to
support calculations in an early stage of development which address, or have the potential to address the scientific
needs of the collaboration; --- No maximum,
but encouraged to be below 2.5 M Jpsi-equivalent core-hours or less on clusters,
or 100 K GPU hours or less on GPU
clusters.
No suggested size for BNL BG/Q requests --- Allocations are for up to
6 months. C) requests for exploratory
calculations, such as those needed to develop
and/or benchmark code, acquire expertise on the use of the
machines, or to perform investigations of limited scope. The
amount of time used by such projects should not exceed 100
K Jpsi core-hours on clusters or 10 K GPU-hours on the GPU-clusters.
Requests for BG/Q at BNL should be handled
on a case basis. Requests
of Type A and B must be made in writing to the Scientific Program Committee and are subject to the policies
spelled out below. These proposals must also specify the amount
of disk and tape storage they will carry forward and the amount of each that
will be created in the coming year. Projects will be charged for new disks and tapes as well as existing disk usage. How this will be
implemented is discussed in section (iii). Requests of Type B can be made anytime of the
year, and will start in the nearest month.
Requests should be sent in an e-mail message to Anna Hasenfratz (anna@eotvos.colorado.edu). Requests of Type C should be made in an e-mail
message to Paul Mackenzie (mackenzie@fnal.gov) for clusters at FNAL, Robert Mawhinney
(rdm@physics.columbia.edu) for the BG/Q at BNL, Chip Watson (Chip.Watson@jlab.org) for clusters at JLAB. Type B requests will be considered up to a
total not exceeding 15% of the available time on
USQCD hardware. Type C requests will be considered up to a total not exceeding 5% of the available time on USQCD hardware. If the demand exceeds such
limits, the Scientific Program Committee will
reconsider the procedures for access. Collaboration members who wish to perform
calculations on USQCD hardware or on resources
awarded to USQCD through the INCITE program can present requests according to procedures specified below. The Scientific Program Committee would like to handle
requests and awards on leadership class
computers and cluster in their respective units, namely Blue Gene core hours or Cray core hours. Requests for the BG/Q will
be handled in BG/Q core hours, and
requests on the GPU clusters will be handled in GPU hours. Conversion factors for clusters, GPUs, and
leadership class computers are given below. As projects usually are not
flexible enough to switch between
running on GPUs, BG/Q, and clusters, we choose to allocate in their respective units. In addition, since the
various GPU clusters have quite different properties, it may be useful if
proposals asking for GPU time included a preference, if any, for particular
USQCD GPU. However, as nominal conversion factors are available, we describe at the
end of the document the total
resources available to USQCD in TFlop-years.
-------------------------------------------- The rest of this message deals with requests
of Types A and B. It is organized as follows: i)
policy directives regarding the usage of awarded resources; ii) guidelines for the format of
the proposals and deadline for submission; iii) procedures that will be followed
to reach a consensus on the research programs and the allocations; iv) policies for handling awards
on leadership-class machines v) description of USQCD resources
at Fermilab and JLAB i) Policy directives. 1) This Call for Proposals is for calculations
that will further the physics goals of the
USQCD Collaboration, as stated in the proposals for funding submitted to the DOE (see /), and have the potential of
benefiting additional research projects by members of the Collaboration. In particular, the scientific goals
are described in the science sections
of the recent SciDAC proposals and in the recent
white papers, which are placed on the same web-site. It is important to
our success in continued funding that we demonstrate continued
importance in helping DoE experiments to succeed. 2) Proposals of Type A are for investigations
of very large scale,which
may require a substantial fraction of the available resources. Proposals
of Type B are for investigations in an early stage of development, and are
medium to large scale which will require a smaller amount of resources.
There is no strict lower limit for requests within Type A proposals, and
there is no upper limit on Type B Proposals. However, Type B requests for
significantly more than 2.5 M Jpsi-equivalent
core-hours on clusters or more than 100 K hours on GPU-clusters, will
receive significant scrutiny. Proposals that request time on the
leadership-class computers at Argonne and Oak Ridge should be of Type A
and should demonstrate that they (i) can
efficiently make use of large partitions of leadership class computers,
and (ii) will run more efficiently on leadership class computers than on
clusters. 3) All Type A and B proposals are expected to
address the scientific needs of the USQCD
Collaboration. Proposals of Type A are for investigations that
benefit the whole USQCD Collaboration. Thus it is expected that the
calculations will either produce data, such as lattice gauge fields or
quark propagators, that can be used by the entire Collaboration, or that
the calculations produce physics results listed among the Collaboration's
strategic goals. Accordingly, proponents planning to
generate multi-purpose data must describe in their proposal what data will
be made available to the whole Collaboration, and how soon, and specify
clearly what physics analyses they would like to perform in an "exclusive
manner" on these data (see below), and the expected time to complete
them. Similarly, proponents planning important
physics analyses should explain how the proposed work meets our strategic
goals and how its results would interest the broader physics community. Projects generating multi-purpose data
are clear candidates to use USQCD's award(s) on leadership-class
computers. Therefore, these proposals must provide additional
information on several fronts: they should - demonstrate the potential
to be of broad benefit, for example by providing a list of other projects
that would use the shared data, or how the strategic scientific needs of
USQCD are addressed; - present a roadmap for
future planning, presenting, for example, criteria for deciding when to
stop with one ensemble and start with another; - discuss how they would
cope with a substantial increase in allocated resources, from the
portability of the code and storage needed to the availability of
competent personnel to carry out the running; Some projects carrying out strategic analyses
are candidates for running on the
leadership-class machines. They should provide the same information as
above. 4) Proposals of Type B are not required to
share data, although if they do so it is a plus.
Type B proposals may also be scientifically valuable even if not
closely aligned with USQCD goals. In that case the proposal should
contain a clear discussion of the physics motivations. If
appropriate, Type B proposals may discuss data-sharing and strategic
importance as in the case of Type A proposals. 5) The data that will be made available to the
whole Collaboration will have to be released
promptly. "Promptly" should be interpreted with common
sense. Lattice gauge fields and propagators do not have to be
released as they are produced, especially if the group is still testing
the production environment. On the other hand, it is not considered
reasonable to delay release of, say, 444 files, just because the last 56
will not be available for a few months. After a period during which such data
will remain for the exclusive use of the members of the USQCD
Collaboration, and possibly of members of other collaborations under
reciprocal agreements, the data will be made available worldwide as
decided by the Executive Committee. 6) The USQCD Collaboration recognizes that the
production of shared data will generally
entail a substantial amount of work by the investigators generating the
data. They should therefore be given priority in analyzing the data,
particularly for their principal physics interests. Thus, proponents
are encouraged to outline a set of physics analyses that they would like
to carry out with these data in an exclusive manner and the amount of time
that they would like to reserve to themselves to complete such
calculations. When using the shared data, all other
members of the USQCD collaboration agree to respect such exclusivity.
Thus, they shall refrain from using the data to reproduce the
reserved or closely similar analyses. In its evaluation of the
proposals the Scientific Program Committee will in particular examine the
requests for exclusive use of the data and will ask the proposers to
revise it in case the request was found too broad or excessive in any
other form. Once an accepted proposal has been posted on the
Collaboration website, it should be deemed by all parties that the request
for exclusive use has been accepted by the Scientific Program
Committee. Any dispute that may arise in regards to the usage of such data
will have to be directed to the Scientific Program Committee for
resolution and all members of the Collaboration should abide by the
decisions of this Committee. 7) Usage of the USQCD software, developed
under our SciDAC grants, is recommended, but not required. USQCD software is
designed to be efficient and portable, and its development leverages
efforts throughout the Collaboration. If you use this software, the
SPC can be confident that your project can use USQCD resources
efficiently. Software developed outside the collaboration must be
documented to show that it performs efficiently on its target
platform(s). Information on portability is welcome, but not mandatory. 8) The investigators whose proposals have been
selected by the Scientific Program
Committee for a possible award of USQCD resources shall agree to have
their proposals posted on a password protected website, available only to
our Collaboration, for consideration during the All Hands' Meeting. 9) The investigators receiving a Type A
allocation of time following this Call for Proposals must maintain a public web page
that reasonably documents their plans, progress, and the availability
of data. These pages should contain information that funding
agencies and review panels can use to determine whether USQCD is a
well-run organization. The public web page need not contain
unpublished scientific results, or other sensitive information. The SPC will not accept new proposals from old
projects that still have no web page.
Please communicate the URL to mackenzie@fnal.gov ii) Format of the proposals and deadline for
submission. The proposals should contain a title page with
title, abstract and the listing of all
participating investigators. The body, including bibliography and
embedded figures, should not exceed 12 pages in length for requests of
Type A, and 10 pages in length for requests of Type B, with font size of
11pt or larger. If necessary, further figures, with captions but
without text, can be appended, for a maximum of 8 additional pages.
CVs, publication lists and similar personal information are not
requested and should not be submitted. Title page, proposal body and
optional appended figures should be submitted as a single pdf file, in an
attachment to an e-mail message sent to anna@eotvos.colorado.edu The deadline for receipt of Type A proposals
is Friday, March 11, 2016. The last sentence of the abstract must state
the total amount of computer time in Jpsi-equivalent core-hours for clusters,
GPU-clusters in GPU-hours, and in BG/Q core hours for those machines.
Proposals lacking this information will be returned without review (but will
be reviewed if the corrected proposal is returned quickly and without
other changes). The body of the proposal should contain the
following information, if possible in the order
below: 1) The physics goals of the calculation. 2) The computational strategy, including such
details as gauge and fermionic actions, parameters, computational methods. 3) The software used, including a description
of the main algorithms and the code base
employed. If you use USQCD software, it is not necessary to document
performance in the proposal. If you use your own code base, then the
proposal should provide enough information to show that it performs
efficiently on its target platform(s). Information on portability is welcome, but not
mandatory. As feedback for the software development team, proposals may include
an explanation of deficiencies of the USQCD software for carrying out
the proposed work. 4) The amount and type of resources requested.
Here one should also state which machine is
most desirable and why, and whether it is feasible or desirable to run
some parts of the proposed work on one machine, and other parts on
another. If relevant, proposals of Type A should indicate
longer-term computing needs here. The Scientific Program Committee will use the
following table to convert: 1 J/psi
core-hour = 1 Jpsi
core-hour
1 12s core-hour
= 2.3 Jpsi core-hour 1 XK7
core-hour = 1.0 Jpsi core-hour 1 BG/Q
core-hour = 1.64 Jpsi core-hour 1 C2050 GPU
hour = 82 Jpsi equivalent
core-hour 1 K20
GPU hour = 172 Jpsi
equivalent core-hour 1 K40
GPU hour = 224 Jpsi
equivalent core-hour 1 Jpsi
core-hour = 1.22 GFlop/sec-hour The above numbers are based on appropriate
averages of asqtad, DWF fermion inverters, and Clover inverters. In the case of
XK7 performance is based on a Clover inverter run on the GPUs
at leadership scale. The conversion of GPU to Jpsi
is based on the average of application performance on user jobs across all
GPU systems at FNAL and JLab (including gamer as
well as non-gamer cards). See performance.html for details. In addition to CPU, proposals must specify how
much mass storage is needed. The resources
section of the proposal should state how much tape and disk storage is
already in use, and how much new storage is needed, for disk and tape, in Tbytes. In addition, please also restate
the storage request in Jpsi-equivalent
core-hours, using the following conversion factor, which reflect the
current replacement costs for disk storage and tapes: 1 Tbyte
disk = 40 K Jpsi-equivalent core-hour 1 Tbyte
tape = 6 K Jpsi-equivalent core-hour Projects using disk storage will be charged
25% of these costs every three months. Projects
will be charged for tape usage when a file is written at the full cost of
tape storage; when tape files are deleted, they will receive a 40% refund
of the charge. Proposals should discuss whether these
files will be used by one, a few, or several project(s). The cost
for files (e.g., gauge configurations) that are used by several projects
will borne by USQCD and not a specific physics project. The charge
for files used by a single project will be deducted from the computing
allocation: projects are thus encouraged to figure out whether it is
more cost-effective to store or re-compute a file. If a few (2-3)
projects share a file, they will share the charge. Projects that expect to have large I/O requirements,
such as those that use eigenvalue and deflation methods, are requested to note
that in their proposal and to work with the site managers to handle these needs
as painlessly as possible. 5) If relevant, what data will be made
available to the entire Collaboration, and the
schedule for sharing it. 6) What calculations the investigators would
like to perform in an "exclusive
manner" (see above in the section on policy directives), and for how
long they would like to reserve to themselves this exclusive right. iii) Procedure for the awards. The Scientific Program Committee will receive
proposals until the deadline of Friday, March
11, 2016. Proposals not stating the total request in the last
sentence of the abstract will be returned without review (but will be
reviewed if the corrected proposal is returned quickly and without other
changes). Proposals that are considered meritorious and
conforming to the goals of the Collaboration will
be posted on the web at /, in the Collaboration's password-protected area. Proposals recommended for awards in previous
years can be found there too. The Scientific Program Committee (SPC) will
make a preliminary assessment of the
proposals. On April 18, 2016, the SPC will send a report to the
proponents raising any concerns about the proposal. A few proposals will be presented and
discussed at the All Hands' Meeting, April
29-30, 2016, at
BNL. Following the All Hands' Meeting the SPC will
determine a set of recommendations on the
awards. The quality of the initial proposal, the proponents' response to
concerns raised in the written report, and the views of the Collaboration
expressed at the All Hands' Meeting will all influence the outcome.
The SPC will send its recommendations to the Executive Committee
after the All Hands' Meeting, and inform the proponents once the
recommendations have been accepted by the Executive Committee. The
successful proposals and the size of their awards will be posted on the
web. The new USQCD allocations will commence July
1, 2016. Scientific publications describing
calculations carried out with these awards should acknowledge the use of USQCD resources, by including
the following sentence in the Acknowledgments: "Computations for this work were carried
out in part on facilities of the USQCD Collaboration, which are funded by the Office of Science
of the U.S. Department of Energy." Projects whose sole source of computing is
USQCD should omit the phrase "in
part". We expect to receive in CY2016
zero-priority time on the BG/Q at Argonne. Based on previous
usage and availability, we will distribute zero-priority time
starting in 2016 as soon as our INCITE allocation has been
consumed and zero-priority time becomes available, according to
the allocations made during last yearŐs, 2015 allocation process.
This is expected to begin in April 2016. These 2015 allocations for
2016 zero-priority time will complete June 30, 2016. As part of the
current, 2016 allocations process, driven by this Call for Proposals, we
will allocate the remainder of CY2016 ANL zero-priority time as well as
that which becomes available to USQCD in the first half of 2017. As
the total amount of time cannot be reliably estimated, we will allocate
percentages of zero-priority usage. The SPC may readjust these
percentage allocations based upon observed usage. The Oak
Ridge facility does not provide a zero-priority queue. The usage of the INCITE allocations should be
monitored by all PIs of INCITE projects on the USQCD WEB-page: http://www.mcs.anl.gov/~osborn/usqcd-spc/2013-14-mira.html v) USQCD computing resources. The Scientific Program Committee will allocate
7200 hours/year to Type A and Type B proposals.
Of the 8766 hours in an average year the facilities are supposed to
provide 8000 hours of uptime. We then reserve 400 hours (i.e., 5%)
for each host laboratory's own use, and another 400 hours for Type C
proposals and contingencies. ================================== At BNL: 60% of a 1024 node BG/Q rack 16 cores/node, up to 4 threads per core 16 GB memory/node 10% of a BNL rack with time donated to
USQCD. 50% of a rack owned by USQCD. total: 7200*1024*16*0.60 =
70.8 M BG/Q core-hours = 116 M Jpsi-equivalent
core-hours There is no tape storage at BNL for USQCD
activities. We have 100+ TBytes of disk
space, which should be ample for users to stage their calculation to/from BNL, but long term storage on tape will continue to be
done at FNAL and JLAB. ================================== At FNAL: 224 node cluster ("Bc") Eight-core, quad-socket 2.8 GHz AMD
Opteron (Abu Dhabi) nodes 32 cores per node 64 GB memory/node 1 core-hour = 1.48 JPsi-equivalent
core-hours total:
7200*224*32*1.48 = 76.4 M Jpsi-equivalent core-hours 314 node cluster ("Pi0") Eight-core, dual-socket 2.6 GHz Intel
Xeon (Ivy Bridge) nodes 16 cores per node 128 GB memory/node 1 core-hour = 3.14 JPsi-equivalent
core-hours total:
7200*314*16*3.14 = 113.6 M Jpsi-equivalent core-hours 32 node cluster ("Pi0g") Eight-core, dual-socket 2.6 GHz Intel
Xeon (Ivy Bridge) nodes 128 GB memory/node 4 GPUs NVIDIA K40m (Kepler
Tesla) per node, GPU rating 2.6 (128 total GPUs available) GPU memory (ECC on) 11.5 GB/GPU Each K40 gpu-hr
is equivalent to 2.6 Fermi-gpu-hr total =
7200*128*2.6 = 2396 K GPU-hours) These clusters will share about 1000 TBytes of disk space in Lustre file
systems. Tape access is also available. For further information see /fnal/ ================================== At JLAB: 276 node cluster ("12s") Eight-core, dual processor Intel Sandy
Bridge nodes 16 cores per node 32 GB memory/node QDR network card, with full
bi-sectional bandwidth network fabric 1 12s core-hour = 2.3 Jpsi cores total: 7200*276*16*2.3 = 73.1 M Jpsi-equivalent
core hours 42 node GPU cluster at JLab
("12k") Dual 8 core Intel Sandy
Bridge host nodes 4 NVIDIA K20m (2012 Kepler
Tesla) GPUs/node 128 GB memory/node FDR network fabric with
full bi-sectional bandwidth 1 K20m = 2 C2050 168 total GPUs available:
in C2050 units -> 336 GPUs total total: 7200*42*4*2 = 2.4
M GPU hours (equivalent to ~ 200 M Jpsi core hours from the listed conversion factors) There is a projected procurement of 49 TFlops in FY16. This might come in the form of 220M as GPU
(equivalent to ~4.4 M GPU hours) plus 80M Jpsi
conventional, or it might come as a pure KNL resource of that same capacity
(300 M Jpsi core hours for this allocation year). At
present we are allocating 250 M Jpsi equivalent core
hours for this projected resource For further information see also http://lqcd.jlab.org . Machine descriptions can be found
at https://wiki.jlab.org/cc/external/wiki/index.php/New_Users_Start_Here At JLAB, the systems will have access to about
1 PBytes of disk space. Tape access is also available. ============================================================== Resource estimates Based upon the performance conversions used
above, the total resources available in this call
are shown below. BG/Q (BNL):
116 M Jpsi
-> 16 TF-yr Clusters (FNAL): 190 M Jpsi
-> 26 TF-yr Clusters (JLab):
73 M Jpsi -> 10 TF-yr New resources(JLAB); 250 M Jpsi
-> 34 TF-yr (for 2nd half of allocation
period) Tesla GPUs (FNAL): 283 M Jpsi
-> 39 TF-yr Total (USQCD):
962 M Jpsi -> 132 TF-yr |
| Andreas Kronfeld | Legal Notices |