General Questions

  1. What is INCITE?
  2. What is new in INCITE 2021?
  3. Who manages INCITE?
  4. Who should I contact if I have questions about the INCITE program, the proposal template, or my application?
  5. When is the next call for proposals?
  6. Can you extend the deadline for proposal submissions?
  7. Will the 2021 proposal deadline be extended due to the COVID-19 pandemic?
  8. Eligibility and Review Process

  9. Who is eligible to participate in INCITE?
  10. Must I have research funding from the Department of Energy?
  11. Are researchers and teams from other federal agencies eligible to apply?
  12. Are foreign researchers and teams eligible to apply?
  13. Does the proposal have to come from a team?
  14. Are teams based around a community-maintained code or end stations eligible to apply?
  15. Does INCITE accept proprietary proposals?
  16. Does the research that I propose have to be in line with the Department of Energy mission, or have DOE research collaborators?
  17. I previously received an INCITE award. Am I eligible to apply again?
  18. How are INCITE projects chosen?
  19. What questions are asked of the reviewers about my proposal?
  20. When will I hear if I have received an award?
  21. Resources Available

  22. What computing resources does INCITE use?
  23. Can I submit a proposal to use resources at both facilities?
  24. Do I have to pay for access?
  25. If I get an INCITE award, how do I get funding for my research labor and other items needed to carry out the INCITE work?
  26. What staff support would an INCITE project receive from the LCFs?
  27. What resources will be allocated in the INCITE program in 2021?
  28. For a three-year proposal, how do I estimate my resource request on Frontier (OLCF) in 2023?
  29. For a three-year proposal, how do I estimate my resource request on Aurora (ALCF) in 2023?
  30. Leadership Computing Definitions

  31. What constitutes a "computationally intensive" research project?
  32. Can I meet the computationally intensive criterion by loosely coupling my jobs?
  33. What constitutes a "Data and/or AI" research project?
  34. What does readiness mean for data-intensive/AI projects?
  35. What software and technical infrastructure exist for data projects?
  36. Do scaling data have to be from the proposed machine?
  37. Award Information

  38. Are INCITE awards for a period of only 1 year?
  39. What kind of protections do the INCITE centers offer for data?
  40. How long can my data be archived after the project ends?
  41. What are my responsibilities if I receive an award?
  1. What is INCITE?

    Each year the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program awards to researchers billions of supercomputer processor hours and 100 trillion bytes of data storage space at the Department of Energy's (DOE's) leadership computing facilities (LCFs) for unclassified supercomputing, which include some of the most powerful computers in the world.

    The program seeks computationally intensive, large-scale research projects that can make high-impact scientific and engineering advances through the use of a substantial allocation of computer time and data storage. The INCITE program specifically encourages proposals from research organizations including universities, national laboratories, and industry.

  2. What is new in INCITE 2021?

    Increasingly, techniques in data science/artificial intelligence (AI) are becoming applicable and critical to large-scale science campaigns. The limited history but dramatic growth and potential impact of data science and analytics/learning oriented applications running at capability scale on LCF resources will be taken into account when evaluating the application readiness of the proposals.

    Renewals that introduce a data science or other methodological component that is new in the context of their initial proposal must be called out explicitly and will be considered on a case-by-case basis.

    From a logistical perspective, three changes have been made to the submission process.

    1. Contributors to the proposal, such as co-investigators (Co-Is), may now edit the proposal information in the submission system, but the lead principal investigator (PI) must still submit the proposal prior to the proposal deadline.
    2. All proposal information will be submitted as a single PDF file rather than individual sections.
    3. The INCITE submission system has changed in recent years and PIs should allow extra time to create a new account if needed.
  3. Who manages INCITE?

    The INCITE program is jointly managed by the LCF centers at Argonne National Laboratory (ANL) and Oak Ridge National Laboratory (ORNL). The peer review of proposals is carried out by external groups of experts from national laboratories, universities, and industry; National Academy members; and senior computational science researchers who have a working knowledge of the current computational challenges and opportunities in their fields.

  4. Who should I contact if I have questions about the INCITE program, the proposal template, or my application?

    Questions about the INCITE program and applications should be directed to the INCITE manager at INCITE@DOEleadershipcomputing.org. The INCITE manager may forward your query to a representative of the Argonne Leadership Computing Facility (ALCF) or Oak Ridge Leadership Computing Facility (OLCF) as appropriate.

  5. Can you extend the deadline for proposal submissions?

    The INCITE Call for Proposals is from mid-April through the end of June (see the call for proposals for exact dates). The INCITE manager will consider extending the deadline if PIs experience documented difficulties submitting a proposal. However, researchers should begin the submission process as early as possible under the assumption that the deadline will not be extended. Proposals may be initiated and saved, allowing the PI to begin uploading content prior to the deadline.

  6. Will the 2021 proposal deadline be extended due to the COVID-19 pandemic?

    The INCITE call for proposals will close on Friday, June 19, 2020 at 8:00pm Eastern Daylight time (EDT). At this time, we are closely monitoring the COVID-19 pandemic, but do not anticipate changing the submission deadline. In the unlikely event that the proposal deadline is extended, we will post an announcement on the INCITE website (www.doeleadershipcomputing.org).

  7. Who is eligible to participate in INCITE?

    The INCITE program is open to US- and non-US-based researchers and research organizations needing large allocations of computer time, supporting resources, and data storage to pursue transformational advances in science and engineering. INCITE considers requests regardless of a PI’s funding source (ex. DOE, NSF, state, private, etc). The intent of INCITE is to support large-scale, computationally intensive projects that would not be possible or productive without petascale computing. Applicants must present evidence that their proposed production simulations can make effective use of a significant fraction, in most cases 20 percent or more, of the high-performance computing (HPC) systems offered for allocation. Applicants planning to execute ensemble jobs are advised to review the Frequently Asked Question "Can I meet the computationally intensive criterion by loosely coupling my jobs?"

  8. Must I have research funding from the Department of Energy?

    No, DOE sponsorship is not required. However, researchers are expected to have all funding and any other elements necessary to ensure the success of their research project in place at the time of their application.

  9. Are researchers and teams from other federal agencies eligible to apply?

    Yes, research applications from other agencies are accepted.

  10. Are foreign researchers and teams eligible to apply?

    Yes, their proposals will be evaluated on the same scientific and technical criteria as those of domestic researchers. U.S. collaborations are encouraged but not required.

  11. Does the proposal have to come from a team?

    No, the solicitation is open to individual researchers, a team of researchers from the same institution, or multi-institutional teams.

  12. Are teams based around a community-maintained code or end stations eligible to apply?

    Yes. If the research objectives of a proposal are best satisfied by the operation of a community-maintained code or suite of codes on INCITE resources, such use is entirely acceptable. If the requested system time is to be part of a long-term community activity, this should be noted in the proposal.

  13. Does INCITE accept proprietary proposals?

    Yes. The peer-review selection process and project reporting requirements described within the INCITE Overview and Policies will be implemented for both proprietary and non-proprietary user proposals. Individuals considering submittal of a proposal for proprietary research must contact the INCITE manager, INCITE@DOEleadershipcomputing.org, before the call for proposals closes to discuss the policy on proprietary work.

  14. Does the research that I propose have to be in line with the Department of Energy mission, or have DOE research collaborators?

    No, alignment with the DOE mission is not required. Collaborators funded by DOE or located at DOE national laboratories are not required.

  15. I previously received an INCITE award. Am I eligible to apply again?

    Yes. A previous recipient of an INCITE award, either a 1-year or multiyear award, may submit a new proposal.

  16. How are INCITE projects chosen?

    INCITE proposals are subjected to two reviews: (1) computational readiness and (2) peer review by an international panel of experts. Awards are based on the quality and impact of the research and the suitability of the proposed simulations for the requested resource.

    In the computational-readiness review, each LCF evaluates all qualified proposals for the readiness and scalability of the code and its algorithms. Experts are drawn from the facility staff and other institutional personnel who are well versed in the unique requirements of the leadership-class systems as well as experts from the computational science community, as needed. Reviewers are given the opportunity to submit a list of questions to the project's PI to clarify vague or incomplete proposal information.

    In the peer review, the INCITE program solicits independent computational science, engineering, and computer science peer reviewers to evaluate each proposal's potential for scientific or technical impact. Proposals are evaluated on scientific/technical merit, expected impact, qualification of the PI, qualification and composition of the proposed research team, and the computational plan. Review panels are composed of application domain experts from national laboratories, universities, and industry; National Academy members; and senior computational science researchers who have a working knowledge of the current computational challenges and opportunities in their fields.

    The INCITE Awards Committee, composed of the LCF management teams, makes selections based on the rankings by the peer-review panel. The readiness ratings are used to determine the capability of each project to effectively use the selected system and are based on proficiency shown through benchmarking data and/or proposed development plans. Proposals are selected that answer a high-impact, key scientific/technical question and/or point to a new area of research. Successful proposal teams demonstrate a very clear understanding of petascale computing and can optimally use these resources to accomplish the stated scientific/technical goals.

  17. What questions are asked of the reviewers about my proposal?

    Details about the reviewer questions and ratings can be found on the proposal template web page.

  18. When will I hear if I have received an award?

    Announcements will be made in early November. All PIs of new and renewed proposals will be contacted by the INCITE manager at that time.

  19. What computing resources does INCITE use?

    DOE is the nation's largest supporter of basic research in the physical sciences and provides a portfolio of LCFs housing some of the world's most advanced supercomputers. It makes several of its supercomputers available for use in tackling large-scale, high-potential research projects in a broad array of science and engineering domains that are judged to be the most scientifically or technically promising in the INCITE peer-review process. For a description of the current systems and any anticipated new systems or system modifications that may be relevant to multiyear proposals, see the INCITE HPC Resources page.

  20. Can I submit a proposal to use resources at both facilities?

    The applicant can request one or more resources. If allocations on more than one resource are requested, justification for the use of each resource must be clearly stated in the Description of Research and Computational Readiness sections of the application. Relevance to the accomplishment of the stated research goals should be the principal determining factor in any resource request.

  21. Do I have to pay for access?

    Nonproprietary research projects selected through the INCITE program are not charged for access to the LCFs. It is expected that users will publish their results in open, peer-reviewed literature. For details, see the user agreements at https://www.aps.anl.gov/Users-Information/Legal-Financial/Argonne-User-Facility-Agreements and https://www.ornl.gov/content/user-agreements for ANL and ORNL, respectively. Proprietary projects are charged full cost recovery for use of LCF resources.

  22. If I get an INCITE award, how do I get funding for my research labor and other items needed to carry out the INCITE work?

    The INCITE award is for allocation of computer time and data-storage resources only. Applicants must have all other resources required for the successful completion of the INCITE work committed to them at the time of their INCITE application. Applicants who cannot demonstrate compliance with this requirement will be deemed ineligible for an INCITE award.

  23. What staff support would an INCITE project receive from the LCFs?

    INCITE key projects have an assigned expert, the ‘catalyst’, to maximize and accelerate research. Catalysts are computational scientists that are experts in their fields: computational chemistry, physics, fluid dynamics, astrophysics, data analytics, workflows, machine/deep learning etc., and actively participate in the research projects. Projects also benefit from the support provided by the scientific and operational staff of the facility. This includes support from performance engineers, as well as visualization and operations team members, who are all available to help the project as needed.

  24. What resources will be allocated in the INCITE program in 2021?

    Up to 60% of the allocable time in calendar year (CY) 2021 on Summit, the 200-petaflop IBM AC922 machine; and Theta, the 12-petaflop Cray XC40 machine will be awarded by the INCITE program. Aurora and Frontier will be available in CY 2023.

  25. For a three-year proposal, how do I estimate my resource request on Frontier (OLCF) in 2023?

    For the 2021 INCITE proposal submission, Frontier allocations in 2023 will be requested in equivalent “Summit node-hours.” For planning purposes for this cycle, we conservatively expect nearly 133 million Summit-equivalent node-hours to be allocated on Frontier per year and that the average INCITE project will be awarded approximately 3-4 million Summit-equivalent node-hours in 2023. Awarded three-year projects will be required to revisit their Summit-equivalent node-hour request for Frontier and the computational readiness for Frontier will be re-evaluated in future renewals.

  26. For a three-year proposal, how do I estimate my resource request on Aurora (ALCF) in 2023?

    For the 2021 INCITE proposal submission, Aurora allocations in 2023 will be requested in equivalent “Theta node-hours.” For planning purposes for this cycle, we conservatively expect nearly 1.78 billion Theta-equivalent node-hours to be allocated on Aurora per year and that the average INCITE project will be awarded approximately 40-50 million Theta-equivalent node-hours in 2023. Awarded three-year projects will be required to revisit their Theta-equivalent node-hour request for Aurora and the computational readiness for Aurora will be re-evaluated in future renewals.

  27. What constitutes a "computationally intensive" research project?

    A computationally intensive research project refers broadly to computing and/or data projects which would require a significant fraction of the processing resources and systems at the facility—in most cases 20 percent or more—for the proposed production science runs. Data-centric and machine-learning/analytics/AI based applications should utilize a significant fraction of the resources but may have a different usage profile across processor, memory, network, or disk usage. Please refer to the Frequently Asked Question “What constitutes a "Data and/or AI" research project?” for additional guidance. The limited history but dramatic growth and potential impact of data and AI oriented applications running at capability scale on LCF resources will be taken into account when evaluating the application readiness of the proposals.

  28. Can I meet the computationally intensive criterion by loosely coupling my jobs?

    Projects planning to execute ensemble jobs to meet the INCITE criterion of utilizing a large fraction (e.g., 20 percent) of the leadership-class system for production simulations will be considered. The mission of the INCITE program is to enable high-impact, grand-challenge research that could not otherwise be performed without access to the leadership-class systems, Theta and Summit. Parameter sweeps, ensembles, design of experiments, and other statistical methods that require large numbers of discrete or loosely coupled simulations can be considered capability-class campaigns if the volume of work is so large that time-to-solution is an untenable pacing issue and if a software workflow solution (e.g., pre- and post-processing scripts that automate run management and analysis) is provided to facilitate this volume of work. Conversely, if by decoupling the simulations the work could be effectively carried out on a smaller resource within a reasonable time-to-solution, the project in all likelihood does not require the INCITE systems and will be assessed in this light. (A smaller resource is defined as a non-leadership-class system available through other allocation programs, e.g. institutional systems, nationally funded computing resources, etc.). Bundling together small jobs solely for the purpose of meeting the criterion for usage of a large fraction of LCF HPC will not be judged as capability computing. INCITE awards are made based on the potential for impact of the research and the need to carry out capability-class simulations.

  29. What constitutes a “Data and/or AI” research project?

    Proposals in these areas should have strong aspects of scalable data processing (data-intensive computing, experimental/observational/simulation data analytics, etc.) and/or machine learning/AI (deep learning, neural networks, discovery of patterns and reduced models for scientific data and/or simulation modeling, etc.). As guidance on relevant research areas for proposals to this call, consider these suggestions: complex and interactive workflows, streaming/real-time data analysis, statistical methods, graph analytics, uncertainty quantification, deep learning, machine learning, hyper-parameter optimization, pattern recognition and classification, and machine learning integrated with applications and/or steering simulations. This guidance includes methodological advances in data science that require use of the LCF capabilities, and also applications that deploy these methods to advance their outcomes. Cross-cutting projects that integrate simulation and large-scale data science are welcome.

  30. What does readiness mean for data-intensive/AI projects?

    See above for “What constitutes a “computationally intensive” research project?” Data analytics and/or AI proposals should demonstrate computational readiness by providing performance data (representative compute, memory, network, or disk usage) which support the required scale and/or the time demands of the problem and demonstrate the need for LCF resources. Performance data should demonstrate that the application is optimized for the resources requested (in terms of efficiency, scalability, throughput, data input/output, workflow tools for ensemble runs, checkpointing etc.). We encourage PIs planning to submit a data analytics and/or AI project to contact INCITE management and the LCFs if there are additional questions, in particular, regarding application software requirements. Because we anticipate that these projects may have unique requirements or challenges compared to traditional model-driven simulation campaigns, these projects are encouraged to collaborate closely with the LCF.

  31. What software and technical infrastructure exist for data projects?

    OLCF and ALCF have infrastructure and software that is directly aimed at data science projects including workflow infrastructure (e.g., Slate at OLCF, Petrel at ALCF, etc.), analytics notebooks, data transfer nodes, etc. See OLCF and ALCF websites for specifics.

  32. Do scaling data have to be from the proposed machine?

    No. Because the applicant may not have previously had access to the machine requested, he or she can provide scaling data from any machine available to him or her. However, scaling data from the proposed machine is highly recommended for demonstrating computational readiness. Upon request, the centers can provide short-term allocations for the purpose of preparing for an INCITE proposal.

  33. Are INCITE awards for a period of only 1 year?

    INCITE awards can be from 1 to 3 years in length. Multiyear awards must submit annual renewal requests, which will undergo peer review. Requests for multiyear awards will be subject to the highest standards of excellence and are expected to yield truly extraordinary results.

  34. What kind of protections do the INCITE centers offer for data?

    The INCITE centers have implemented cyber-security programs at a moderate baseline with compensatory controls in accordance with National Institute of Standards and Technology Special Publication 800-53. Project-specific security requirements can also be discussed and potentially addressed; the proposer should contact the center directly for details. Risk is currently mitigated using available enterprise infrastructure and network-based tools to provide perimeter protection and vulnerability resolution. System- and application-level security controls are implemented in accordance with industry and federal best practices. All non-temporary user data reside on centralized file and archival storage systems that are regularly backed up. These data are protected and segmented from other users through standard access controls.

    The HPC centers cannot guarantee zero risks to information stored on their systems. Requests for processing sensitive (e.g., proprietary or export-controlled) data must therefore be clearly identified in the proposal. Information-sensitivity concerns apply to the intent of the research, storing and compiling of source code, loading and execution of application software, input data for the application software, output data generated by the application software, and data resulting from analysis of output data. If, for example, the application software has an Export Control Classification Number (ECCN), prospective users should state this fact in the proposal and provide the ECCN number. Prospective users should work with the center to identify appropriate levels of data protection. Greater levels of protection not typically offered may be provided at a cost borne by the project.

  35. How long can my data be archived after the project ends?

    The facilities reserve the right to limit the amount and duration of archival storage once a project has been completed. See the LCF data management policies at https://www.alcf.anl.gov/support-center/facility-policies/data-policy for ANL and https://docs.olcf.ornl.gov/data/index.html for ORNL.

  36. What are my responsibilities if I receive an award?

    The PI of each awarded project will be expected to sign terms-of-use agreements with the center and provide periodic reports (e.g. quarterly, end of year). Project participants will also sign terms-of-use agreements and be asked to respond to an annual user survey. All INCITE users are expected to acknowledge the center and the program in publications resulting from their award. The appropriate acknowledgement is listed below for INCITE and the respective center.

    INCITE/ALCF Acknowledgement:
    An award of computer time was provided by the INCITE program. This research used resources of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357.

    INCITE/OLCF Acknowledgement:
    An award of computer time was provided by the INCITE program. This research also used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.

    INCITE/ALCF/OLCF Acknowledgement:
    An award of computer time was provided by the INCITE program. This research used resources of both the Argonne and Oak Ridge Leadership Computing Facilities, which are DOE Office of Science User Facilities supported under contracts DE-AC02-06CH11357 and DE-AC05-00OR22725

    Non-compliance with the Terms of Use Agreement may result in the suspension of computer access.