2023 INCITE Call for Proposals
Renewal Preparation Instructions for Authors
Projects requesting multiyear allocations are reviewed annually over the lifetime of the project to re-evaluate and prioritize allocation requests for the upcoming allocation year. The renewal application requires detailed and accurate answers about current year accomplishments and allocation usage as well as the plans for the next year. These responses will be used to determine which projects will be granted continued access and the extent of the new allocation for the upcoming allocation year.
You may use the renewal application form only if the INCITE program manager has instructed the principal investigator (PI) that the project is eligible to submit a renewal. A multiyear award must submit a renewal proposal to receive allocations for year 2 and year 3 (if appropriate). Once the term of the current award is complete, the principal investigator (PI) must submit a new proposal in order to continue the research. The INCITE program does not “extend” projects beyond an approved term of award.
Up to sixty percent of the allocatable time on the Frontier exascale system and the IBM/AC922 Summit at the Oak Ridge Leadership Computing Facility (OLCF), Polaris and the Intel/Cray Theta machine, along with the exascale system, Aurora, at the ALCF will be allocated for calendar year (CY) 2023 through the INCITE program. For each resource, allocations are anticipated to be between 500K and 1M Summit node-hours and 1.5M and 2.5M Theta node-hours. Individual awards may be higher.
INCITE seeks research enterprises for capability computing: production simulations – including ensembles – and data analytics and artificial intelligence (AI) applications that necessarily require our leadership class resources. Please note that requests smaller than 20 percent of the average award noted above may be considered too small for the INCITE program, unless the proposal is (i) for algorithm development or other computer science activities requiring a large fraction of the machine resource (one or more of computing, memory, network, etc.) but not a large amount of time or (ii) clearly articulates the need for other aspects of the architecture and/or infrastructure associated with leadership computing resources. Please contact the INCITE program manager (INCITE@DOEleadershipcomputing.org) if you have questions regarding the allocation request for your proposed project allocation size.
INCITE provides several resources to help you prepare your renewal.
- The INCITE Overview and Policies includes a description of the basis for award decisions, among other things.
- The questions used by reviewers to assess renewals are available here, as are templates for sections of the renewal.
- The slides from previous Informational Webinars about the INCITE program are available.
Experts will assess the project’s execution to date, determining whether the milestones originally set have been accomplished. Successful renewals clearly tie the usage to-date with the milestones and science/technological impact originally proposed. Significant changes to the original project scope should be discussed with the INCITE Program Manager prior to submittal.
Revisions for 2023 INCITE Renewal Proposals
- Introduced in 2021, the Early Career Track for new proposals will continue. For renewals, this is not expected to be a particular advantage, but renewal PIs may self-identify.
- All submission materials must be concatenated together into a single PDF file for submission. Individual files will not be permitted. For renewal proposals, the required elements are a Project Status Summary, the Project Achievements and Plans, Publications results from this and prior INCITE awards, and an updated Milestone Table.
- The INCITE program has significantly changed its proposal submission site from prior years. Please allow extra time to create an account on this site and submit your proposal.
- The INCITE submission site now allows proposal contributors (e.g. Co-Is) to edit the proposal information in the submission system. However, the lead PI must still submit the proposal prior to the proposal submission deadline.
Before submitting their renewals, it is strongly recommended that authors comply with the guidelines established below, because they will be used to assist in the review of renewals. Templates for all sections are available.
The renewal must be clear and readily legible, and it must conform to the following requirements:
- Header/Footer/Page Number: Each section of the application must be paginated. Footers should be used for paginating all files. Also, headers should be used to indicate the title of the renewal and the lead PI.
- Title: Renewal titles may not be longer than 80 characters.
- Font: One of the two typefaces must be used: Arial or Times New Roman (font size 11). A font size of less than 11 points may be used for mathematical formulas or equations; figure, table, or diagram captions; or when using a symbol font to insert Greek letters or special characters. PIs are cautioned, however, that the text must still be readable.
- Margins: Margins must be at least 1 inch in all directions. These requirements apply to all sections of the proposal, including supplementary documentation.
- Spacing: Proposals should be prepared using single line spacing. The proposal elements should not exceed the specified page count limits.
- References: References should be gathered at the end of the narrative. References are not included in the total page count.
- File Size: The total file size should be limited to 15 MB.
- Support Information: No letters of collaboration or letters of support will be accepted with the application. Current and pending support documentation is not required for this solicitation.
Adherence to type size and spacing requirements is necessary to ensure readability and that no proposer will have an unfair advantage by using smaller type or spacing to accommodate more text.
All renewal proposals must be submitted electronically via the INCITE renewal submission website. The INCITE program manager will also contact the PI directly with the site information.
Electronic applications will be accepted starting mid-April 2022 thru 5:00 pm EDT on Friday, July 22, 2022. All proposals must follow these instructions. INCITE reserves the right to decline consideration of proposals not compliant with these instructions and the above guidelines. Hardcopies will not be accepted for review.
Any questions should be directed to the INCITE program manager at INCITE@DOEleadershipcomputing.org.
- Project Status Summary (1-page limit): Briefly summarize the goals of the project. It is unnecessary to repeat the executive summary from the original proposal. The project status summary should include an overview of the achievements to date. Industry organizations should also summarize the economic or strategic business impact of the accomplishments to date.
- Project Achievements and Plans (14-page limit): Visual materials—such as charts, graphs, pictures, etc.—are included in the 14-page limit. References do not count toward the 14-page limit and should be included at the end of the Project Achievements and Plans. URLs that provide information related to the application should not be included. The 14-page limit will be strictly enforced.
- Project Achievements: The Project Achievements should address the following points. This section is typically about 10 pages.
- Significance of Accomplishments to Date: Explain what advances you accomplished through the INCITE award (e.g., impact on community paradigms, valuable insights into or solving a long-standing challenge). Place the proposed research in the context of competing work in your discipline or business. Reiterate the milestones of your proposal and discuss the accomplishments (planned or unplanned) achieved this year relative to those milestones and allocation use. Summarize the impact of the results achieved: What conclusions can be drawn and what is solved because of these results? What new and follow-on investigations have these results motivated?
- Allocation Use: Summarize your project’s allocation use to-date this year: percent of allocated node-hours used on each platform, job size distribution, number of users, etc. Associate the resource use with particular results where possible. Also summarize your project’s projected use from now until the end of December (i.e., end of current allocation year): anticipated percent of allocated node-hours used on each platform, job size distribution, etc. Associate this resource usage with anticipated results. Do you expect your usage to be evenly distributed throughout the remainder of this year? If not, explain.
- Application Parallel Performance: Summarize the performance (percent of peak, scalability) of your project’s production application used in the allocations this year. What progress was made in improving the application’s performance on this architecture? What challenges (if any) remain? List the technical risks and challenges that were confronted by your project (overcome or not) this year. Were they anticipated?
- Data Storage: What is the current cumulative size of stored data? What is your projection for the cumulative size of stored data at the end of the project? What tools and/or plans do you have to reduce the data? To share the data?
- Project Achievements: The Project Achievements should address the following points. This section is typically about 10 pages.
The LCF data management policies can be found at:
- Project Plans for Next Year: The Project Plans should address the following points. This section is typically about 4 pages.
- Summarize the Project Plans: Briefly explain what advances you expect to accomplish through the next award period and associate these with the overarching goals of your project. Clearly explain the relationships between the milestones, planned production simulations, and expected compute time required for these sets of simulations. Explain any change in the scope of the project (research objectives, computational approach, personnel, etc.) relative to the plans and approach articulated in the original proposal. If resource requirements differ from those of the previous year, provide details on the differences (platform, increased/decreased node-hours, file system and archival storage, networking) and the reasons for them. If you are requesting a new resource, you must provide evidence that your project is optimized to run on that resource. See the New Code Applications section below. Summarize the requirements that are driving the differences and what science/technology outcomes are expected. Significant changes to the original project scope should be discussed with the INCITE Program Manager prior to submittal.
- Development Work: Describe what, if any, development work has been carried out and the outcome of this work. Describe what additional development work will be executed, and when. Provide an estimate for the percentage of project time you will spend on development computing (i.e., porting, performance analysis) and other nonproduction runs.
- New Code Applications (where relevant): Are you planning to use any new production codes next year that were not included in your original proposal? Or are you proposing use of a new resource not included in your original proposal? If so, provide direct evidence, including supporting quantitative data, for your production application’s parallel performance for the intended research simulations and analysis. Ideally, the proposing team will have generated the data. If you cite work by others, explain why it is applicable here. You should use the application code you intend for the production work, not a related code. Data for sample systems not related to the intended research is undesirable. Scaling and performance data in either strong or weak scaling mode should be provided. Explain how the strong or weak scaling applies to the proposed work. For data analytics and AI proposals, a description of the compute, memory, networking, and storage needs in the context of the LCF resources will help reviewers understand the strong or weak scaling characteristics. See the examples at the end of this page.
NOTE: You may apply for a startup account at one of the centers to conduct performance studies. Director’s Discretion applications are available at
- Publications Resulting from INCITE Awards: Provide a list of publications resulting from your current and previous INCITE awards to this project team for work related to the proposal under consideration. Only those publications that include an acknowledgement to INCITE and/or the LCF may be included. This list may not be used in lieu of references in Section 1, “Project Achievements and Plans.” If applicable, list the citations in both Sections 1 and 2. (Does not count toward the 14-page Project Achievements and Plans limit).
- Milestone Table: Renewal must be accompanied by a summary table of the originally planned milestones and the status of the work. Use the template provided. Below is a description of what should be provided for each column in the table. (Does not count toward the 14-page Project Achievements and Plans limit.)
Milestones: Update the scientific and technical (e.g., development) milestones for each year of the proposed work.
Details: Update the following details as appropriate for each milestone.
- Resource: List the primary computing resource.
- Node hours: List the number of node-hours associated with this milestone.
- File system storage: Provide an estimate in terabytes of the required temporary storage needed on the file system while the data is analyzed and reduced or before it is moved to archival storage. Also provide the estimated time period when this storage is needed.
- Archival storage: Provide an estimate in terabytes of the required archival storage associated with this milestone for the duration of the project. Also provide the estimated time period when this storage is needed.
- Application: List the software application or code needed for this milestone.
- Tasks: Briefly summarize the work that has been or will be done by listing the subtasks, computational runs, and data analysis and reduction tasks associated with each milestone.
- Dependencies: Note dependencies between the scientific milestones and on proposed development work.
Date: List the projected start and completion date for each milestone.
Status: Provide a brief description of the status of each milestone (e.g. In progress, Completed, Not Started). If a milestone has been significantly delayed from what was originally proposed, provide a brief explanation. If a milestone for the coming year(s) was not included in the original proposal, please indicate with “New.”
Examples of Performance Required Materials
Quantitative data for production application performance should be provided in either tabular or graphical form or both. This data should reflect the performance of the application for the production simulations proposed, and should include all I/O requirements of the production simulations.
Where appropriate, characterize the production application’s single-node performance (e.g., percent of peak). For example, describe the most computationally expensive portion of your algorithm and describe the on-node parallelization scheme employed. Any optimization strategy to improve single-node performance, including code restructuring, GPU strategy, and exploiting OpenMP/Pthreads and vectorized instructions, should be described. For Summit provide any evidence of speedup using GPUs compared to using all CPU cores. For Theta, demonstrate the efficient use of many CPU cores per node, and if not all cores can be used effectively, explain why. For all architectures indicate the degree of utilization of any architecture-specific features. Data analytics and AI proposals may provide performance data which demonstrates scalability, efficiency, node or GPU utilization, network bandwidth estimation, or power profiling for node, CPU, GPU and/or memory.
Weak-scaling behaviors are probed by holding per-node computational work constant (e.g., the size of the mesh on a processor is held constant) as the total problem size grows with processor count. Strong-scaling behaviors are probed by holding the total problem size constant as the processor count grows, thereby decreasing the per-processor computational work. Note that in the examples provided, a logarithmic scale is preferred.
For data analytics and AI applications that may weak scale for improved convergence, present a justification for the scientific impact of the expected improvements when running at large scale. Address, if necessary, overcoming difficulties of training with respect to the relevant hyperparameter list for deep learning applications.
In addition, where appropriate the proposal should describe the entire computational workflow of the proposed project. In particular, the project should demonstrate that the entire workflow is tenable on the proposed architecture or supporting analysis resources at the LCFs or discuss where any pre- and post-processing or data analysis will be conducted.