Process for Evaluating Ideas Submitted to the NEI Audacious Goals Initiative
The NEI solicited ideas for this initiative by two means: 1) the NEI Challenge to Identify Audacious Goals in Vision Research and Blindness Rehabilitation, in accordance with a new prize authority granted by America COMPETES Reauthorization Act of 2010. 2) Ideas solicited by email from people ineligible for the prize competition, such as employees of the National Institutes of Health and non-US citizens or permanent residents.
NEI received a total 548 entries: 476 through the prize competition and 72 by email. To ensure thoughtful and efficient review, NEI established the following procedure for selecting winning ideas for audacious goals. All entries were de-identified before any of the following review procedures were undertaken.
First level of review: Responsiveness. Entries were reviewed by three NEI scientists to determine if they were responsive to the Challenge criteria. Submissions were deemed "non-responsive" if any of the following applied:
- Did not describe a goal
- Did not fit within NEI's mission
- Did not address feasibility
- Rationale or facts supporting a goal were incorrect or misleading
- Did not have a broad scope (e.g., should be beyond the work of a single research grant)
474 entries were deemed responsive (86% of 548 submitted entries).
Second level of review: Technical Advisor Input. A number of clinicians and scientists (total number = 81) with broad expertise in relevant disciplines provided input to the panel of judges on the submitted ideas based on the criteria announced in the NEI Challenge prize competition. The input consisted of numerical scores, letter grades, and technical comments. The technical advisor scores and grades were merged using several methods (See Methods). This winnowed the number from 474 to 81 for evaluation by the Federal judging panel.
Final Judging: As required by the rules of the NEI Challenge, a panel of Federal employees evaluated the entries and selected the winning entries (see www.challenge.gov ) based on the input from the technical advisors. The judges met on January 11, 2013 and selected ten winners. The scoring and grading methods used by the judges were identical to those used by the technical advisors. Of the 81 entries evaluated, the judges specifically discussed the 34 top scoring entries first, and then examined the bottom 47 before selecting 10 winning entries. The 10 winning entries were discussed and presented at the NEI Audacious Goals Development Meeting in February, 2013.
The following five selection criteria are reproduced from the Federal Register notice (Vol. 77, No. 156, Aug 13 2012):
1. Relevance to the NEI Mission: Each entry will be rated on how the goal would further the NEI mission to conduct and support research, training, health information dissemination, and other programs with respect to blinding eye diseases, visual disorders, mechanisms of visual function, preservation of sight, and the special health problems and requirements of the blind.
2. Audaciousness: Each entry will be rated on whether the proposed goal is bold, daring, original or unconventional, exceptionally innovative, creative, novel, or any combination.
3. Feasibility: Although it is recommended that contestants consider about a 10 year time period for achieving a proposed goal, NEI recognizes that estimates of the timeframe for an audacious goal could vary considerably depending on the nature of the goal. Thus, audacious goals with shorter or longer time periods may be acceptable. Each entry will be rated on how well it describes the technological, scientific, or other advances that are needed to reach the goal.
4. Scope: Each entry will be rated on the extent to which it is broad and/or far-reaching. Goals can include basic, translational, clinical research, or any combination. Goals may also encompass training or health information dissemination as appropriate within the NEI Mission. The goal could have multiple components, for example research requiring multidisciplinary approaches or involvement of multiple laboratories. Even a goal that addresses a disease affecting a relatively small number of patients may be considered broad and far-reaching if it requires the development of tools and techniques that can be applied to other problems.
5. Impact: Each entry will be rated on its transformative potential; its value in exerting a positive and powerful influence on the NEI mission.
6. After Selecting the Winners: The winners and about 200 experts in vision research met to discuss the winning ideas at the NEI Audacious Goals Development Meeting on February 24-26, 2013.
Last Reviewed: June 2013