Methods for technical advisor input for consideration by the Federal judges.

NEI Challenge to Identify Audacious Goals in Vision Research and Blindness Rehabilitation

  1. Total number of responsive ideas for evaluation = 474
  2. 81 technical advisors were assigned to evaluate and score up to 40 submissions using a numerical scale from 0 to 100 (scale 0 – 100) on the basis of their scientific and/or clinical expertise. Each advisor also was asked to choose their top 5 entries selected from any of the 474 and grade them A -E. They also had the option of providing comments about any entry. Many, but not all, advisors examined all submissions in order to find their top 5 and therefore did not limit their evaluation solely to their assigned entries.
  3. Each submission was scored by at least four but not more than seven advisors (average = 5.4 advisors per entry).
  4. Each entry was evaluated using three separate rankings:
    a. Mean score was calculated for each of the 474 submissions
    b. Adjusted rank score was calculated for each reviewer. They each scored anywhere from 11 to 40 entries, so their scores were then normalized to 40 as a maximum and ranked from top to bottom.
    c. Grade score sum was computed by transforming letter grades to numbers (i.e., grade A = 5; grade B=4, etc.). The values were summed for all reviewers and then ranked.
  5. The top 10% of the entries were identified using each of the three scoring methods. 74 entries appeared in at least one of these three groups.
  6. We examined all grades (A –E) from outside the 40 assigned entries that each advisor was asked to score. Seven submissions were graded “A” that were not included in the 74 from the rankings.
  7. These 7 were added to the 74 from the top 10% of any of the three rankings for a total of 81 entries for evaluation by the Federal judges.

** The Federal judges evaluated the 81 top entries designated by the technical advisors using the same methods. However, in step 5, the top 25% of the entries were identified for each of the three scoring methods. This gave a total of 32 entries. Additional “A” grades added two more for a total of 34 of the 81 as the top entries for discussion by the panel.

Last Reviewed: 
January 2013