National RadioAstronomy Observatory
National RadioAstronomy Observatory
National Radio|
Role and Responsibilities Review Criteria Numerical Grades Percent of Time Recommended Comments Conflicts of Interest Past Proposals |
Role and ResponsibilitiesThe purpose of the entire proposal-selection process for NRAO telescopes is to select the proposals that potentially are most valuable for the advancement of scientific knowledge. This does not necessarily mean recommending only those proposals that will provide sure results; it also includes a careful consideration of well-reasoned proposals that may be unconventional, but provide opportunities for new discoveries. In the evaluation of proposals, we ask that the referees think about how best to exploit the full capability of the unique scientific instruments that NRAO operates for the community, namely the EVLA, VLBA, and GBT. In this context, we ask the referees to take a constructive approach. For instance, the Proposal Selection Committees are eager to receive comments about ways in which proposals might be implemented to enhance their scientific value, such as the allocation of additional time, additional telescope configurations, or even additional NRAO telescopes. The goal should not be to provide the bare minimum of observing time to get the maximum number of observers on the telescopes, but rather to maximize the scientific impact of the observations undertaken with the telescopes.Review CriteriaThere is no formal weighting of the criteria described below. Scientific merit is most important, but a good scientific goal is (of course) not worth much if an observation is technically infeasible!Scientific MeritWe ask referees to grade proposals primarily based on their scientific merit, thinking of the question "How scientifically valuable is this use of EVLA/VLBA/GBT time, compared to the other proposals under review?" We seek proposals that may have high scientific impact, not just "sure things" with modest impact. For the VLBA, proposals that require extra resources such as the GBT also should be held to higher standards, and the need for these resources must be explicitly justified. We leave the exact details of this to the individual referees. But if proposals that require a lot of resources to accomplish their goals are judged to be very valuable scientifically, we strongly encourage the referees to give them a very good grade, and comment on the importance of doing the entire project, rather than cutting it back in any way. Particularly in such cases, it is quite reasonable for referees to evaluate whether the proposal team can marshal the resources and expertise to complete and publish the observations.Technical FeasibilityThis generally is considered by the Proposal Selection Committee or the NRAO scientific and technical staff. However, assessments by the referees also are valuable. For example, they may believe that a proposal requires more resources than requested, or can be done in half the time requested, or is not feasible because the proposers may have miscalculated the sensitivity. Justification of the amount of GBT time needed for short-spacing data in interferometer images, and assessment of RFI and confusion issues, also may be important. Clear comments from the referees about possible technical flaws are particularly helpful.Other Review CriteriaPublication record. Due recognition should be given to teams who have an established publication record from past related proposals. In contrast, some observers may have considerable data that clearly has not been digested or published yet. Referees should take these positive and negative factors into account when making recommendations about granting additional observing time.Possibility of acquiring more appropriate data. Referees also may wish to consider the possibility that other data may be more appropriate for reaching the scientific goals of the proposal. For instance, the relevant data might already exist in the VLBA archive, or in the Canadian Galactic Plane Survey. Selection of resources. For the EVLA and VLBA, is the right mix of configurations and/or telescopes requested? For example, referees may consider that a particular EVLA proposal is better done in a configuration other than the one proposed, or may not need one of the configurations that was proposed. (Note that proposals should be reviewed and graded even if they are for a future EVLA configuration, since proposers sometimes will submit a proposal a trimester early so that they might have an opportunity to revise it, if necessary, in response to referees' comments.) For the VLBA, referees may note that a request for the GBT is highly justified, or that this antenna is not really needed. For some scientific goals, the GBT might be a better instrument than a compact EVLA configuration. Student status. We encourage the use of NRAO telescopes for student research, particularly for Ph.D. dissertations. In this case, it is most helpful if the proposal will say in some manner how the proposed observations will be used in the dissertation, and whether they are a sidelight or a main focus of the thesis. Referees are encouraged to comment on this topic, and may choose to support proposals more strongly if there appears to be a well-thought-out program of student research. Proposal length. The justification must obey page limits, as follows: The cover information form is not counted as part of these page limits. Resource requirements. Referees can and should take into account the amount of resources requested. For example, a particular proposal or scientific goal may be viewed as a valuable use of 4 hours of telescope time, but may not be as valuable if it requires 80 hours of time, considering the other proposals that it would displace.Numerical GradesWe use a scale from 1 to 9, with 1 being an outstanding proposal, and 9 being a very poor proposal. Decimal scores, from 0.1 to 9.9, are permissible. It is helpful if referees give a fairly wide spread in their grades, rather than giving every proposal a grade between 2.9 and 3.1! Grades are re-normalized so that each referee has the same normalized median rating; this process works best if there is some spread in the grades.Percent of Time RecommendedSince the NRAO telescopes are oversubscribed, it can be especially useful for referees to use this field to indicate whether the percentage of time recommended is less (or more!) than was proposed. It is most useful if these recommendations are accompanied by specific comments (see below). For instance, if a referee recommends 50% of the time requested, it is helpful to know if he or she thinks the integration time should be cut in half on each object, or if half the objects (which ones?) should be eliminated.CommentsComments by the referees are very important for the process. It is common for different referees to come to quite different conclusions about proposals. In resolving this, it is important to the scheduling committee that the referees comment about what leads them to their rating. Do not think "I'm giving it a very good rating, so I don't have to say why." Let us know what specific benefits are likely to accrue. Specific comments are, moreover, of great benefit to the proposers in understanding the disposition of their proposal. This is especially true for proposals that are not successful, because the referee comments often are used as a basis for revising a proposal for resubmittal. Comments should observe some norms of professional courtesy. For example, rather than saying "These proposers are idiots," it is more constructive to say, "This proposal appears to be based on presumptions contrary to those established in the general literature." A mechanism is provided for the referee to comment directly to the scheduling committee, without its being seen by the proposers. This is primarily intended for remarks that, rendered openly, would compromise the anonymity of the referee. For example, "I observed this source two years ago, and found that the peak flux density was a factor of five below that assumed by the proposers." It is not intended primarily as a channel for remarks such as "These proposers are idiots".Conflicts of InterestReferees should not evaluate proposals by teams which include themselves, their spouses, significant others, or other immediate family members. Beyond that, we have no formal rules with respect to competitors, past students or advisors, collaborators on other projects, and so on. It is permissible to evaluate proposals from investigators at the same institution if the referee believes that he or she can evaluate the proposal fairly. We leave this issue to the referee's discretion and conscience. Based on the above criteria, if the referee feels that they have a conflict of interest, we ask that he/she supply no grade and no comments, beyond noting a conflict of interest in a remark to the scheduling committee. The committee does take note if a referee tends generally to evaluate proposals by a particular group as being exceptionally good or bad. This can be a matter of professional opinion about a particular research direction, and this is why we have referees! However, in such cases, the referees should take care that their comments clearly indicate the scientific issues that are the basis for the numerical grades.Past ProposalsThe disposition of past proposals is posted on the web. For the EVLA and VLBA, see http://www.aoc.nrao.edu/~schedsoc for the list of proposals accepted at the last deadline, or for proposals currently in the VLBA dynamic scheduling queue. For the GBT, see http://gbrescal.gb.nrao.edu/gbtobs/proposals.dbw for all accepted proposals since the telescope was first opened for operations.
Copyright © 2009 Associated Universities, Inc. |