The purpose of the entire proposal-selection process for NRAO telescopes
is to select the proposals that potentially are most valuable for the
advancement of scientific knowledge. This does not necessarily mean
recommending only those proposals that will provide sure results; it also
includes a careful consideration of well-reasoned proposals that may be
unconventional, but provide opportunities for new discoveries. In the
evaluation of proposals, we ask that the referees think about how best to
exploit the full capability of the unique scientific instruments that NRAO
operates for the community, namely the VLA, VLBA, and GBT. In this
context, we ask the referees to take a constructive approach. For
instance, the Proposal Selection Committees are eager to receive comments
about ways in which proposals might be implemented to enhance their
scientific value, such as the allocation of additional time, additional
telescope configurations, or even additional NRAO telescopes. The goal
should not be to provide the bare minimum of observing time to get the
maximum number of observers on the telescopes, but rather to maximize the
scientific impact of the observations undertaken with the telescopes.
There is no formal weighting of the
criteria described below. Scientific merit is most important,
but a good scientific goal is (of course) not worth much if
an observation is technically infeasible!
We ask referees to grade proposals primarily
based on their scientific merit, thinking of the question
"How scientifically valuable is this use of VLA/VLBA/GBT time,
compared to the other proposals under review?"
We seek proposals that may have high scientific
impact, not just "sure things" with modest impact.
For the VLBA, proposals that require extra resources such as
the phased VLA and the GBT also should be held to higher standards,
and the need for these resources must be explicitly justified.
We leave the exact details of this to the individual referees.
But if proposals that require a lot of resources to accomplish
their goals are judged to be very valuable scientifically, we
strongly encourage the referees to give them a very good grade,
and comment on the importance of doing the entire project, rather
than cutting it back in any way.
Particularly in such cases, it is quite reasonable for referees
to evaluate whether the proposal team can marshal the resources
and expertise to complete and publish the observations.
This generally is considered by
the scheduling committee or the NRAO scientific and technical
staff. However, assessments by the referees also are valuable.
For example, they may believe that a proposal requires more
resources than requested, or can be done in half the time
requested, or is not feasible because the proposers may have
miscalculated the sensitivity. Justification of the amount of
GBT time needed for short-spacing data in interferometer
images, and assessment of RFI and confusion issues, also may
be important. Clear comments from the referees about possible
technical flaws are particularly helpful.
Publication record. Due
recognition should be given to teams who have an established publication
record from past related proposals. In contrast, some observers may have
considerable data that clearly has not been digested or published yet.
Referees should take these positive and negative factors into account
when making recommendations about granting additional observing
time. Possibility of acquiring more appropriate data.
Referees also may wish to consider the possibility that other
data may be more appropriate for reaching the scientific goals
of the proposal. For instance, the relevant data might already
exist in the VLA archive, or in the Canadian Galactic Plane
Survey. Selection of resources.
For the VLA and VLBA, is the right mix of configurations
and/or telescopes requested? For example, referees may consider
that a particular VLA proposal is better done in a configuration
other than the one proposed, or may not need one of the configurations
that was proposed. (Note that proposals should be reviewed and
graded even if they are for a future VLA configuration, since
proposers sometimes will submit a proposal a trimester early
so that they might have an opportunity to revise it, if necessary,
in response to referees' comments.) For the VLBA, referees may note that a
request for a single VLA antenna is highly justified, or that
this antenna is not really needed. For some scientific goals,
the GBT might be a better instrument than a compact VLA configuration. Student status.
We encourage the use of NRAO telescopes for student research,
particularly for Ph.D. dissertations. In this case, it is most
helpful if the proposal will say in some manner how the
proposed observations will be used in the dissertation, and
whether they are a sidelight or a main focus of the thesis.
Referees are encouraged to comment on this topic, and may
choose to support proposals more strongly if there appears
to be a well-thought-out program of student research. Proposal length.
The justification should obey page limits. Regular or rapid-response proposals will be allowed a maximum of
four (4) one-sided pages (US letter sized) with 11 point font
(minimum) to present the scientific justification and the
technical feasibility of the project, including all figures,
tables and references. Large proposals will be allowed a maximum of ten (10) one-sided
pages (US letter sized) with 11 point font (minimum) to present
the scientific justification and the technical feasibility of the
project, including all figures, tables and references. The cover information form is not counted as part of these page limits.
Resource requirements.
Referees can and should take into account the amount of resources
requested. For example, a particular proposal or scientific
goal may be viewed as a valuable use of 4 hours of telescope time,
but may not be as valuable if it requires 80 hours of time,
considering the other proposals that it would displace.
We use a scale from 1 to 9, with 1 being an outstanding proposal,
and 9 being a very poor proposal. Decimal scores, from 0.1 to
9.9, are permissible. It is helpful if referees give a fairly
wide spread in their grades, rather than giving every proposal
a grade between 2.9 and 3.1! Grades are re-normalized so that
each referee has the same mean and standard deviation; this
process works best if there is some spread in the grades.
Since the NRAO telescopes are oversubscribed, it can be
especially useful for referees to use this field to indicate
whether the percentage of time recommended is less (or more!)
than was proposed. It is most useful if these recommendations
are accompanied by specific comments (see below). For instance,
if a referee recommends 50% of the time requested, it is
helpful to know if he or she thinks the integration time should
be cut in half on each object, or if half the objects
(which ones?) should be eliminated.
Comments by the referees are very important for the process.
It is common for different referees to come to quite different
conclusions about proposals. In resolving this, it is important
to the scheduling committee that the referees comment about what
leads them to their rating. Do not think "I'm giving it a very
good rating, so I don't have to say why." Let us know what specific
benefits are likely to accrue. Specific comments are, moreover,
of great benefit to the proposers in understanding the disposition
of their proposal. This is especially true for proposals that are
not successful, because the referee comments often are used as a
basis for revising a proposal for resubmittal. Comments
should observe some norms of professional courtesy. For example,
rather than saying "These proposers are idiots," it is more
constructive to say, "This proposal appears to be based on presumptions
contrary to those established in the general literature." A mechanism is
provided for the referee to comment directly to the scheduling committee,
without its being seen by the proposers. This is primarily intended
for remarks that, rendered openly, would compromise the anonymity of the
referee. For example, "I observed this source two years ago, and
found that the peak flux density was a factor of five below that
assumed by the proposers." It is not intended primarily as a channel
for remarks such as "These proposers are idiots".
Referees should not evaluate proposals by teams which include
themselves, their spouses, significant others, or other immediate
family members. Beyond that, we have no formal rules with respect
to competitors, past students or advisors, collaborators on other
projects, and so on. It is permissible to evaluate proposals from
investigators at the same institution if the referee believes
that he or she can evaluate the proposal fairly. We leave this
issue to the referee's discretion and conscience.
Based on the above criteria, if the referee feels that they have a
conflict of interest, we ask that he/she supply no grade and no
comments, beyond noting a conflict of interest in a remark to the
scheduling committee. The committee does take note if a referee
tends generally to evaluate proposals by a particular group as being
exceptionally good or bad. This can be a matter of professional
opinion about a particular research direction, and this is why we
have referees! However, in such cases, the referees should take
care that their comments clearly indicate the scientific issues
that are the basis for the numerical grades.
The disposition of past proposals is posted on the web. For the VLA
and VLBA, see
http://www.aoc.nrao.edu/~schedsoc for the list of proposals
accepted at the last deadline, or for proposals currently in
the VLBA dynamic scheduling queue. For the GBT, see
http://www.gb.nrao.edu/gbtprops/acceptedproposals.shtml for
all accepted proposals since the telescope was first opened for
operations.
Modified on Tuesday, 05-Dec-2006 14:46:26 EST by Pat Murphy (minor
edits for Dale Frail)