ALBERT

Notes: This article discusses the peer review system in the context of countries with small scientific communities. It is argued that the traditional peer review system is inadequate and probably dangerous for such countries. The reasoning is that in scientifically small countries personal relations and politics dominate the scene and objective impartial evaluation is not possible. The novel peer review systems adopted in the Netherlands and in South Africa are outlined and suggestions for improvements are made.〈section xml:id="abs1-2"〉〈title type="main"〉SUMMARY AND SUGGESTIONS FOR IMPROVEMENTThe thrust of this article is the argument that the traditional peer-review system is inadequate and probably dangerous for scientifically small countries. Although there is no hard empirical evidence to support the thesis that the peer-review system is functioning better in scientifically large countries, a priori reasoning and circumstantial evidence indicate that the system is inefffective in countries with small scientific communities. The reasoning is that in small countries personal relations and politics dominate the scene and objective, impartial evaluation is not possible. As a result science policy can be misguided and societal loss can occur as valuable research effort is spent on ineffective peer-evaluations.A variety of measures can be used to compensate for the ineffectiveness of the peer-review system. The prime goal, however, is not only to increase the effectiveness of the system, but also to minimize societal costs due to misallocation of resources.The first step towards improvement of the system is to monitor it. Referees' reports and advice can be kept in files and properly analysed periodically. In this way biases concerning researchers, institutes, scientific subjects and approaches can be identified and appropriately remedied. Refereeing the referees could elimate intentional biases and improve the quality control system by anticipation. Referees knowing that their reports would be checked would take care to prepare proper evaluations.Increasing the number of referees is another way to increase reliability. Scientists from other disciplines can and should be incorporated in the evaluation juries. This way relevance of research and cross-fertilisation of ideas can be incorporated in the system, partly ‘disinterested’ referees will be included in the juries (i.e. referees and applicants would not compete for the same research funds) and the lack of scientists in scientifically small countries will be alleviated. Care should be taken, however, not to over-burden the system with unnecessary evaluation effort. In requiring researchers with demonstrated ability to go through the process of making formal applications for funding every year, valuable research time is wasted in preparing and evaluating proposals which, after all, will go through due to the status of the applicant. Attempting to regulate the expansion or contraction of the forest and not of the trees may provide a better solution. Evaluating researchers and disciplines instead of individual projects and funding them accordingly may save valuable research time and increase research productivity and effectiveness.Finally, one of the most promising areas for improvement in the decision making process is the incorporation of quantitative techniques in the tool-kit of evaluation criteria. Publication and citation analyses can reveal the past performance (which is the only indicator of success for future research) of individuals and research groups. In a sense, bibliometric anaiysis can be seen as an expansion in time and panel size of the conventional peer reviews. Instead of asking a limited number of referees about a scientist's work, we count the citations that the particular scientist received by the whole scientific community and over an extended time period. Provided that care is taken to exclude self-citations, negative citations, compare the same with the same, citation analysis can identify the pecking order in the scientific community. As bibliometric measures correlate well with non-bibliometric measures (Table 2) the use of such criteria can ameliorate personal biases, improve objective evaluations and reduce time spent on evaluation purposes considerably.