Abstract
Assessing the research of individual scholars is currently a matter of serious concern and worldwide debate. In order to gauge the long-term efficacy and efficiency of this practice, we carried out a limited survey of the operation and outcome of Mexico’s 30-year old National System of Investigators or SNI, the country’s main instrument for stimulating competitive research in science and technology. A statistical random sample of researchers listed in the area of Humanities and Behavioral Sciences—one of SNI’s first and better consolidated academic divisions comprising a wide range of research disciplines, from philosophy to pedagogy to archaeology to experimental brain research—was screened comparing individual ranks or "Levels of distinction" to actual compliance with the SNI’s own evaluation criteria, as reflected in major public databases of scholarly production. The same analysis was applied to members of a recent Review Committee, integrated by top-level researchers belonging to that general area of knowledge, who have been in charge of assessing and ranking their colleagues. Our results for both sets of scholars show wide disparity of individual productivity within the same SNI Level, according to all key indicators officially required (books issued by prestigious publishers, research articles appeared in indexed journals, and formation of new scientists), as well as in impact estimated by numbers of citations. Statistical calculation from the data indicates that 36% of members in the Review Committee and 53% of researchers in the random sample do not satisfy the official criteria requested for their appointed SNI Levels. The findings are discussed in terms of possible methodological errors in our study, of relevance for the SNI at large in relation to independent appraisals, of the cost-benefit balance of the organization as a research policy tool, and of possible alternatives for its thorough restructuring. As it currently stands SNI is not a model for efficient and effectual national systems of research assessment.
Highlights
In today's knowledge-based societies within an intensely competitive global economy, and in the face of increasing environmental and cultural challenges, scientific and technological research has become a topic of national security for most countries, both large and small
A workable starting glimpse of the overall situation can be attempted by restricting the inspection to just one of the System of Investigators (SNI)’s general areas of knowledge that by official design is itself widely diverse in disciplines and specialties, resembling in this regard the SNI’s full universe. Such endeavor provides an opportunity to gauge the long-term efficacy and efficiency of regularly assessing the research of individual scholars at a national scale. With this admittedly limited and approximate approach, here we report a quantitative analysis of the correspondence between individual SNI ranks and scholarly production, in terms of the SNI’s own evaluation criteria, for a random sample of professors and a Review Committee belonging in Area IV, which incorporates the humanities and the behavioral sciences
The total production of research books is strikingly irregular among the individual scholars constituting each group in our random sample, and even in the Review Committee itself where differences between members vary by up to an order of magnitude
Summary
In today's knowledge-based societies within an intensely competitive global economy, and in the face of increasing environmental and cultural challenges, scientific and technological research has become a topic of national security for most countries, both large and small. The Review Committees are in charge of devising and revising specific criteria for evaluating the performance of their colleagues in the respective areas of knowledge, according to their own distinct working methods and traditions Based upon these specific criteria, the SNI Review Committees discuss and assess the individual applications received for each period of evaluation. Applicants are appointed as National Investigators in one of four academic ranks or “Levels of distinction”: Candidate (beginner), I (junior), II (intermediate), and III (senior, Emeritus, or occasionally young outstanding) scientist Through this process over 21,000 scholars currently enrolled in SNI—about half of all active in Mexico—are periodically evaluated, with mandatory reviewing intervals established according to their SNI Level and the date of their most recent appointment
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
