Peer Review Method Research Articles

The Church and Peer Review: Was 'Peer' Review Fairer, More Honest Then Than Now?

The traditional thought regarding peer review tends to be that it started with the establishment of the academy, sometime around 1650. It is a reasonable presumption that to have peer review one needs first to have peers. However, the actual review of works certainly occurred long before 1650. Of some importance is the nature of that review that took place prior to the appearance of universities in Bologna and Paris. The standard (and misapplied) logic is that the Church wielded a heavy hand on all publishing, acting as a restraint on inappropriate works prior to their publication. This is not wholly true, however. The Church is best known for its suppression of works post-publication. In a way, it acted as a critic, offering its advice to authors who it found proposed errant ideas and suggesting they might wish to recant and return to good standing. This is interesting when cast in today's peer-review environment. The author suggests that much can be learned from the Church's method of dealing with scholarship, especially in a world of e-reserves. Should we ditch the traditional peer-review method and go back to a publish-then-evaluate system used by the Holy See? In large part, the author argues that unless the academy is willing to cure the perceived ills of peer review and do so soon, the question will be answered in the affirmative, with or without our agreement.

The Church and Peer Review: Was ‘Peer’ Review Fairer, More Honest Then Than Now?

The traditional thought regarding peer review tends to be that it started with the establishment of the academy, sometime around 1650. It is a reasonable presumption that to have peer review one needs first to have peers. However, the actual review of works certainly occurred long before 1650. Of some importance is the nature of that review that took place prior to the appearance of universities in Bologna and Paris. The standard (and misapplied) logic is that the Church wielded a heavy hand on all publishing, acting as a restraint on inappropriate works prior to their publication. This is not wholly true, however. The Church is best known for its suppression of works post-publication. In a way, it acted as a critic, offering its advice to authors who it found proposed errant ideas and suggesting they might wish to recant and return to good standing. This is interesting when cast in today's peer-review environment. The author suggests that much can be learned from the Church's method of dealing with scholarship, especially in a world of e-reserves. Should we ditch the traditional peer-review method and go back to a publish-then-evaluate system used by the Holy See? In large part, the author argues that unless the academy is willing to cure the perceived ills of peer review and do so soon, the question will be answered in the affirmative, with or without our agreement.

Effects of large-scale research funding programs: a Japanese case study

This study investigates the effects of large-scale research funding from the Japanese government on the research outcomes of university researchers. To evaluate the effects, we use the difference-in-differences estimator and measure research outcomes in terms of number of papers and citation counts per paper. Our analysis shows that the funding program led to an increase in the number of papers in some fields and an increase in the citation counts in the other fields. A comparison of our estimation results with assessment data obtained from peer reviews showed important differences. Since the characteristics of research vary according to the field, bibliometrics analysis should be used along with the peer review method for a more accurate analysis of research impact.

Directed Peer Review in Surgical Pathology

Second pathologist peer review is used in many surgical laboratory quality-assurance programs to detect error. Directed peer review is 1 method of second review and involves the selection of specific case types, such as cases from a particular site of anatomic origin. The benefits of using the directed peer review method are unique and directed peer review detects both errors in diagnostic accuracy and precision and this detection may be used to improve practice. We utilize the Lean quality improvement A3 method of problem solving to investigate these issues. The A3 method defines surgical pathology diagnostic error and describes the current state in surgical pathology, performs root cause analysis, hypothesizes an ideal state, and provides opportunities for improvement in error reduction. Published data indicate that directed peer review practices may be used to prevent active cognitive errors that lead to patient harm. Pathologists also may use directed peer review data to target latent factors that contribute to error and improve diagnostic precision.

Revisiting the scaling of citations for research assessment

Over the past decade, national research evaluation exercises, traditionally conducted using the peer review method, have begun opening to bibliometric indicators. The citations received by a publication are assumed as proxy for its quality, but they require standardization prior to use in comparative evaluation of organizations or individual scientists: the citation data must be standardized, due to the varying citation behavior across research fields. The objective of this paper is to compare the effectiveness of the different methods of normalizing citations, in order to provide useful indications to research assessment practitioners. Simulating a typical national research assessment exercise, he analysis is conducted for all subject categories in the hard sciences and is based on the Thomson Reuters Science Citation Index-Expanded®. Comparisons show that the citations average is the most effective scaling parameter, when the average is based only on the publications actually cited.

An evidence based protocol for peer review of radiographer musculoskeletal plain film reporting

AimsMedical image interpretation by non-medically trained staff continues to court controversy. This article aims to show that any potential risks associated with radiographer reporting can be monitored and mitigated if a robust peer review system is introduced. A search of the evidence base illustrates a paucity of guidance on how reporting radiographers should be audited or how a peer review process should be implemented. A practical framework for designing a reporting radiographer peer review process is provided. MethodsFollowing a literature review, key issues faced when designing a peer review protocol were identified. The following questions are answered: How frequent should peer review take place? How many reports should be reviewed? How are reports selected for review? Who should peer review the radiographer reports? How should radiographer's reporting performance be measured? What standard of reporting is acceptable? ResultsDetails are provided of the process that has been used for over three years at a busy inner-city teaching hospital for auditing musculoskeletal plain film radiographer reporting. The peer review method presented is not intended to produce robust statistical data; it is a practical method of locally assessing the reporting competency. As such, our protocol should be viewed as part of a larger programme for continuing professional development. ConclusionIt is hoped that this practical protocol will encourage radiology departments to engage in a programme of peer review for reporting radiographers.

A little idea

A little idea

How to improve the peer review method: Free-selection vs assigned-pair protocol evaluated in a computer networking course

This study provides field research evidence on the efficiency of a “free-selection” peer review assignment protocol as compared to the typically implemented “assigned-pair” protocol. The study employed 54 sophomore students who were randomly assigned into three groups: Assigned-Pair (AP) (the teacher assigns student works for review to student pairs), Free-Selection (FS) (students are allowed to freely explore and select peer work for review), and No Review (NR) (control group). AP and FS student groups studied and reviewed peer work in the domain of Computer Networking, supported by a web-based environment designed to facilitate the two peer review protocols. Our results indicate that students following the Free Selection protocol demonstrate (a) better domain learning outcomes, and (b) better reviewer skills, compared to the AP condition. Overall, the study analyzes the benefits and shortcomings of the FS vs AP review assignment protocol, providing evidence that the FS condition can be multiply beneficial to students who engage in peer review activities.

Peer Review Verfahren in der Pathologie – mehr als zehn Jahre Erfahrungen im Freistaat Sachsen

The positive experiences regarding peer review practice go back to the 1990s and can be traced in the position paper of the German Medical Association concerning quality assurance in the field of pathology. This evolved into the initiative of pathologists in Saxony to implement the peer review practice within the remit of their State Medical Association. Since the 14(th) of May 1999 various institutes as well as medical group practices in pathology and cytology have joined forces to undertake voluntary inter-institutional quality control measures. The aim was to improve the process and especially the outcome quality by reviewing each other's quality (analysing samples) on the basis of defined criteria. A positive review outcome report, which will have to be unanimously agreed upon by all participants, will lead to a positive recommendation for a certificate issued by the Medical Association of Saxony. Between 1999 and 2011 a total of 56 peer review proceedings took place. The potentials for improvement concern the problem of "kitchen pathology" still being used in the context of macroscopic description, the quality of microscopic description including the use of correct nomenclature, the quality of tumour classification and immune histological investigations. Statements concerning the advantage of the peer review method in particular refer to the integration of resident pathologists. Due to the long period of implementation of the peer review system und the small number of proceedings, it is not possible to come to a clear conclusion about the improvement or deterioration of quality. During a period of more than 10 years three peer review proceedings were not successful. In this situation, it is appropriate to ask the question of how restrictive peer reviews should be. The three most important aspects for the pathologists in Saxony are: self-determination, learning from each other, and a commitment to quality improvement. So this method set an example of how a quality management method can persist for more than 10 years through voluntary engagement. And the initiative of pathologists in Saxony has given an important impetus to the development of other peer review systems in various health care organisations or disciplines in Germany.

Citation analysis as a measure of article quality, journal influence and individual researcher performance

The research-related performance of universities, as well as that of individual researchers, is increasingly evaluated through the use of objective measures, or metrics, which seek to support or in some cases even replace more traditional methods of peer review. In particular there is a growing awareness in research communities, government organisations and funding bodies around the concept of using evaluation metrics to analyse research citations. The tools available for ‘citation analysis’ are many and varied, enabling a quantification of scientific quality, academic impact and prestige. However there is increasing concern regarding the potential misuse of such tools, which have limitations in certain research disciplines.This article uses ‘real world’ examples from radiography research and scholarship to illustrate the range of currently available citation analysis tools. It explores the academic debate surrounding their strengths and limitations, and identifies the potential impact of citation analysis on the radiography research community.The article concludes that citation analysis is a valuable tool for researchers to use for personal reflection and research planning, yet there are inherent dangers if it is used inappropriately. Whilst citation analysis can give objective information regarding an individual, research group, journal or higher education institution, it should not be used as a total substitute for traditional qualitative review and peer assessment.

Peer Review, Judgment, and Reading

Evaluating new forms of scholarship for tenure and promotion requires taking those forms, and the methods of peer review they bring with them, on their own terms. Even more, it requires exercising the critical judgment on which our profession relies instead of outsourcing that judgment to others. Such evaluation requires reading both the work and the available evidence of the ways that scholars have responded to that work. (KF)

Blood Transfusion's first "impact factor".

2.519 This is the value of the impact factor that the Journal Citation Reports/Science Edition attributed Blood Transfusion for the year 2010 after Thomson Reuters had reported, in January 2011, that Blood Transfusion had been selected for coverage in products and services and that, beginning with volume 6 (1) 2008, our journal would be indexed and abstracted in Science Citation Index Expanded (also known as SciSearch®) and the Journal Citation Reports/Science Edition. This is a wonderful result, on the one hand because it is the first time that Blood Transfusion has been awarded an impact factor and on the other hand because the value is particularly high, indicating that our scientific journal is among the most important in the field of Transfusion Medicine. This is a goal of a process started in 2003, when the Italian Society of Transfusion Medicine and Immunohaematology (SIMTI) started the project of modernisation and re-launching of its journal, by flanking the Italian articles with their English versions, a step indispensable for entry into the international scientific arena. Another major step was made in 2007 with the difficult decision to renounce the use of Italian and publish the journal entirely in English, as well as fully embracing the scientific method of peer review, developing a web site and allowing free access to it for the whole scientific community1. In 2008 Blood Transfusion was selected and included by the National Library of Medicine in the PubMed-MEDLINE database and the journal was also indexed in the Embase and Scopus databases, in PubMed Central and in Google Scholar2. In this very editorial section, we had explicitly set out the aim of obtaining an impact factor for Blood Transfusion. Another fact of great relevance is that Blood Transfusion is no long the official organ of only SIMTI, but has become the patrimony of four other European scientific societies. This important achievement is due to the commitment of the Board and Affiliated Editors and of the International Editorial Board, to SIMTI’s support -also financial- of the project, to the development of the website, to the team in the Editorial Office who have put so much enthusiasm into carrying out their work and, finally, to the numerous referees around the world who, in some cases, have become fervent supporters of the journal with their diligence and suggestions. Blood Transfusion today has different tasks and roles. The affiliation of the European scientific societies has been a good way of stimulating scientific collaboration and awareness. We must now broaden this participation (the procedure to include the Croatian Society of Haematology and Transfusion Medicine is underway) and find new ways for a more direct participation by everyone in the management of Blood Transfusion. We must develop this process because it will enable people to feel that Blood Transfusion is their “own” journal: a journal in which they can print what they write (provided they write well!), but also a vehicle for publishing specific supplements containing the proceedings of their national congresses, courses and meetings of local or scientific value that can appear in PubMed. Obtaining an impact factor has been a great achievement, but inevitably also sets a new starting point because it obliges us to take on and maintain the responsibility of scientific warranty in an international context: obtaining the impact factor was a long and tasking process, improving it or even only maintaining it will be even more difficult and challenging. This is our new aim.

Paired peer review of university classroom teaching in a school of nursing and midwifery

Peer review of university classroom teaching can increase the quality of teaching but is not universally practiced in Australian universities. To report an evaluation of paired peer-review process using both paper and web based teaching evaluation tools. Twenty university teachers in one metropolitan Australian School of Nursing and Midwifery were randomly paired and then randomly assigned to a paper based or web-based peer review tool. Each teacher reviewed each other's classroom teaching as part of a peer review program. The participants then completed an 18 question survey evaluating the peer review tool and paired evaluation process. Responses were analyzed using frequencies and percentages. Regardless of the tool used, participants found this process of peer review positive (75%), collegial (78%), supportive (61%) and non-threatening (71%). Participants reported that the peer review will improve their own classroom delivery (61%), teaching evaluation (61%) and planning (53%). The web-based tool was found to be easier to use and allowed more space than the paper-based tool. Implementation of a web-based paired peer review system can be a positive method of peer review of university classroom teaching. Pairing of teachers to review each other's classroom teaching is a promising strategy and has the potential to improve teaching in teaching universities.

SU-E-T-366: Dosimetric Characterization of Brachytherapy Surface Applicators

Purpose: Conical surface applicators with diameters ranging from 10mm to 45mm have been developed by Varian Medical Systems, Inc. These applicators are designed to be used with the GammaMedplus iX and VariSource iX high-dose rate Ir-192 afterloaders, allowing for conformal dose delivery for the treatment of surface lesions. Clinical use of these applicators has become widespread, yet no peer-reviewed method of output verification exists. Traditional brachytherapy dosimetry protocols (AAPM TG-43U1) cannot be applied directly, creating a gap in the chain of calibration traceability. The ultimate goal of this work is the creation of a dosimetry protocol for the calibration of applicators like those under investigation. An initial step towards this includes Monte Carlo and experimental assessment of the dose-rate and distributions for each applicator. Methods: Each applicator was modeled in MCNP5 based on specifications provided by the manufacturer. A collision kerma tally was used to determine the kerma distributions at the surface and at depth in water. Surface dose profiles and fractional depth dose curves in water were generated for each applicator. Results: Initial assessment of the surface dose distributions show a peak at the center of the applicator with a rapid fall off to the edges. A prominent dip in the surface dose profile is observed with the vertical entry applicators due to the shorter source-to-surface (SSD) distance. The penumbra for both the horizontal and vertical applicators was 1.04cm, despite differences in the SSD. Conclusions: The surface dose distributions and fractional depth dose curves generated from this initial Monte Carlo investigation show characteristics typical for applicators of this type. This work expands the knowledge about output characteristics of these kinds of applicators, leading to the ultimate goal of a clinically relevant dosimetry protocol for clinicians. Conflict of Interest: Varian Medical Systems, Inc. provided sources and applicators.

Assessing the Assignation of Public Subsidies: Do the Experts Choose the Most Efficient R&D Projects?

The implementation of public programs to support business R&D projects requires the establishment of a selection process. This selection process faces various difficulties, which include the measurement of the impact of the R&D projects as well as selection process optimization among projects with multiple, and sometimes incomparable, performance indicators. To this end, public agencies generally use the peer review method, which, while presenting some advantages, also demonstrates significant drawbacks. Private firms, on the other hand, tend toward more quantitative methods, such as Data Envelopment Analysis (DEA), in their pursuit of R&D investment optimization. In this paper, the performance of a public agency peer review method of project selection is compared with an alternative DEA method.

The first Italian research assessment exercise: A bibliometric perspective

In December 2003, seventeen years after the first UK research assessment exercise, Italy started up its first-ever national research evaluation, with the aim to evaluate, using the peer review method, the excellence of the national research production. The evaluation involved 20 disciplinary areas, 102 research structures, 18,500 research products and 6661 peer reviewers (1465 from abroad); it had a direct cost of 3.55 millions Euros and a time length spanning over 18 months. The introduction of ratings based on ex post quality of output and not on ex ante respect for parameters and compliance is an important leap forward of the national research evaluation system toward meritocracy. From the bibliometric perspective, the national assessment offered the unprecedented opportunity to perform a large-scale comparison of peer review and bibliometric indicators for an important share of the Italian research production. The present investigation takes full advantage of this opportunity to test whether peer review judgements and (article and journal) bibliometric indicators are independent variables and, in the negative case, to measure the sign and strength of the association. Outcomes allow us to advocate the use of bibliometric evaluation, suitably integrated with expert review, for the forthcoming national assessment exercises, with the goal of shifting from the assessment of research excellence to the evaluation of average research performance without significant increase of expenses.

Citation analysis as a measure of the impact of research

The research-related performance of universities, as well as that of individual researchers, is increasingly evaluated through the use of objective measures, or metrics, which seek to support or in some cases even replace more traditional methods of peer review. In particular there is a growing awareness in research communities, government organisations and funding bodies around the concept of using evaluation metrics to analyse research citations. The tools available for ‘citation analysis’ are many and varied, enabling a quantification of scientific quality, research impact and prestige. However there is increasing concern regarding the potential misuse of such tools, which all have potential limitations in certain research disciplines. This article uses ‘real world’ examples from radiography research and scholarship to illustrate the range of currently available citation analysis tools. It explores the academic debate surrounding their strengths and limitations, and identifies the potential impact of citation analysis on the radiography research community. The article concludes that citation analysis is a valuable tool for researchers to use for personal reflection and research planning, yet there are inherent dangers if it is used inappropriately. Whilst citation analysis can give objective information regarding an individual, research group, journal or higher education institution, it should not be used as a total substitute for traditional qualitative review and peer assessment.

A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments

National exercises for the evaluation of research activity by universities are becoming regular practice in ever more countries. These exercises have mainly been conducted through the application of peer-review methods. Bibliometrics has not been able to offer a valid large-scale alternative because of almost overwhelming difficulties in identifying the true author of each publication. We will address this problem by presenting a heuristic approach to author name disambiguation in bibliometric datasets for large-scale research assessments. The application proposed concerns the Italian university system, comprising 80 universities and a research staff of over 60,000 scientists. The key advantage of the proposed approach is the ease of implementation. The algorithms are of practical application and have considerably better scalability and expandability properties than state-of-the-art unsupervised approaches. Moreover, the performance in terms of precision and recall, which can be further improved, seems thoroughly adequate for the typical needs of large-scale bibliometric research assessments.

OMIPs—Orchestrating multiplexity in polychromatic science

CYTOMETRY is spearheading advances in quantitative cell based research with its technological emphasis on flow and image based cytometric assays. However, it is not only the innovative technologies but also innovations in increasing the quality and reliability of scientific publications. The prominent example is the MIFlowCyt standard that has been published 2 years ago (1) defining the minimal information that is needed to understand and repeat flow cytometry-based experiments. As of today, two manuscripts were published that were prepared in compliance with MIFlowCyt. One by Blimkie et al. (2) and the second now by Heimbeck and colleagues (this issue: page 823). Both manuscripts serve as prototype papers for authors to make their upcoming work in agreement with the rules. To ease their burden, our team has developed a checklist (this issue: page 813) that simplifies MIFlowCyt adaptation of a new manuscript. In this issue is the inauguration of a new publication type for Cytometry Part A, Optimized Multicolor Immunofluorescence Panel (OMIP). OMIPs are designed to fill a gap in the literature: a mechanism for communicating and disseminating complex panels of reagents designed to interrogate phenotypic or functional aspects of cells. Since panels are basically methodological in nature, it is extravagant to devote an entire research paper to a single panel and most journals are appropriately loathe to do so. Similarly, including panel development in the eventual biological research publication is made difficult by publication constraints. Nonetheless, the development and optimization of these panels can be very complex, time-consuming, and laborious, akin to optimization of any new technique in a laboratory; disseminating that information is highly valuable. As proposed, OMIPs will occupy only two printed pages, with a significant amount of supplemental information. Hence, the OMIP becomes a parsimonious publication mechanism by which information about the panel optimization and use can be communicated. By doing so in a peer-reviewed, citable publication, those who have carried out this significant effort can be suitably recognized for their work. In addition, the OMIP can serve to dramatically shorten development efforts in labs that are initiating a new immunophenotyping study, by providing a suitably-optimized panel or, at a minimum, a significant body of information from which a suitable panel can be built. OMIPs embrace a broad field of scientific and clinical applications including clinical immunology (3; and Alamooti et al., this issue: page 890), leukemia (Xie et al. this issue: page 840), cytokine production (4), phenotyping of stem and progenitor cells (5; and Estes et al., this issue: page 831), and leukocytes (Heimbeck et al., this issue: page 823). It should be stressed that OMIPs are not solely in the realm of flow cytometry. Polychromatic microscopes like the laser scanning cytometer (Zoog et al.; this issue: page 849) are becoming more common; panels optimized for fluorescence microscopy are timely and welcome. Image cytometry technologies may also include labeling techniques that apply sequential restaining to obtain highly complex patterns of cellular phenotype and function such as Toponomics (6) or chip based explorative cytometry (7).

Complex research on undeclared work : theoretical aspects and empirical application in Lithuania

Relevance of undeclared work as a scientific issuebecomes evident through the fact that this phenomenon isinsufficiently investigated (no single concept is agreedupon), besides, it is difficult to estimate its extent inquantitative terms. The extent of undeclared work is notreflected in official statistics, thus measurement of changein the officially registered level of unemployment mightproduce the image of unemployment as an economic andsocial issue, which fails to represent reality.The article addresses the issue pertaining toambiguous treatment of the concept “undeclared work” inscientific and special researches, analyses cases when theconcept is treated as identical to other employment andunemployment ideas, and explores quantitative assessmentaspects of undeclared work.Scientific literature and applied research uses differentterminology while describing undeclared work (namely,undeclared work, illegal work and hidden unemployment)and provides various definitions. The chosen concept ofundeclared work (as a structural part of shadow economy)depends on objectives of a research as well as the conceptof shadow economy.In this article, the concept of the “shadow economy”contains economic activities undertaken in spite oflegislative requirements and restrictions even though suchactivities are subject to rigorous regulation and control(Krumplyte 2008, 2009a, 2009b, 2009c, 2010). In thisresearch, undeclared work is considered in the context oftax non-compliance, disassociating from economic activitiesattributable to unregulated economy (self-employed naturalpersons, legally not required to register their activity andpay taxes) as well as criminal activities. This article definesthe undeclared work as an economic activity (comprisinglabour relations in enterprises as well as self-employed),which is carried out in violation of legislative requirementswith the purpose of tax non-compliance.Phenomena of undeclared work may be classified intotax avoidance and tax evasion categories. Tax avoidance isperceived as reduction or nonpayment of payable taxes inthe form of reduction or elimination of tax base of acertain tax with the help of artificially created businessschemes or sequences of transactions Tax evasion isattributed to the group of the most serious offences, i.e. taxnon-compliance in the form of concealed economic activity(Krumplyte, 2009b, 2009c). Under the given context, aclassification of undeclared work phenomena in Lithuaniawas designed in the course of the empiric research.Evaluating the trends of undeclared work in a complexmanner, the authors suggest identifying indicatorspertaining to distribution of each phenomenon and theextent of formally undocumented income. The distributionindicator of a phenomenon indicates a portion of economicoperators who were involved in a certain economic activityand participated in shadow economy. The indicatorshowing the extent of officially undocumented incomeindicates the portion of income within a certain type oftotal income per one economic operator, not included intoofficial accounting documents.The article offers the results of the empiric research(delivered following the peer review method) ondistribution of undeclared work phenomena in Lithuaniaand trends (of the periods 2006-2008 and 2009-2011)pertaining to the extent of undeclared income (employmentincome and earnings received from self-employment).