Peer Review Method Research Articles

Assessing the assignation of public subsidies: do the experts choose the most efficient R&D projects?

The implementation of public programmes to support business research and development (R&D) projects requires the establishment of a selection process. This selection process faces various difficulties, which include the measurement of the impact of the R&D projects as well as selection process optimisation among projects with multiple, and sometimes incomparable, performance indicators. To this end, public agencies generally use the peer review method, which, while presenting some advantages, also demonstrates significant drawbacks. Private firms, on the other hand, tend toward more quantitative methods, such as data envelopment analysis (DEA), in their pursuit of R&D investment optimisation. In this paper, the performance of a public agency peer review method of project selection is compared with an alternative DEA method, with the objective to show some shortcomings of current procedures of ex-ante and ex-post evaluation of R&D subsidies.

Regulatory Models and the Environment: Practice, Pitfalls, and Prospects

Computational models support environmental regulatory activities by providing the regulator an ability to evaluate available knowledge, assess alternative regulations, and provide a framework to assess compliance. But all models face inherent uncertainties because human and natural systems are always more complex and heterogeneous than can be captured in a model. Here, we provide a summary discussion of the activities, findings, and recommendations of the National Research Council's Committee on Regulatory Environmental Models, a committee funded by the U.S. Environmental Protection Agency to provide guidance on the use of computational models in the regulatory process. Modeling is a difficult enterprise even outside the potentially adversarial regulatory environment. The demands grow when the regulatory requirements for accountability, transparency, public accessibility, and technical rigor are added to the challenges. Moreover, models cannot be validated (declared true) but instead should be evaluated with regard to their suitability as tools to address a specific question. The committee concluded that these characteristics make evaluation of a regulatory model more complex than simply comparing measurement data with model results. The evaluation also must balance the need for a model to be accurate with the need for a model to be reproducible, transparent, and useful for the regulatory decision at hand. Meeting these needs requires model evaluation to be applied over the "life cycle" of a regulatory model with an approach that includes different forms of peer review, uncertainty analysis, and extrapolation methods than those for nonregulatory models.

Peer Review and Manuscript Management in Scientific Journals: Guidelines for Good Practice

Peer Review and Manuscript Management in Scientific Journals: Guidelines for Good Practice

Learning effects of thematic peer-review: A qualitative analysis of reflective journals on spiritual care

This study describes the learning effects of thematic peer-review discussion groups (Hendriksen, 2000. Begeleid intervisie model, Collegiale advisering en probleemoplossing, Nelissen, Baarn.) on developing nursing students' competence in providing spiritual care. It also discusses the factors that might influence the learning process. The method of peer-review is a form of reflective learning based on the theory of experiential learning (Kolb, 1984. Experiential learning, Experience as the source of learning development. Englewoods Cliffs, New Jersey, Prentice Hill). It was part of an educational programme on spiritual care in nursing for third-year undergraduate nursing students from two nursing schools in the Netherlands. Reflective journals (n=203) kept by students throughout the peer-review process were analysed qualitatively The analysis shows that students reflect on spirituality in the context of personal experiences in nursing practice. In addition, they discuss the nursing process and organizational aspects of spiritual care. The results show that the first two phases in the experiential learning cycle appear prominently; these are 'inclusion of actual experience' and 'reflecting on this experience'. The phases of 'abstraction of experience' and 'experimenting with new behaviour' are less evident. We will discuss possible explanations for these findings according to factors related to education, the students and the tutors and make recommendations for follow-up research.

An organizational framework and strategic implementation for system-level change to enhance research-based practice: QUERI Series.

BackgroundThe continuing gap between available evidence and current practice in health care reinforces the need for more effective solutions, in particular related to organizational context. Considerable advances have been made within the U.S. Veterans Health Administration (VA) in systematically implementing evidence into practice. These advances have been achieved through a system-level program focused on collaboration and partnerships among policy makers, clinicians, and researchers.The Quality Enhancement Research Initiative (QUERI) was created to generate research-driven initiatives that directly enhance health care quality within the VA and, simultaneously, contribute to the field of implementation science. This paradigm-shifting effort provided a natural laboratory for exploring organizational change processes. This article describes the underlying change framework and implementation strategy used to operationalize QUERI.Strategic approach to organizational changeQUERI used an evidence-based organizational framework focused on three contextual elements: 1) cultural norms and values, in this case related to the role of health services researchers in evidence-based quality improvement; 2) capacity, in this case among researchers and key partners to engage in implementation research; 3) and supportive infrastructures to reinforce expectations for change and to sustain new behaviors as part of the norm. As part of a QUERI Series in Implementation Science, this article describes the framework's application in an innovative integration of health services research, policy, and clinical care delivery.ConclusionQUERI's experience and success provide a case study in organizational change. It demonstrates that progress requires a strategic, systems-based effort. QUERI's evidence-based initiative involved a deliberate cultural shift, requiring ongoing commitment in multiple forms and at multiple levels. VA's commitment to QUERI came in the form of visionary leadership, targeted allocation of resources, infrastructure refinements, innovative peer review and study methods, and direct involvement of key stakeholders. Stakeholders included both those providing and managing clinical care, as well as those producing relevant evidence within the health care system. The organizational framework and related implementation interventions used to achieve contextual change resulted in engaged investigators and enhanced uptake of research knowledge. QUERI's approach and progress provide working hypotheses for others pursuing similar system-wide efforts to routinely achieve evidence-based care.

Research evaluation per discipline: a peer-review method and its outcomes

This paper describes the method for ex-post peer review evaluation per research discipline used at the Vrije Universiteit Brussel (VUB) and summarizes the outcomes obtained from it. The method produces pertinent advice and triggers responses - at the level of the individual researcher, the research team and the university's research management - for the benefit of research quality, competitivity and visibility. Imposed reflection and contacts during and after the evaluation procedure modify the individual researcher's attitude, improve the research teams' strategies and allow for the extraction of general recommendations that are used as discipline-dependent guidelines in the university's research management. The deep insights gained in the different research disciplines and the substantial data sets on their research, support the university management in its policy decisions and in building policy instruments. Moreover, the results are used as a basis for comparison with other assessments, leading to a better understanding of the possibilities and limitations of different evaluation processes. The peer review method can be applied systematically in a pluri-annual cycle of research discipline evaluations to build up a complete overview, or it can be activated on an ad hoc basis for a particular discipline, based on demands from research teams or on strategic or policy arguments.

Peer Review of Abstracts Submitted to the Graduate Student and New Evaluators Topical Interest Group for the 2006 American Evaluation Association Conference

Peer review is an umbrella term that refers to a class of selection and oversight practices, including the familiar mechanisms of the review of proposals submitted for funding, of manuscripts for scholarly publications, and of personnel qualifications and portfolios for selection and promotion. Peer review has long been a cornerstone of modern scientific method premised on the assumption that those within a discipline are best suited to assess the work of others within that field. As such, it is also frequently employed to evaluate proposals submitted for professional meetings such as the annual conference of the American Evaluation Association (AEA). This paper presents a blind peer review method developed by AEA’s Graduate Student & New Evaluators (GS&NE) Topical Interest Group (TIG) in an effort to construct an impartial and reliable process in proposal selection. Implications for conference review processes, AEA, and the field of evaluation in general are discussed.

Applying Total-Quality-Management Principles to Improving Engineering Education

Peer review of teaching is assumed to be an excellent tool, next to student evaluation, to measure the quality of a professor. This argument builds on the fact that only peers can understand the material of the engineering discipline and are up-to-date with the practice. On the other hand, virtually no engineering professor has had any training in being an educator, never having passed a course in pedagogy or instruction. This paper will first review this inconsistency before introducing the substance of scholarly teaching. The main section, however, discusses a new and dynamic model for a peer review of teaching that introduces a process of constant improvement. It is believed that replacing the old method of peer review is vital in persuading the engineering professoriate to implement the changes called for by many blue-ribbon committees. However, this change has to be framed by an institutional culture that provides the incentives for continuous improvements of teaching by an ongoing increase of the competence level of teachers who, today, lack any significant training in education.

Pay for Performance: Rationale and Potential Implications for Urology

Pay for performance represents a new paradigm for physician reimbursement based on the value based purchasing of health care services. Government and private payers have expressed an interest in moving toward this system with several pay for performance programs already in place. The rationale behind this initiative and what it means for the practicing urologist are discussed. MEDLINE and Internet based research focusing on the topics of health care quality, measures used to implement pay for performance, and private and public sector experience with pay for performance to date were reviewed. Health care quality can be assessed through 3 types of measures, including structure, process and outcome. Structure measures involve the environment where services are provided, whereas process measures capture how a particular provider delivers health care. Outcome assessment involves the results of the services provided. These measures are best used when they are used in coordination with each other, and when they are risk adjusted. Most pay for performance systems in use today are based on these measures. However, there are little data that show whether this reimbursement paradigm actually improves the quality of heath care provided. Many questions remain regarding the implementation of a pay for performance system in the field of urology. Government and private payers are motivated to implement pay for performance. However, specific evidence based metrics for urology that fairly and accurately define quality are currently lacking. Given that implementation of a nationwide pay for performance system appears to be inevitable, urology involvement in the development and implementation of these health care quality metrics is essential.

Screen media practice research and peer review

This paper is based upon reflections arising from the preparation of ScreenWork as the first practice based Screen Media publication DVD (Vol. 1, Intellect Press, forthcoming at the time of writing). It addresses the development of appropriate methods of peer review for screen media practice.

Industry-Sponsored Research: Author's Reply

In his letter [1] Dr. Mansfield has raised a debatable issue: “If the CRC [Collegiate Research Council] is a single international monopoly how could the risk of corruption and inefficiency be managed?” By no means would the CRC be an international monopoly; in the proposal it was stated that the CRC would be certified by the appropriate health authorities of any country. Moreover, several CRCs could be established, as long as the participants were nominated by prestigious academic institutions and the CRC was certified by the federal regulatory agencies. The risk of corruption and inefficiency would be remote for the following reasons. The constituents of the CRC would be distinguished scholars, appointed independently by leading medical institutions and universities. Contrary to Dr. Mansfield's and Dr. Prabhakar's fears [1,2], this council would not require a substantial infrastructure, since its main goal would be to meet periodically, to select the investigators and institutions that would conduct the clinical protocol submitted during that period, and to review the results generated by previous trials. This task could be expedited through the peer review method. The administrative implementation necessary to carry out the decisions of the CRC might be done by existing institutions (e.g., the US Food and Drug Administration), without the formation of an additional bureaucratic body. In this way, the CRC would be strictly maintained as an academic and scientific core whose opinions and sanctions were respected by the individuals and institutions involved in the execution of any given protocol of drug testing. The CRC would not have “bureaucratic bodies,” as it would function only as a collegiate board to select who would do the testing and where a given protocol of drug testing would be conducted. The research funds provided by the pharmaceutical company would not be administered by the CRC, but by the corresponding institution in charge of the specific research or by the administrators appointed by the CRC. To the question of who would fund the CRC the answer is simple: a very small percentage of the total cost of the protocol would be used to create a fund to cover travel expenses and stipends for the scholars participating in the CRC. In response to the statement that the CRC “will consume significant amounts of time and human resources” [2], I do not see why this should be the case. I believe that the members of the CRC could meet once or twice a month and propose, in a single day, one or various candidates for a given protocol. The operation of the CRC would be somewhat similar to that of the editorial board of scientific journals who meet periodically to select reviewers for scientific papers and to evaluate the results of reviews. Once the scholars of the CRC had selected the potential investigators or institutions they would be invited to conduct the research; this task could be accomplished by commissioned personnel independent from the regulatory agency. With regard to the prices of drugs, I think that Dr. Prabhakar has not chosen a comparable example. In fact, hospital charges and physicians' consultations are not controlled in most countries; that is because these prices are usually set—very much as for many other services and goods—by production costs and competition. As mentioned in my proposal, this is not the case for pharmaceutical substances.

Effects of visitation among allied health professionals

Visitation is a method for external peer review. The goal is to improve the quality of patient care by giving feedback on quality of competence and performance of a professional during a practice visit. Feedback is offered as recommendations for improvement. This study aims to evaluate the effects of visitation and to determine which factors are related to the effectiveness of visitation. Members of seven allied health professions in the Netherlands: dieticians, exercise therapists, physiotherapists, dental hygienists, occupational therapists, podiatrists, and radiology assistants. Evaluation questionnaires were sent to 151 allied health professionals who had participated in visitation. The questions included all practice management aspects that had been assessed during the practice visit. The effects of visitation were studied at three levels: change in awareness of weak and strong aspects of competence and performance, intention to carry out recommendations, and actual improvements. Results showed effects of visitation on all three levels. Respondents intended to carry out two-thirds of the recommendations. Visitation led to a better awareness of weak points on 36% of the aspects and better awareness of strong points on 53% of the aspects of practice management. Young respondents reported more changes in awareness than older respondents. Actual improvements were carried out on 33% of the aspects. Visitation is an effective method to stimulate quality improvement in allied health professionals. Although changes in awareness more often occurred in younger respondents, actual improvements were made by all respondents.

ECrystals: a route for open access to small molecule crystal structure data

Recently the funding councils in the UK stated that ‘the data underpinning the published results of publically-funded research should be made available as widely and rapidly as possible’. Thirty years ago a research student would present about five crystal structures as their PhD thesis, however with modern technologies and good crystals this can now be achieved in the timespan of a single morning. This increase in pace of generation further exacerbates a problem in the communication of the results. Additionally, the general route for the publication of a crystal structure report is coupled with and often governed by the underlying chemistry and is therefore subject to the lengthy peer review process and tied to the timing of the publication as a whole. This bottleneck in the dissemination of crystal structure data hinders the potential growth of databases and the data mining studies that are reliant on these collections. Just 500,000 small unit cell crystal structures are available in the CSD, ICSD & CRYSMET databases, while it is estimated that at least twice this number have been determined in research laboratories and are likely to remain unpublished. In addition, publication in the mainstream literature still offers only indirect (and often subscription controlled) access to this data. The work of the eBank-UK project (http://www.ukoln.ac.uk/projects/ebank-uk/) has addressed this problem by establishing an institutional data repository that supports, manages and disseminates metadata relating to the crystal structure data it contains (i.e. all the files generated during a crystal structure determination). This process alters the traditional method of peer review by openly providing crystal structure data where the reader or user may directly check correctness and validity. The repository (http://ecrystals.chem.soton.ac.uk) makes available all the raw, derived and results data from a crystallographic experiment with little further researcher effort after the creation of a normal completed structure in a laboratory archive. Not only does this approach allow rapid release of crystal structure data into the public domain, but it can also provide mechanisms for value added services that allow rapid discovery of the data for further studies and reuse, whilst ownership of the data is retained by the creator. The details of the preparation of data, upload process, files supported and automatic report generation will be presented. Additionally, the process whereby metadata relating to each archive entry is disseminated, using current Digital Libraries technologies, for discovery and reuse by will be summarised. Strategies for the installation of archives at new sites, the construction of harvesting and aggregator services and the interaction with crystallographic data holding bodies, such as IUCr and CCDC, will also be outlined. Additionally links to educational tools, specifically the Schools eMalaria project (http://emalaria.soton.ac.uk), will also be presented.

Using Peer Review to Improve Student Writing in Business Courses

Student peer review has proven an effective technique for improving student writing in both English and business communication classes, yet the technique is not widely used in business courses other than business communication. In this article, the author discusses using student peer review in business classes to improve students' final written submissions. In doing so, the author discusses the definition of peer review, the benefits of peer review, the reasons peer review helps improve student writing, the common peer review methods, the ways to incorporate peer review into business courses, and the disadvantages of peer review. Samples of different types of peer review are also included.

A method for peer review, a peer review for methods

A method for peer review, a peer review for methods

The Impact of E-Learning in Medical Education

The authors provide an introduction to e-learning and its role in medical education by outlining key terms, the components of e-learning, the evidence for its effectiveness, faculty development needs for implementation, evaluation strategies for e-learning and its technology, and how e-learning might be considered evidence of academic scholarship. E-learning is the use of Internet technologies to enhance knowledge and performance. E-learning technologies offer learners control over content, learning sequence, pace of learning, time, and often media, allowing them to tailor their experiences to meet their personal learning objectives. In diverse medical education contexts, e-learning appears to be at least as effective as traditional instructor-led methods such as lectures. Students do not see e-learning as replacing traditional instructor-led training but as a complement to it, forming part of a blended-learning strategy. A developing infrastructure to support e-learning within medical education includes repositories, or digital libraries, to manage access to e-learning materials, consensus on technical standardization, and methods for peer review of these resources. E-learning presents numerous research opportunities for faculty, along with continuing challenges for documenting scholarship. Innovations in e-learning technologies point toward a revolution in education, allowing learning to be individualized (adaptive learning), enhancing learners' interactions with others (collaborative learning), and transforming the role of the teacher. The integration of e-learning into medical education can catalyze the shift toward applying adult learning theory, where educators will no longer serve mainly as the distributors of content, but will become more involved as facilitators of learning and assessors of competency.

Technology and New Directions in Professional Development: Applications of Digital Video, Peer Review, and Self-Reflection

Pedagogical applications of technology—including digital video, course management systems, online discussion forums, and CD-ROM compilations—for development of professional skills were tested in three distinct professional graduate programs. Role-playing, peer review, and self-reflection instructional methods were technologically enhanced by creating customized interfaces for use in counseling, library science, and teacher education graduate programs. Implications for further iterations of integrating technology and professional development, potential uses in a variety of disciplines, and ways in which the process can be improved are also discussed.

A Method to Increase Student Interaction Using Student Groups and Peer Review over the Internet

A method of peer review for student groups is proposed. In this method, groups of students publish their assignments results over the Internet. A fellow student group reviews their work and publishes their findings (on the Internet). Finally, the two groups debate their points of view in front of the class. The debate and healthy competition among groups give the students a chance to learn how to give and receive criticism in a constructive way. This should increase the students' ability to interact and work in groups, an important skill for computer science professionals.

Interval Cancer Peer Review in East Anglia: Implications for Monitoring Doctors as Well as the NHS Breast Screening Programme

AIM: To devise a method for reviewing interval cancers that will both educate radiologists and facilitate improvement of breast screening practice. To evaluate different methods for case classification to find one that best serves this purpose.METHOD: The method of peer review and the means by which interval cancers are classified is described. The way in which cases are designated false-negative is an issue of acceptability for radiologists, and so three different methods are evaluated. Each is applied to the data set collected in this region over a 3-year period.RESULTS: For cases read by five readers, when a consensus method was used for classifying cases, the proportion of cases classified as false-negative was 14%. Using a method in which only one of the five readers had to classify a case as false-negative for it to be categorized as such, the proportion of false-negative cases rose to a maximum of 38%. The minimum proportion of cases that could be considered to be false-negative was 6% and was obtained when all five readers had to classify a case as false-negative for it to be so categorized. Consistent with its majority viewpoint, the consensus method gave results for proportions of total cases classified as false-negative which were similar to those given by methods in which cases are classified as false-negative if either three of five readers, or at least 60% of readers, classified it as such.CONCLUSION: For the peer review method to achieve its dual aims of educating radiologists and auditing performance, the participating radiologists must share ownership of the results and view the analysis as fair. The method used to classify interval cancers as false-negative will influence the number so classified. A consensus method has been found to give a result that is both fair and acceptable to our radiologist. Using this method 16% of all reviewed cases were classified as false-negative and 60% as true interval cancers.Britton, P. D. (2001). Clinical Radiology56, 44–49.

35th anniversary of the Journal of Pediatric Surgery (1966-2001): A look into the future

35th anniversary of the Journal of Pediatric Surgery (1966-2001): A look into the future