Multiple Evaluation Methods Research Articles

Developing a decision support system for tobacco use counselling using primary care physicians

Clinical decision support systems (CDSS) have the potential to improve adherence to guidelines, but only if they are designed to work in the complex environment of ambulatory clinics as otherwise physicians may not use them. To gain input from primary care physicians in designing a CDSS for smoking cessation to ensure that the design is appropriate to a clinical environment before attempts to test this CDSS in a clinical trial. This approach is of general interest to those designing similar systems. Design and approach: We employed an iterative ethnographic process that used multiple evaluation methods to understand physician preferences and workflow integration. Using results from our prior survey of physicians and clinic managers, we developed a prototype CDSS, validated content and design with an expert panel, and then subjected it to usability testing by physicians, followed by iterative design changes based on their feedback. We then performed clinical testing with individual patients, and conducted field tests of the CDSS in two primary care clinics during which four physicians used it for routine patient visits. The CDSS prototype was substantially modified through these cycles of usability and clinical testing, including removing a potentially fatal design flaw. During field tests in primary care clinics, physicians incorporated the final CDSS prototype into their workflow, and used it to assist in smoking cessation interventions up to eight times daily. A multi-method evaluation process utilising primary care physicians proved useful for developing a CDSS that was acceptable to physicians and patients, and feasible to use in their clinical environment.

Towards a Multiple Criteria Decision Making framework for common stock portfolio selection

An integrated multiple criteria methodological framework is proposed to support decisions that concern the construction and selection of common stock portfolios. The proposed framework combines discrete multiple criteria evaluation methods with a non-linear continuous optimisation model. The preferences and experiences of professionals and experts in the field of portfolio management are strongly taken into consideration, through all the stages of the process. The validity of the above methodology is tested through a large scale illustrative application on the stocks that constitute the FTSE-140 index of the Athens Stock Exchange.

Survey: State of the Art in NDE Data Fusion Techniques

As the requirements on the accuracy of nondestructive testing and evaluation are increasing, multiple nondestructive evaluation (NDE) methods are often employed to increase the reliability and reduce the uncertainty of the testing and evaluation. The need to process and analyze data from multiple sources uses the technique named data fusion, which has been growing rapidly in the NDE community in recent years. This paper reviews the progress in NDE data fusion techniques and examines the mathematical fusion algorithms, which include but are not limited to optimization methods, multiresolution approaches, heuristic methods, probabilistic methods, and scientific visualization. In the light of recent applications in NDE data fusion, important issues in handling the acquired data and applying the fusion algorithms are identified and discussed. A generic framework to apply the NDE data fusion process is described. The study on fusion of advanced NDE and sensor techniques opens up prospects for the diagnostic and prognostic health monitoring applications in the future.

Testing of Intraocular Drugs for Clinical Use

In recent times, the use of intraocular drugs has become frequent in ophthalmology, in particular in the treatment of retinal diseases. Such treatments are most often off-label use of compounds that have been developed for systemic use, not as primary intraocular agents. The modern history of this off-label use began with drugs for endophthalmitis and then progressed to anticytomegaloviral drugs for human immunodeficiency virus infection‐associated acute retinal necrosis syndrome. More recently, triamcinolone has come into widespread use for patients with macular edema from a variety of causes. In addition to therapeutic intravitreal agents, intravitreal drugs have become popular as surgical adjuvants to pars plana vitrectomy. Compounds such as ICG, trypan blue, and triamcinolone have been used to assist visualization of the internal limiting membrane, epiretinal membranes, and vitreous cortex and have been widely considered useful adjuncts to vitreoretinal surgery. One of the problems with the use of intraocular drugs, particularly if not primarily developed for this purpose, is the conflicting publications in the scientific literature about their safety. For example, a search of PubMed for the use of ICG in vitrectomy surgery returned 146 papers published in the area. These included clinical studies in patients, in vivo studies in animals, and in vitro studies of the dye in tissue culture. The conclusions were widely conflicting, with some finding that ICG is safe and others that it is toxic. Some investigators have decided that if the retina is intact (i.e., no macular hole), then the use of ICG is safe, but if the substance comes in contact with the retinal pigment epithelium (RPE), toxicity ensues. The multiple methods of evaluation of drugs or compounds to be used in the eye lead to much confusion about their toxicity. For example, ICG is widely used to aid in visualizing the internal limiting membrane of the retina so that it can be removed in eyes with macular holes and macular edema, yet there is controversy regarding its potential for toxicity. In tissue culture systems and in some studies in which in vivo evaluation systems were used, the dye has been found to be either safe or toxic at similar concentrations. 1,2 Some investigators have attributed toxicity to the osmolarity of the solution and have suggested that isosmolar preparations of ICG are not toxic. Others have concluded that toxicity may be light dependent, and still others have concluded that the compound is toxic only to the RPE cells and not the retina. We are in the uncomfortable position of having a dye in use as a surgical adjuvant whose safety seems to be controversial. Standardized methods of evaluation that are universally acceptable seem to be lacking. One of the problems in evaluating such drugs and compounds is the lack of a standard agreed-on methodology. Certainly tissue culture is one way to evaluate such drugs, but the results may be misleading. The intraocular milieu is complex and difficult to replicate in a tissue culture system. For example, testing of the toxicity of drugs injected into the vitreous using RPE cell lines may be irrelevant, as most of the drug is in contact with the retinal surface, not the RPE. It is difficult to replicate the clinical situation in vivo in tissue culture, and the relevancy of such results is questionable. Using in vivo systems or tissue culture cell lines, several groups have made efforts to evaluate the effect of triamcinolone. 3‐7 One problem is that triamcinolone is water insoluble. When injected into the human vitreous cavity, the compound may not even directly contact the retinal surface, because much of it is suspended in the liquid or formed vitreous. The effect of the drug on the retina has been reported by some observers to be beneficial in cases of macular edema due to several different diseases. Such an effect is presumably mediated by the solubilized fraction of drug; the portion of triamcinolone crystals that dissolve in the vitreous fluid. It is difficult to know if, in vivo, triamcinolone crystals are in equilibrium, what the soluble fraction of triamcinolone is in the vitreous, and precisely what concentration of soluble drug is available to

Quantitative evaluation of enterprise marketing effectiveness

Abstract Enterprises operating in a complicated, constantly changing environment should pay much more attention to the market and the user, ie to marketing. Despite the fact that the main subject of the state economy is an enterprise, the problems of marketing are mostly analysed at the level of activities. At the enterprises, these problems, which are of great practical and theoretical importance, are paid little attention. One of the main problems in this context is quantitative evaluation of marketing effectiveness. The nature of the considered processes implies that this problem can be solved by multiple criteria evaluation methods. It allows us to offer a particular methodology verified by calculations. The evaluation of some actual enterprises has shown that marketing considerably affects their commercial activities in a positive way, increasing their effectiveness (profitability).

QUANTITATIVE EVALUATION OF ENTERPRISE MARKETING EFFECTIVENESS

Enterprises operating in a complicated, constantly changing environment should pay much more attention to the market and the user, ie to marketing. Despite the fact that the main subject of the state economy is an enterprise, the problems of marketing are mostly analysed at the level of activities. At the enterprises, these problems, which are of great practical and theoretical importance, are paid little attention. One of the main problems in this context is quantitative evaluation of marketing effectiveness. The nature of the considered processes implies that this problem can be solved by multiple criteria evaluation methods. It allows us to offer a particular methodology verified by calculations. The evaluation of some actual enterprises has shown that marketing considerably affects their commercial activities in a positive way, increasing their effectiveness (profitability).

Hormesis Outperforms Threshold Model in National Cancer Institute Antitumor Drug Screening Database

Which dose-response model best explains low-dose responses is a critical issue in toxicology, pharmacology, and risk assessment. The present paper utilized the U.S. National Cancer Institute yeast screening database that contains 56,914 dose-response studies representing the replicated effects of 2189 chemically diverse possible antitumor drugs on cell proliferation in 13 different yeast strains. Multiple evaluation methods indicated that the observed data are inconsistent with the threshold model while supporting the hormetic model. Hormetic response patterns were observed approximately four times more often than would be expected by chance alone. The data call for the rejection of the threshold model for low-dose prediction, and they support the hormetic model as the default model for scientific interpretation of low-dose toxicological responses.

Evaluating Education Effectiveness

I would like to clear up a misconception that readers might derive from your News Focus story “Is the education directorate headed for a failing grade?” by J. Mervis (24 Feb., p. [1092][1]). In the story, I am quoted as saying, “Maybe NSF [National Science Foundation] education programs need to be rethought.” In fact, I believe that NSF education programs have been instrumental in creating a series of outstanding curricula for school science in the United States and have served an important role in making both scientists and schools aware of the urgent need to rethink what we mean by “science education” ([1][2]). In my conversation with your reporter, I stressed the need to create a robust body of high-quality research on science education in our schools. In this context, I suggested that the NSF rethink its requirement for formal project evaluations, by which I meant that it support a broad range of methodologies aimed at producing useful and transferable knowledge. I suggested, for example, greater attention to multisite research that is designed to discover what does and does not work, and why. I also questioned an NSF tradition of discontinuing even the best programs after 5 years, with the expectation that school districts (or others) will be able to cover the expense of continuing the programs thereafter. 1. 1.[↵][3] 1. B. Alberts , Cell 123, 739 (2005). [OpenUrl][4][CrossRef][5][PubMed][6][Web of Science][7] # {#article-title-2} The news story “Is the education directorate headed for a failing grade?” by J. Mervis (24 Feb., p. [1092][1]) confounds a number of important issues pertaining to the National Research Council's 2004 study “On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations,” chaired by Professor Confrey. First, the story blurred an important distinction between quality of evaluations and quality of curricula: We did not say that none of the curricula are effective or that none of the evaluations had merit. Rather, we found evaluations that ranged in quality from highly flawed to very imaginative, but that taken together could not support broad claims—positive or negative—about the quality of the curricula. Second, the story failed to note that many of the studies we reviewed were not evaluations but progress reports, descriptive summaries, and advocacy pieces; that they included evaluations of commercially developed as well as NSF-sponsored curricula; and that the committee did not directly examine the 19 curricula in question. Third, the article misstated Professor Confrey's views about the importance of sustained and significant federal investment in curriculum evaluation. Contrary to the story's suggestion that past investments yielded little knowledge, her point was that they have informed us about what to avoid, about how to design rigorous evaluations, and about the need for greater attention to judging effectiveness through multiple evaluation methods. [1]: /lookup/doi/10.1126/science.311.5764.1092 [2]: #ref-1 [3]: #xref-ref-1-1 View reference 1. in text [4]: {openurl}?query=rft.jtitle%253DCell%26rft.stitle%253DCell%26rft.aulast%253DAlberts%26rft.auinit1%253DB.%26rft.volume%253D123%26rft.issue%253D5%26rft.spage%253D739%26rft.epage%253D741%26rft.atitle%253DA%2Bwakeup%2Bcall%2Bfor%2Bscience%2Bfaculty.%26rft_id%253Dinfo%253Adoi%252F10.1016%252Fj.cell.2005.11.014%26rft_id%253Dinfo%253Apmid%252F16325564%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [5]: /lookup/external-ref?access_num=10.1016/j.cell.2005.11.014&link_type=DOI [6]: /lookup/external-ref?access_num=16325564&link_type=MED&atom=%2Fsci%2F312%2F5770%2F54.atom [7]: /lookup/external-ref?access_num=000233814100003&link_type=ISI

Usability as a challenge in agricultural engineering

One of the primary goals of research activities has always been to apply the newest potential results into practise. This is the case especially in engineering research. Recently, as productivity has gained importance as a quality measure for research, rapid application of the results has become even more important. Consequently, applicability has been lifted as a major criterion in the application for public funding thus promoting product development as an integrated part of research projects. No product (or research result) does have any impact if it is not taken into practical use. The end customers of research are supposed to take the developed products into active practical use. Without that phase all our efforts are put in vain. Usability is an important part of user acceptance. According to Nielsen (1993) system acceptability includes a social and a practical part. To be practically acceptable the product should be e.g. economical, compatible, reliable and useful. To be useful the product should be functionally suitable for the user’s tasks and usable. Usability includes easy learn ability, efficiency in use, remember ability, lack of errors in operation and subjective pleasure. In agricultural research there are distinct problems in usability when considering the phase oftaking the results into practical use. Of course, there are challenges also in the initial phase of research such as choosing research topics and later in the initial phase of product development. Usability, however, by far dictates user experience and thus decides if the product is taken into wide use or not. Consequently, MTT Agricultural Engineering Research has set usability and acceptability research as an important research topic. Usability in agricultural engineering is a complex issue since the context of use is variable. Mobile work is typical to agricultural producers.In this study, an example of usability evaluation is presented. Mobile work in the context of electronic control of precision combined drilling is evaluated. The research themes were:1. How great a challenge is usability in Precision Agriculture (PA)?- is it the cause for poor market penetration?2. Which usability evaluation methods are applicable to PA?- are there special issues in PA, or agricultural engineering generally, which limit the feasibilityof some methods?3. Which kind of usability problems can be detected with one selected method (heuristicevaluation)?- a demonstration of usability evaluation: the Human-Machine Interface (HMI) of a precisioncombined drillAccording to literature usability issues have not been a central issue in electronics development in agriculture. Poor experiences of unacceptable operation could be one reason for the customers not relying on new electronic control systems such as those of PA. There are multiple potential usability evaluation methods for agricultural engineering. The results from the case study performed show that heuristic evaluation is a suitable method for detecting design deficiencies in electronic control of mobile PA. To get a wider picture, further studies with other methods and applications should be done.

Physical Therapist Student Learning Through Authentic Experiences in Management: One Programʼs Interpretation of Service Learning

Background and Purpose. This article describes the implementation of a unique educational model for teaching management content explicitly utilizing experiential and service learning principles. It also highlights physical therapist student and faculty learning and growth that have occurred in the last 3 years as a result of implementing these strategies into the program's curriculum. This model was developed in an effort to meet the Nazareth College mission of community engagement, to achieve program goals of infusing more curricular integration and experiential learning into the program, and to enhance program objectives that were previously only met minimally. Method/Model Description and Evaluation. In this model, physical therapist students assume dual roles as student clinicians and student managers in the same semester to provide physical therapy services in 3 notfor-profit on-campus and off-campus learning experiences. Data was collected each year through multiple means: course evaluations; a student satisfaction survey; end of semester feedback sessions, including reviews by both faculty and students; required reflective journaling; client satisfaction surveys; focus groups; and graduate surveys conducted 1 year post graduation. Outcomes. Multiple methods of evaluation yielded the following outcomes: Student learning objectives were met and often exceeded; students identified and resolved management challenges; faculty learning occurred; community demand for services was greater than predicted; an appreciation for service learning as a teaching method improved; students defined views of “success”; and empowerment promoted professional development. Discussion and Conclusion. Innovative experiential learning experiences were instrumental in building skills, such as leadership, teamwork, communication, marketing, finance, and conflict resolution in one program. Faculty, students, and community providers have gained an appreciation for experiential learning experiences and identified areas for growth and expansion. This model presents an opportunity for changing perspectives among physical therapy educators and physical therapy practitioners regarding management content in professional (entry-level) physical therapist education. The model has potential for replication in other programs, with modification for specific community needs.

Establishing an Internet-Based Tobacco-Control Network for Czech Health Professionals

This article describes the background, implementation, and evaluation of an Internet-based health promotion network in the Czech Republic. The network was intended to build the indigenous capacity of Czech health professionals in the use of the Internet for tobacco control to organize and coordinate public health campaigns, heighten public awareness, and catalyze policy change. Multiple methods of evaluation included three needs assessment surveys, three training workshop evaluations, an Internet survey of participating member organizations, and quarterly reports from the project partners. The evaluation showed that an Internet-based network was established with 40 health professionals from 30 organizations. A grassroots heart disease prevention and tobacco-control advocacy group was formed. Network participants utilized and continue to utilize the Internet to communicate, research, coordinate outreach, and advocate tobacco-control issues.

Site selection for waste disposal through spatial multiple criteria decision analysis

This article deals with the application of spatial multiple criteria evaluation (SMCE) concepts and methods to support identification and selection of proper sites for waste disposal. The process makes use of a recently developed SMCE module, integrated into ITC`s existing geographic information system called ILWIS. This module supports application of SMCE in planning and decision making processes through several compensatory and non-compensatory approaches, allowing inclusion of the spatial and thematic priority of decision makers. To demonstrate the process, a landfill site selection problem around the town of Chinchina, in Colombia, is used as an example. Based on different objectives, a spatial data set consisting of several map layers, e.g., land use, geological, landslide distribution, etc., is made available and used for modeling the site selection process.

The Relationship Between Awake Positioning and Motor Performance Among Infants Who Slept Supine

The purpose of this study was to describe the gross motor development of infants who slept supine and spent different amounts of time in the prone position when awake. Thirty infants who were six months old and slept supine were evaluated using the Alberta Infant Motor Scale (AIMS) and then divided into two groups: prone (n = 16) and no prone (n = 14) based on the number of times per day they were placed in prone position while awake. The prone group scored higher than the no prone group on the AIMS total scores (U = 36, p = 0.004), total percentile scores (U = 42, p = 0.003), and prone subscale scores (U = 25, p < 0.001). Gross motor performance as measured by the AIMS was more advanced in infants who slept supine and had been placed in the prone position when awake than in infants who slept supine but had limited or no experience in the prone position while awake. The influence of awake positioning needs to be considered when interpreting the developmental motor performance of infants who are six months old.

"You Talking to Me?" Exploring Voice in Self-Service User Interfaces

Automated teller machines (ATMs) are a classic example of ubiquitous computing as they pervade our everyday life and, for many, are typical artifacts of convenient, modern living. More important, most people are unaware that in using an ATM they are connected via a computer to a powerful network. Through our research reported within this article, we examine the user acceptance and usability aspects of a novel, "contactless" ATM. This prototype ATM has iris identification, speech recognition, speech synthesis, and communication to a user's personal digital assistant (PDA). In this article, we describe our experiences of and explorations with "Stella," a contactless ATM. First, we briefly discuss the background to the study in terms of technology and trends, and then overview previous usability research in the area. We report our findings from a cognitive walkthrough, the heuristic review of the prototype, focus groups, and a user trial. These results underline the need for multiple evaluation methods for novel concepts. In general, from the findings of this work, consumers believe that speech is a technology that will create more usable applications, even when faced with the relative failure of current technology to live up to their expectations or fulfill their practical needs. People are often initially resistant to the concept of iris identification and the PDA interaction, yet basic experience with a usable prototype quickly overcame people's reservations. Finally, we outline some of the underlying challenges facing voice-based interaction approaches to this form of ubiquitous, public self-service computing.

The evaluation of the effectiveness of health education interventions in clinical practice: a continuing methodological challenge.

This paper examines the methodological issues arising from an evaluation of the effectiveness of a health education project undertaken to increase Hong Kong Chinese women's knowledge of the prevention of cervical cancer and the uptake of screening. The significance of health promotion to the prevention of diseases currently affecting contemporary society has become increasingly recognized. Within the context of health promotion health education continues to provide an important preventive strategy. Indeed the leading causes of mortality such as coronary heart disease and cancer lend themselves well to health education interventions. However the evaluation of the effectiveness of health education remains complex and raises some important methodological issues. The project used a health education intervention as the major preventive strategy and employed multiple methods of evaluation to assess its effectiveness. Outcome evaluation consisted of a confidential questionnaire administered at two points in time to measure changes in health-related behaviour and knowledge. It also included the collection of data from service providers to assess changes in the uptake of cervical cancer screening. Process evaluation involved the use of focus groups with randomly selected groups of women who had participated in the health education intervention and a diary kept by the project nurse. Methodological issues identified in the evaluation of the project included the extent to which changes in health-related knowledge and behaviour could be attributed to the intervention, the sensitivity of outcome measures and challenges in developing methods of process evaluation appropriate to the target population. The findings highlight the complexity of designing effective evaluation strategies for health education and the need to consider these issues in the development of both process and outcome evaluation.

Evaluating the health‐promoting school

Examines some of the constraints on the effective evaluation of the health‐promoting school, particularly where a whole‐school approach is used, and outcomes are not behaviour‐specific. Considers the components of a successful school health intervention, and identifies some of the difficulties in achieving comprehensive evaluation. Explores the advantages of randomisation and controlled studies and identifies some limitations of these approaches, when the main aims are to examine process and non‐specific behavioural outcomes. Identifies some common errors that have been made in school‐based evaluations in the past and argues that the use of multiple methods of evaluation are necessary for a multi‐factorial whole‐school approach to teaching and learning about health. Discusses the recommendations for future evaluation by the WHO European Working Group on Health Promotion Evaluation.

Proceduralizing Teamwork in Aviation Crews

This project investigated whether appropriate procedural interventions can facilitate team process and outcomes. Specific crew procedures were designed to fit general Crew Resource Management (CRM) principles and the specific needs of a commercial airline. The procedures were trained and implemented as Standard Operating Procedure (SOP) for one fleet while a control fleet did not have the training or SOP interventions. Multiple evaluation methods used different samples of evaluators and different evaluation instruments. Evaluation data converged to confirm the positive effects of the interventions in the experimental fleet.

Design, analysis and comparison of robot learners

This paper outlines some ideas as to how robot learning experiments might best be designed. There are three principal findings: (i) in order to evaluate robot learners we must employ multiple evaluation methods together; (ii) in order to measure in any absolute way the performance of a learning algorithm we must characterise the complexity of the underlying decision task formed by the interaction of the agent, task and environment; (iii) that in fact this goal is too difficult to attain in practice, and progress in robot learning must rely on comparative work. Four methods for agent analysis are presented. These are used to analyse a robot that learns to push boxes from reinforcement. Using these techniques we have been able to show that Q(λ)-learning outperforms one-step Q-learning on a typical robot task. The differences are statistically significant. We emphasise the importance of experimental design in order to integrate the various forms of evaluation.

A Renaissance of Group Supervision in Practicum

This paper presents an educational model of group field instruction. The model enhances the quality of training social work students by offering a wider variety of teaching-learning experiences and enabling the field instructor to rely on multiple evaluation methods to assess student performances. Social work service units from a large Midwestern DVA Medical Center using this model describe the strengths and limitations of providing field supervision in the manner.

Psychosocial adaptation of orthopedically disabled Mexican children and their siblings

This article reports a study of psychosocial adaptation of orthopedically disabled children and their siblings in Mexicali, Baja California, Mexico to determine if the psychodynamic of disability documented in U.S. studies is also found in this group. This was achieved by an examination of disabled-nondisabled differences through comparison of children with disabilities and their siblings on a wide range of psychosocial variables, and assessment of the relationship of disability, school and family factors to behavior problems and self-esteem. The investigation used multiple methods of evaluation, including assessment of physical, behavioral, social, and psychological adaptation. Findings include many predicted differences between disabled and non-disabled children on measures of adjustment, self-esteem and professional expectations; negative correlation of self-esteem with disability; an increase in behavior problems with poor self-esteem; and a decrease in behavior problems with improved self-esteem and increased family and social activities.