Agile Evaluation Research Articles

Concept and Pictogram-Based User-Interface Design of a Helper Tool for People with Aphasia.

Aphasia describes the lack of the already gained ability to use language in a common way. "Language" here covers all variations of forming or understanding messages. The APH-Alarm project aims to develop a service concept that provides alternative communication options for people with Aphasia to trigger timely help when needed. It considers that a typical user may not be familiar with modern technologies and offers several simple and intuitive options. The approach is based on event detection of gestures (during daytime or in bed), movement pattern recognition in bed, and an easy-to-use pictogram-based smartphone app. Agile evaluation of the smartphone app showed a promising outcome. The idea of a versatile and comprehensive solution for aphasic people to easily contact private or public helpers based on their actions or automatic detection is promising and will be further investigated in an upcoming field trial.

Health information communication technology evaluation frameworks for pharmacist prescribing: A systematic scoping review.

Information communication technology (ICT) is instrumental in pharmacists' current practice and emerging roles. One such role is prescribing, which requires the use of clinical guidelines and documentation of decision-making, commonly via ICT. Development and refinement of ICT should be guided by evaluation frameworks that describe or measure features of ICT and its implementation. In the context of pharmacist prescribing, these evaluation frameworks should be specific to health stakeholders and the pharmacy setting. To identify ICT evaluation frameworks from health-related literature and review frameworks relevant to the development, implementation, and evaluation of pharmacist prescribing. A database search of CINAHL, Cochrane Library, EMBASE, Medline (Ovid), ProQuest, Scopus, Web of Science and grey literature was conducted, using combinations of keywords relating to 'ICT', 'utilization', 'usability', and 'evaluation framework'. Abstracts and titles were screened according to inclusion criteria. Identified evaluation frameworks were critiqued for relevance to pharmacy practice. Twenty-two articles were identified, describing the development or application of 20 evaluation frameworks. None of the frameworks was developed specifically for pharmacy practice. The Technology Acceptance Model (TAM), describing use behavior, behavior intention, perceived usefulness, and perceived ease of use, was the most widely utilized framework. The Information System Success (ISS) and Human-Organization and Technology Fit (HOT-fit) are notable evaluation frameworks that address user and organizational influences in health ICT utility, and factors of both can address the limitation of TAM. The findings call for development of an agile evaluation framework for the system under review; however, this can prove difficult due to the heterogenicity and complexity of the healthcare system, particularly contemporary pharmacy practice. While the TAM appears useful to evaluate user attitudes and intentions towards ICT, its relevance to ICT in contemporary community pharmacy practice requires exploration.

Viestinnän mittaaminen ja strateginen viestintä

Communication measurement and evaluation 
 – Goal-orientation, agility or fumbling in the dark?
 The interest of organizations in assessing stakeholder relationships and expectations, intangible assets, and social impact has grown strongly in the 21st century. This change is linked to new views on the strategic role of communication and strategic management. We examine the development and current state of the measurement and evaluation (M&E) of communication through research literature and empirical material. We suggest that the models of M&E used in organizations can also tell us about their views of strategic communication and the connection to organizational strategy. Present practices of M&E are dominated by logical models as well as summative evaluation, which are connected with a functionalist view of strategic communication. The data also provide indications of agile evaluation. Many organizations also engage in practices that we call blind measurement: metrics are not directly linked to the goals of communication.

Conversations that count: Lessons from evaluating a men’s digital mental health response during COVID-19

This paper details disruption and innovation in digital evaluation practice at Movember, as a result of the COVID-19 pandemic. The paper examines a men’s digital health intervention (DHI) – Movember Conversations – and the product pivot that was necessary to ensure it could respond to the pandemic. The paper focuses on the implications of the pivot for the evaluation and how the evaluation was adapted to the COVID-19 exigencies. It details the redesign of the evaluation to ensure methods wrapped around the modified product and could deliver real-time, practical insights. The paper seeks to fill knowledge gaps in the DHI evaluation space and outlines four key principles that support evaluation re-design in an agile setting. These include a user-centred approach to evaluation design, proportionate data collection, mixed (and flexible) methodologies, and agile evaluation reporting. The paper concludes with key lessons and reflections from the evaluators about what worked at Movember, to support other evaluators planning digital evaluations.

An Education Framework for Effective Implementation of a Health Information System: Scoping Review.

BackgroundTo optimize their use of a new Health Information System (HIS), supporting health care providers require effective HIS education. Failure to provide this education can significantly hinder an organization’s HIS implementation and sustainability efforts.ObjectiveThe aim of this review is to understand the most effective educational strategies and approaches to enable health care providers to optimally use an HIS.MethodsOvid MEDLINE, Ovid Embase, EBSCO Cumulative Index to Nursing and Allied Health Literature, and EBSCO Education Resources Information Center were searched to identify relevant papers. Relevant studies were systematically reviewed and analyzed using a qualitative thematic analysis approach.ResultsOf the 3539 studies screened, 17 were included for data extraction. The literature on the most effective approaches to enable health care providers to optimally use an HIS emphasized the importance of investing in engaging and understanding learners in the clinical context, maximizing the transfer of learning to care, and designing continuous and agile evaluation to meet the emerging demands of the clinical environment.ConclusionsThis review supports the advancement of a new HIS learning framework that organizational leaders and educators can use to guide HIS education design and development. Future research should examine how this framework can be translated into practice.

Analysis of the transferability and robustness of GANs evolved for Pareto set approximations

The generative adversarial network (GAN) is a good example of a strong-performing, neural network-based generative model, even though it does have some drawbacks of its own. Mode collapsing and the difficulty in finding the optimal network structure are two of the most concerning issues. In this paper, we address these two issues at the same time by proposing a neuro-evolutionary approach with an agile evaluation method for the fast evolution of robust deep architectures that avoid mode collapsing. The computation of Pareto set approximations with GANs is chosen as a suitable benchmark to evaluate the quality of our approach. Furthermore, we demonstrate the consistency, scalability, and generalization capabilities of the proposed method, which shows its potential applications to many areas. We finally readdress the issue of designing this kind of models by analyzing the characteristics of the best performing GAN specifications, and conclude with a set of general guidelines. This results in a reduction of the many-dimensional problem of structural manual design or automated search.

Agile communication evaluation and measurement

PurposeThe purpose of this paper is to analyze what the concept of agility means for communication evaluation and measurement and to challenge assumptions of goal-oriented and organization-centric approaches to evaluation and measurement.Design/methodology/approachThis paper is a development debate based on a literature review, regarding agility, evaluation theory, communication evaluation approaches and what agility means for communication measurement.FindingsAgility teaches that what works is more important than what was agreed upon in advance, so it is with more emphasis on needs rather than objectives. Regarding evaluation, the findings show that in today’s communication evaluation theory, evaluation is equated with summative evaluation of smart designed and fixed objectives. In agility, evaluation is always formative, to foster development and improvement within an ongoing activity. Consequently smart objectives are no longer valid as fixed benchmarks and ex ante and ex post evaluations do not exist; instead evaluation is an on-going and forward looking activity during action. Regarding measurement, the basic focus in agility on user needs implies that qualitative methods are more obvious than quantitative. The classic Weberian idea of “Verstehen” is helpful to understand how to focus on needs rather than objectives. This paper finally explores the merits of action research and sense-making methodology as applicable measurements in which “Verstehen” is the basis.Research limitations/implicationsAgility is a very radical concept. The practical and theoretical implications of agile evaluation and measurement mean a total change for practice as well as for communication measurement and evaluation theory building.Originality/valueThe value of this paper is that it is the first to include agility into communication evaluation and measurement and that it, consequently, moves beyond organization-centric concepts of evaluation and measurement by bringing the often overlooked user needs into the game.

A Decade of Veteran Voices: Examining Patient Portal Enhancements Through the Lens of User-Centered Design.

BackgroundHealth care systems have entered a new era focused on patient engagement. Patient portals linked to electronic health records are recognized as a promising multifaceted tool to help achieve patient engagement goals. Achieving significant growth in adoption and use requires agile evaluation methods to complement periodic formal research efforts.ObjectiveThis paper describes one of the implementation strategies that the Department of Veterans Affairs (VA) has used to foster the adoption and sustained use of its patient portal, My HealtheVet, over the last decade: an ongoing focus on user-centered design (UCD). This strategy entails understanding the users and their tasks and goals and optimizing portal design and functionality accordingly. Using a case study approach, we present a comparison of early user demographics and preferences with more recent data and several examples to illustrate how a UCD can serve as an effective implementation strategy for a patient portal within a large integrated health care system.MethodsVA has employed a customer experience analytics (CXA) survey on its patient portal since 2007 to enable ongoing direct user feedback. In a continuous cycle, a random sample of site visitors is invited to participate in the Web-based survey. CXA model questions are used to track and trend satisfaction, while custom questions collect data about users’ characteristics, needs, and preferences. In this case study, we performed analyses of descriptive statistics comparing user characteristics and preferences from FY2008 (wherein “FY” means “fiscal year”) to FY2017 and user trends regarding satisfaction with and utilization of specific portal functions over the last decade, as well as qualitative content analysis of user’s open-ended survey comments.ResultsUser feedback has guided the development of enhancements to core components of the My HealtheVet portal including available features, content, interface design, prospective functional design, and related policies. Ten-year data regarding user characteristics and portal utilization demonstrate trends toward greater patient engagement and satisfaction. Administration of a continuous voluntary Web-based survey is an efficient and effective way to capture veterans’ voices about who they are, how they use the patient portal, needed system improvements, and desired additional services.ConclusionsLeveraging “voice-of-the-customer” techniques as part of patient portal implementation can ensure that such systems meet users’ needs in ways that are agile and most effective. Through this strategy, VA has fostered significant adoption and use of My HealtheVet to engage patients in managing their health.

Collaborative quality enhancement in engineering education: an overview of operational models at a programme level

ABSTRACT This article discusses the tension between quality assurance and quality enhancement in engineering education at a programme level. It acknowledges that accreditation has evolved for many years, but does not agilely support innovation or implement changes in educational programmes. Existing quality assurance systems, institutional collaboration networks, as well as new innovative quality enhancement models and processes are described, contrasted and synthesised. Quality enhancement is analysed based on its function as a source of inspiration and dissemination of good practice. The article reflects on a novel and more collaborative approach to quality enhancement, built on the foundations of specific pedagogical standards and rubrics (e.g. CDIO). One solution leading to real continuous quality enhancement could be flexible and agile evaluation processes. These are founded on measurement and rating frameworks and complemented with quality assurance for engineering education. Incremental enhancement is based on relevant needs identified collaboratively between programmes.

Balancing Product and Process Assurance for Evolving Security Systems

At present, security-related engineering usually requires a big up-front design (BUFD) regarding security requirements and security design. In addition to the BUFD, at the end of the development, a security evaluation process can take up to several months. In today's volatile markets customers want to be able to influence the software design during the development process. Agile processes have proven to support these demands. Nevertheless, there is a clash between traditional security design and evaluation processes. In this paper, the authors propose an agile security evaluation method for the Common Criteria standard. This method is complemented by an implementation of a change detection analysis for model-based security requirements. This system facilitates the agile security evaluation process to a high degree. However, the application of the proposed evaluation method is limited by several constraints. The authors discuss these constraints and show how traditional certification schemes could be extended to better support modern industrial software development processes.

Agile Supply Chain Performance Evaluation of Automobile Manufacturing Based on Regression Analysis

Considering the current agile supply chain performance evaluation methods of automobile manufacturing has defects such as low accuracy, complex computing. This paper proposes the agile supply chain performance evaluation of automobile manufacturing based on regression analysis. Firstly, the index system of agile supply chain performance evaluation of automobile manufacturing is established. Secondly, the agile supply chain performance evaluation model is build based on regression analysis. Finally, the feasibility and availability of this model are verified combined with the numerical examples.

Evaluación de materiales escritos de autoaprendizaje para educación a distancia

The value of self-learning written material on Distance Education is based on its pedagogic quality necessary to achieve a meaningful learning. Objective. To design both a model and a reliable valid instrument to evaluate the pedagogic quality of the self-learning written materials. Methodology. The search was carried out at Universidad Industrial de Santander (Bucaramanga, Colombia). Firstly, a model was designed for evaluating the materials mentioned above. Thereafter, the instrument was elaborated and it was defined the concept of pedagogical quality of the self-learning written materials. They were also analyzed the validity of content (Factorial analysis), the facial validity and confidentiality of the alpha coefficient of Cronbrach instrument). Results. It was obtained an agile evaluation model and an instrument composed by 28 items, all constituted into one only factor; alpha coefficient of Cronbrach of 0.96 and a power to recognize the differences between the pedagogic elements of a self-learning written constituted from the inductive approach and another from the constructivism approach.

A Methodology for Building Up an Infrastructure of Haptically Enhanced Computer-Aided Design Systems

This paper presents an infrastructure that integrates a haptic interface into a mainstream computer-aided design (CAD) system. A haptic interface, by providing force feedback in human-computer interaction, can improve the working efficiency of CAD/computer-aided manufacturing (CAM) systems in a unique way. The full potential of the haptic technology is best realized when it is integrated effectively into the product development environment and process. For large manufacturing companies this means integration into a commercial CAD system (Stewart, et al., 1997, “Direct Integration of Haptic User Interface in CAD Systems,” ASME Dyn. Syst. Control Div., 61, pp. 93–99). Mainstream CAD systems typically use constructive solid geometry (CSG) and boundary representation (B-Rep) format as their native format, while internally they automatically maintain triangulated meshes for graphics display and for numerical evaluation tasks such as surface-surface intersection. In this paper, we propose to render a point-based haptic force feedback by leveraging built-in functions of the CAD systems. The burden of collision detection and haptic rendering computation is alleviated by using bounding spheres and an OpenGL feedback buffer. The major contribution of this paper is that we developed a sound structure and methodology for haptic interaction with native CAD models inside mainstream CAD systems. We did so by analyzing CAD application models and by examining haptic rendering algorithms. The technique enables the user to directly touch and manipulate native 3D CAD models in mainstream CAD systems with force/touch feedback. It lays the foundation for future tasks such as direct CAD model modification, dynamic simulation, and virtual assembly with the aid of a haptic interface. Hence, by integrating a haptic interface directly with mainstream CAD systems, the powerful built-in functions of CAD systems can be leveraged and enhanced to realize more agile 3D CAD design and evaluation.