Evaluative Inquiry Research Articles

Implementation of task-based language teaching in a Spanish language program: Instructors’ and students’ perceptions

Most research on task-based language teaching (TBLT) has focused on specific factors that play a role in task-based performance and learning, whereas considerably fewer studies have paid attention to how TBLT curricula have been developed and delivered in second language (L2) teaching contexts. However, it has been argued that the latter type of evaluative inquiry is crucial in order to advance the educational significance of the approach. While more evaluation studies have been published in recent years, few of them adopt a multi-methodological, longitudinal and cyclical perspective. The current study examines the planning and implementation of task-based instruction in a university-level Spanish as a foreign language program over a five-year period, with a particular emphasis on instructors’ and students’ perceptions about the approach. Data sources consisted of notes from meetings with instructors, classroom observations, students’ perceptions collected through journals, focus groups and questionnaires, and instructors’ perceptions collected through a questionnaire. The qualitative and quantitative analysis of these data revealed critical aspects of the planning phase, positive and challenging components of the approach, modifications made in response to participants’ perceptions, as well as a gradual increase regarding the level of acceptance of task-based instruction throughout the implementation. Implications for the implementation and evaluation of TBLT in other second language educational contexts are discussed.

Editorial Presentation - Perspectives on innovation governance: challenges and dilemmas

Editorial Presentation - Perspectives on innovation governance: challenges and dilemmas

81 Autism Symptoms Reported and Observed During Neuropsychological Assessment of Children with Congenital Heart Disorder

Objective:Children with congenital heart disease (CHD) have increased likelihood for Autism Spectrum Disorder (AuSD; Sigmon, Kelleman, Susi, Nylung & Oster, 2020). Even those not meeting full criteria remain at greater risk for problems with social processing/communication (Cassidy et al, 2018). The current study examined what symptoms indicative of potential AuSD were qualitatively reported by parents, and what symptoms were noted behaviorally by clinicians. These behaviors may be targets for both further clinical inquiry and intervention.Participants and Methods:A retrospective chart review of CHD patients seen for neuropsychological assessment between the ages of 6-18 years and between 2016-2021 was conducted. The final sample included 88 patients (Ethnicity: 14% Hispanic; Race: 76% White, 17% Black/African American, 5% Asian, 1% Native American, 1% Unknown). A coding system for AuSD symptoms was derived by the authors, who are experienced in AuSD diagnosis and assessment, based on DSM-5 criteria. A comprehensive list of behaviors consistent with each symptom category was generated, and assessment reports were subsequently reviewed noting the presence, absence or “no mention” of each symptom. A second coding system was derived to assess for AuSD symptoms documented in each report’s behavioral observations. Three pediatric neuropsychologists, one post-doctoral fellow, and one psychometrist were coders, with very good reliability (k=.854 (95% CI, .827 to .881), p < .0001).Results:Twelve patients (14%) were diagnosed with AuSD. Age of diagnosis ranged from 3-14 years (M=7.82, SD=3.92). Main concerns parents expressed included difficulty with reciprocal conversation (75%), making friends (75%), initiating and maintaining social interactions (67%), and restrictive/intense interests (58%). During testing, providers noted variable eye contact (67%), appropriate responses to questions but minimal social conversation (67%), and exaggerated prosody (42%). Of those who did not have an AuSD diagnosis, the most frequent parent concerns included difficulty making friends (38%), difficulty initiating or maintaining social interactions (33%), atypical affect (25%), and restrictive/intense interests (18%). In this sample, providers noted concerns with answering questions but not maintaining conversation (26%), flat affect (16%), loud (17%) or soft (21%) speech volume, and socially immature behaviors (10%). Within this No Diagnosis group, general social concerns were highly correlated (point biserial) with more specific autism symptoms (e.g., intense interests, difficulty with transitions, sensory sensitivities r = .986 - .784), although most often the presence or absence of these concerns were not documented.Conclusions:We examined qualitative parent-reported and provider-observed behaviors indicative of potential AuSD as detailed in a comprehensive neuropsychological evaluation report. Behaviors in children with formal AuSD diagnoses were consistent with that diagnosis, based on both parent and provider description. Of note, in children without AuSD, though, both parents and providers reported AuSD-like concerns (e.g., social communication/interaction problems, atypical interests, atypical affect, atypical speech volume) in a large minority of children. It is important that if general social concerns are present, that providers follow-up on, and document, a broader constellation of AuSD symptoms. These behaviors deserve further exploration and study within the CHD population and are important areas of inquiry in any clinical evaluation, as they should directly inform intervention.

Armchair Evaluative Knowledge and Sentimental Perceptualism

We seem to be able to acquire evaluative knowledge by mere reflection, or “from the armchair.” But how? This question is especially pressing for proponents of sentimental perceptualism, which is the view that our evaluative knowledge is rooted in affective experiences in much the way that everyday empirical knowledge is rooted in perception. While such empirical knowledge seems partially explained by causal relations between perceptions and properties in the world, in armchair evaluative inquiry, the relevant evaluative properties are typically not even present. The paper shows how sentimental perceptualists can ultimately provide a broadly causal explanation of our reflective evaluative knowledge.

The Last 20 Years of Empirical Research on Government Utilization of Academic Social Science Research: A State-of-the-Art Literature Review

We organize and critique the last 20 years of empirical research on policy utilization of academic social science, offering eight recommendations for future research: (1) improvement in utilization measures; (2) greater alignment on constructs of interest; (3) greater address of different national contexts; 4) study of mechanisms connecting academic research to policymakers; (5) incorporation of knowledge on information search; (6) attention to differences between utilization of policy, social, and physical science in different sectors; (7) evaluative inquiry on the effects of research utilization; and (8) investigation of the value of research in addressing different classes of societal problems.

Teaching specification of evaluative criteria: A guide for evaluation education

AbstractEvaluative criteria describe the attributes that define a high‐quality intervention and represent values about which intervention characteristics or results are desirable. There are many types of criteria, including those focused on intervention outcomes or impact, design and implementation, and relevance. Criteria may remain implicit and assumed in evaluation practice, yet they nonetheless direct evaluative inquiry and provide the value basis for evaluative conclusions. It is therefore important for novice evaluators to understand why criteria matter and how to thoughtfully consider which and whose criteria to use. We introduce an approach for teaching criteria specification to guide novice evaluators in considering, selecting, and articulating criteria. We introduce key concepts, outline an instructional approach, and provide a downloadable teaching activity for graduate students, undergraduates, and working professionals.

Translating tools and indicators in territorial RRI.

By a series of calls within the Horizon 2020 framework programme, the EU funded projects intended to deploy Responsible Research and Innovation (RRI) at a territorial level, in regional research and innovation ecosystems. This paper presents efforts to document and evaluate the achievements in TRANSFORM, one of these projects. Evaluative inquiry and theoretical reasoning. Noting the need for a general principle to be interpreted, adapted and translated in order to be rendered meaningful at a local level, we studied precisely these multiple territorial translations of RRI, the organizational and institutional orderings with which they co-emerge and the challenges that come with these translations. An important shared feature is that RRI work does not start from zero, but rather builds on pre-existing relationships and repertoires of collaboration. The RRI project is hence a way to continue ongoing work and follow pre-set purposes, aims and objectives, as a form of "maintenance work". In this very human sense, RRI is deployed with a logic of care in the regional context, while the Horizon 2020 calls and proposals above all are formulated in a logic of choice, to be assessed by indicators. We warn against undue standardization of RRI by toolification and use of quantitative indicators, and recommend that RRI performance is monitored by methods of evaluative inquiry.

The Case for Enhancing the Quality of Primary Evaluations through Metaevaluation in Developing Country Contexts

This article makes a case for metaevaluation in the context of further evaluative inquiry beyond primary evaluation level and advances the plausible perspective that metaevaluation improves the quality of primary evaluations and the evidence used to support decision-making. It argues that metaevaluation is a form of evaluative enquiry that is aimed at assessing the strengths and weaknesses of primary evaluations. Through so doing, metaevaluation makes a valuable contribution toward enhancing the credibility and reliability of primary evaluations and the evidence provided to decision-makers. Metaevaluation has been utilised in different contexts in some developed countries to assess the quality of evaluations of development interventions, although scanty evidence exists of its use in developing country contexts.

Evaluation through narratives: A practical case of Participatory Narrative Inquiry in women empowerment evaluation in Niger

This study explores the use of Participatory Narrative Inquiry (PNI) in evaluating development interventions by contributing to the debate of using participatory narrative methods. Stories on personal experience are used to evaluate the project’s effects as with similar methods such as Most Significant Change and Sensemaking. To the authors’ knowledge, this is one of the early applications of PNI to the evaluation of international development programmes. The study discusses advantages and limitations, and provides a scholarly reflection based on an application of PNI in the evaluation of gender and women’s empowerment in Niger. The study concludes that PNI is a powerful alternative to existing qualitative and participatory narrative evaluation methods. Within mixed-method approaches, PNI allows for greater inclusion of project beneficiaries in the evaluation process, while helping to elaborate a thorough theory of change, understand the complexity of the context, identify and assess outcome pathways, and provide an evidence-based evaluation.

The Los Angeles 1984 Olympics: Using a logic model as a tool to understand soft power

This article argues that a logic model is an effective analytical tool to examine the opening ceremony of the Los Angeles 1984 Olympics as an exercise in soft power. Specifically, it uses a logic model as a guide to emphasize how the Los Angeles 1984 Olympics provided a structure to share values and culture with the intention to influence. Although this is not an evaluative inquiry, the evaluative framework lends itself to identifying the resources required for the creation of the opening ceremony and the formal presentation. For instance, a review of interurban competition (competition between cities or regions that attract economic activities), management and boycotts as inputs of the Los Angeles Games reveal the presentation of American culture and values during the Olympic opening ceremony. The purpose of this article is to implement a logic model as means for understanding the cultural programming of the Olympics by first identifying the unique inputs for the Los Angeles 1984 Olympics and analysing the visual presentation of the opening ceremony.

Embedding researchers into organisations: a study of the features of embedded research initiatives

Background: ‘Embedded research’ (co-locating researchers within non-academic organisations) is advocated as a way of developing more effective services through better creation and application of knowledge. Aims and objectives: The existing literature on embedded initiatives has largely been descriptive. There has been less in the way of analysis, for example, disaggregating the components of such schemes, unpacking underpinning logics, or comparing the diverse ways in which schemes are instantiated. We aimed to explore the nature and organisation of such schemes in health settings in the UK, with the objective of providing a systematised means of understanding their makeup. Methods: This study uses a focused literature review combined with a systematic scoping exercise of extant initiatives. We assembled documentation on each scheme (n=45) and conducted in-depth interviews in twelve of them (n=17). Analytically, we focused on surfacing and articulating the key features of embedded research initiatives in relation to their intent, structure and processes. Findings were then tested and validated during a co-production workshop with embedded researchers and their managers. Findings: We identified 26 ‘clusters’ of peer-reviewed papers detailing specific embedded research initiatives, and we explored 45 extant initiatives. The initiatives were varied in intent, structure and processes, but we were able to surface ten themes representing common features: intended outcomes, power dynamics, scale, involvement, proximity, belonging, functional activities, skill and expertise, relational roles, and learning and reflection. Discussion and conclusion: The themes uncovered can be used as a framework for guiding further systematic and evaluative enquiry on embedded research initiatives.

The production of scientific and societal value in research evaluation: a review of societal impact assessment methods

AbstractOver the past two decades, several methods have been developed to evaluate the societal impact of research. Compared to the practical development of the field, the conceptual development is relatively weak. This review article contributes to the latter by elucidating the theoretical aspects of the dominant methods for evaluating societal impact of research, in particular, their presuppositions about the relationship between scientific and societal value of research. We analyse 10 approaches to the assessment of the societal impact of research from a constructivist perspective. The methods represent different understandings of knowledge exchange, which can be understood in terms of linear, cyclical, and co-production models. In addition, the evaluation methods use a variety of concepts for the societal value of research, which suggest different relationships with scientific value. While some methods rely on a clear and explicit distinction between the two types of value, other methods, in particular Evaluative Inquiry, ASIRPA, Contribution Mapping, Public Value Mapping, and SIAMPI, consider the mechanisms for producing societal value integral to the research process. We conclude that evaluation methods must balance between demarcating societal value as a separate performance indicator for practical purposes and doing justice to the (constructivist) science studies’ findings about the integration of scientific and societal value of research. Our analytic comparison of assessment methods can assist research evaluators in the conscious and responsible selection of an approach that fits with the object under evaluation. As evaluation actively shapes knowledge production, it is important not to use oversimplified concepts of societal value.

The engine of evaluative inquiry: Social learning processes

AbstractAlthough originated in the 1990s, few empirical studies exist that examine the Evaluative Inquiry model, particularly, the social learning processes within the model. This study examined the social learning processes that represent the forms of communication that drive learning within the Evaluative Inquiry model: dialog, reflection, asking questions and identifying and challenging values, beliefs, assumptions and knowledge. Results indicated that social learning processes promoted individual and team learning, as well as provided the beginnings of organizational learning. In addition, results confirmed that dialog served as the central process through which all other social learning processes occur. Our findings present a new conceptualization of the social learning processes at the center of the Evaluative Inquiry model.

AN EXPLORATORY SURVEY ON ORGANIZATIONAL READINESS FOR EVALUATION IN TURKEY

In designing Evaluation Capacity Building (ECB) programs in a country, it is vital to know the baseline situation. A necessary step is to reveal the status of current understanding and practices of evaluation. The objective of the paper is to explore the evaluation readiness of organizations in Turkey. Following the literature on organizational culture on evaluative inquiry, the study reports on the development and administration of a structured questionnaire with particular focus on various dimensions of organizational readiness for evaluation such as culture, leadership, resources and existing evaluation practices, if any. The survey sample includes 71 representatives of government agencies and, domestic and international NGOs in Turkey. Although evaluation is regarded and perceived as a tool for organizational learning and improved organizational performance, the culture, leadership and structure of the organizations do not show supporting characteristics on the implementation and use of evaluation in organizations. The results reveal a necessity of positioning evaluation as a major tool for efficient use of resources and organizational learning in Turkish organizations.

BIBLIOMETRIC TECHNIQUES AND THEIR USE IN BUSINESS AND ECONOMICS RESEARCH

AbstractBibliometric indicators and bibliometric techniques have been used in the domains of business and economics (BE) in the context of network studies and the evaluation of universities, journals and researchers, among other purposes. Using a strengths, weaknesses, opportunities and threats analysis as a general framework, we present an overview of the strengths, weaknesses, opportunities and threats related to the use of mainstream bibliometrics in BE. Several weaknesses in the use of bibliometric indicators for research evaluation in BE, and, in general, are discussed. Misuse of bibliometric indicators may lead to goal displacement and gender bias. Yet, being aware of possible problems leads to opportunities for further developments and applications of bibliometrics in the field of BE. We propose that traditional research evaluation should evolve into evaluative inquiry, a method where bibliometric indicators play a role, but in which all aspects of life and work of a scientific professional can be taken into account and assessment is relabelled in positive terms such as organizational and individual development, care and support.

The Improvement of Prospective Teachers' Life-long Learning during the Plant Diversity Course with 5E+e Inquiry

Not a few students who have difficulty interpreting the material being studied in terms of life encourage the need for a paradigm shift in science learning, which is directed at lifelong learning. In this paper, an entrepreneurship-oriented 5E (Engagement, Explain, Exploration, Elaboration, and Evaluation) inquiry learning program is developed, which is also often called 5E+e. This research investigates the improvement of lifelong learning for the prospective teachers during the 5E+e inquiry in the plant diversity course. The quasi-experimental method with a one-group pretest-postest design was used. The research subjects were taken by purposive sampling, namely the fifth-semester students of the Biology Education Study Program at one of the Teachers Institution in Central Java with 31 participants. Data were gathered using a questionnaire filled out by students before (pre) and after (post) the treatment of the 5E+e inquiry learning and observation. The data analysis was carried out in both quantitative and qualitative-descriptive manners to make a comprehensive conclusion. The results revealed that the average lifelong learning score of prospective teachers compiled with rubrics had increased between before (2.68) and after being given the program (3.26) with a maximum score of 4.00. Thus, it can be concluded that the entrepreneurship-oriented inquiry learning program applied to the plant diversity course can increase the lifelong learning of prospective teachers.

TÜRKİYE’DE KURUMLARIN DEĞERLENDİRME KONUSUNDA HAZIR OLMA DURUMLARI İLE İLGİLİ KEŞİFSEL BİR ARAŞTIRMA

In designing Evaluation Capacity Building (ECB) programs in a country, it is vital to know the baseline situation. A necessary step is to reveal the status of current understanding and practices of evaluation. The objective of the paper is to explore the evaluation readiness of organizations in Turkey. Following the literature on organizational culture on evaluative inquiry, the study reports on the development and administration of a structured questionnaire with particular focus on various dimensions of organizational readiness for evaluation such as culture, leadership, resources and existing evaluation practices, if any. The survey sample includes 71 representatives of government agencies and, domestic and international NGOs in Turkey. Although evaluation is regarded and perceived as a tool for organizational learning and improved organizational performance, the culture, leadership and structure of the organizations do not show supporting characteristics on the implementation and use of evaluation in organizations. The results reveal a necessity of positioning evaluation as a major tool for efficient use of resources and organizational learning in Turkish organizations.

An evaluative inquiry into continuing professional development: understanding faculty perceptions

ABSTRACT This article is comprised of the results of a developmental evaluation inquiry into continuing professional development (CPD) at a college in Western Canada. Since 2015, despite having one of the most generous CPD budgets among all colleges in the region, the faculty participation rate in CPD has ranged between approximately 48% and 56%. Patton’s ‘What? So What? Now What?’ inquiry framework was used to investigate why more faculty were not accessing CPD funds. This article also includes an examination of the intersection between professional identity and self, and offers some recommendations that could positively impact faculty participation in CPD. Key recommendations include creating a stronger workplace for faculty members, recognizing CPD as a key factor in supporting excellence in teaching and learning, capitalizing on mid-career academics, and exploring self-reflection as a tool for faculty members to identify and pursue meaningful CPD.

Applying Evaluative Inquiry: Experience and Recommendations from the Field

Evaluative Inquiry (EI) is a process for how evaluation and social interactions provide a structured model which promotes individual, team, and organizational learning. We share experiences and guidance from the field which highlight benefits, challenges, and lessons learned. These include guiding principles for promoting, planning, implementing, and maintaining EI. Finally, we share additional items to consider that are not part of the formal EI model that we found important through our experience.

Critical reflections on erotic content in Star FM’s Couples quality time and Ya FM’s Moto mubhurugwa (fire in underwear)

ABSTRACT In Shona tradition, sex education customarily occurs in gendered, compartmentalised and private contexts. Accordingly, sexually explicit representations in the public sphere are morally obnoxious. However, there has been an unpredicted upsurge of sexually explicit content in Zimbabwe’s Shona language broadcasts. This study is an evaluative and explanatory enquiry of erotic content in Star FM’s Couples quality time and Ya FM’s Moto mubhurugwa, and an account of radio producers/presenters’ intensified interest in the subject of sex, against the Shona people’ ethics on sexual deliberations. Using interviews, data were collected from purposively selected sample comprising radio presenters and listeners, and content analysis of conveniently picked episodes of radio broadcasts in question. Deploying the indigenous knowledge systems viewpoint, this study shows that, the framing and presentation of Couples quality time and Moto Mubhurugwa generally flouts Shona moral standards on sex. However, we argue that, while preserving culture is important, in the absence of sound cultural institutions and spaces, African language radio is an expedient medium for sexual knowledge dissemination. Using the culture industry thesis, the study also demonstrates the commodification and marketisation of sex as one of the new generation radio stations’ strategies to penetrate the broadcasting market and compete with well-established players. The study provides insights into the conundrums of integrating sexual issues in African language broadcasting, and broadcasters’ dilemma in mediating cultural and commercial functions.