Research Assessment Processes Research Articles

The marginalisation of responsible management in business schools: a consideration of future trajectories

Purpose Business schools are vital in promoting responsible management (RM) – a management grounded in ethics and values beneficial to a wide array of stakeholders and overall society. Nevertheless, due to deeply embedded institutional modernistic dynamics and paradigms, RM is, despite its importance, repeatedly marginalised in business school curricula. If students are to engage with RM thinking, then its occlusion represents a pressing issue. Drawing on the United Kingdom (UK) business school context, this paper aims to examine this issue through a framework of institutional theory and consider the role played by (modernistic) institutional accreditation and research assessment processes in marginalisation of RM. Design/methodology/approach This study used an exploratory qualitative research method. Data were collected from 17 RM expert participants from 15 UK business schools that were signatories to the United Nations Principles for Responsible Management Education through semi-structured in-depth interviews and analysed using the six phases of Braun and Clarke’s thematic analysis. Findings The study identifies a potent institutional isomorphic amalgam resulting in conservative impacts for RM. This dynamic is termed multiple institutional isomorphic marginalisation (MIIM) – whereby a given domain is occluded and displaced by hegemonic institutional pressures. In RM’s case, MIIM operates through accreditation-driven modernistic-style curricula. This leads business schools to a predilection towards “mainstream” representations of subject areas and a focus on mechanistic research exercises. Consequently, this privileges certain activities over RM development with a range of potential negative effects, including social impacts. Originality/value This study fills an important gap concerning the need for a critical, in-depth exploration of the role that international accreditation frameworks and national institutional academic research assessment processes such as the Research Excellence Framework in the UK play in affecting the possible growth and influence of RM. In addition, it uses heterotopia as a conceptual lens to reveal the institutional “mask” of responsibility predominantly at play in the UK business school context, and offers alternative pathways for RM careers.

Valuing Preprints Must be Part of Responsible Research Assessment

Comments on papers by Schönbrodt et al. (2022) and Gärtner et al. (2022) proposing reforms to the research assessment process. Given the prominent role of preprints in contemporary scientific practice, they must be an accepted and central component of research assessment.

Technology assisted research assessment: algorithmic bias and transparency issues

PurposeTechnology is sometimes used to support assessments of academic research in the form of automatically generated bibliometrics for reviewers to consult during their evaluations or by replacing some or all human judgements. With artificial intelligence (AI), there is increasing scope to use technology to assist research assessment processes in new ways. Since transparency and fairness are widely considered important for research assessment and AI introduces new issues, this review investigates their implications.Design/methodology/approachThis article reviews and briefly summarises transparency and fairness concerns in general terms and through the issues that they raise for various types of Technology Assisted Research Assessment (TARA).FindingsWhilst TARA can have varying levels of problems with both transparency and bias, in most contexts it is unclear whether it worsens the transparency and bias problems that are inherent in peer review.Originality/valueThis is the first analysis that focuses on algorithmic bias and transparency issues for technology assisted research assessment.

Indicators of research quality, quantity, openness, and responsibility in institutional review, promotion, and tenure policies across seven countries

Abstract The need to reform research assessment processes related to career advancement at research institutions has become increasingly recognized in recent years, especially to better foster open and responsible research practices. Current assessment criteria are believed to focus too heavily on inappropriate criteria related to productivity and quantity as opposed to quality, collaborative open research practices, and the socioeconomic impact of research. Evidence of the extent of these issues is urgently needed to inform actions for reform, however. We analyze current practices as revealed by documentation on institutional review, promotion, and tenure (RPT) processes in seven countries (Austria, Brazil, Germany, India, Portugal, the United Kingdom and the United States). Through systematic coding and analysis of 143 RPT policy documents from 107 institutions for the prevalence of 17 criteria (including those related to qualitative or quantitative assessment of research, service to the institution or profession, and open and responsible research practices), we compare assessment practices across a range of international institutions to significantly broaden this evidence base. Although the prevalence of indicators varies considerably between countries, overall we find that currently open and responsible research practices are minimally rewarded and problematic practices of quantification continue to dominate.

A framework for assessing the impact of health research from a broad population perspective

Abstract
 
 
 Background: research in the health sector is an activity associated with an uncertain return on investment. This study aims to outline a multidimensional framework for assessing the imact of research funded by public (or mixed public and private) sponsors. 
 Methods: a focus group representing different stakeholders was created to develop a sense of purpose that would be shared by the intended beneficiaries of the research assessment process, and others with a direct or indirect interest in a program’s implementation. 
 Results: a framework was designed that takes a holistic approach to the impact of research in which not only technical, but also economic, social and political dimensions are interrelated with the results achieved for the population and health services. 
 Conclusions: the framework reflected the views of different stakeholders, favoring the development of the positive, proactive relationships essential to orienting the process of transforming the evidence emerging from research into action for health. 

Valuing and evaluating musical practice as research in ethnomusicology and its implications for research assessment

ABSTRACT In this article, we argue that ethnomusicology holds valuable epistemic insights for considering how to measure and evaluate research for academics, as well as for research policy and management professionals. We focus on two notable instances of standardised national research assessment frameworks: the UK’s Research Excellence Framework (REF), and Excellence in Research for Australia (ERA) and identify the advantages of practice as research models for music research by considering the benefits of ethnomusicology’s reflexive and relativist methodologies to formal research assessment processes. To support our argument, we refer to published case studies of ethnomusicological research that reach beyond Western practice and thought to highlight the advantages recognising practice as research as a more inclusive modality of original knowledge production. We call upon ethnomusicologists to pro-actively engage with the formal processes of research assessment to make them more equitable and representative of our discipline’s broad commitment to decolonising academic practice.

Measuring research excellence amongst economics lecturers in the UK

AbstractUsing a rich new data source, we explore the selection of economics lecturers into the last UK Research Excellence Framework (REF) exercise. Only some one‐in‐two (54%) of these lecturers were submitted to REF2014; 57% of men and 46% of women. The decision making of institutions is found to be well approximated by a simplified selection approach; focusing on working papers and higher quality journal publications. Our results also reveal sizeable conditional differences in the probability of selection, especially so in departments with higher research rankings. More than half of the variance in selection probability remains unexplained, revealing considerable idiosyncrasies in the management of submissions and uncertainty across the discipline in this research assessment process.

Measuring Research Excellence Amongst Economics Lecturers in the UK

Using a rich new data source, we explore the selection of economics Lecturers into the last UK Research Excellence Framework (REF) exercise. Only some one-in-two (54%) of these Lecturers were submitted to REF2014; 57% of men and 46% of women. The decision making of Institutions is found to be well approximated by a simplified selection approach; focusing on working papers and higher quality journal publications. Our results also reveal sizeable conditional differences in the probability of selection, especially so in departments with higher research rankings. More than half of the variance in selection probability remains unexplained, revealing considerable idiosyncrasies in the management of submissions and uncertainty across the discipline in this research assessment process.

How Evaluative Informetrics Relates to Scientific, Socio-Historical, Political, Ethical and Personal Values

Policy highlights:If evaluation is defined as “a systematic determination of a subject’s merit, worth and significance, using criteria governed by a set of standards”, there is no evaluation without an evaluative framework specifying these criteria and standards.On the other hand, evaluative informetrics itself, defined as the study of evaluative aspects of science and scholarship using citation analysis, altmetrics and other indicators, does not evaluate.However, informetric indicators are often used in research assessment processes. To obtain a better understanding of their role, the links between evaluative informetrics and ‘values’ are investigated, and a series of practical guidelines are proposed.Informetricians should maintain in their evaluative informetric studies a neutral position toward the policy issues addressed and the criteria specified in an evaluative framework.As professional experts, informetricians’ competence lies primarily in the development and application of analytical models within the context of a given evaluative framework.Informetric researchers could propose that evaluators and policy makers incorporate fundamental scientific values such as openness and adopting a critical attitude in assessment processes.Informetricians could also promote and participate in an overall discussion within the academic community and the research policy domain about the objectives and criteria in research assessment processes and the role of informetric tools therein.Evaluative informetrics is defined as the study of evaluative aspects of science and scholarship using informetric data and methodologies, such as citation analysis and altmetrics. Following the main lines of an article by the Dutch philosopher O.D. Duintjer, nine interfaces are distinguished between quantitative science studies, especially evaluative informetrics, and the domain of values, including scientific, socio-historical, political, ethical and personal norms and objectives. Special attention is given to the “principle of value neutrality” at the meta-level of methodological rules guiding scientific inquiry and to the crucial, independent role of evaluative frameworks in research evaluation. The implications of the various relationships between science and values for research practices in evaluative informetrics and for its application in research assessment are considered.

Position Statement and Recommendations on Research Assessment Processes

Position Statement and Recommendations on Research Assessment Processes

Libraries and the REF: how do librarians contribute to research excellence?

Librarians support research in a wide variety of ways. However, librarians may not always receive recognition for the valuable contributions they make to the research life cycle or research environments. In higher education institutions (HEIs), librarians face competition from other professional support services in addition to external organizations and suppliers. This article provides an analysis of submissions made by HEIs to the UK’s Research Excellence Framework (REF) in 2014 and demonstrates that the work of librarians is rarely acknowledged in substantive ways, particularly in STEM disciplines. It provides an overview of the multiple ways in which librarians contribute to research environments as well as how they can practically contribute to the REF 2021 submission process and other research assessment processes. Librarians have developed a wide range of responsive and innovative support services over the last decade and REF 2021 is demonstrated to be an opportunity for librarians to gain recognition for the important role they play in UK HEIs.

The decline of the professionally-qualified accounting academic: Recruitment into the accounting academic community

The profile of the accounting academic has changed in recent years. The earliest academics were often recruited from the accountancy profession. Now the typical accounting academic recruit has a profile similar to the rest of the university, with the PhD being the qualification of choice. The reasons for this trend are examined using a cultural and institutional logics framework. The recruitment context and the institutional changes impacting on recruitment in accounting in academia are explored through the views of heads of department who have knowledge of both their institution’s recruitment policies and of the requirements of their discipline. As the research assessment process appears to be a driver of changing recruitment patterns, recruitment is considered in contrasting contexts: Scotland, where periodic research assessment takes place in both old and new universities, and the Republic of Ireland which does not have such a process. Despite differences in the views expressed by heads in these different contexts and differences in their research environments, the trend in all sectors is towards the recruitment (Scotland) or development (Republic of Ireland) of PhD holders rather than professionally qualified staff. The consequences for the nature of the discipline are discussed.

The Journal Impact Factor and its discontents: steps toward responsible metrics and better research assessment

A small, self-selected discussion group was convened to consider issues surrounding impact factors at the first meeting of the Open Scholarship Initiative in Fairfax, Virginia, USA, in April 2016, and focused on the uses and misuses of the Journal Impact Factor (JIF), with a particular focus on research assessment. The group’s report notes that the widespread use, or perceived use, of the JIF in research assessment processes lends the metric a degree of influence that is not justified on the basis of its validity for those purposes, and retards moves to open scholarship in a number of ways. The report concludes that indicators, including those based on citation counts, can be combined with peer review to inform research assessment, but that the JIF is not one of those indicators. It also concludes that there is already sufficient information about the shortcomings of the JIF, and that instead actions should be pursued to build broad momentum away from its use in research assessment. These actions include practical support for the San Francisco Declaration on Research Assessment (DORA) by research funders, higher education institutions, national academies, publishers and learned societies. They also include the creation of an international “metrics lab” to explore the potential of new indicators, and the wide sharing of information on this topic among stakeholders. Finally, the report acknowledges that the JIF may continue to be used as one indicator of the quality of journals, and makes recommendations how this should be improved.OSI2016 Workshop Question: Impact FactorsTracking the metrics of a more open publishing world will be key to selling “open” and encouraging broader adoption of open solutions. Will more openness mean lower impact, though (for whatever reason—less visibility, less readability, less press, etc.)? Why or why not? Perhaps more fundamentally, how useful are impact factors anyway? What are they really tracking, and what do they mean? What are the pros and cons of our current reliance on these measures? Would faculty be satisfied with an alternative system as long as it is recognized as reflecting meaningfully on the quality of their scholarship? What might such an alternative system look like?

Multidimensional assessment of scholarly research impact

This article introduces the Multidimensional Research Assessment Matrix of scientific output. Its base notion holds that the choice of metrics to be applied in a research assessment process depends on the unit of assessment, the research dimension to be assessed, and the purposes and policy context of the assessment. An indicator may by highly useful within one assessment process, but less so in another. For instance, publication counts are useful tools to help discriminate between those staff members who are research active, and those who are not, but are of little value if active scientists are to be compared with one another according to their research performance. This paper gives a systematic account of the potential usefulness and limitations of a set of 10 important metrics, including altmetrics, applied at the level of individual articles, individual researchers, research groups, and institutions. It presents a typology of research impact dimensions and indicates which metrics are the most appropriate to measure each dimension. It introduces the concept of a “meta‐analysis” of the units under assessment in which metrics are not used as tools to evaluate individual units, but to reach policy inferences regarding the objectives and general setup of an assessment process.

Perceptions of the UK’s Research Excellence Framework 2014: a media analysis

This paper explores perceptions of the UK’s Research Excellence Framework (REF) and its implications for individuals, institutions and wider academia through an analysis of media coverage of the REF over a 2-year period. In recent years, the importance attached to the REF has become an increasing focus of concern for academics and other commentators, particularly vis-à-vis issues such as staff morale, funding, ‘impact’, working practices and institutional inequalities. In examining media stories related to the REF, we uncover three key findings in terms of the main ways in which the REF is presented and perceived. First, media discussions of the REF are overwhelmingly negative, although this appears to have abated over time. Second, there are significant differences in the how the REF is discussed; this variation stems from characteristics such as the disciplinary background, institutional type and professional seniority of commentators. Third, although wide range of themes is discussed in relation to the REF, several tend to dominate media portrayals: these include ‘impact’, ‘funding’ and ‘marketisation’. The implications for the role and legitimacy of research assessment processes are discussed.

Empleo del método Delphi y su empleo en la investigación en comunicación y educación

Se analizan las posibilidades que la metodología Delphi ofrece para la investigación educativa y de la comunicación al mismo tiempo se ofrece algunas referencias históricas, se presentan cuáles son sus características fundamentales, las posibilidades y limitaciones que tiene su aplicación, las diferentes versiones existentes para su incorporación en los procesos de evaluación en investigación, y las fases operativas para su aplicación. También se estudia una de las problemáticas fundamentales que presenta esta técnica y que se refiere al proceso de selección de los expertos, respecto al cual se ofrecen algunas sugerencias y aportaciones para su una correcta identificación. Al mismo tiempo se presenta una de las estrategias más novedosas que últimamente se están utilizado para la selección de los informadores: el denominado “coeficiente de conocimiento experto”.

Managing researcher development: ‘drastic transition’?

Academics are expected to write for publication and meet publication targets in research assessment processes. These targets are set by national bodies and institutions, and they can be daunting for academics at the start of a research career. This article reports on an intervention designed to address this issue, writer’s retreat, where academics simultaneously engaged with research assessment and developed writing projects. Using Gardner’s model of researcher development, an evaluation study showed that retreat participants quickly adopted the programmatic element, made some progress in the relational element, but made little or no progress with the personal element. This analysis shows that the structured, facilitated writer’s retreat is an effective method of researcher development, and points to specific areas where development did and did not occur. It shows that this new retreat format allows emerging researchers to negotiate their relationship with research assessment through writing, but it also identifies areas where further support is needed.

Do age and professional rank influence the order of authorship in scientific publications? Some evidence from a micro-level perspective

Scientific authorship has important implications in science since it reflects the contribution to research of the different individual scientists and it is considered by evaluation committees in research assessment processes. This study analyses the order of authorship in the scientific output of 1,064 permanent scientists at the Spanish CSIC (WoS, 1994–2004). The influence of age, professional rank and bibliometric profile of scientists over the position of their names in the byline of publications is explored in three different research areas: Biology and Biomedicine, Materials Science and Natural Resources. There is a strong trend for signatures of younger researchers and those in the lower professional ranks to appear in the first position (junior signing pattern), while more veteran or highly-ranked ones, who tend to play supervisory functions in research, are proportionally more likely to sign in the last position (senior signing pattern). Professional rank and age have an effect on authorship order in the three fields analysed, but there are inter-field differences. Authorship patterns are especially marked in the most collaboration-intensive field (i.e. Biology and Biomedicine), where professional rank seems to be more significant than age in determining the role of scientists in research as seen through their authorship patterns, while age has a more significant effect in the least collaboration-intensive field (Natural Resources).

The Assessment of Research Quality in UK Universities: Peer Review or Metrics?

This paper investigates the extent to which the outcomes of the 2008 Research Assessment Exercise in the UK, determined by peer review, can be explained by a set of quantitative indicators. Three cognate units of assessment are examined in detail: business and management, economics and econometrics, and accounting and finance. The main finding is that each of the three components of research activity (namely, research output, esteem and research environment) is highly correlated with various quantitative indicators. A further finding is that the judgement of the Research Assessment Exercise panels was biased in favour of Russell Group universities. There is also evidence of bias by the economics and econometrics panel. The results support the use of quantitative indicators in the research assessment process, particularly a journal quality index. Requiring the panels to take bibliometric indicators into account should help not only to reduce the workload of panels but also to mitigate the problem of implicit bias.

Establishing best practice university research funding strategies using mixed-mode modelling

This paper develops a model representing the university research funding problem under a performance based research funding (PBRF) scheme during the ‘lead-up period’ using a mixed-mode modelling approach (involving soft and hard models) and suggests a solution heuristic. The resultant model facilitates the development of ‘best practice’ strategies to assist in raising the level of research quality and participation, thus placing the university (or academic unit) in the best possible position for facing the final hurdle, the formal research assessment process. This assessment process constitutes the ‘positioning problem’, for which models already exist to assist individual universities to adopt the most favourable strategy. However, the ultimate position of the university depends on the results from the lead-up period. The suggested model facilitates ‘research enhancement’ strategy formulation, evaluation and revision and actively involves the researchers themselves. Benchmarking, an Expert Panel (operating on panel consensus) and subjective strategy impact evaluation are the key tools used. The suggested methodology relies on the knowledge, wisdom and experience of the researchers themselves and will hopefully facilitate the achievement of an academic unit's research goals over the lead-up period preparing them for the ‘positioning problem’. The paper also offers some suggestions as to how to establish and obtain maximum usefulness from the expert panel. Lastly, the suggested solution to the suggested model is user friendly, requiring no more than the use of a spreadsheet.