Role Of Metrics Research Articles

Rhetorical Metrics Building Securities Regulation in America’s Era of Booms and Busts, 1890-1940

AbstractHow do regulators regulate with metrics? This article offers a rhetorical approach to this question, using early U.S. securities regulation as a case in point, and reliance on credit ratings as empirical illustration. A rhetorical approach challenges economists’ claim that metrics are limited to providing technical guidance to policy formation: the fact is that the role of metrics in regulation can be appreciated only if technical and social aspects are considered together. A rhetorical perspective also fills an important gap in sociological studies of “co-production” that claim that procedural deliberation enhances the legitimacy of regulation but underemphasize the role of quantification when procedural rules are lacking. This article suggests that rhetoric is not suboptimal or irrational but a vital form of deliberation in contexts of uncertainty, when decision-making requires some amount of persuasion outside a procedural context. I observe that metrics can be a powerful vehicle of rhetorical change. Two components of rhetorical metrics are highlighted. First are cognitive clutches, or the capacity to shift prevailing models of attention. Second are actionable arguments, the capacity to embed cognitive deviance into a compelling argument for change. I conclude with reflections on the legacy of rhetorical decisions on current policy debates.

Can reporting more lead to less? The role of metrics in assessing liver transplant program performance.

Appropriate metrics for performance analysis is an active topic of debate within the transplant community. This study explores current proposals on metric expansion as well as potential metrics and prospective collaborations that have not received widespread discussion within the transplant community. The premature introduction of additional, nonvalidated metrics risks behaviors that may undermine donor utilization and patient access to transplantation.

The Role of Metrics in Food Policy: Lessons from a Decade of Experience in New York City

In the last decade, New York City developed food policies designed to improve access to healthy food, reduce food insecurity, support community development, promote sustainable food systems, and improve conditions for food workers. Since 2012, the New York City Council has mandated the Mayor’s Office to prepare annual Food Metrics Reports to present data on selected food system indicators. This article uses these reports to assess how the metrics describe the city’s progress in implementing municipal food policies set in the last decade. Our analysis examines: (1) changes in the indicators that the city reports; (2) strengths and weaknesses of the Food Metrics Reports as a tool for monitoring policy enactment and impact; and (3) opportunities for improvements to the indica­tors and the development and implementation of future metrics. We found that the reports show improvements in 51% of the 37 indicators and sub-indicators, declines in 40% and no change or no assessment in the remaining indicators. While the food metrics process has provided valuable data on the implementation of selected city food policies, it has several limitations. By adding new indicators, tapping into additional data sources, and engaging additional constituencies in the process, New York City food metrics could play a more useful role in helping New York City to set goals and monitor progress towards the development of a more equitable, efficient, and sustainable munici­pal food system. The experience with food metrics in New York City suggests lessons for the use of food policy monitoring to improve food systems in other cities.

Can innovation be measured? A framework of how measurement of innovation engages attention in firms

Abstract Many firms manage the innovation process by using metrics. Yet, whether measurement supports or hinders innovation continues to be a topic of debate. To shed new light on this debate, this paper presents a conceptual framework of how measurement engages attention in firms. We draw on attention based theory and conceptualize innovation measurement as an attention-focusing device. We identify two ideal types of measurement practices. i) Directional Measurement: which is based on few and unidirectional metrics and encourages exploitative innovation efforts. ii) Conversational Measurement: which is based on multiple and ambiguous metrics and encourages exploration. We extend theory building in the technology and accounting literatures by theorizing the role of metrics and measurement for attention and by discussing the implications of such attentional engagement for innovation performance. In so doing, we engage closely with the managerial task of managing innovation while simplifying its conditions, thereby providing actionable advice.

Counting Clicks: Quantification and Variation in Web Journalism in the United States and France

Sociological studies often emphasize the role of metrics in broader processes of convergence and homogenization. Yet numbers can take on different meanings depending on their contexts. This article focuses on the case of journalism, a field transformed by quantification in the form of “clicks.” Drawing on ethnographic material gathered at two news websites—one in New York, the other in Paris—it documents important differences in the uses and meanings assigned to audience metrics in the United States and France. At the U.S. website, editors make significant decisions based on metrics, but staff journalists are relatively unconcerned by them. At the French website, however, editors are conflicted about metrics, but staff writers fixate on them. To understand these differences, this article analyzes how the trajectories of the U.S. and French journalistic fields affect newsroom dynamics. It shows how cultural differences can be reproduced at a time of technological convergence.

Co-evolving with self-tracking technologies

Seen in a longitudinal perspective, Quantified Self-inspired self-tracking sets up “a laboratory of the self,” where people co-evolve with technologies. By exploring ways in which self-tracking technologies energize everyday aims or are experienced as limiting, we demonstrate how some aspects of the self are amplified while others become reduced and restricted. We suggest that further developing the concept of the laboratory of the self renews the conversation about the role of metrics and technologies by facilitating comparison between different realms of the digital, and demonstrating how services and devices enlarge aspects of the self at the expense of others. The use of self-tracking technologies is inscribed in, but also runs counter to, the larger political-economy landscape. Personal laboratories can aid the exploration of how the techno-mediated selves fit into larger structures of the digital technology market and the role that metrics play in defining them.

From compliance to co-production: Emergent forms of agency in Sustainable Wine Production in New Zealand

This article engages with non-human agency through the interrogation of the emerging role of metrics in the governance of sustainability in the New Zealand primary sector. In it, we argue that the agency of the metrics builds on previous work that has elaborated the impact of audited best practice on the subjectivities of producers and processors, including the recent examination of the active influence of metrics that engenders unexpected and uncontrolled change in social networks of production. In this case, the analysis of the influence of metrics shifts to those used within a recently introduced ‘learning’ tool (Wine Industry Sustainability Engine) that can be classified as an effort in transition management. The capacity of metrics as agents is already apparent in the perceived interactions and engagements with the Wine Industry Sustainability Engine tool as expressed by likely users during assessments of the usability of initial pilot software. Using their response, we demonstrate that, despite intentions to use the tool to foster particular sets of practices and ethics through benchmarking, the metrics have multiple roles in production worlds—compelling compliance to regulations, creating new ways to communicate complex relations and practices, and generating information for reflexive self-evaluation. Through these roles, we argue, metrics clearly operate as both a material and ontological non-human actor, expressed in different ways in different assemblages. This conclusion has implications for the application of transition management more broadly, and helps us to better understand what we want metrics to accomplish, what they can accomplish, and the possible gap between the two.

Data literacies for the postdemographic social media self

In a postdemographic world, characterized by the continuous production and calculation of social data in the form of likes, comments, shares, keywords, locations or hashtags, social media platforms are designed with techniques of market segmentation in mind. “Datafication” challenges the agency of participatory social media practices and traditional accounts of the presentation of self in the use of social media. In the process, a tension or paradox arises between the personal, curative or performative character of social media practices and the calculative design and commercial usefulness of platforms and apps. In this paper I interrogate this paradox, and explore the potential role of metrics and analytics for emergent data literacies. By drawing together common self-oriented metrics across dominant platforms, the paper emphasizes analytics targets around a) profile, b) activity, c) interactivity and d) visibility, as a step toward developing new data literacies.

Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective

Inspired by The Metric Tide report (2015) on the role of metrics in research assessment and management, and Lord Nicholas Stern’s report Building on Success and Learning from Experience (2016), which deals with criticisms of REF2014 and gives advice for a redesign of REF2021, this article discusses the possible implications for other countries. It also contributes to the discussion of the future of the REF by taking an international perspective. The article offers a framework for understanding differences in the motivations and designs of performance-based research funding systems (PRFS) across countries. It also shows that a basis for mutual learning among countries is more needed than a formulation of best practice, thereby both contributing to and correcting the international outlook in The Metric Tide report and its supplementary Literature Review.

Uncertainty Quantification Metrics with Varying Statistical Information in Model Calibration and Validation

Test-analysis comparison metrics are mathematical functions that provide a quantitative measure of the agreement (or lack thereof) between numerical predictions and experimental measurements. While calibrating and validating models, the choice of a metric can significantly influence the outcome, yet the published research discussing the role of metrics, in particular, varying levels of statistical information the metrics can contain, has been limited. This paper calibrates and validates the model predictions using alternative metrics formulated based on three types of distance-based criteria: 1) Euclidian distance (i.e., the absolute geometric distance between two points), 2) Mahalanobis distance (i.e., the weighted distance that considers the correlations of two point clouds), and 3) Bhattacharyya distance (i.e., the statistical distance between two point clouds considering their probabilistic distributions). A comparative study is presented in the first case study, where the influence of various metrics, and the varying levels of statistical information they contain, on the predictions of the calibrated models is evaluated. In the second case study, an integrated application of the distance metrics is demonstrated through a cross-validation process with regard to the measurement variability.

Scholarly Metrics Baseline: A Survey of Faculty Knowledge, Use, and Opinion about Scholarly Metrics

This article presents the results of a faculty survey conducted at the University of Vermont during academic year 2014–2015. The survey asked faculty about: familiarity with scholarly metrics, metric-seeking habits, help-seeking habits, and the role of metrics in their department’s tenure and promotion process. The survey also gathered faculty opinions on how well scholarly metrics reflect the importance of scholarly work and how faculty feel about administrators gathering institutional scholarly metric information. Results point to the necessity of understanding the campus landscape of faculty knowledge, opinion, importance, and use of scholarly metrics before engaging faculty in further discussions about quantifying the impact of their scholarly work.

Science deserves to be judged by its contents, not by its wrapping: Revisiting Seglen's work on journal impact and research evaluation.

The scientific foundation for the criticism on the use of the Journal Impact Factor (JIF) in evaluations of individual researchers and their publications was laid between 1989 and 1997 in a series of articles by Per O. Seglen. His basic work has since influenced initiatives such as the San Francisco Declaration on Research Assessment (DORA), the Leiden Manifesto for research metrics, and The Metric Tide review on the role of metrics in research assessment and management. Seglen studied the publications of only 16 senior biomedical scientists. We investigate whether Seglen’s main findings still hold when using the same methods for a much larger group of Norwegian biomedical scientists with more than 18,000 publications. Our results support and add new insights to Seglen’s basic work.

Metrology and sustainability: Using sustainability audits in New Zealand to elaborate the complex politics of measuring

The study of metrology has emerged in the last couple of years as a useful new approach to understanding economic practices and networks. In this paper, we draw on research experience with the New Zealand kiwifruit and wine sectors as well as the development of an indigenous branding project to examine the role of metrics in promoting sustainable practice. We first identify two more commonly theorised aspects in which metrics operate: as measures (i.e., simple representations of uncontested values, and as tools (i.e., signifiers of the power of wider institutions or structures. We also argue, however, that metrics have operated in a third, potentially more controversial, manner in exerting their own power as 'material agents' within economic networks. In each case, there are elements of the development of new economic practices that suggest that metrics can work as material agents in re-organising economic activity and reordering social networks

Scholarly Metrics Baseline: A Survey of Faculty Knowledge, Use, and Opinion about Scholarly Metrics

This article presents the results of a faculty survey conducted at the University of Vermont during academic year 2014–2015. The survey asked faculty about: familiarity with scholarly metrics, metric-seeking habits, help-seeking habits, and the role of metrics in their department’s tenure and promotion process. The survey also gathered faculty opinions on how well scholarly metrics reflect the importance of scholarly work and how faculty feel about administrators gathering institutional scholarly metric information. Results point to the necessity of understanding the campus landscape of faculty knowledge, opinion, importance, and use of scholarly metrics before engaging faculty in further discussions about quantifying the impact of their scholarly work.

Role of Public Organic Procurement Policies (POPP’s) in the implementation of organic food and farming strategies:

Targeting public procurement policies has become an important goal for European policy makers since it is believed to contribute to more sustainable food systems and food consumption patterns. Food purchased by the public for institutions such as hospitals, care homes, universities, prisons, schools, armed forces and canteens represent a significant part of the food economy in European countries. As a result governments at different levels have adopted policies that promote organic food and farming in many countries. Since agencies and institutions of the public can be expected to be influential by setting good examples of food consumption, the food for the public plate has come to take a much more visible role in public service provision policymaking over the past decades. However, for such strategies to be effective the right policy mix is essential. This paper reports on the Danish case of Public Organic Procurement Policy (POPP’s) and its role in organic food and farming strategies. It gives a brief account on component of the policy mix including the setting up of a labelling system, a monitoring system for sales volumes and a foodservice workers training program. The paper examines the first results from the monitoring system and provides insight in the progress of POPP’s. It details the implementation and maintenance of the monitoring program and discusses the role of metrics in relation to other policy tools as well as the contribution that monitoring can make in terms of policy implementation.

The changing role of metrics in research institute evaluations undertaken by the Chinese Academy of Sciences (CAS)

AbstractAs an efficient S&T management tool, the Chinese Academy of Sciences (CAS) institute evaluations have experienced four major systems in which quantitative indicators have played a great role. To better understand the roles of quantitative indicators in CAS institute evaluations and institute development, this article aims to discuss the history and development of the CAS institute evaluation systems. It is concluded that institute evaluation has played a significant role not only in promoting scientific progress but also in guiding the direction and focus of the institutes’ development, thereby aligning it with the CAS’s strategy at different phases. Our intention is that this article offers useful insights for a wide scientific community into CAS’ attitudes on the application of metrics in research institute evaluations. This article is published as part of a collection on the future of research assessment.

Metrics and Assessment

An important and timely plenary session at the 2015 UKSG Conference and Exhibition focused on the role of metrics in research assessment. The two excellent speakers had slightly divergent views.Todd Carpenter from NISO (National Information Standards Organization) argued that altmetrics aren’t alt anymore and that downloads and other forms of digital interaction, including social media reference, reference tracking, personal library saving, and secondary linking activity now provide mainstream approaches to the assessment of scholarly impact. James Wilsdon is professor of science and democracy in the Science Policy Research Unit at the University of Sussex and is chair of the Independent Review of the Role of Metrics in Research Assessment commissioned by the Higher Education Funding Council in England (HEFCE). The outcome of this review will inform the work of HEFCE and the other UK higher education funding bodies as they prepare for the future of the Research Excellence Framework. He is more circumspect arguing that metrics cannot and should not be used as a substitute for informed judgement. This article provides a summary of both presentations.

We need a measured approach to metrics.

Metrics evoke a mixed reaction from the research community. A commitment to using data and evidence to inform decisions makes many of us sympathetic to, even enthusiastic about, the prospect of granular, real-time analysis of our own activities. Yet we only have to look at the blunt use of metrics such as journal impact factors, h-indices and grant-income targets to be reminded of the pitfalls. Some of the most precious qualities of academic culture resist simple quantification, and individual indicators can struggle to do justice to the richness and plurality of our research. This article gives an overview of the findings of the UK's independent review of the role of metrics in research assessment, which published its final report on 9 July 2015.

Business model metrics: an open repository

Development of successful business models has become a necessity in turbulent business environments, but compared to research on business modeling tools, attention to the role of metrics in designing business models in literature is limited. Building on existing approaches to business models and performance measurement literature, we develop a generic open repository of metrics related to core business model concepts. We validate and assess the practical value of the repository based on four e-Business model cases. The repository can be utilized in designing business models, specifically for networked enterprises. It is neither complete nor universally applicable, but needs adjustment with each design. Ultimately, with this paper, we hope measurement will become an integral part of the business model innovation discussion.

Finding “good enough” metrics for the UK’s research excellence framework

Purpose – The purpose of this paper is to encourage discussion about the potential role of metrics in research assessment. Design/methodology/approach – The paper considers the results of the UK’s 2014 Research Excellence Framework, and the potential of metrics to reduce the size of future exercises. Findings – The paper suggests that a battery of non-robust metrics selected by the higher education institutions could support a greatly reduced peer-reviewed exercise. Practical implications – Policy makers should reconsider the role of metrics in research assessment. Originality/value – The paper provides an alternative vision of the role of metrics in research assessment, and will be of interest to policy makers, bibliometricians, and anyone interested in HEI research assessment.