Taxonomy Of Metrics Research Articles

Bias and Fairness in Large Language Models: A Survey

Abstract Rapid advancements of large language models (LLMs) have enabled the processing, understanding, and generation of human-like text, with increasing integration into systems that touch our social sphere. Despite this success, these models can learn, perpetuate, and amplify harmful social biases. In this article, we present a comprehensive survey of bias evaluation and mitigation techniques for LLMs. We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing, defining distinct facets of harm and introducing several desiderata to operationalize fairness for LLMs. We then unify the literature by proposing three intuitive taxonomies, two for bias evaluation, namely, metrics and datasets, and one for mitigation. Our first taxonomy of metrics for bias evaluation disambiguates the relationship between metrics and evaluation datasets, and organizes metrics by the different levels at which they operate in a model: embeddings, probabilities, and generated text. Our second taxonomy of datasets for bias evaluation categorizes datasets by their structure as counterfactual inputs or prompts, and identifies the targeted harms and social groups; we also release a consolidation of publicly available datasets for improved access. Our third taxonomy of techniques for bias mitigation classifies methods by their intervention during pre-processing, in-training, intra-processing, and post-processing, with granular subcategories that elucidate research trends. Finally, we identify open problems and challenges for future work. Synthesizing a wide range of recent research, we aim to provide a clear guide of the existing literature that empowers researchers and practitioners to better understand and prevent the propagation of bias in LLMs.

A taxonomy of metrics for human representations in digital engineering

AbstractAs humans play a significant role in maintaining, supporting, and operating systems, it is important to include humans in digital engineering models to ensure that the sociotechnical system is capable of meeting stakeholder requirements. The current research proposes a taxonomy of metrics and taxonomic nomenclature useful for representing humans in systems engineering and integration. This taxonomy is intended to provide a structure for considering metrics in frameworks such as the Objectives and Metrics portion of the human view architecture, as well as in trade studies which may be represented in MBSE models. The proposed taxonomy categorizes human‐centered metrics into four categories, including those affecting system performance, system readiness, interface fitness, and occupational health and safety. It is further recognized that many of these metrics can be affected by the mental or physical state of the human, which are commonly captured in human factors metrics, such as workload or situation awareness. It is proposed that this taxonomy can guide the selection of human‐centered metrics which affect the system's ability to meet system stakeholder requirements.

A taxonomy of metrics for GUI-based testing research: A systematic literature review

GUI-based testing is a sub-field of software testing research that has emerged in the last three decades. GUI-based testing techniques focus on verifying the functional conformance of the system under test (SUT) through its graphical user interface. However, despite the research domains growth, studies in the field have low reproducibility and comparability. One observed cause of these phenomena is identified as a lack of research rigor and commonly used metrics, including coverage metrics.

A new metric for the assessment of spatial resolution in satellite imagers

• Spatial resolution quantifies the smallest scene detail captured by an image. • About thirty spatial resolution metrics have been used for satellite imagers. • Most popular spatial resolution metrics are shown to give biased results. • A new unbiased spatial resolution metric is developed from first principles. • This new metric generates a paradigm shift in spatial resolution assessment. A three-criterion taxonomy of metrics to compute the spatial resolution of satellite imagers is presented and used to classify about thirty spatial resolution metrics found in the literature. A new metric, the “Spatial Resolution Function”, is proposed by applying the two-point source resolution criterion to sampled images. This new metric computes resolution distance as a function of the resolving contrast in the image plane, using the sensor’s Point Spread Function as input, and demanding that the output agrees with the Ground Sampling Distance metric for very low optical factors. Further, the application of the Rose criterion to the Spatial Resolution Function allows the dependence of resolution distance on signal to noise ratio to be considered. The procedure to compute the Spatial Resolution Function is applied to three quality types of generic satellite imaging scanners, computing their resolution distances in the along scan and across scan directions, for a wide range of optical factors encountered in Earth observation satellites. The Spatial Resolution Function is used to assess twelve commonly used spatial resolution metrics, the results indicate that all these metrics are biased, showing significant errors when used to compare different satellite imagers. The overall utility of spatial resolution metrics that use a property of the Point Spread Function to roughly estimate resolution distance is challenged, as by using this same function the Spatial Resolution Function allows its exact calculation.

PROFILE-ORIENTED ASSESSMENT OF SOFTWARE REQUIREMENTS QUALITY: MODELS, METRICS, CASE STUDY

The article is devoted to profile-oriented approach to assessing the software requirements profile quality. The process of assessment is implemented in accordance with software requirements profile quality model for external quality of software requirements profile and internal quality of software requirements profile. For assessment of quality of each element of such model (characteristics and attributes of software requirements and their classification features; characteristics and attributes of software requirements profile and their classification features; structure of software requirements profile; semantics and syntax of software requirements) taxonomy of metrics and indicators are introduced, which is a part of software requirements profile quality assessment model. All metrics are divided into qualitative and quantitative groups. Additive convolution method is used to go from the multi-vector values of set of connected indicators or metrics to the single scalar value. Received values of metrics and indicators are visualized by radial-metrics diagrams. An example is discussed, which represents the assessment of external quality of software requirements profile for requirements profile from new standard “Requirements to computer security of NPP Instrumentation and Control Systems (NPP I&C)” developed by Ukrainian state regulatory body.

МОДЕЛІ ТА ОЦІНЮВАННЯ ЯКОСТІ ЗРУЧНОСТІ ВИКОРИСТАННЯ ІНТЕРФЕЙСУ ПРОГРАМНОГО ЗАБЕЗПЕЧЕННЯ ДЛЯ ЛЮДИНО-КОМП'ЮТЕРНОЇ ВЗАЄМОДІЇ

The software quality model describes software quality in terms of non-functional requirements. The most well-known and authoritative quality model ISO/IEC 25010, includes 8 related characteristics: functionality, performance, compatibility, usability, reliability, security, maintainability, and portability. The article materials are limited only by the quality of the software in terms of usability characteristics. The characteristic of the usability of the software should include subcharacteristics inherent in the quality of the user interface, on the one hand, as a static object, and on the other hand, subcharacteristics of the process of interaction with the user – human-computer interaction. Existing quality models and usability assessments do not combine the quality elements of the user interface itself and the user experience. The article proposes models of quality and quality assessment of the usability of the software interface of human-computer interaction, which combine the characteristics inherent directly to the user interface and the characteristics of human-computer interaction. Such models are interconnected due to a single nomenclature of subcharacteristics. The model for assessing the quality of software usability consists of two parts and includes many metrics and indicators that correspond to the indicated sub-characteristics. The purpose of the article is to develop a quality model of the usability of the software interface of human-computer interaction and a corresponding model for assessing its quality, which would combine the subcharacteristics of the quality of the user interface and the subcharacteristics of the quality of its interaction with the user. The object of the research is the subcharacteristics of the usability of the software interface of human-computer interaction. The idea of developing the model is based on the results of the analysis of the following standards: ISO / IEC 25010, ISO / IEC 25022, and ISO / IEC 25022. The provisions (subcharacteristics and metrics) of these standards were taken into account when forming the main material of this article. The taxonomy of metrics and indicators was formed by combining metrics from ISO / IEC 25022, ISO / IEC 25023, and proprietary metrics. As a result, this paper proposes a model for the quality of the usability of the software interface of human-computer interaction and a model for assessing the quality of the usability of the software interface of human-computer interaction.

Impending Collision Judgment from an Egocentric Perspective in Real and Virtual Environments: A Review.

The human egocentric perception of approaching objects and the related perceptual processes have been of interest to researchers for several decades. This article gives a literature review on numerous studies that investigated the phenomenon when an object approaches an observer (or the other way around) with the goal to single out factors that influence the perceptual process. A taxonomy of metrics is followed by a breakdown of different experimental measurement methods. Thereinafter, potential factors affecting the judgment of approaching objects are compiled and debated while divided into human factors (e.g., gender, age, and driving experience), compositional factors (e.g., approaching velocity, spatial distance, and observation time), and technical factors (e.g., field of view, stereoscopy, and display contrast). Experimental findings are collated, juxtaposed, and critically discussed. With virtual-reality devices having taken a tremendous developmental leap forward in the past few years, they have been able to gain ground in experimental research. Therefore, special attention in this article is also given to the perception of approaching objects in virtual environments and put in contrast to the perception in reality.

Using the Taxonomy and the Metrics: What to Study When and Why Comment on "Metrics and Evaluation Tools for Patient Engagement in Healthcare Organization- and System-Level Decision-Making: A Systematic Review".

Dukhanin and colleagues’ taxonomy of metrics for patient engagement at the organizational and system levels has great potential for supporting more careful and useful evaluations of this ever-growing phenomenon. This commentary highlights the central importance to the taxonomy of metrics assessing the extent of meaningful participation in decision-making by patients, consumers and community members; discusses how the purpose of an evaluation and the organizational relationships among key evaluation stakeholders is likely to influence the choice of metrics in important ways; and suggests a recasting of the metrics in the form of a logic model that supports the selection of metrics that are appropriate for a program given its stage of development and the purposes of the study

Metrics for Sustainable Data Centers

There are a multitude of metrics available to analyze individual key performance indicators of data centers. In order to predict growth or set effective goals, it is important to choose the correct metric and be aware of their expressivity and potential limitations. As cloud based services and the use of ICT infrastructure are growing globally, continuous monitoring and measuring of data center facilities are becoming essential to ensure effective and efficient operations. In this work, we explore the diverse metrics that are currently available to measure numerous data center infrastructure components. We propose a taxonomy of metrics based on core data center dimensions. Based on our observations, we argue for the design of new metrics considering factors such as age, location, and data center typology (e.g., co-location center), thus assisting in the strategic data center design and operations processes.

Development and Verification of a Taxonomy of Assessment Metrics for Surgical Technical Skills

To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature. In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables. The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92%) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were "time," "manual techniques" (objective and subjective), "errors," and "procedural steps." Only one new metric, "learning curve," emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance. Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.

A taxonomy of metrics for testing the industrial ecology hypotheses and application to design of freezer insulation

A taxonomy of metrics for testing the industrial ecology hypotheses and application to design of freezer insulation

A taxonomy of metrics for testing the industrial ecology hypotheses and application to design of freezer insulation

Industrial ecology is sometimes referred to as the science of sustainability, but unlike mature sciences, it has yet to establish reliable modes of inquiry or a uniform framework for dialogue—partly because sustainability is difficult to characterize, and partly because industry–environment interactions are difficult to model quantitatively. A number of sustainability metrics have recently been proposed that partially address the quantification problem, but inevitably oversimplify the problem of sustainability characterization. This paper proposes an overarching taxonomy for classifying the quantitative criteria of sustainability as financial, thermodynamic, environmental, ecological, socio-political or aggregated. Several examples are presented. Pollution potential, which is an environmental metric related to the ideal thermodynamic work (of mixing) per mole required to remove pollutants from the environment, is discussed in particular. However, because no single metric can capture sustainability per se, industrial ecology may be limited to elucidating the trade-offs that exist between different types of complementary (rather than substitutable) criteria. The life cycle of polyurethane foam insulation in freezer applications is presented as a quantitative example of how different environmental criteria can lead to different design recommendations. Lastly, the framework proposed herein suggests a research agenda for industrial ecology, especially regarding the linkages between different measures of sustainability and cross-comparison of the investigative methods found in different sciences related to sustainability.

Measures of technology transfer effectiveness: key dimensions and differences in their use by sponsors, developers and adopters

Federally funded R&D has been viewed as a key source of advanced technologies that, if successfully transferred to the private sector, could help rebuild America's global competitiveness. The growing perception that the nation is not getting an adequate return from its federal R&D budget is accompanied by a growing demand for more measurable technology transfer results. Yet measures of technology transfer effectiveness are neither well defined nor universally accepted. This exploratory study focused on defining and describing the measures or metrics used in the process of transferring government-funded technologies to private sector firms. The paper presents an initial conceptual framework and an exploratory, empirically based taxonomy of metrics used in technology transfer. This taxonomy and specific measures were used to help determine which technology transfer metrics were used by various players across the federal technology transfer process. Individuals who played roles as either sponsors, developers or adopters of federally funded technologies were surveyed on their roles and the measures of transfer effectiveness used in their work units. The data showed statistically significant differences in frequency of use of the transfer measures by the three roles. Secondly, a broad set of measures were used in varying degrees by all roles. Most importantly, all three roles used most measures rather infrequently. Recommendations to guide future research are included. Recommendations are also made for technology transfer practitioners. >