Fair Test Research Articles

Bias Behind the Wheel: Fairness Testing of Autonomous Driving Systems

Bias Behind the Wheel: Fairness Testing of Autonomous Driving Systems

Investigating measurement invariance of the IDS-2 intelligence scales between migrant and non-migrant groups

Intelligence plays a crucial role in various aspects of human life, impacting health, academic achievement, and socio-economic success. However, cultural and linguistic disparities in intelligence testing pose challenges, particularly for individuals from migrant backgrounds. This study replicates and extends the landmark study by Wicherts and Dolan (2010) exploring measurement invariance of the German language intelligence test, the Intelligence and Development Scales – 2 (IDS-2), between children and adolescents from migrant (N = 132) and non-migrant (N = 1898) groups. The results revealed partial strict measurement invariance in the IDS-2 intelligence scale subtests across the examined groups. The breach of full strict measurement invariance is primarily due to intercept differences on the verbally loaded subtests—Naming Categories, Naming Opposites, and Story Recall—highlighting the confounding impact of language complexity on test outcomes. These discrepancies resulted in a cumulative intercept difference disadvantaging migrant participants of approximately 4 IQ points on the Full-Scale IQ Score. The findings indicate that while the IDS-2 scales generally assess intelligence consistently across diverse groups, the influence of language complexity on the verbal subtests may result in a disadvantage for children and adolescents with migration backgrounds. To address these biases, we propose the development of non-verbal and culturally fair intelligence tests.

RE-EXAMINING NOZICK’S EXPERIENCE MACHINE IN VIEW OF EMERGING AI COMPANIONS

The aim of this paper is to examine Robert Nozick’s experience machine, assess where his thought experiment could have been framed differently or where it failed to consider other relevant aspects people view as important, such as their values and duties. Additionally, I provide what I consider to be a more fair test, still based on Nozick’s initial thought experiment, primarily changing the duration spent in the experience machine and adding the option to lengthen the sessions according to one’s desires, together with preserving one’s memory of their actual lives, so as to accommodate the issues identified in Nozick’s original setting. In the second half of the paper, I correlate these findings with emerging AI technologies, which promise users companionship and constant support. Finally, the paper concludes that what matters is not what people think or say they will choose, but what they actually choose, calling for a need to lower one’s potentially deluded sense of self-control in relation to the kind of content we consume, especially online.

Fairness Testing of Machine Translation Systems

Machine translation is integral to international communication and extensively employed in diverse human-related applications. Despite remarkable progress, fairness issues persist within current machine translation systems. In this article, we propose FairMT, an automated fairness testing approach tailored for machine translation systems. FairMT operates on the assumption that translations of semantically similar sentences, containing protected attributes from distinct demographic groups, should maintain comparable meanings. It comprises three key steps: (1) test input generation, producing inputs covering various demographic groups; (2) test oracle generation, identifying potential unfair translations based on semantic similarity measurements; and (3) regression, discerning genuine fairness issues from those caused by low-quality translation. Leveraging FairMT, we conduct an empirical study on three leading machine translation systems–Google Translate, T5, and Transformer. Our investigation uncovers up to 832, 1,984, and 2,627 unfair translations across the three systems, respectively. Intriguingly, we observe that fair translations tend to exhibit superior translation performance, challenging the conventional wisdom of a fairness-performance tradeoff prevalent in the fairness literature.

Fairness Testing: A Comprehensive Survey and Analysis of Trends

Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedicated to conducting fairness testing of ML software, and this article offers a comprehensive survey of existing studies in this field. We collect 100 papers and organize them based on the testing workflow (i.e., how to test) and testing components (i.e., what to test). Furthermore, we analyze the research focus, trends, and promising directions in the realm of fairness testing. We also identify widely adopted datasets and open-source tools for fairness testing.

Vox populi, vox dei? The effect of sociotropic and egocentric incongruence on democratic preferences

AbstractSystemic congruence between the whole legislature and the whole electorate (‘many‐to‐many’, or sociotropic congruence) should be the benchmark to evaluate a democratic system. Yet, most studies link shifts in democratic preferences to individual‐level representation (‘many‐to‐one’, or egocentric incongruence), since individual‐level representation failures should be more salient and visible for individual citizens. We argue that the sociotropic incongruence hypothesis has not been appropriately tested to date, because the measure does not vary at individual level in observational data. Using an experiment conducted in France, we manipulate various sociotropic (in)congruence scenarios at the individual level. In addition to the incongruence hypotheses, our original experiment tests whether offering expertise‐based justifications to incongruence attenuates the backlash against representatives. We find that, even when giving sociotropic incongruence a fair test, egocentric incongruence still consistently shapes democratic preferences, while the effect of sociotropic incongruence remains negligible. Furthermore, we find that narratives rooted in expertise claims do not attenuate the effect of representation failure on backlash against representative democracy: they exacerbate it.

Distribution-aware fairness test generation

Ensuring that all classes of objects are detected with equal accuracy is essential in AI systems. For instance, being unable to identify any one class of objects could have fatal consequences in autonomous driving systems. Hence, ensuring the reliability of image recognition systems is crucial. This work addresses how to validate group fairness in image recognition software. We propose a distribution-aware fairness testing approach (called DISTROFAIR) that systematically exposes class-level fairness violations in image classifiers via a synergistic combination of out-of-distribution (OOD) testing and semantic-preserving image mutation. DISTROFAIR automatically learns the distribution (e.g., number/orientation) of objects in a set of images. Then it systematically mutates objects in the images to become OOD using three semantic-preserving image mutations – object deletion, object insertion and object rotation. We evaluate DISTROFAIR using two well-known datasets (CityScapes and MS-COCO) and three major, commercial image recognition software (namely, Amazon Rekognition, Google Cloud Vision and Azure Computer Vision). Results show that about 21% of images generated by DISTROFAIR reveal class-level fairness violations using either ground truth or metamorphic oracles. DISTROFAIR is up to 2.3× more effective than two main baselines, i.e., (a) an approach which focuses on generating images only within the distribution (ID) and (b) fairness analysis using only the original image dataset. We further observed that DISTROFAIR is efficient, it generates 460 images per hour, on average. Finally, we evaluate the semantic validity of our approach via a user study with 81 participants, using 30 real images and 30 corresponding mutated images generated by DISTROFAIR. We found that images generated by DISTROFAIR are 80% as realistic as real-world images.

Participating in Competitions to Go to University: The Discourse of Ballet Competitions in South Korea

This study unravels the intricate dynamics of ballet competitions in South Korea, highlighting the interplay between domestic and international competitions. Expected to function as fair and credible tests, domestic competitions in South Korea are entwined with academicism, neoliberalism, and private education. Meanwhile, international ballet competitions are perceived as more prestigious and occupy a hierarchical status over domestic events, which influences and, at times, compromises local practices and discourses. This research examines the interplay of nationalism, commercialism, and genre rivalry in dance competition to emphasize how these events in South Korea serve diverse entities and play both symbolic and practical roles.

Can the late dark energy parameterization reconcile the Hubble tension?* *Ming-Jian Zhang is Supported by the Natural Science Foundation of Shandong Province, China (ZR2021MA075), and the Talent Research Project of Qilu University of Technology (2023RCKY029). Li Chen is Supported by the Natural Science Foundation of Shandong Province, China (ZR2019MA033)

In this study, we constructed ten dark energy models to test whether they can reconcile the Hubble tension and how much it is affected by parameterization. To establish a fair test, the models are diverse, encompassing fractional, logarithmic, exponential, and inverse exponential forms as well as several non-parameterized models. The dataset we used includes the NPIPE pipeline of cosmic microwave background (CMB) power-spectrum data from Planck2020, Pantheon+ samples from Supernovae Type Ia, and baryon acoustic oscillations. The MCMC calculations imply dark energy transferring from to for the four parameterized dark energy models. However, these models cannot adequately reconcile the Hubble tension. Notably, we found that phantom-like dark energy with can achieve the greatest reduction in the Hubble tension to . However, AIC analysis indicates that this alleviation is at the cost of high AIC. We also investigated the effect of constructions on the derivative of the equation of state , cosmic density parameter, CMB power spectrum , and matter spectra . We also found that the Hubble tension may be related to the reionization process.

THE EFFECT OF ORGANIZATIONAL FAIRNESS AND JOB CHARACTERISTICS ON EMPLOYEE ATTACHMENT

The purpose of this study is to determine the Effect of Organizational Justice and Job Characteristics on Employee Attachment to PT. Ching Luh Indonesia, either partially or simultaneously at PT. Ching Luh Indonesia. This study used associative method with quantitative approach. A lot of sampling was 78 people. Descriptive data analysis techniques are used to determine the picture of respondents' answer responses and data distribution, while inferential statistical analysis is for hypothesis testing. Data processing is assisted by Microsoft excel and SPSS software version 25. Based on the correlation test of Organizational Fairness (X1), and Job Characteristics (X2) with Employee Attachment (Y) have a strong relationship. Organizational Fairness and Job Characteristics can project Organizational Fairness with a positive influence, this can be seen from the equation Y = 2.881 + 0.622X11 + 0.304X21 which means that the higher or increased the Employee Attachment. Organizational Fairness and Job Characteristics contributed 55.30% while the remaining 44.70% was influenced by other variables classified in epsilon, namely other factors that affect Employee Attachment but were not careful in this study.

Generalizability of the Swedish WISC-V to the Finland-Swedish minority – the FinSwed study

ABSTRACT International guidelines highlight the importance of using appropriate and culturally fair test materials when conducting clinical psychological assessments. In the present study, the generalizability of the Swedish WISC-V with Scandinavian normative data was explored in 6–16-year-old Swedish-speaking children in Finland (N = 134), as no local test versions or norms are available for this minority. First, metric measurement invariance was established, i.e., the constructs measured were equivalent between the standardization data and the present sample. Second, the performance of this minority group on the Swedish WISC-V was compared to the Scandinavian normative mean. The findings showed that the Finland-Swedish children performed overall higher than the normative mean on the Swedish WISC-V, with an FSIQ of 103. The performance was significantly higher also in the indexes VSI, FRI, and WMI as well as in several subtests. However, in the subtest Vocabulary, the Finland-Swedish children achieved significantly lower scores than the Scandinavian mean. Further analyses showed significant associations between cognitive performance and age as well as parental education. For the VCI and the FSIQ, performance increased significantly with age, despite the use of age-standardized scaled scores. The general high performance was suggested to relate to the overall high educational level of the Finland-Swedes as well as to other cultural and test-related factors. The results have implications for clinicians conducting assessments with this minority, but also highlight the importance of establishing test fairness by validating tests when used in different cultural groups.

Math items about real-world content lower test-scores of students from families with low socioeconomic status

In many countries, standardized math tests are important for achieving academic success. Here, we examine whether content of items, the story that explains a mathematical question, biases performance of low-SES students. In a large-scale cohort study of Trends in International Mathematics and Science Studies (TIMSS)—including data from 58 countries from students in grades 4 and 8 (N = 5501,165)—we examine whether item content that is more likely related to challenges for low-SES students (money, food, social relationships) improves their performance, compared with their average math performance. Results show that low-SES students scored lower on items with this specific content than expected based on an individual’s average performance. The effect sizes are substantial: on average, the chance to answer correctly is 18% lower. From a hidden talents approach, these results are unexpected. However, they align with other theoretical frameworks such as scarcity mindset, providing new insights for fair testing.

Face-Inception-Net for Recognition

Face recognition in general scenarios has been saturated in recent years, but there is still room to enhance model performance in extreme scenarios and fairness situations. Inspired by the successful application of Transformer and ConvNet in computer vision, we propose a FIN-Block, which gives a more flexible composition paradigm for building a novel pure convolution model and provides a foundation for constructing a new framework for general face recognition in both extreme scenarios and fairness situations. FIN-Block-A uses a combination of stacked large-size convolution kernels and parallel convolution branches to ensure a large spatial receptive field while improving the module’s deep feature embedding and extraction capabilities. FIN-Block-B takes advantage of stacked orthogonal convolution kernels and parallel branches to balance model size and performance. By applying FIN-Block with an adapted convolution kernel size in different stages, we built a reasonable and novel framework Face-Inception-Net, and the performance of the model is highly competitive with ConvNeXt and InceptionNeXt. The models were trained on CASIA-WebFace and MS-wo-RFW databases and evaluated on 14 mainstream benchmarks, including LFW, extreme scene, and fairness test sets. The proposed Face-Inception-Net achieved the highest average TAR@FAR0.001 of 95.9% in all used benchmarks, fully demonstrating effectiveness and generality in various scenarios.

BiasTrap: Runtime Detection of Biased Prediction in Machine Learning Systems

Machine Learning (ML) systems are now widely used across various fields such as hiring, healthcare, and criminal justice, but they are prone to unfairness and discrimination, which can have serious consequences for individuals and society. Although various fairness testing methods have been developed to tackle this issue, they lack the mechanism to continuously monitor ML system behaviour at runtime. In this study, a runtime verification tool called BiasTrap is proposed to detect and prevent discrimination in ML systems. The tool combines data augmentation and bias detection components to create and analyse instances with different sensitive attributes, enabling the detection of discriminatory behaviour in the ML model. The simulation results demonstrate that BiasTrap can effectively detect discriminatory behaviour in ML models trained on different datasets using various algorithms. Therefore, BiasTrap is a valuable tool for ensuring fairness in ML systems in real-time.

Reassessing Proportionality: Implied Limitations and Judicial Review under Section 6 of the British Nationality Act 1981

Abstract There exists a lively debate regarding whether proportionality outside of the European Convention on Human Rights is an independent ground for judicial review or overlaps with the Wednesbury Unreasonableness or irrationality ground and is indistinguishable from it.1 This Article contends that proportionality, as a fair balance test, is an implied limitation to a statutory power and, therefore, constitutes an independent test within section 6 of the British Nationality Act 1981 (BNA 1981). Consequently, if a decision is disproportionate, it is amenable to judicial review on the existing ground of illegality. In that case, it is not necessary to consider an independent Wednesbury unreasonableness or irrationality test.

Effectiveness of Interferential Therapy with Hip Muscle Strengthening Exercises and Ultrasound Therapy with Hip Muscle Strengthening Exercises for Piriformis Syndrome

Background: Sciatic nerve irritation caused by the piriformis muscle ,which results in Buttock pain, sciatica, or bothis known as piriformis syndrome. The prevalence of PS with sciatic nerve piercing is 17% and 81% respectively,while it was present in 17.2% suffering from low back discomfort.Purpose: The purpose of this study is to find the effectiveness of interferential therapy with hip muscle strengtheningexercises and ultrasound therapy with hip muscle strengthening exercises for piriformis syndrome.Materials and Methods: About 220 participants were selected by using FAIR test and visual analogue scale oninclusion and exclusion criteria. Written informed consent was collected before the commencement of the study. The220 participants were allocated into 2 groups. Interferential therapy group (n=110) and ultrasound therapy group(n=110). Interferential therapy group were given interferential therapy with hip muscle strengthening exercisesand the ultrasound therapy group were given ultrasound therapy with hip muscle strengthening exercises. Theintervention was given 5 times a week for a total duration of 2 weeks. Study Period: October 2022-June 2023.Results: The collected data was statistically analyzed using paired and unpaired t test, when comparing theInterferential therapy group to the ultrasound therapy, the Interferential therapy group indicates significanteffects P<0.0001 in piriformis syndrome, as assessed by VAS.Conclusion: This study concluded that the Interferential therapy group was found to be more effective than theultrasound therapy group in patients with Piriformis Syndrome.

Pinch force sense test–retest reliability evaluation using contralateral force matching task

A high test–retest reliability in measurement of pinch force sense is required to assess a clinical parameter accurately over a longitudinal study. Ipsilateral reproduction (IR) task and contralateral matching (CM) task have commonly been used for the assessment of force sense. To date, there has been little research on the test–retest reliability of pinch force sense utilizing the contralateral force matching task. This research aimed to explore this phenomenon across a spectrum of reference force levels (10, 30, and 50 percent maximum voluntary isometric contraction (MVIC)) using a contralateral matching task. Every participant in the study was tested twice by the same skilled experts, with each session separated by one week. Although normalized variable error indicated a poor level of reliability (intraclass correlation coefficient (ICC) = − 0.25 to 0.05) for these force sense tests, normalized constant error (ICC = 0.76–0.85) and normalized absolute error (ICC = 0.61–0.81) results indicated a fair to good of reliability. The lower bound of 95% CI of ICC for NAE and NCE indicated fair test–retest reliability (0.41–0.69). These findings suggest that investigators can reasonably obtain a fair to good test–retest reliability when investigating pinch force sense using the contralateral matching task. The Bland–Altman plots, SEM, and MDD95% were lower at these lower reference force level (10% MVIC) compared to the level of higher reference forces (30% and 50% MVIC). Therefore, when the reference force level increases, the participant needs a larger NAE or NCE decrease to show that their pinch force sense has indeed improved.

AN EVALUATION OF STEREO AND MULTIVIEW ALGORITHMS FOR 3D RECONSTRUCTION WITH SYNTHETIC DATA

Abstract. The reconstruction of 3D scenes from images has usually been addressed with two different strategies, namely stereo and multiview. The former requires rectified images and generates a disparity map, while the latter relies on the camera parameters and directly retrieves a depth map. For both cases, deep learning architectures have shown an outstanding performance. However, due to the differences between input and output data, the two strategies are difficult to compare on a common scene. Moreover, for remote sensing applications multi-view data is hard to acquire and the ground truth is either sparse or affected by outliers. Hence, in this article we evaluate the performance of stereo and multi-view architectures trained on synthetic data resembling remote sensing images. The data has been and processed and organized to be compatible with both kind of neural networks. For a fair comparison, training and testing are done only with two views. We focus on the accuracy of the reconstruction, as well as the impact of the depth range and the baseline of the stereo array. Results are presented for deep learning architectures and non-learning algorithms.

Diversity-aware fairness testing of machine learning classifiers through hashing-based sampling

Context:There are growing concerns about algorithmic fairness, as some machine learning (ML)-based algorithms have been found to exhibit biases against protected attributes such as gender, race, age and so on. Individual fairness requires an ML classifier to produce similar outputs for similar individuals. Verification Based Testing (Vbt) is a state-of-the-art black-box testing algorithm for individual fairness that leverages constraint solving to generate test cases. Objective:Generating diverse test cases is expected to facilitate efficient detection of diverse discriminatory data instances (i.e., cases that violate individual fairness). Hashing-based sampling techniques draw a sample approximately uniformly at random from the set of solutions of given Boolean constraints. We propose Vbt-X, which improves Vbt with hashing-based sampling, aiming to improve its testing performance. Method:We realize hashing-based sampling for Vbt. The challenge is that the off-the-shelf hashing-based sampling techniques cannot be integrated in a straightforward manner because the constraints in Vbt are generally not Boolean. Moreover, we propose several enhancement techniques to make Vbt-X more efficient. Results:To evaluate our method, we conduct experiments, where Vbt-X is compared to Vbt, Sg and ExpGA (other well-known fairness testing algorithms) over a set of configurations consisting of several datasets, protected attributes, and ML classifiers. The results show that, with each configuration, Vbt-X detects more discriminatory data instances with higher diversity than Vbt and Sg. Vbt-X detects discriminatory data instances with higher diversity than ExpGA, though the number of discriminatory data instances detected by Vbt-X is lesser than ExpGA. Conclusion:Our proposed method performs better than other state-of-the-art black-box fairness testing algorithms, particularly in terms of diversity. Our method can serve to efficiently identify flaws in ML classifiers with respect to individual fairness for subsequent improvements of an ML classifier. On the other hand, although our method is specific to individual fairness, it could work for testing other aspects of a software system such as security and counterfactual explanations with some technical adaptations, which remains for future work.

A study of performance comparison of databases for HALF data archiving system

The Hefei Advanced Light Facility (HALF) is a 2.2 GeV fourth-generation synchrotron radiation light source. The HALF control system is a distributed control system based on Experimental Physics and Industrial Control System (EPICS). As the essential part of the HALF control system, the Historical Data Archiving System (HDAS) is responsible to store operational data for the entire facility including the accelerator and beamlines, and provides the functions for data query and analysis. According to the estimation based on the HALF scale, approximately 25,000 EPICS PVs will be stored in HDAS, and these accumulated massive data require a dedicated database for persistent storage and management. Under the EPICS PV data scenario of HALF, a fair database test platform is designed and built to test the read-write performance of databases commonly used in the particle accelerator field. The tested objects include EPICS Archiver Appliance and the five databases MongoDB, HBase, InfluxDB, TimescaleDB, and Cassandra. Additionally, research on the five databases software ecosystem is conducted for the development and extension of HDAS. The test results indicate that TimescaleDB has the fastest read performance, and 1.4×10^6 items of data can be read per second. TimescaleDB also has superior write performance. 1.2 × 105 items of data can be written per second, which can fully meet the writing requirements of HDAS. Regarding the software ecosystem, TimescaleDB is compatible with a larger number of data visualization software and big data processing frameworks. Therefore, TimescaleDB is more suitable as the HDAS database. In the future, a TimescaleDB distributed cluster will be designed and deployed, and an HDAS prototype system will be developed based on this cluster.