Algorithmic Decision-making Systems Research Articles

Benefits and harms associated with the use of AI-related algorithmic decision-making systems by healthcare professionals: a systematic review

Despite notable advancements in artificial intelligence (AI) that enable complex systems to perform certain tasks more accurately than medical experts, the impact on patient-relevant outcomes remains uncertain. To address this gap, this systematic review assesses the benefits and harms associated with AI-related algorithmic decision-making (ADM) systems used by healthcare professionals, compared to standard care. In accordance with the PRISMA guidelines, we included interventional and observational studies published as peer-reviewed full-text articles that met the following criteria: human patients; interventions involving algorithmic decision-making systems, developed with and/or utilizing machine learning (ML); and outcomes describing patient-relevant benefits and harms that directly affect health and quality of life, such as mortality and morbidity. Studies that did not undergo preregistration, lacked a standard-of-care control, or pertained to systems that assist in the execution of actions (e.g., in robotics) were excluded. We searched MEDLINE, EMBASE, IEEE Xplore, and Google Scholar for studies published in the past decade up to 31 March 2024. We assessed risk of bias using Cochrane's RoB 2 and ROBINS-I tools, and reporting transparency with CONSORT-AI and TRIPOD-AI. Two researchers independently managed the processes and resolved conflicts through discussion. This review has been registered with PROSPERO (CRD42023412156) and the study protocol has been published. Out of 2,582 records identified after deduplication, 18 randomized controlled trials (RCTs) and one cohort study met the inclusion criteria, covering specialties such as psychiatry, oncology, and internal medicine. Collectively, the studies included a median of 243 patients (IQR 124-828), with a median of 50.5% female participants (range 12.5-79.0, IQR 43.6-53.6) across intervention and control groups. Four studies were classified as having low risk of bias, seven showed some concerns, and another seven were assessed as having high or serious risk of bias. Reporting transparency varied considerably: six studies showed high compliance, four moderate, and five low compliance with CONSORT-AI or TRIPOD-AI. Twelve studies (63%) reported patient-relevant benefits. Of those with low risk of bias, interventions reduced length of stay in hospital and intensive care unit (10.3 vs. 13.0 days, p=0.042; 6.3 vs. 8.4 days, p=0.030), in-hospital mortality (9.0% vs. 21.3%, p=0.018), and depression symptoms in non-complex cases (45.1% vs. 52.3%, p=0.03). However, harms were frequently underreported, with only eight studies (42%) documenting adverse events. No study reported an increase in adverse events as a result of the interventions. The current evidence on AI-related ADM systems provides limited insights into patient-relevant outcomes. Our findings underscore the essential need for rigorous evaluations of clinical benefits, reinforced compliance with methodological standards, and balanced consideration of both benefits and harms to ensure meaningful integration into healthcare practice. This study did not receive any funding.

Bias Mitigation Methods: Applicability, Legality, and Recommendations for Development

As algorithmic decision-making systems (ADMS) are increasingly deployed across various sectors, the importance of research on fairness in Artificial Intelligence (AI) continues to grow. In this paper we highlight a number of significant practical limitations and regulatory compliance issues associated with the application of existing bias mitigation methods to ADMS. We present an example of an algorithmic system used in recruitment to illustrate these limitations. Our analysis of existing methods indicates a pressing need for a change in the approach to the development of new methods. In order to address the limitations, we provide recommendations for key factors to consider in the development of new bias mitigation methods that aim to be effective in real-world scenarios and comply with legal requirements in the European Union, United Kingdom and United States, such as non-discrimination, data protection and sector-specific regulations. Further, we suggest a checklist relating to these recommendations that should be included with the development of new bias mitigation methods.

Decolonial Artificial Intelligence; Algorithmic Fairness in Alignment with Turkish and Islamic Values

Research works on topics such as; Fairness in Agreement with European Values: An Interdisciplinary Perspective on AI Regulation or Decolonial AI Alignment: Openness, Viśesa -Dharma and Including Excluded Knowledges, generated the motivation to search for Artificial intelligence (AI) alignment with Turkish and Islamic Values. The driving force to this research work is the fact that all of the algorithmic decision-making systems include bias to some extend and the non-western world needs to construct its own value-based technological and sociological development models since there is not much belief left in the so-called international justice or the so-called democratic values. This research includes examination of brief information about the fundamentals of big data, algorithms, and artificial intelligence. Importance of thick data and digital anthropology is emphasized. Misuse and abuse AI has been identified as one of the most important challenges. Vygotsky’s arguments on social learning, social construction of technology theory and worldview theory may provide some of the arguments to construct the idea of an AI approach may be developed in accordance with Turkish and Islamic values. Decolonial AI arguments and fairness in AI approaches have also been utilized to empower our argument. Lastly, brief information on Turkish and Islamic values were presented, limited to the scope of this research.

Insights into Algorithmic Decision-Making Systems via a Decolonial-Intersectional Lens: A Cross-Analysis Case Study

As the line between governance, technology, and decision-making becomes ever-blurred, advocates and disparagers of Artificial Intelligence (AI) debate the impacts of its societal integration. Despite efforts to promote Fairness, Accountability, Transparency, and Ethics (FATE), vulnerable populations continue to be systematically marginalized and made “invisible” by the racialised, classed, and colonial institutions buttressing Algorithmic Decision-Making Systems (ADMS). To address these sociotechnical risks and acknowledge our privileged, Western “standpoint epistemology,” we employ a “metaparadigm perspective” to engage the literature and case studies through a critical theory lens. The cross-analysis of three case studies: Systems Theory, the United States’ “Blueprint for an AI Bill of Rights,” and predictive policing demonstrate that our current risk mitigation frameworks are incapable of addressing transhistorical injustices. Therefore, we call for operationalizing intersectional risk theory via the decolonial turn to address ADMS risks. On that note, we propose three algorithmic accountability recommendations. We theorize that through a reformulation of FATE principles to that of the algorithmic (decolonial) self, understanding power imbalances through marginalized risk discourses, and advancing intersectional ML decision-making practices, policymakers may be better positioned to enact equitable and just ADMS.

EU’s AI Regulation Approaches and Their Implication for Human Rights

The rapid development of Artificial Intelligence (AI) technologies presents significant opportunities, but also introduces critical challenges, particularly concerning the protection of human rights. In response, the European Union (EU) has developed comprehensive AI regulations, most notably the AI Act, positioning itself as a leader in global AI governance. However, questions remain regarding the effectiveness of these laws in safeguarding key human rights such as privacy, equality, autonomy, and freedom. This study aims to assess the sufficiency of the EU’s current AI regulations in protecting human rights, analyze the specific impacts of AI technologies on privacy and equality, and explore emerging legal trends in AI regulation within the EU and their broader global implications. Utilizing a mixed-method approach, this research integrates legislative analysis, comparative research, scientific forecasting, and interdisciplinary inquiry. The study critically examines the provisions of the AI Act and other relevant EU regulations, comparing them with international legal frameworks to identify gaps and future challenges. The findings reveal that, while the EU’s AI regulations offer foundational protections, there are significant deficiencies in addressing complex issues such as privacy, algorithmic bias, and transparency, particularly in high-risk areas like healthcare, law enforcement, and autonomous systems. The study also highlights that existing regulations are inadequate in managing the rapid integration of AI technologies across sectors and that marginalized groups are especially vulnerable to the risks posed by algorithmic decision-making systems. Furthermore, the EU’s AI regulatory framework is poised to become a global benchmark, but there is an urgent need for more adaptive and flexible legal mechanisms to keep pace with the dynamic nature of AI advancements. In conclusion, the study emphasizes the need for legal reforms to bridge the current gaps in privacy protection and address algorithmic bias, ensuring the robust protection of human rights in the age of AI.

Multiple fairness criteria in decision tree learning

The use of algorithmic decision-making systems based on machine learning models has led to a need for fair (unbiased) and explainable classification outcomes. In particular, machine learning algorithms can encode biases, which might result in discriminatory decisions for certain groups such as gender, race, or age. Although a number of works on decision tree learning have been proposed to decrease the chance of discrimination, they usually focus on the use of a single fairness metric. In general, creating a model based on a single fairness metric is not a sufficient way to mitigate discrimination since bias can originate from various sources—e.g., the data itself or the optimization process. In this paper, we propose a novel decision tree learning process that utilizes multiple fairness metrics to address both group and individual discrimination. This is achieved by extending the attribute selection procedure to consider not only information gain but also gain in fairness. Computational experiments on fourteen different datasets with various sensitive features demonstrate that the proposed Fair-C4.5 models improve fairness without a loss in predictive accuracy when compared to the well-known C4.5 and the fairness-aware FFTree algorithms.

Black-Box Testing and Auditing of Bias in ADM Systems

For years, the number of opaque algorithmic decision-making systems (ADM systems) with a large impact on society has been increasing: e.g., systems that compute decisions about future recidivism of criminals, credit worthiness, or the many small decision computing systems within social networks that create rankings, provide recommendations, or filter content. Concerns that such a system makes biased decisions can be difficult to investigate: be it by people affected, NGOs, stakeholders, governmental testing and auditing authorities, or other external parties. Scientific testing and auditing literature rarely focuses on the specific needs for such investigations and suffers from ambiguous terminologies. With this paper, we aim to support this investigation process by collecting, explaining, and categorizing methods of testing for bias, which are applicable to black-box systems, given that inputs and respective outputs can be observed. For this purpose, we provide a taxonomy that can be used to select suitable test methods adapted to the respective situation. This taxonomy takes multiple aspects into account, for example the effort to implement a given test method, its technical requirement (such as the need of ground truth) and social constraints of the investigation, e.g., the protection of business secrets. Furthermore, we analyze which test method can be used in the context of which black box audit concept. It turns out that various factors, such as the type of black box audit or the lack of an oracle, may limit the selection of applicable tests. With the help of this paper, people or organizations who want to test an ADM system for bias can identify which test methods and auditing concepts are applicable and what implications they entail.

Algorithmic discrimination: examining its types and regulatory measures with emphasis on US legal practices.

Algorithmic decision-making systems are widely used in various sectors, including criminal justice, employment, and education. While these systems are celebrated for their potential to enhance efficiency and objectivity, they also pose risks of perpetuating and amplifying societal biases and discrimination. This paper aims to provide an indepth analysis of the types of algorithmic discrimination, exploring both the challenges and potential solutions. The methodology includes a systematic literature review, analysis of legal documents, and comparative case studies across different geographic regions and sectors. This multifaceted approach allows for a thorough exploration of the complexity of algorithmic bias and its regulation. We identify five primary types of algorithmic bias: bias by algorithmic agents, discrimination based on feature selection, proxy discrimination, disparate impact, and targeted advertising. The analysis of the U.S. legal and regulatory framework reveals a landscape of principled regulations, preventive controls, consequential liability, self-regulation, and heteronomy regulation. A comparative perspective is also provided by examining the status of algorithmic fairness in the EU, Canada, Australia, and Asia. Real-world impacts are demonstrated through case studies focusing on criminal risk assessments and hiring algorithms, illustrating the tangible effects of algorithmic discrimination. The paper concludes with recommendations for interdisciplinary research, proactive policy development, public awareness, and ongoing monitoring to promote fairness and accountability in algorithmic decision-making. As the use of AI and automated systems expands globally, this work highlights the importance of developing comprehensive, adaptive approaches to combat algorithmic discrimination and ensure the socially responsible deployment of these powerful technologies.

Identifying Reasons for Bias: An Argumentation-Based Approach

As algorithmic decision-making systems become more prevalent in society, ensuring the fairness of these systems is becoming increasingly important. Whilst there has been substantial research in building fair algorithmic decision-making systems, the majority of these methods require access to the training data, including personal characteristics, and are not transparent regarding which individuals are classified unfairly. In this paper, we propose a novel model-agnostic argumentation-based method to determine why an individual is classified differently in comparison to similar individuals. Our method uses a quantitative argumentation framework to represent attribute-value pairs of an individual and of those similar to them, and uses a well-known semantics to identify the attribute-value pairs in the individual contributing most to their different classification. We evaluate our method on two datasets commonly used in the fairness literature and illustrate its effectiveness in the identification of bias.

Trust in algorithmic decision-making systems in health: A comparison between ADA health and IBM Watson.

Algorithmic decision-making systems (ADMs) support an ever-growing number of decision-making processes. We conducted an online survey study in Flanders (n = 1,082) to understand how laypeople perceive and trust health ADMs. Inspired by the ability, benevolence, and integrity trustworthiness model (Mayer et al., 1995), this study investigated how trust is constructed in health ADMs. In addition, we investigated how trust construction differs between ADA Health (a self-diagnosis medical chatbot) and IBM Watson Oncology (a system that suggests treatments for cancer in hospitals). Our results show that accuracy and fairness are the biggest predictors of trust in both ADMs, whereas control plays a smaller yet significant role. Interestingly, control plays a bigger role in explaining trust in ADA Health than IBM Watson Oncology. Moreover, how appropriate people evaluate data-driven healthcare and how concerned they are with algorithmic systems prove to be good predictors for accuracy, fairness, and control in these specific health ADMs. The appropriateness of data-driven healthcare had a bigger effect with IBM Watson Oncology than with ADA Health. Overall, our results show the importance of considering the broader contextual, algorithmic, and case-specific characteristics when investigating trust construction in ADMs.

The Unfairness of Fair Machine Learning: Leveling Down and Strict Egalitarianism by Default

In recent years, fairness in machine learning (ML), artificial intelligence (AI), and algorithmic decision-making systems has emerged as a highly active area of research and development. To date, most measures and methods to mitigate bias and improve fairness in algorithmic systems have been built in isolation from policymaking and civil societal contexts and lack serious engagement with philosophical, political, legal, and economic theories of equality and distributive justice. Many current measures define “fairness” in simple terms to mean narrowing gaps in performance or outcomes between demographic groups while preserving as much of the original system’s accuracy as possible. This oversimplified translation of the complex socio-legal concept of equality into fairness measures is troubling. Many current fairness measures suffer from both fairness and performance degradation—or “leveling down”—where fairness is achieved by making every group worse off or by bringing better-performing groups down to the level of the worst off. Leveling down is a symptom of the decision to measure fairness solely in terms of equality, or disparity between groups in performance and outcomes, that ignores other relevant concerns of distributive justice (e.g., welfare or priority), which are more difficult to quantify and measure. When fairness can only be measured in terms of distribution of performance or outcomes, corrective actions can likewise only target how these goods are distributed between groups. We refer to this trend as “strict egalitarianism by default.” Strict egalitarianism by default runs counter to both the stated objectives of fairness measures and the presumptive aim of the field: to improve outcomes for historically disadvantaged or marginalized groups. When fairness can only be achieved by making everyone worse off in material or relational terms–through injuries of stigma, loss of solidarity, unequal concern, and missed opportunities for substantive equality—something has gone wrong in translating the vague concept of “fairness” into practice. Leveling down should be rejected in fairML because it (1) unnecessarily and arbitrarily harms advantaged groups in cases where performance is intrinsically valuable, such as medical applications of AI; (2) demonstrates a lack of equal concern for affected groups, undermines social solidarity, and contributes to stigmatization; (3) fails to live up to the substantive aims of equality law and fairML and squanders the opportunity afforded by interest in algorithmic fairness to substantively address longstanding social inequalities; and (4) fails to meet the aims of many viable theories of distributive justice including pluralist egalitarian approaches, prioritarianism, sufficientarianism, and others. This paper critically scrutinizes these initial observations to determine how fairML can move beyond mere leveling down and strict egalitarianism by default. We examine the causes and prevalence of leveling down across fairML and explore possible justifications and criticisms based on philosophical and legal theories of equality and distributive justice as well as equality-law jurisprudence. We find that fairML does not currently engage in the type of measurement, reporting, or analysis necessary to justify leveling down in practice. The types of decisions for which ML and AI are currently used, as well as inherent limitations on data collection and measurement, suggest leveling down is rarely justified in practice. We propose a first step toward substantive equality in fairML: “leveling up” systems by enforcing minimum acceptable harm thresholds, or “minimum rate constraints,” as fairness constraints at the design stage. We likewise propose an alternative harms-based framework to counter the oversimplified egalitarian framing currently dominant in the field and push future discussion more towards substantive equality opportunities and away from strict egalitarianism by default.

The General Equal Treatment Act and protection against discrimination by algorithmic decision-making systems

The General Equal Treatment Act and protection against discrimination by algorithmic decision-making systems

Objectives for algorithmic decision-making systems in childhood asthma: Perspectives of children, parents, and physicians.

To identify with children, parents and physicians the objectives to be used as parameters for algorithmic decision-making systems (ADMSs) adapting treatments in childhood asthma. We first conducted a qualitative study based on semi-structured interviews to explore the objectives that children aged 8-17 years, their parents, and their physicians seek to achieve when taking/giving/prescribing a treatment for asthma. Following the grounded theory approach, each interview was independently coded by two researchers; reconciled codes were used to assess code frequency, categories were defined, and the main objectives identified. We then conducted a quantitative study based on questionnaires using these objectives to determine how children/parents/physicians ranked these objectives and whether their responses were aligned. We interviewed 71 participants (31 children, 30 parents and 10 physicians) in the qualitative study and identified seven objectives associated with treatment uptake and five objectives associated with treatment modalities. We included 291 participants (137 children, 137 parents, and 17 physicians) in the quantitative study. We found little correlation between child, parent, and physician scores for each of the objectives. Each child's asthma history influenced the choice of scores assigned to each objective by the child, parents, and physician. The identified objectives are quantifiable and relevant to the management of asthma in the short and long term. They can therefore be incorporated as parameters for future ADMS. Shared decision-making seems essential to achieve consensus among children, parents, and physicians when choosing the weight to assign to each of these objectives.

Transparency in Algorithmic Decision-making: Interpretable Models for Ethical Accountability

Concerns regarding their opacity and potential ethical ramifications have been raised by the spread of algorithmic decisionmaking systems across a variety of fields. By promoting the use of interpretable machine learning models, this research addresses the critical requirement for openness and moral responsibility in these systems. Interpretable models provide a transparent and intelligible depiction of how decisions are made, as opposed to complicated black-box algorithms. Users and stakeholders need this openness in order to understand, verify, and hold accountable the decisions made by these algorithms. Furthermore, interpretability promotes fairness in algorithmic results by making it easier to detect and reduce biases. In this article, we give an overview of the difficulties brought on by algorithmic opacity, highlighting how crucial it is to solve these difficulties in a variety of settings, including those involving healthcare, banking, criminal justice, and more. From linear models to rule-based systems to surrogate models, we give a thorough analysis of interpretable machine learning techniques, highlighting their benefits and drawbacks. We suggest that incorporating interpretable models into the design and use of algorithms can result in a more responsible and moral application of AI in society, ultimately benefiting people and communities while lowering the risks connected to opaque decision-making processes.

Path To Gain Functional Transparency In Artificial Intelligence With Meaningful Explainability

Artificial Intelligence (AI) is rapidly integrating into various aspects of our daily lives, influencing decision-making processes in areas such as targeted advertising and matchmaking algorithms. As AI systems become increasingly sophisticated, ensuring their transparency and explainability becomes crucial. Functional transparency is a fundamental aspect of algorithmic decision-making systems, allowing stakeholders to comprehend the inner workings of these systems and enabling them to evaluate their fairness and accuracy. However, achieving functional transparency poses significant challenges that need to be addressed. In this paper, we propose a design for user-centered compliant-by-design transparency in transparent systems. We emphasize that the development of transparent and explainable AI systems is a complex and multidisciplinary endeavor, necessitating collaboration among researchers from diverse fields such as computer science, artificial intelligence, ethics, law, and social science. By providing a comprehensive understanding of the challenges associated with transparency in AI systems and proposing a user-centered design framework, we aim to facilitate the development of AI systems that are accountable, trustworthy, and aligned with societal values.

When AI is Perceived to Be Fairer than a Human: Understanding Perceptions of Algorithmic Decisions in a Job Application Context

This study investigates people’s perceptions of AI decision-making as compared to human decision-making within the job application context. It takes into account both favorable and unfavorable outcomes, employing a 2 × 2 experimental design (decision-making agent: AI algorithm vs. human; outcome: favorable vs. unfavorable). Upon evaluating a job seeker’s suitability for a position, participants viewed algorithmic decisions as fairer, more competent, more trustworthy, and more useful than those made by humans. Interestingly, when a candidate was deemed unsuitable for hiring, people reacted more negatively to the verdict given by a human than to the same judgment offered by AI. Moreover, participants credited algorithmic decisions with greater sensitivity to both quantitative and qualitative qualifications, thus indicating algorithmic appreciation. Our findings shed light on the psychological basis of perceptions surrounding Algorithmic Decision-Making (ADM) and the responses to the decisions rendered by ADM systems.

Social context of the issue of discriminatory algorithmic decision-making systems

Social context of the issue of discriminatory algorithmic decision-making systems

Labour market digitalization and social class: evidence of mobility and reproduction from a European survey of online platform workers

Abstract The type of work we do as adults is significantly influenced by our parents’ social class. However, digital technologies are transforming the way labour markets work. Candidates are screened using algorithmic decision-making systems. Skills are validated with online tests and feedback ratings. Communications take place online. Could these transformations undermine the advantages that have accrued to workers with privileged backgrounds or reproduce this privilege through digital divides? We examine this question with survey evidence from the online (remote) platform economy, a labour market segment where these digital transformations have progressed furthest (N = 1,001). The results reveal that online platform workers come predominantly from privileged class backgrounds, but we find less evidence of parental class shaping what types of online work they do. We conclude that digital transformations of labour markets may reproduce disparities in access to work but attenuate some class-based differences in the selection of workers by employers.

The public perceptions of algorithmic decision-making systems: Results from a large-scale survey

The public perceptions of algorithmic decision-making systems: Results from a large-scale survey

Managing unintended consequences of algorithmic decision-making: The case of Robodebt

In 2016, the Australian welfare agency Centrelink implemented an information system to automate the identification and recollection of welfare overpayments. Such algorithmic decision-making systems are increasingly leveraged to improve the efficiency of public administration. However, Centrelink’s scheme went horribly wrong: the system, branded as “Robodebt” by the popular media, generated debt notices that were inaccurate and based on insufficient evidence. Numerous vulnerable citizens who received the debt notices suffered a great deal of distress. While public controversy ensued, the ruling government continued to defend the flawed system until a court decision ruled it unlawful in 2019. This teaching case challenges one to analyze what went wrong in the implementation and management of the Robodebt system through the lens of sociotechnical systems.