Algorithmic Decision Research Articles

Non-empirical problems in fair machine learning

The problem of fair machine learning has drawn much attention over the last few years and the bulk of offered solutions are, in principle, empirical. However, algorithmic fairness also raises important conceptual issues that would fail to be addressed if one relies entirely on empirical considerations. Herein, I will argue that the current debate has developed an empirical framework that has brought important contributions to the development of algorithmic decision-making, such as new techniques to discover and prevent discrimination, additional assessment criteria, and analyses of the interaction between fairness and predictive accuracy. However, the same framework has also suggested higher-order issues regarding the translation of fairness into metrics and quantifiable trade-offs. Although the (empirical) tools which have been developed so far are essential to address discrimination encoded in data and algorithms, their integration into society elicits key (conceptual) questions such as: What kind of assumptions and decisions underlies the empirical framework? How do the results of the empirical approach penetrate public debate? What kind of reflection and deliberation should stakeholders have over available fairness metrics? I will outline the empirical approach to fair machine learning, i.e. how the problem is framed and addressed, and suggest that there are important non-empirical issues that should be tackled. While this work will focus on the problem of algorithmic fairness, the lesson can extend to other conceptual problems in the analysis of algorithmic decision-making such as privacy and explainability.

The legitimacy gap of algorithmic decision-making in the public sector: Why it arises and how to address it

Algorithmic decision-making (ADM) systems are increasingly adopted by the state to support various administrative functions and improve the effectiveness and efficiency of public services, such as in unemployment services or policing. While these systems create challenges of opaqueness, unfairness, and value trade-offs, the present paper argues that a more fundamental challenge lies in the way these systems alter the epistemic bases of decision-making. It contributes to the literature by highlighting why procedural standards of legitimacy in operative decision-making no longer suffice for certain applications and by discussing how the resulting legitimacy gap can be addressed through stakeholder involvement. By adapting research on participatory technology assessments to the particularities of ADM system design, it is possible to identify the core challenges of such a stakeholder process and the necessary steps to deal with them.

Can I perform? Affective responses to algorithmic decision-making in the recruitment process

Companies increasingly use artificial intelligence (AI) and algorithmic decision-making for their recruitment and selection process for cost and efficiency reasons. However, knowledge about the affective responses to the selection process is still limited when AI supports different selection process stages (i.e., preselection, telephone, and video interview). Using a scenario-based between-subject design (N = 160) in Study 1, we found that AI-support at later stages of the selection process (i.e., telephone and video interview) decreases the opportunity to perform and increases emotional creepiness. In turn, the opportunity to perform and emotional creepiness mediate the association of AI-support in telephone/ video interviews and organizational attractiveness. Conversely, we did not find negative affective responses to AI-support in the preselection. In Study 2, we examined whether hiding the information about AI-support in preselection negatively influences applicant reactions after the selection process has ended. The hide of information about AI-support decreases procedural justice, trust, and organizational attractiveness. As we offer evidence for possible adverse reactions, this study provides important practical and theoretical implications.

Machine Learning in Strategic Search: Alleviating and Aggravating Local Search

Machine learning technologies could direct executives’ scarce attention to improve the quality of decision making and, in particular, improve the search for strategic options. However, the use of such technologies could also amplify bias or induce local search. We draw from literature on local search as well as literature on managerial implications of artificial intelligence in general and machine learning in particular to specify how the use of machine learning could affect search in the context of strategic decision-making. We propose how machine learning applications can broaden or distort search space, change how cues are noted, and affect the interpretation of alternatives. We further detail what factors are likely to shape these dynamics. We thus contribute to the literature on heuristics in management by detailing how technologies might alleviate or aggravate local search, and to the literature on algorithmic decision-making by suggesting how the use of these technologies influences the quality of search.

AI-Powered Decentralized Recruitment System on the Blockchain

By combining artificial intelligence (AI) algorithms with blockchain technology, the AI-powered decentralized Recruitment System on the Blockchain (ADRSoB) offers a cutting-edge method for completely changing the hiring process. This study aims to assess the effectiveness and impact of ADRSoB in several areas, such as user satisfaction, bias mitigation, candidate selection quality, efficiency, and transparency. From a methodological standpoint, case studies, research papers, and current literature pertinent to ADRSoB were compiled using a secondary data-based review article approach. Principal results show that ADRSoB increases user happiness while promoting transparency, mitigating biases, improving applicant selection quality, and streamlining recruiting processes. However, policy implications such as encouraging technology use, guaranteeing data privacy legislation, and fostering fairness in algorithmic systems are required due to obstacles to technology adoption, data privacy issues, and bias in algorithmic decision-making. ADRSoB has enormous potential to change hiring procedures while addressing critical issues and encouraging moral and responsible hiring in the digital era.

The Impacts of Algorithmic Work Assignment on Fairness Perceptions and Productivity

Since algorithms are widely adopted by businesses and often require human involvement, understanding how humans perceive algorithms is instrumental to the success of algorithm design. Particularly, the growing concern that algorithms may reproduce inequality historically exhibited by humans calls for research about how people perceive the fairness of algorithmic decision-making relative to traditional human-based decision-making and, consequently, adjust their work behaviors. We study how algorithmic (vs. human-based) task assignment processes change task recipients’ fairness perceptions and productivity. In a 15-day-long field experiment with Alibaba Group in a warehouse where workers pick products following orders (or “pick lists”), we randomly assigned half of the workers to receive pick lists from a machine that ostensibly relied on an algorithm to distribute pick lists, and the other half to receive pick lists from a human distributor. Despite that we used the same underlying rule to assign pick lists in both groups, workers perceived the algorithmic (vs. human-based) assignment process as fairer by 0.94-1.02 standard deviations. This yielded productivity benefits: receiving tasks from an algorithm (vs. a human) increased workers’ picking efficiency by 17.35%-19.39%. The algorithmic assignment produced larger productivity gains among workers for whom perceived fairness has a stronger effect on productivity, including more educated workers and workers who cared more about the difficulty of their pick lists. We replicated the main results in another field experiment and showed via online experiments that people in the U.S. also view algorithmic task assignment as fairer. We demonstrate that algorithms can have broader impacts beyond offering greater efficiency and accuracy than humans: introducing algorithmic assignment processes may enhance fairness perceptions and productivity. This insight can be utilized by managers to better design and implement algorithm-based decision making at work.

Improving fairness of artificial intelligence algorithms in Privileged-Group Selection Bias data settings

An increasing number of decisions regarding the daily lives of human beings are being controlled by artificial intelligence (AI) algorithms. Since they now touch on many aspects of our lives, it is crucial to develop AI algorithms that are not only accurate but also objective and fair. Recent studies have shown that algorithmic decision-making may be inherently prone to unfairness, even when there is no intention for it. In this paper, we study the fairness of AI algorithms in data settings for which unprivileged groups are extremely under-represented compared to privileged groups. A typical domain which often presents such Privileged Group Selection Bias (PGSB) is AI-based hiring, which stems from an inherent lack of labeled information for rejected applicants. We first demonstrate that such a selection bias can lead to a high algorithmic bias, even if privileged and unprivileged groups are treated exactly the same. We then propose several methods to overcome this type of bias. In particular, we suggest three in-process and pre-process fairness mechanisms, combined with both supervised and semi-supervised learning algorithms. An extensive evaluation that was conducted using two real-world datasets, reveals that the proposed methods are able to improve fairness considerably, with only a minimal compromise in accuracy. This is despite the limited information available for unprivileged groups and the inherent trade-off between fairness and accuracy.

How many angels can dance on the head of a pin? Understanding ‘alien’ thought

Initially coined by Weizenbaum in 1976, ‘alien' thought refers to the radical difference with which ‘thinking machines’ approach the process of thinking. The contemporary paradox of over-determination and indeterminacy—caused largely by algorithmic decision-making in the civic realm—makes these differences both more entangled and more difficult to navigate. In this essay, I trace over-determination to Leibniz and Turing’s axiomatic procedures and to instrumental rationality, and I trace indeterminacy to the mid-twentieth century co-development of computers and neurosciences to advance the following proposition: understanding alien thought requires understanding incomputability, temporal swarming, and inscriptive-significational errance. Understanding these phenomena in turn requires understanding thinking by doing, distributed thinking, and ontological indeterminacy. All are present in machinic operations as well as in the twentieth century experimental artistic practices of artists such as Duchamp, Cage, and Xu. These practices rely on indeterminate procedures and function as diagrammatic machines. A diagrammatic machine is neither abstract nor particular; neither an idea that is determining in the supreme instance, nor an infrastructure that is determining in the last instance, but rather instantiates a real yet to come (Deleuze and Guattari in A thousand plateaus: capitalism and schizophrenia: Trans. Massumi B, University of Minnesota Press, Minneapolis, 1987). In this essay, indeterminate artistic practices are used as an entry into alien thought and its correlates—infinity and complexity—by way of aesthetic analogy.

ENGAGING END USERS IN AN AI-ENABLED SMART SERVICE DESIGN - THE APPLICATION OF THE SMART SERVICE BLUEPRINT SCAPE (SSBS) FRAMEWORK

AbstractArtificial Intelligence (AI) has expanded in a diverse context, it infiltrates our social lives and is a critical part of algorithmic decision-making. Adopting AI technology, especially AI-enabled design, by end users who are non-AI experts is still limited. The incomprehensible, untransparent decision-making and difficulty of using AI become obstacles which prevent these end users to adopt AI technology. How to design the user experience (UX) based on AI technologies is an interesting topic to explore.This paper investigates how non-AI-expert end users can be engaged in the design process of an AI-enabled application by using a framework called Smart Service Blueprint Scape (SSBS), which aims to establish a bridge between UX and AI systems by mapping and translating AI decisions based on UX. A Dutch mobility service called ‘stUmobiel ’ was taken as a design case study. The goal is to design a reservation platform with stUmobiel end users. Co-creating with case users and assuring them to understand the decision-making and service provisional process of the AI-enabled design is crucial to promote users’ adoption. Furthermore, the concern of AI ethics also arises in the design process and should be discussed in a broader sense.

Data analytics (ab) use in healthcare fraud audits

This study explores how government-adopted audit data analytic tools promote the abuse of power by auditors enabling politically sensitive processes that encourage industry-wide normalization of behavior. In an audit setting, we investigate how a governmental organization enables algorithmic decision-making to alter power relationships to effect organizational and industry-wide change. While prior research has identified discriminatory threats emanating from the deployment of algorithmic decision-making, the effects of algorithmic decision-making on inherently imbalanced power relationships have received scant attention. Our results provide empirical evidence of how systemic and episodic power relationships strengthen each other, thereby enabling the governmental organization to effect social change that might be too politically prohibitive to enact directly. Overall, the results suggest that there are potentially negative effects caused by the use of algorithmic decision-making and the resulting power shifts, and these effects create a different view of the level of purported success attained through auditor use of data analytics.

Theorising Algorithmic Justice

ABSTRACT The mounting evidence of unintended harmful social consequences of automated algorithmic decision-making (AADM), powered by AI and big data, in transformative services (e.g., welfare services), is startling. The algorithmic harm experienced by individuals, communities and society-at-large involves new injustice claims and disputes that go beyond issues of social justice. Drawing from the theory of “abnormal justice” in this paper we articulate a new theory of algorithmic justice that addresses the questions: WHAT is the matter of algorithmic justice? WHO counts as a subject of algorithmic justice? HOW are algorithmic justices performed? and How to address and resolve disputes about the WHAT, WHO and HOW of algorithmic justice? We illustrate the theory of algorithmic justice by drawing from a case of AADM in social welfare services, widely adopted by governments around the world. Our research points to datafication, technological inscribing and the systemic nature of injustices as important IS-specific aspects of algorithmic justice. Our main practical contribution comes from the articulation of algorithmic justice as a framework that (1) makes visible the injustices related to the “what”, “who”, and “how” of AADM in transformative services, and (2) provides further insights into how we might address and resolve these algorithmic injustices.

How to use algorithmic decision-making to promote inclusiveness in organizations

Organizations are increasingly employing algorithms to recruit candidates suitable for the jobs on offer. One important responsibility that organizations in their recruitment efforts need to achieve is that decisions are made in ways that promote inclusiveness and diversity of the workforce. Recruitment efforts today are increasingly being automated, due to a widespread belief that algorithmic decision-making will reveal less biased evaluations, as such increasing the likelihood that the ethical value of inclusiveness will not be violated. The reality, however, is that algorithmic decision-making is also biased and therefore we introduce solutions for data scientists to ensure that inclusiveness is optimized in designing algorithms. We conclude that organizations employing algorithms in their recruitment efforts need to create awareness about the importance of inclusiveness and ensure that data scientists and business experts work together in fine-tuning algorithmic decision-making to achieve inclusiveness.

FairLens: Auditing black-box clinical decision support systems

The pervasive application of algorithmic decision-making is raising concerns on the risk of unintended bias in AI systems deployed in critical settings such as healthcare. The detection and mitigation of model bias is a very delicate task that should be tackled with care and involving domain experts in the loop. In this paper we introduce FairLens, a methodology for discovering and explaining biases. We show how this tool can audit a fictional commercial black-box model acting as a clinical decision support system (DSS). In this scenario, the healthcare facility experts can use FairLens on their historical data to discover the biases of the model before incorporating it into the clinical decision flow. FairLens first stratifies the available patient data according to demographic attributes such as age, ethnicity, gender and healthcare insurance; it then assesses the model performance on such groups highlighting the most common misclassifications. Finally, FairLens allows the expert to examine one misclassification of interest by explaining which elements of the affected patients’ clinical history drive the model error in the problematic group. We validate FairLens’ ability to highlight bias in multilabel clinical DSSs introducing a multilabel-appropriate metric of disparity and proving its efficacy against other standard metrics.

Investigating Algorithmic Decision-Making with Applications to Student Diversity at Illinois Tech


 
 
 
 This study focuses on algorithmic decision-making and how structural inequalities can introduce inadvertent bias in an algorithm. There are a lot of efforts by universities in the United States to diversify the student body. Most of these efforts focus on outreach, but few focus on improving any internal processes first.
 This study aimed to look for room for improvement to make internal processes more equitable so the rest of the efforts to diversify a student body would be more effective. Specifically, Illinois Tech admissions data was observed and patterns found were tested for statistical significance.
 
 
 

Fairness in credit scoring: Assessment, implementation and profit implications

The rise of algorithmic decision-making has spawned much research on fair machine learning (ML). Financial institutions use ML for building risk scorecards that support a range of credit-related decisions. Yet, the literature on fair ML in credit scoring is scarce. The paper makes three contributions. First, we revisit statistical fairness criteria and examine their adequacy for credit scoring. Second, we catalog algorithmic options for incorporating fairness goals in the ML model development pipeline. Last, we empirically compare different fairness processors in a profit-oriented credit scoring context using real-world data. The empirical results substantiate the evaluation of fairness measures, identify suitable options to implement fair credit scoring, and clarify the profit-fairness trade-off in lending decisions. We find that multiple fairness criteria can be approximately satisfied at once and recommend separation as a proper criterion for measuring the fairness of a scorecard. We also find fair in-processors to deliver a good balance between profit and fairness and show that algorithmic discrimination can be reduced to a reasonable level at a relatively low cost. The codes corresponding to the paper are available on GitHub.

Algorithmic abstractions of ‘fashion identity’ and the role of privacy with regard to algorithmic personalisation systems in the fashion domain

This paper delves into the nuances of ‘fashion’ in recommender systems and social media analytics, which shape and define an individual’s perception and self-relationality. Its aim is twofold: first, it supports a different perspective on privacy that focuses on the individual’s process of identity construction considering the social and personal aspects of ‘fashion’. Second, it underlines the limitations of computational models in capturing the diverse meaning of ‘fashion’, whereby the algorithmic prediction of user preferences is based on individual conscious and unconscious associations with fashion identity. I test both of these claims in the context of current concerns over the impact of algorithmic personalisation systems on individual autonomy and privacy: creating ‘filter bubbles’, nudging the user beyond their conscious awareness, as well as the inherent bias in algorithmic decision-making. We need an understanding of privacy that sustains the inherent reduction of fashion identity to literal attributes and protects individual autonomy in shaping algorithmic approximations of the self.

Algorithmic pollution: Making the invisible visible

In this article, we focus on the growing evidence of unintended harmful societal effects of automated algorithmic decision-making in transformative services (e.g. social welfare, healthcare, education, policing and criminal justice), for individuals, communities and society at large. Drawing from the long-established research on social pollution, in particular its contemporary ‘pollution-as-harm’ notion, we put forward a claim – and provide evidence – that these harmful effects constitute a new type of digital social pollution, which we name ‘algorithmic pollution’. Words do matter, and by using the term ‘pollution’, not as a metaphor or an analogy, but as a transformative redefinition of the digital harm performed by automated algorithmic decision-making, we seek to make it visible and recognized. By adopting a critical performative perspective, we explain how the execution of automated algorithmic decision-making produces harm and thus performs algorithmic pollution. Recognition of the potential for unintended harmful effects of algorithmic pollution, and their examination as such, leads us to articulate the need for transformative actions to prevent, detect, redress, mitigate and educate about algorithmic harm. These actions, in turn, open up new research challenges for the information systems community.

Algorithmic Pollution: Making the Invisible Visible

In this paper, we focus on the growing evidence of unintended harmful societal effects of automated algorithmic decision-making (AADM) in transformative services (e.g., social welfare, healthcare, education, policing and criminal justice), for individuals, communities and society at large. Drawing from the long-established research on social pollution, in particular its contemporary ‘pollution-as-harm’ notion, we put forward a claim, and provide evidence, that these harmful effects constitute a new type of digital social pollution, which we name ‘algorithmic pollution’. Words do matter, and by using the term ‘pollution’, not as a metaphor, but as a transformative redefinition of the digital harm performed by AADM, we seek to make it visible and recognized. By adopting a critical performative perspective, we explain how the execution of AADM produces harm and thus performs algorithmic pollution. Recognition of the potential for unintended harmful effects of algorithmic pollution, and their examination as such, leads us to articulate the need for transformative actions to prevent, detect, redress, mitigate, and educate about algorithmic harm. These actions, in turn, open up new research challenges for the information systems community.

Threat of racial and economic inequality increases preference for algorithm decision-making

Artificial intelligence (AI) algorithms hold promise to reduce inequalities across race and socioeconomic status. One of the most important domains of racial and economic inequalities is medical outcomes; Black and low-income people are more likely to die from many diseases. Algorithms can help reduce these inequalities because they are less likely than human doctors to make biased decisions. Unfortunately, people are generally averse to algorithms making important moral decisions—including in medicine—undermining the adoption of AI in healthcare. Here we use the COVID-19 pandemic to examine whether the threat of racial and economic inequality increases the preference for algorithm decision-making. Four studies (N = 2819) conducted in the United States and Singapore show that emphasizing inequality in medical outcomes increases the preference for algorithm decision-making for triage decisions. These studies suggest that one way to increase the acceptance of AI in healthcare is to emphasize the threat of inequality and its negative outcomes associated with human decision-making.

Why a Right to an Explanation of Algorithmic Decision-Making Should Exist: A Trust-Based Approach

Businesses increasingly rely on algorithms that are data-trained sets of decision rules (i.e., the output of the processes often called “machine learning”) and implement decisions with little or no human intermediation. In this article, we provide a philosophical foundation for the claim that algorithmic decision-making gives rise to a “right to explanation.” It is often said that, in the digital era, informed consent is dead. This negative view originates from a rigid understanding that presumes informed consent is a static and complete transaction. Such a view is insufficient, especially when data are used in a secondary, noncontextual, and unpredictable manner—which is the inescapable nature of advanced artificial intelligence systems. We submit that an alternative view of informed consent—as an assurance of trust for incomplete transactions—allows for an understanding of why the rationale of informed consent already entails a right to ex post explanation.