Search bias quantification: investigating political bias in social media and web search

Muhammad Bilal Zafar,Saptarshi Ghosh,Karrie Karahalios,Johnnatan Messias,Motahhare Eslami,Juhi Kulshrestha,Krishna P Gummadi

doi:10.1007/s10791-018-9341-2

Abstract

Users frequently use search systems on the Web as well as online social media to learn about ongoing events and public opinion on personalities. Prior studies have shown that the top-ranked results returned by these search engines can shape user opinion about the topic (e.g., event or person) being searched. In case of polarizing topics like politics, where multiple competing perspectives exist, the political bias in the top search results can play a significant role in shaping public opinion towards (or away from) certain perspectives. Given the considerable impact that search bias can have on the user, we propose a generalizable search bias quantification framework that not only measures the political bias in ranked list output by the search system but also decouples the bias introduced by the different sources—input data and ranking system. We apply our framework to study the political bias in searches related to 2016 US Presidential primaries in Twitter social media search and find that both input data and ranking system matter in determining the final search output bias seen by the users. And finally, we use the framework to compare the relative bias for two popular search systems—Twitter social media search and Google web search—for queries related to politicians and political events. We end by discussing some potential solutions to signal the bias in the search results to make the users more aware of them.

Highlights

Algorithmic systems have become ubiquitous in our modern lives, and they exert great influence on many aspects of our daily lives, including shaping news and information we are exposed to via information retrieval algorithms
Our results indicate that our bias inferred using our source based scheme has a higher (70% or more) match with the bias of the tweets, and performs better than content based schemes. (ii) We included new results on the temporal variation of bias for political queries on Twitter social media search in Sect. 4.4.3. (iii) And we have applied our bias quantification framework to study the relative bias of two popular search systems—Twitter social media search and Google web search
We propose a search bias quantification framework which quantifies the bias in the output ranked list shown to the users, but it discerns to what extent is this bias due to the ranking system of the search system, or the input data to the ranking system

Summary

Introduction

Algorithmic systems have become ubiquitous in our modern lives, and they exert great influence on many aspects of our daily lives, including shaping news and information we are exposed to via information retrieval algorithms. Algorithms have been shown to create discriminatory ads based on gender (Datta et al 2015) or race (Sweeney 2013), to show different prices for the same products/service to different users (Hannak et al 2014), to skew users’ ratings to benefit low-rated hotels (Eslami et al 2017) and to mistakenly label a black man as an ape (Hern 2015) These issues have lead researchers, organizations and even governments towards a new avenue of research called “auditing algorithms”, which endeavors to understand if and how an algorithmic system can cause biases, when they are misleading or discriminatory to users (Sandvig et al 2014; Executive Office of the President 2016). Search engines are an important set of algorithms that users interact with on daily basis and these algorithms’ susceptibility to bias has resulted in several audit studies in recent years These audits cover a wide range of search platforms including Web search and social media search. We give an overview of prior work on examining the bias for search platforms, and discuss how our work adds to this line of existing research

Objectives

Results

Discussion

Conclusion