Image Search Results Research Articles

Patient Photographs and Google Images: An AI Ethics Case Study

Over 35% of case reports published in medical journals contain patient photographs. With widespread online academic publishing, there is growing concern about patient photographs that were originally published in medical journals appearing in online image search results. Research has shown that from a random sample of case reports indexed in PubMed, at least one image was found on Google Images for 76.3% of the publications. Inspired by recent examples, this case study explores the ethical implications of patient photographs circulating outside of the original journal website, including attention to patient privacy and informed consent.

Investigating What Factors Influence Users’ Rating of Harmful Algorithmic Bias and Discrimination

There has been growing recognition of the crucial role users, especially those from marginalized groups, play in uncovering harmful algorithmic biases. However, it remains unclear how users’ identities and experiences might impact their rating of harmful biases. We present an online experiment (N=2,197) examining these factors: demographics, discrimination experiences, and social and technical knowledge. Participants were shown examples of image search results, including ones that previous literature has identified as biased against marginalized racial, gender, or sexual orientation groups. We found participants from marginalized gender or sexual orientation groups were more likely to rate the examples as more severely harmful. Belonging to marginalized races did not have a similar pattern. Additional factors affecting users’ ratings included discrimination experiences, and having friends or family belonging to marginalized demographics. A qualitative analysis offers insights into users' bias recognition, and why they see biases the way they do. We provide guidance for designing future methods to support effective user-driven auditing.

Finding Medical Photographs of Patients Online: Randomized, Cross-Sectional Study.

Photographs from medical case reports published in academic journals have previously been found in online image search results. This means that patient photographs circulate beyond the original journal website and can be freely accessed online. While this raises ethical and legal concerns, no systematic study has documented how often this occurs. The aim of this cross-sectional study was to provide systematic evidence that patient photographs from case reports published in medical journals appear in Google Images search results. Research questions included the following: (1) what percentage of patient medical photographs published in case reports were found in Google Images search results? (2) what was the relationship between open access publication status and image availability? and (3) did the odds of finding patient photographs on third-party websites differ between searches conducted in 2020 and 2022? The main outcome measure assessed whether at least 1 photograph from each case report was found on Google Images when using a structured search. Secondary outcome variables included the image source and the availability of images on third-party websites over time. The characteristics of medical images were described using summary statistics. The association between the source of full-text availability and image availability on Google Images was tested using logistic regressions. Finally, we examined the trend of finding patient photographs using generalized estimating equations. From a random sample of 585 case reports indexed in PubMed, 186 contained patient photographs, for a total of 598 distinct images. For 142 (76.3%) out of 186 case reports, at least 1 photograph was found in Google Images search results. A total of 18.3% (110/598) of photographs included eye, face, or full body, including 10.9% (65/598) that could potentially identify the patient. The odds of finding an image from the case report online were higher if the full-text paper was available on ResearchGate (odds ratio [OR] 9.16, 95% CI 2.71-31.02), PubMed Central (OR 7.90, 95% CI 2.33-26.77), or Google Scholar (OR 6.07, 95% CI 2.77-13.29) than if the full-text was available solely through an open access journal (OR 5.33, 95% CI 2.31-12.28). However, all factors contributed to an increased risk of locating patient images online. Compared with the search in 2020, patient photographs were less likely to be found on third-party websites based on the 2022 search results (OR 0.61, 95% Cl 0.43-0.87). A high proportion of medical photographs from case reports was found on Google Images, raising ethical concerns with policy and practice implications. Journal publishers and corporations such as Google are best positioned to develop an effective remedy. Until then, it is crucial that patients are adequately informed about the potential risks and benefits of providing consent for clinicians to publish their images in medical journals.

I-Cubid: a nonlinear cubic graph-based approach to visualize and in-depth browse Flickr image results.

The existing image search engines allow web users to explore images from the grids. The traditional interaction is linear and lookup-based. Notably, scanning web search results is horizontal-vertical and cannot support in-depth browsing. This research emphasizes the significance of a multidimensional exploration scheme over traditional grid layouts in visually exploring web image search results. This research aims to antecedent the implications of visualization and related in-depth browsing via a multidimensional cubic graph representation over a search engine result page (SERP). Furthermore, this research uncovers usability issues in the traditional grid and 3-dimensional web image search space. We provide multidimensional cubic visualization and nonlinear in-depth browsing of web image search results. The proposed approach employs textual annotations and descriptions to represent results in cubic graphs that further support in-depth browsing via a search user interface (SUI) design. It allows nonlinear navigation in web image search results and enables exploration, browsing, visualization, previewing/viewing, and accessing images in a nonlinear, interactive, and usable way. The usability tests and detailed statistical significance analysis confirm the efficacy of cubic presentation over grid layouts. The investigation reveals improvement in overall user satisfaction, screen design, information & terminology, and system capability in exploring web image search results.

‘I’m not bad, I’m just … drawn that way’: media and algorithmic systems logics in the Italian Google Images construction of (cr)immigrants’ communities

ABSTRACT The paper aims at creating a bridge between media and migration studies and critical algorithm studies. By adopting a media ecological approach and a mutual shaping of technology and society perspective, in this paper, we explore the factors that lead, especially in Italy, to discriminant and stigmatizing image search results, related to specific groups of immigrants living in the country. We performed a content analysis of Google-Images search results with regard to the largest immigrant communities hosted in France, Germany, Italy, and the United Kingdom. Results show that the depiction of Romanian, Albanian, Moroccan, and Algerian immigrant communities on Google.it is flattened on a univocal stigmatized representation that shows them as criminals, which is not the case in other countries. Most of these stigmatizing images derive from local online newspapers, which questions the interplay between newsmaking choices and routines, and algorithms logics.

Development and validation of the Alcoholic Beverage Identification Deep Learning Algorithm version 2 for quantifying alcohol exposure in electronic images.

Seeing alcohol in media has been demonstrated to increase alcohol craving, impulsive decision-making, and hazardous drinking. Due to the exponential growth of (social) media use it is important to develop algorithms to quantify alcohol exposure efficiently in electronic images. In this article, we describe the development of an improved version of the Alcoholic Beverage Identification Deep Learning Algorithm (ABIDLA), called ABIDLA2. ABIDLA2 was trained on 191,286 images downloaded from Google Image Search results (based on search terms) and Bing Image Search results. In Task-1, ABIDLA2 identified images as containing one of eight beverage categories (beer/cider cup, beer/cider bottle, beer/cider can, wine, champagne, cocktails, whiskey/cognac/brandy, other images). In Task-2, ABIDLA2 made a binary classification between images containing an "alcoholic beverage" or "other". An ablation study was performed to determine which techniques improved algorithm performance. ABIDLA2 was most accurate in identifying Whiskey/Cognac/Brandy (88.1%) followed by Beer/Cider Can (80.5%), Beer/Cider Bottle (78.3%), and Wine (77.8%). Its overall accuracy was 77.0% (Task-1) and 87.7% (Task-2). Even the identification of the least accurate beverage category (Champagne, 64.5%) was more than five times higher than random chance (12.5%=1/8 categories). The implementation of balanced data sampler to address class skewness and the use of self-training to make use of a large, secondary, weakly labeled dataset particularly improved overall algorithm performance. With extended capabilities and a higher accuracy, ABIDLA2 outperforms its predecessor and enables the screening of any kind of electronic media rapidly to estimate the quantity of alcohol exposure. Quantifying alcohol exposure automatically through algorithms like ABIDLA2 is important because viewing images of alcoholic beverages in media tends to increase alcohol consumption and related harms.

An Image of Society

Algorithmically-mediated content is both a product and producer of dominant social narratives, and it has the potential to impact users' beliefs and behaviors. We present two studies on the content and impact of gender and racial representation in image search results for common occupations. In Study 1, we compare 2020 workforce gender and racial composition to that reflected in image search. We find evidence of underrepresentation on both dimensions: women are underrepresented in search at a rate of 42% women for a field with 50% women; people of color are underrepresented with 16% in search compared to an occupation with 22% people of color (the latter being proportional to the U.S. workforce). We also compare our gender representation data with that collected in 2015 by Kay et al., finding little improvement in the last half-decade. In Study 2, we study people's impressions of occupations and sense of belonging in a given field when shown search results with different proportions of women and people of color. We find that both axes of representation as well as people's own racial and gender identities impact their experience of image search results. We conclude by emphasizing the need for designers and auditors of algorithms to consider the disparate impacts of algorithmic content on users of marginalized identities.

Social image retrieval based on topic diversity

Image search re-ranking is one of the most important approaches to enhance the text-based image search results. Extensive efforts have been dedicated to improve the accuracy and diversity of tag-based image retrieval. However, how to make the top-ranked results relevant and diverse is still a challenging problem. In this paper, we propose a novel method to diversify the retrieval results by latent topic analysis. We first employ NMF (Non-negative Matrix Factorization) Lee and Seung (Nature 401(6755):788–791, 1999) to estimate the initial relevance score to the query q. Then, the initial relevance score is fed into an adaptive multi-feature fusion model to learn the final relevance score. Next, the diversification process is conducted. We group all the images by semantic clustering and estimate the topic distribution of each cluster by topic analysis. The clusters are ranked based on the topic distribution vector and the final retrieval image list is obtained by a greedy selection mechanism based on the estimated relevances. Experimental results on the NUS-Wide dataset show the effectiveness of the proposed approach.

Implicit Diversity in Image Summarization

Studies have shown that the people depicted in image search results tend to be of majority groups with respect to socially salient attributes such as gender or race. This skew goes beyond that which already exists in the world - i.e., the search results for images of people are more imbalanced than the ground truth would suggest. For example, Kay et al. showed that although 28% of CEOs in the U.S. are women, only 10% of the top 100 results for "CEO" in Google Image Search are women. Similar observations abound across search terms and across socially salient attributes. Most existing approaches to correct for this kind of bias assume that the images of people include labels denoting the relevant socially salient attributes. These labels are explicitly used to either change the dataset, adjust the training of the algorithm, and/or in the execution of the method. However, such labels are often unknown. Further, using machine learning techniques to infer these labels may often not be possible within acceptable accuracy ranges and may not be desirable due to the additional biases this process could incur. As observed in prior work, alternate approaches consider the diversity of image features, which often do not translate to images of visibly diverse people. We develop a novel approach that takes as input a visibly diverse control set of images of people and uses this set as part of a procedure to select a set of images of people in response to a query. The goal is to have a resulting set that is more visibly diverse in a manner that emulates the diversity depicted in the control set. It accomplishes this by evaluating the similarity of the images selected by a black-box algorithm with the images in the diversity control set, and incorporating this "diversity score" into the final selection process. Importantly, this approach does not require images to be labelled at any point; effectively, it gives a way to implicitly diversify the set of images selected. We provide two variants of our approach: the first is a modification of the well known MMR algorithm to incorporate the diversity scores, and the second is a more efficient variant that does not consider within-list redundancy. We evaluate these approaches empirically on two image datasets: 1) a new dataset we collect which contains the top 100 Google Image results for 96 occupations, for which we evaluate gender and skin-tone diversity with respect to occupations and 2) the well-known CelebA dataset containing images of celebrities for which we can evaluate gender diversity with respect to facial features such as "smiling" or "glasses". Both of our approaches produce image sets that significantly improve the visible diversity of the results (i.e., include a larger fraction of anti-stereotypical images) with respect to current Google Image Search results and other state-of-the-art algorithms for diverse image summarization. Further, they seem to come at a minimal cost to accuracy. These empirical results demonstrate the effectiveness of simple label-independent interventions to diversify image search.

Exploiting textual queries for dynamically visual disambiguation

Due to the high cost of manual annotation, learning directly from the web has attracted broad attention. One issue that limits the performance of current webly supervised models is the problem of visual polysemy. In this work, we present a novel framework that resolves visual polysemy by dynamically matching candidate text queries with retrieved images. Specifically, our proposed framework includes three major steps: we first discover and then dynamically select the text queries according to the keyword-based image search results, we employ the proposed saliency-guided deep multi-instance learning (MIL) network to remove outliers and learn classification models for visual disambiguation. Compared to existing methods, our proposed approach can figure out the right visual senses, adapt to dynamic changes in the search results, remove outliers, and jointly learn the classification models. Extensive experiments and ablation studies on CMU-Poly-30 and MIT-ISD datasets demonstrate the effectiveness of our proposed approach.

Hypergraph-based image search reranking with elastic net regularized regression

Image search reranking is emerging as an effective technique to refine the text-based image search results using visual information. In this paper, we introduce a novel hypergraph-based image search reranking method that accounts for both relevance and diversity of search results. Namely, the text-based image search results are taken as vertices in a probabilistic regression-based hypergraph model and reranking is formulated as a hypergraph ranking problem with absorbing nodes. More specifically, to discover related samples and characterize the relationships among them, we bring the Elastic Net regularized regression model into the hypergraph construction. Exceeding the conventional hypergraph construction schemes, our scheme is able to describe the high-order relationships and the local manifold structure among visual samples while ensuring the datum-adaptiveness. Afterward, we apply a hypergraph-based ranking with absorbing nodes to ensure a diversified reranking. That is, during the reranking process, previously-ranked samples are transformed into absorbing nodes at each iteration, thereby redundant ones are prevented from receiving high ranking scores. Extensive experiments on real-world data from Flickr suggest our proposed reranking method achieves promising results compared to existing reranking methods.

Color associations in abstract semantic domains

The embodied cognition paradigm has stimulated ongoing debate about whether sensory data – including color – contributes to the semantic structure of abstract concepts. Recent uses of linguistic data in the study of embodied cognition have been focused on textual corpora, which largely precludes the direct analysis of sensory information. Here, we develop an automated approach to multimodal content analysis that detects associations between words based on the color distributions of their Google Image search results. Crucially, we measure color using a transformation of colorspace that closely resembles human color perception. We find that words in the abstract domains of academic disciplines, emotions, and music genres, cluster in a statistically significant fashion according to their color distributions. Furthermore, we use the lexical ontology WordNet and crowdsourced human judgments to show that this clustering reflects non-arbitrary semantic structure, consistent with metaphor-based accounts of embodied cognition. In particular, we find that images corresponding to more abstract words exhibit higher variability in colorspace, and semantically similar words have more similar color distributions. Strikingly, we show that color associations often reflect shared affective dimensions between abstract domains, thus revealing patterns of aesthetic coherence in everyday language. We argue that these findings provide a novel way to synthesize metaphor-based and affect-based accounts of embodied semantics.

On Annotation Methodologies for Image Search Evaluation

Image search engines differ significantly from general web search engines in the way of presenting search results. The difference leads to different interaction and examination behavior patterns, and therefore requires changes in evaluation methodologies. However, evaluation of image search still utilizes the methods for general web search. In particular, offline metrics are calculated based on coarse-fine topical relevance judgments with the assumption that users examine results in a sequential manner. In this article, we investigate annotation methods via crowdsourcing for image search evaluation based on a lab-based user study. Using user satisfaction as the golden standard, we make several interesting findings. First, instead of item-based annotation, annotating relevance in a row-based way is more efficient without hurting performance. Second, besides topical relevance, image quality plays a crucial role when evaluating the image search results, and the importance of image quality changes with search intent. Third, compared to traditional four-level scales, the fine-grain annotation method outperforms significantly. To our best knowledge, our work is the first to systematically study how diverse factors in data annotation impact image search evaluation. Our results suggest different strategies for exploiting the crowdsourcing to get data annotated under different conditions.

A novel multimodal clustering framework for images with diverse associated text

With the enormous growth in the number of images on the web, image clustering has become an essential part of any image retrieval system. Since web images are often accompanied by related text or tags, both visual and textual features can be exploited to improve the precision of web image clustering. Existing clustering methods either utilize them separately in a specific order, or use them simultaneously, but independently. In this work, we propose a new framework, Multimodal Hierarchical Clustering for Images (MHCI), which exploits the coexistence of both visual and textual patterns to establish a relationship between them. We propose textual and visual weights to quantify the relationship established between images and their features. The proposed framework can be applied to a wide variety of image datasets with different characteristics, viz., search results with noisy surrounding text, and tagged images. It can also cluster image search queries and their corresponding clicked images. The respective datasets used include image search results, Flicker (NUS-WIDE), and Clickture (Bing query-log). The proposed framework is shown to be versatile on Clickture dataset, which has not been examined by any of the previous approaches. The experimental results show that MHCI significantly improves the quality of image clusters as compared to existing methods.

Clustering of near duplicate images using bundled features

Clustering the images is generally based on the image’s visual features. Selection of relevant features is the most essential task. A clustering approach based on the bundled features is presented in this paper. Bundling of affine scale invariant feature transform (ASIFT) feature helps to cluster the near duplicates. When the local features are combined with the ASIFT features, the clustering efficiency is increased. Clustering the results from the web image search engines is very essential to help users narrow their search. We applied our idea of clustering with bundled features over Google Image search results. The results obtained show that the presented approach outperforms compared to the clustering done only with local features.

HyperSSR: A hypergraph based semi-supervised ranking method for visual search reranking

Abstract Recently, considerable efforts have been made in visual search reranking towards refining initial text-based image search results. In this paper, we propose a hypergraph based semi-supervised ranking method called HyperSSR for image search reranking. According to the basic visual consistency principle that visually similar images should have similar ranking scores, we introduce the hypergraph to capture the intrinsic geometrical structure of the data distribution. To build a robust hypergraph, a novel hypergraph construction approach is developed to incorporate relevance and pseudo relevance degree information from labeled and unlabeled samples, respectively. Based on the premise that a ranking model should work better with the prior pairwise preferences, we jointly incorporate the hypergraph regularizer and the prior pairwise preferences information into a unified ranking learning framework. Experimental results on MSRA-MM 1.0 dataset suggest our proposed approach produces superior performances compared with several state-of-the-art methods.

Exploration of Image Search Results Quality Assessment

Image retrieval plays an increasingly important role in our daily lives. There are many factors which affect the quality of image search results, including chosen search algorithms, ranking functions, and indexing features. Applying different settings for these factors generates search result lists with varying levels of quality. However, no setting can always perform optimally for all queries. Therefore, given a set of search result lists generated by different settings, it is crucial to automatically determine which result list is the best in order to present it to users. This paper aims to solve this problem and makes four main innovations. First, a preference learning model is proposed to quantitatively study and formulate the best image search result list identification problem. Second, a set of valuable preference learning related features is proposed by exploring the visual characters of returned images. Third, a query-dependent preference learning model is further designed for building a more precise and query-specific model. Fourth, the proposed approach has been tested on a variety of applications including reranking ability assessment, optimal search engine selection, and synonymous query suggestion. Extensive experimental results on three image search datasets demonstrate the effectiveness and promising potential of the proposed method.

PRISM

Most existing tag-based social image search engines present search results as a ranked list of images, which cannot be consumed by users in a natural and intuitive manner. In this demonstration, we present a novel concept-preserving image search results summarization system called prism . prism exploits both visual features and tags of the search results to generate high quality summary , which not only breaks the results into visually and semantically coherent clusters but it also maximizes the coverage of the original top- k search results. It first constructs a visual similarity graph where the nodes are images in the top- k search results and the edges represent visual similarities between pairs of images. This graph is optimally decomposed and compressed into a set of concept-preserving subgraphs based on a set of summarization criteria. One or more exemplar images from each subgraph is selected to form the exemplar summary of the result set. We demonstrate various innovative features of prism and the promise of superior quality summary construction of social image search results.

Learning Image Re-Rank: Query-DependentImage Re-Ranking Using Semantic Signature

Image re-ranking, is an effective way to improve the results of web-based image search and has been adopted by current commercial search engines such as Bing and Google. When a query keyword is given, a list of images are first retrieved based on textual information given by the user. By asking the user to select a query image from the pool of images, the remaining images are re-ranked based on their index with the query image. A major challenge is that sometimes semantic meanings may interpret user’s search intention. Many people recently proposed to match images in a semantic space which used attributes or reference classes closely related to the semantic meanings of images as basis. In this paper, we propose a novel image re-ranking framework, in which automatically offline learns different semantic spaces for different query keywords and displays with the image details in the form of augmented images. The images are projected into their related semantic spaces to get semantic signatures with the help of one click feedback from the user. At the online stage, images are re-ranked by comparing their semantic signatures obtained from the semantic space specified by the query keyword given by the user. The proposed query-specific semantic signatures significantly improve both the accuracy and efficiency of image re-ranking. Experimental results show that 25-40 percent relative improvement has been achieved on re-ranking precisions compared with the state-of-the-art methods.

Query Difficulty Estimation for Image Search With Query Reconstruction Error

Current image search engines suffer from a radical variance in retrieval performance over different queries. It is therefore desirable to identify those “difficult” queries in order to handle them properly. Query difficulty estimation is an attempt to predict the performance of the search results returned by an image search system. Most existing methods for query difficulty estimation focus on investigating statistical characteristics of the returned images only, while neglecting very important information , i.e., the query and its relationship with returned images. This relationship plays a crucial role in query difficulty estimation and should be explored further. In this paper we propose a novel query difficulty estimation method with query reconstruction error. This method is proposed based on the observation that, given the images returned for an unknown query, we can easily deduce what the query is from those images if the search results are high quality (i.e., lots of relevant images returned); otherwise, it is difficult to deduce the original query. Therefore, we propose to predict the query difficulty by measuring to what extent the original query can be recovered from the image search results. Specifically, we first reconstruct a visual query from the returned images to summarize their visual theme, and then use the reconstruction error, i.e., the distance between the original textual query and the reconstructed visual query, to estimate the query difficulty. We conduct extensive experiments on two real-world Web image datasets and demonstrate the effectiveness of the proposed method.