Visual Analytics Research Articles

An Efficient Search on Cloud Images with Blockchain Technology

A key task in computer vision is image retrieval, which has wide-ranging applications across multiple domains. Using query picture features, this abstract proposes an image retrieval approach that focuses on the widely used Scale-Invariant Feature Transform (SIFT) algorithm for feature extraction and distance calculation. The suggested method starts by extracting SIFT features from a set of photos, building a keypoints and descriptor database. By capturing the unique qualities of nearby image regions, these features enable reliable matching and retrieval. The security of private cloud data, which includes query queries, the search tree, and outsourced photographs, is another major issue. Use a feature extraction method first for integrated picture features, which are composed of fundamental components like colour and shape. In particular, because the proposed method uses a balancing index tree, it can achieve logarithmic search time. Second, the picture and query feature are encrypted using the secure inner product. Include a system for determining duplicate picture material as well. When given a query image, SIFT feature extraction is applied to it, producing keypoints and descriptors that correspond to its visual characteristics. Next, a distance computation method like Euclidean distance is used to compare the features of the database photos with the query image features. The similarity between the query image and the database images is measured through this comparison. The retrieved photos are ranked in order of similarity to the query image based on the calculated distances. Search results with lesser distances between images are deemed more similar and are displayed first. To locate visually related images in huge databases, the image retrieval system that uses SIFT feature extraction and distance calculation provides a reliable and effective solution. It contributes to developments in multimedia retrieval, visual analytics, and picture understanding by enabling applications like content-based image search, image recommendation systems, and image clustering.

Exploring Data Input Problems in Mixed Reality Environments: Proposal and Evaluation of Natural Interaction Techniques

Data input within mixed reality environments poses significant interaction challenges, notably in immersive visual analytics applications. This study assesses five numerical input techniques: three benchmark methods (Touch-Slider, Keyboard, Pinch-Slider) and two innovative multimodal techniques (Bimanual Scaling, Gesture and Voice). An experimental design was employed to compare these techniques’ input efficiency, accuracy, and user experience across varying precision and distance conditions. The findings reveal that multimodal techniques surpass slider methods in input efficiency yet are comparable to keyboards; the voice method excels in reducing cognitive load but falls short in accuracy; and the scaling method marginally leads in user satisfaction but imposes a higher physical load. Furthermore, this study outlines these techniques’ pros and cons and offers design guidelines and future research directions.

Is medieval distant viewing possible? : Extending and enriching annotation of legacy image collections using visual analytics

Abstract Distant viewing approaches have typically used image datasets close to the contemporary image data used to train machine learning models. To work with images from other historical periods requires expert annotated data, and the quality of labels is crucial for the quality of results. Especially when working with cultural heritage collections that contain myriad uncertainties, annotating data, or re-annotating, legacy data is an arduous task. In this paper, we describe working with two pre-annotated sets of medieval manuscript images that exhibit conflicting and overlapping metadata. Since a manual reconciliation of the two legacy ontologies would be very expensive, we aim (1) to create a more uniform set of descriptive labels to serve as a “bridge” in the combined dataset, and (2) to establish a high-quality hierarchical classification that can be used as a valuable input for subsequent supervised machine learning. To achieve these goals, we developed visualization and interaction mechanisms, enabling medievalists to combine, regularize and extend the vocabulary used to describe these, and other cognate, image datasets. The visual interfaces provide experts an overview of relationships in the data going beyond the sum total of the metadata. Word and image embeddings as well as co-occurrences of labels across the datasets enable batch re-annotation of images, recommendation of label candidates, and support composing a hierarchical classification of labels.

EDAR 4.0: Machine Learning and Visual Analytics for Wastewater Management

Wastewater treatment plant (WWTP) operations manage massive amounts of data that can be gathered with new Industry 4.0 technologies such as the Internet of Things and Big Data. These data are critical to allow the wastewater treatment industry to improve its operation, control, and maintenance. However, the data available need to be improved and enriched, partly due to their high dimensionality and low reliability, and the lack of appropriate data analysis and processing tools for such systems. This paper presents a visual analytics-based platform for WWTP that allows users to identify relationships among data through data inspection. The results show that the tool developed and implemented for a full-scale WWTP allows operators to construct machine learning (ML) models for water quality and other water treatment process variables. Consequently, analyzing and optimizing plant operation scenarios can enhance key variables, including energy, reagent consumption, and water quality. This improvement facilitates the development of a more sustainable WWTP, contributing to a beneficial environmental impact. Domain experts validated the variables influencing the created ML models and proved their appropriateness.

TSSeer: a visual analytics approach for exploring the correlation between teachers’ multimodal emotions and students’ behaviors in massive open online courses

TSSeer: a visual analytics approach for exploring the correlation between teachers’ multimodal emotions and students’ behaviors in massive open online courses

BPCoach: Exploring Hero Drafting in Professional MOBA Tournaments via Visual Analytics

Hero drafting for multiplayer online arena (MOBA) games is crucial because drafting directly affects the outcome of a match. Both sides take turns to "ban"/"pick" a hero from a roster of approximately 100 heroes to assemble their drafting. In professional tournaments, the process becomes more complex as teams are not allowed to pick heroes used in the previous rounds with the "best-of-N" rule. Additionally, human factors including the team's familiarity with drafting and play styles are overlooked by previous studies. Meanwhile, the huge impact of patch iteration on drafting strengths in the professional tournament is of concern. To this end, we propose a visual analytics system, BPCoach, to facilitate hero drafting planning by comparing various drafting through recommendations and predictions and distilling relevant human and in-game factors. Two case studies, expert feedback, and a user study suggest that BPCoach helps determine hero drafting in a rounded and efficient manner.

ProtoExplorer: Interpretable forensic analysis of deepfake videos using prototype exploration and refinement

In high-stakes settings, Machine Learning models that can provide predictions that are interpretable for humans are crucial. This is even more true with the advent of complex deep learning based models with a huge number of tunable parameters. Recently, prototype-based methods have emerged as a promising approach to make deep learning interpretable. We particularly focus on the analysis of deepfake videos in a forensics context. Although prototype-based methods have been introduced for the detection of deepfake videos, their use in real-world scenarios still presents major challenges, in that prototypes tend to be overly similar and interpretability varies between prototypes. This paper proposes a Visual Analytics process model for prototype learning, and, based on this, presents ProtoExplorer, a Visual Analytics system for the exploration and refinement of prototype-based deepfake detection models. ProtoExplorer offers tools for visualizing and temporally filtering prototype-based predictions when working with video data. It disentangles the complexity of working with spatio-temporal prototypes, facilitating their visualization. It further enables the refinement of models by interactively deleting and replacing prototypes with the aim to achieve more interpretable and less biased predictions while preserving detection accuracy. The system was designed with forensic experts and evaluated in a number of rounds based on both open-ended think aloud evaluation and interviews. These sessions have confirmed the strength of our prototype-based exploration of deepfake videos while they provided the feedback needed to continuously improve the system.

Improved DNN-assisted Customer Behavior Analysis with Smart Visual Analytics

Improved DNN-assisted Customer Behavior Analysis with Smart Visual Analytics

Differential effects of visual complexity in firm-generated content on consumer engagements: a deep learning approach

PurposeThis research explores the differential effects of pixel-level and object-level visual complexity in firm-generated content (FGC) on consumer engagement in terms of the number of likes and shares, and further investigates the moderating role of image brightness.Design/methodology/approachDrawing on a deep learning analysis of 85,975 images on a social media platform in China, this study investigates visual complexity in FGC.FindingsThe results indicate that pixel-level complexity increases both the number of likes and shares. Object-level complexity has a U-shaped relationship with the number of likes, while it has an inverted U-shaped relationship with the number of shares. Moreover, image brightness mitigates the effect of pixel-level complexity on likes but amplifies the effect on shares; contrarily, it amplifies the effect of object-level complexity on likes, while mitigating its effect on shares.Originality/valueAlthough images play a critical role in FGC, visual data analytics has rarely been used in social media research. This study identified two types of visual complexity and investigated their differential effects. We discuss how the processing of information embedded in visual content influences consumer engagement. The findings enrich the literature on social media and visual marketing.

Interactive visual query of density maps on latent space via flow‐based models

AbstractVisual querying of spatiotemporal data has become a dominant mode in the field of visual analytics. Previous studies have utilized well‐designed data structures to speed up the querying of spatiotemporal data. However, reducing storage overhead while improving the querying efficiency of data distribution remains a significant challenge. We propose a flow‐based neural representation method for efficient visual querying. First, we transform spatiotemporal data into density maps through kernel density estimation. Then, we leverage the data‐driven modeling capabilities of a flow‐based neural network to achieve a highly latent representation of the data. Various computations and queries can be performed on the latent representation to improve querying efficiency. Our experiments demonstrate that our approach achieves competitive results in visually querying spatiotemporal data in terms of storage overhead and real‐time interaction efficiency.

Leveraging Visualization and Machine Learning Techniques in Education: A Case Study of K-12 State Assessment Data

As data-driven models gain importance in driving decisions and processes, recently, it has become increasingly important to visualize the data with both speed and accuracy. A massive volume of data is presently generated in the educational sphere from various learning platforms, tools, and institutions. The visual analytics of educational big data has the capability to improve student learning, develop strategies for personalized learning, and improve faculty productivity. However, there are limited advancements in the education domain for data-driven decision making leveraging the recent advancements in the field of machine learning. Some of the recent tools such as Tableau, Power BI, Microsoft Azure suite, Sisense, etc., leverage artificial intelligence and machine learning techniques to visualize data and generate insights from them; however, their applicability in educational advances is limited. This paper focuses on leveraging machine learning and visualization techniques to demonstrate their utility through a practical implementation using K-12 state assessment data compiled from the institutional websites of the States of Texas and Louisiana. Effective modeling and predictive analytics are the focus of the sample use case presented in this research. Our approach demonstrates the applicability of web technology in conjunction with machine learning to provide a cost-effective and timely solution to visualize and analyze big educational data. Additionally, ad hoc visualization provides contextual analysis in areas of concern for education agencies (EAs).

Towards an understanding and explanation for mixed-initiative artificial scientific text detection

Large language models (LLMs) have gained popularity in various fields for their exceptional capability of generating human-like text. Their potential misuse has raised social concerns about plagiarism in academic contexts. However, effective artificial scientific text detection is a non-trivial task due to several challenges, including (1) the lack of a clear understanding of the differences between machine-generated and human-written scientific text, (2) the poor generalization performance of existing methods caused by out-of-distribution issues, and (3) the limited support for human-machine collaboration with sufficient interpretability during the detection process. In this paper, we first identify the critical distinctions between machine-generated and human-written scientific text through a quantitative experiment. Then, we propose a mixed-initiative workflow that combines human experts’ prior knowledge with machine intelligence, along with a visual analytics system to facilitate efficient and trustworthy scientific text detection. Finally, we demonstrate the effectiveness of our approach through two case studies and a controlled user study. We also provide design implications for interactive artificial text detection tools in high-stakes decision-making scenarios.

Developing campus digital twin using interactive visual analytics approach

AbstractDigital Twins (DTs) are increasingly recognized for their potential to improve efficiency and decision-making in various domains of the built environment. Despite their promise, challenges like cost, complexity, interoperability, and data integration remain. This paper introduces a novel interactive visual analytics system that tackles these issues, using a case study of simulating class distribution and campus building capacity at a large public university. The system leverages enrollment data, converting it into a spatial-temporal format for interactive exploration and analysis of class distribution and resource utilization. Through case studies, we demonstrate the system's effectiveness, adaptability, and real-world applicability, highlighting its role in practical DT implementation for built environments.

Visual Analytics in Explaining Neural Networks with Neuron Clustering

Deep learning (DL) models have achieved state-of-the-art performance in many domains. The interpretation of their working mechanisms and decision-making process is essential because of their complex structure and black-box nature, especially for sensitive domains such as healthcare. Visual analytics (VA) combined with DL methods have been widely used to discover data insights, but they often encounter visual clutter (VC) issues. This study presents a compact neural network (NN) view design to reduce the visual clutter in explaining the DL model components for domain experts and end users. We utilized clustering algorithms to group hidden neurons based on their activation similarities. This design supports the overall and detailed view of the neuron clusters. We used a tabular healthcare dataset as a case study. The design for clustered results reduced visual clutter among neuron representations by 54% and connections by 88.7% and helped to observe similar neuron activations learned during the training process.

An integrated visual analytics system for studying clinical carotid artery plaques

An integrated visual analytics system for studying clinical carotid artery plaques

Multiple Cubes Growth Algorithms for Simple Representative Elementary Volume Determination on 3D Binary Images

Porosity analysis is fundamental for understanding various material properties and transport phenomena in scientific and engineering disciplines. This study delves into the challenging task of determining the representative elementary volume (REV) in porous media, crucial for accurate analyses. Two novel algorithms, Center-Corner Cubes Growing (3CG) and Random Cubes Growing (RCG), were proposed and tested on synthetic body-centered cubic (BCC) sphere packing and natural porous structures of Berea sandstone and Indiana limestone, obtained using µCT. First algorithm (3CG) operates by analyzing porosity within cubes growing from each of the eight corners and a central region of a 3D binarized stack. In contrast, the Random Cube Growing (RCG) algorithm randomly selects seed points within the 3D stack and grows cubic regions around them. Both algorithms systematically compute porosity for various cube sizes, determining the average porosity and standard deviation for each extent. These visual analytics tools contribute to identifying the specific size ranges where porosity curves converge and stabilize, indicating potential REV within the material. While 3CG simplifies the approach by focusing on a limited number of curves, RCG provides a broader view, capturing diverse porosity patterns. The absence of consistent local minima in certain cases indicates high porosity heterogeneity and the impossibility of achieving REV in certain sample sizes.

Визуализация в задачах реконструкции данных

Many application tasks of multidimensional data analysis which describe the state of real physical or other systems face with difficulties. This is a consequence of the low-quality source data, including missing values, the probability of errors or unreliability of measurements. Incomplete data can become an obstacle for research using many modern informational methods. The current work examines the potential and capabilities of visual analytics tools for preliminary preparation, correction or complete analysis of primary data volumes. A promising area of application of the approach discussed in the study is the targeted use of visualization capabilities as a data analysis tool. The implementation of specialized visual metaphors is used to solve problems of processing and interpreting data, the sources of which are cyberphysical systems of different complexity levels. Such systems operate in an autonomous or partially controlled mode. A characteristic feature of these systems is the presence of a large number of sensors that collect various types of data. Such data differ in the capacity of the corresponding information channels, their speed and reliability. Examples of such cyberphysical systems are unmanned aerial vehicles (UAVs), robotic stations, and multimodal monitoring systems. These systems can function in conditions where it is difficult to obtain objective observation experience (deep-sea robots). The effective use of data collected by cyberphysical monitoring systems is a condition for solving a large number of application and research tasks.

Explainable Deep Learning: A Visual Analytics Approach with Transition Matrices

The non-transparency of artificial intelligence (AI) systems, particularly in deep learning (DL), poses significant challenges to their comprehensibility and trustworthiness. This study aims to enhance the explainability of DL models through visual analytics (VA) and human-in-the-loop (HITL) principles, making these systems more transparent and understandable to end users. In this work, we propose a novel approach that utilizes a transition matrix to interpret results from DL models through more comprehensible machine learning (ML) models. The methodology involves constructing a transition matrix between the feature spaces of DL and ML models as formal and mental models, respectively, improving the explainability for classification tasks. We validated our approach with computational experiments on the MNIST, FNC-1, and Iris datasets using a qualitative and quantitative comparison criterion, that is, how different the results obtained by our approach are from the ground truth of the training and testing samples. The proposed approach significantly enhanced model clarity and understanding in the MNIST dataset, with SSIM and PSNR values of 0.697 and 17.94, respectively, showcasing high-fidelity reconstructions. Moreover, achieving an F1m score of 77.76% and a weighted accuracy of 89.38%, our approach proved its effectiveness in stance detection with the FNC-1 dataset, complemented by its ability to explain key textual nuances. For the Iris dataset, the separating hyperplane constructed based on the proposed approach allowed for enhancing classification accuracy. Overall, using VA, HITL principles, and a transition matrix, our approach significantly improves the explainability of DL models without compromising their performance, marking a step forward in developing more transparent and trustworthy AI systems.

Money market insights in China: Evidence from visual analytics approach

This research employs visual analytics approaches to demystify the complex dynamics of China’s money market, spanning from 1984 to 2020. Our objective is to transform intricate financial data into intuitive visual representations, thereby enhancing understanding and decision-making. We utilize advanced visual analytics techniques to analyze key aspects like money supply, deposits, loans, and foreign exchange. The study reveals significant trends and insights, contributing to a more comprehensive understanding of financial dynamics in China. These findings serve as valuable tools for economists and policymakers, guiding more informed decisions in financial governance.

An Expository Examination of Temporally Evolving Graph-Based Approaches for the Visual Investigation of Autonomous Driving

With the continuous advancement of autonomous driving technology, visual analysis techniques have emerged as a prominent research topic. The data generated by autonomous driving is large-scale and time-varying, yet more than existing visual analytics methods are required to deal with such complex data effectively. Time-varying diagrams can be used to model and visualize the dynamic relationships in various complex systems and can visually describe the data trends in autonomous driving systems. To this end, this paper introduces a time-varying graph-based method for visual analysis in autonomous driving. The proposed method employs a graph structure to represent the relative positional relationships between the target and obstacle interferences. By incorporating the time dimension, a time-varying graph model is constructed. The method explores the characteristic changes of nodes in the graph at different time instances, establishing feature expressions that differentiate target and obstacle motion patterns. The analysis demonstrates that the feature vector centrality in the time-varying graph effectively captures the distinctions in motion patterns between targets and obstacles. These features can be utilized for accurate target and obstacle recognition, achieving high recognition accuracy. To evaluate the proposed time-varying graph-based visual analytic autopilot method, a comparative study is conducted against traditional visual analytic methods such as the frame differencing method and advanced visual analytic methods like visual lidar odometry and mapping. Robustness, accuracy, and resource consumption experiments are performed using the publicly available KITTI dataset to analyze and compare the three methods. The experimental results show that the proposed time-varying graph-based method exhibits superior accuracy and robustness. This study offers valuable insights and solution ideas for developing deep integration between intelligent networked vehicles and intelligent transportation. It provides a reference for advancing intelligent transportation systems and their integration with autonomous driving technologies.