Visual Framework Research Articles

GIS-based multi-criteria decision making under Silica Saturation Index (SSI) for selecting the best direct use scenarios for geothermal resources in Central and Southern Rift Valley, Kenya

Direct use (DU) of geothermal energy is site-specific and majorly dictated by economics. Even with the availability of high enthalpy geothermal fluids, DU could still be curtailed by fluid pH, salinity, and silica content. Selecting economically viable best direct use (DU) scenarios for a given geothermal resource needs meticulous matching of geothermal resource characteristics to the nearby economic activities and local infrastructure. Silica saturation index (SSI) analysis should also be factored in to avoid silica scaling of DU facilities and reinjection well. There is a lack of a visual framework for incorporating all the necessary criteria for selecting the best DU scenarios, hence the motivation of this research. The current study employed a GIS-based multi-criteria decision making (MCDM) method: a hybrid of the analytical hierarchy process (AHP) method with the weighted aggregated sum product assessment (WASPAS) method. The model analyzed nine utilization scenarios against eleven criteria to select the best DU scenarios for six high-enthalpy quaternary volcanoes in the central and southern Kenyan rift: Menengai, Elementaita, Eburru, Olkaria, Longonot, and Suswa. The best typical DU schemes identified were horticultural greenhouse warming, spa, and crop drying. The model was validated with existing DU case studies to gauge its ability to predict suitable utilization methods. There was a good matching between the model's selection and existing case studies implicating the synergistic merger of GIS-based MCDM methods and SSI as a robust tool for planning and carrying out the feasibility study for DU of geothermal energy.

Cooperative multi-camera vehicle tracking and traffic surveillance with edge artificial intelligence and representation learning

Traffic surveillance cameras are the eyes of the Intelligent Transportation Systems (ITS). However, they are currently isolated and can only extract information from each of their fixed views. To track vehicles across multiple cameras and help public agencies collect link travel time and speed information, an Edge-empowered Cooperative Multi-camera Sensing (ECoMS) System is proposed. ECoMS system presents a novel algorithmic and edge-server cooperative system construct to push edge computing and multi-camera re-identification workflow serving for traffic sensing based on Internet of Things (IoT) architecture. On the algorithm side, ECoMS system proposes a featherlight edge-based computer vision framework for vehicle detection, tracking, and features selection process in a real-time manner. Then, by only sending the objects’ representations to the server, the high-bandwidth data transmission and the heavy post-processing system can be abandoned. Furthermore, a hierarchical clip-based deep vehicle re-identification framework is proposed and integrated into the ECoMS system, and significantly outperforms other state-of-the-art methods by 4%–8% on Rank-1 accuracy. Finally, to balance the accuracy level of different camera pairs, a collaborative cross-camera traffic information estimation framework based on kernel density estimation with kernel smoother is implemented, which can get the precise link and region traffic information together with distributions (less than 1.01 KL distance). By maximizing the cooperation of the computational resources, orchestrating the data transmission and integrating road network graph features in the system, the ECoMS can precisely model the network-scale traffic information in a flexible, cost-effective, and scalable workflow. To the author’s best knowledge, ECoMS is the first multi-camera vehicle tracking and traffic monitoring system based on cooperative IoT architecture.

Analysing cerebrospinal fluid with explainable deep learning: From diagnostics to insights.

Analysis of cerebrospinal fluid (CSF) is essential for diagnostic workup of patients with neurological diseases and includes differential cell typing. The current gold standard is based on microscopic examination by specialised technicians and neuropathologists, which is time-consuming, labour-intensive and subjective. We, therefore, developed an image analysis approach based on expert annotations of 123,181 digitised CSF objects from 78 patients corresponding to 15 clinically relevant categories and trained a multiclass convolutional neural network (CNN). The CNN classified the 15 categories with high accuracy (mean AUC 97.3%). By using explainable artificial intelligence (XAI), we demonstrate that the CNN identified meaningful cellular substructures in CSF cells recapitulating human pattern recognition. Based on the evaluation of 511 cells selected from 12 different CSF samples, we validated the CNN by comparing it with seven board-certified neuropathologists blinded for clinical information. Inter-rater agreement between the CNN and the ground truth was non-inferior (Krippendorff's alpha 0.79) compared with the agreement of seven human raters and the ground truth (mean Krippendorff's alpha 0.72, range 0.56-0.81). The CNN assigned the correct diagnostic label (inflammatory, haemorrhagic or neoplastic) in 10 out of 11 clinical samples, compared with 7-11 out of 11 by human raters. Our approach provides the basis to overcome current limitations in automated cell classification for routine diagnostics and demonstrates how a visual explanation framework can connect machine decision-making with cell properties and thus provide a novel versatile and quantitative method for investigating CSF manifestations of various neurological diseases.

VVAFER — Versatile visual analytics framework for exploration and research

The development of interactive visualization applications that are applicable to many real-world problems is a challenging affair. For every new project, developers need to follow the same repetitive steps of fetching the raw data, transforming the data into processable form, defining visual structures and then displaying them appropriately. To accelerate this, we propose the Versatile Visual Analytics Framework for Exploration and Research (VVAFER). VVAFER is planned to be an extensible visual analytics framework, upon which different applications can be developed with minimum overload at the development side. Through modular architecture, unified data formats, reusable templates and software components, developers will be able to quickly deploy and create their visualization applications by configuring existing templates with their own specific functionalities. In this paper, we describe our motivation for this future framework and its architectural design.

FCLWebVis: A flexible cross-language web-based data visualization framework

We present a new web-based, client-server data processing and visualization framework that supports a flexible workflow, enabling the user to customize different data processing and visualization tasks with tools implemented in different programming languages. Our framework supports server-side applications developed with different languages, allowing visualization researchers to easily make their new techniques available to the target users. The client-side of our framework is implemented in the web browser environment with customizable interface and visualizations. We describe the design of the architecture of our framework and the process of adding new user-defined tasks, followed by the demonstration of the proposed framework on a number of data processing and visualization tasks.

INPHOVIS: Interactive visual analytics for smartphone-based digital phenotyping

Digital phenotyping is the characterization of human behavior patterns based on data from digital devices such as smartphones in order to gain insights into the users’ state and especially to identify ailments. To support supervised machine learning, digital phenotyping requires gathering data from study participants’ smartphones as they live their lives. Periodically, participants are then asked to provide ground truth labels about their health status. Analyzing such complex data is challenging due to limited contextual information and imperfect health/wellness labels. We propose INteractive PHOne-o-typing VISualization (INPHOVIS), an interactive visual framework for exploratory analysis of smartphone health data to study phone-o-types. Prior visualization work has focused on mobile health data with clear semantics such as steps or heart rate data collected using dedicated health devices and wearables such as smartwatches. However, unlike smartphones which are owned by over 85 percent of the US population, wearable devices are less prevalent thus reducing the number of people from whom such data can be collected. In contrast, the “low-level” sensor data (e.g., accelerometer or GPS data) supported by INPHOVIS can be easily collected using smartphones. Data visualizations are designed to provide the essential contextualization of such data and thus help analysts discover complex relationships between observed sensor values and health-predictive phone-o-types. To guide the design of INPHOVIS, we performed a hierarchical task analysis of phone-o-typing requirements with health domain experts. We then designed and implemented multiple innovative visualizations integral to INPHOVIS including stacked bar charts to show diurnal behavioral patterns, calendar views to visualize day-level data along with bar charts, and correlation views to visualize important wellness predictive data. We demonstrate the usefulness of INPHOVIS with walk-throughs of use cases. We also evaluated INPHOVIS with expert feedback and received encouraging responses.

Research on Semantic Extraction of Online Mathematics Guidance Videos in Colleges and Universities Based on Intelligent Information Technology

This article proposes a general solution for multi-granularity semantic analysis and extraction of video data based on the online instructional video of mathematics in colleges and universities of intelligent information technology. In this scheme, multi-level semantic analysis and multi-modal information fusion technology are unified and applied in the same model. This paper first proposes a method for detecting the gradual change of shots based on statistical distribution, and uses a key frame selection strategy with temporal semantic context constraints to represent the temporal content. After basic visual semantic recognition, a hierarchical approach is obtained. The multi-granularity visual semantic analysis extraction framework then uses the sound spectrum obtained by the time-frequency transformation as the observable feature, and constructs a hidden Markov model for semantic recognition of mathematical videos, which improves the efficiency by 7.93%.

EscheR: unified multi-dimensional visualizations with Gestalt principles.

The creation of effective visualizations is a fundamental component of data analysis. In biomedical research, new challenges are emerging to visualize multi-dimensional data in a 2D space, but current data visualization tools have limited capabilities. To address this problem, we leverage Gestalt principles to improve the design and interpretability of multi-dimensional data in 2D data visualizations, layering aesthetics to display multiple variables. The proposed visualization can be applied to spatially-resolved transcriptomics data, but also broadly to data visualized in 2D space, such as embedding visualizations. We provide an open source R package escheR, which is built off of the state-of-the-art ggplot2 visualization framework and can be seamlessly integrated into genomics toolboxes and workflows. The open source R package escheR is freely available on Bioconductor (https://bioconductor.org/packages/escheR).

ManiVault: A Flexible and Extensible Visual Analytics Framework for High-Dimensional Data.

Exploration and analysis of high-dimensional data are important tasks in many fields that produce large and complex data, like the financial sector, systems biology, or cultural heritage. Tailor-made visual analytics software is developed for each specific application, limiting their applicability in other fields. However, as diverse as these fields are, their characteristics and requirements for data analysis are conceptually similar. Many applications share abstract tasks and data types and are often constructed with similar building blocks. Developing such applications, even when based mostly on existing building blocks, requires significant engineering efforts. We developed ManiVault, a flexible and extensible open-source visual analytics framework for analyzing high-dimensional data. The primary objective of ManiVault is to facilitate rapid prototyping of visual analytics workflows for visualization software developers and practitioners alike. ManiVault is built using a plugin-based architecture that offers easy extensibility. While our architecture deliberately keeps plugins self-contained, to guarantee maximum flexibility and re-usability, we have designed and implemented a messaging API for tight integration and linking of modules to support common visual analytics design patterns. We provide several visualization and analytics plugins, and ManiVault's API makes the integration of new plugins easy for developers. ManiVault facilitates the distribution of visualization and analysis pipelines and results for practitioners through saving and reproducing complete application states. As such, ManiVault can be used as a communication tool among researchers to discuss workflows and results. A copy of this paper and all supplemental material is available at osf.io/9k6jw, and source code at github.com/ManiVaultStudio.

A Comparative Visual Analytics Framework for Evaluating Evolutionary Processes in Multi-Objective Optimization.

Evolutionary multi-objective optimization (EMO) algorithms have been demonstrated to be effective in solving multi-criteria decision-making problems. In real-world applications, analysts often employ several algorithms concurrently and compare their solution sets to gain insight into the characteristics of different algorithms and explore a broader range of feasible solutions. However, EMO algorithms are typically treated as black boxes, leading to difficulties in performing detailed analysis and comparisons between the internal evolutionary processes. Inspired by the successful application of visual analytics tools in explainable AI, we argue that interactive visualization can significantly enhance the comparative analysis between multiple EMO algorithms. In this paper, we present a visual analytics framework that enables the exploration and comparison of evolutionary processes in EMO algorithms. Guided by a literature review and expert interviews, the proposed framework addresses various analytical tasks and establishes a multi-faceted visualization design to support the comparative analysis of intermediate generations in the evolution as well as solution sets. We demonstrate the effectiveness of our framework through case studies on benchmarking and real-world multi-objective optimization problems to elucidate how analysts can leverage our framework to inspect and compare diverse algorithms.

ANTECEDENTES, RESULTADOS E LIMITES DA GESTÃO DE RECURSOS HUMANOS VERDE: UMA REVISÃO BIBLIOGRÁFICA

ABSTRACT Green human resource management (GHRM) is a crucial element of the globally trending topic of green management. Although GHRM has received considerable research attention in recent years, confusion remains regarding its antecedents, the social, and psychological processes through which GHRM influences employee behavior, and the potential outcomes to be derived from its use. This paper aims to disentangle and summarize the components that have been explored in the GHRM-performance relationship. To achieve these goals, we conduct a systematic review based on the preferred reporting items method and then present a series of theoretical approaches to stimulate new debates on theory building and its subsequent use in GHRM research. We also carefully explain information on organizational and employee-level factors that motivate and hinder GHRM and display this information in a visual framework. The implications for practice provide focused recommendations to help managers understand how to create favorable conditions for enhancing sustainability performance. The gaps identified should open the way to new lines of research that still need attention.

Smart Vernacular Architecture: A Framework for Assessment and Virtual Reality-based Visualisation of Indigenous Toda Dwellings

Indigenous communities and their way of life are facing significant challenges due to globalisation, industrialisation and urbanisation, which have forced them to gradually adopt the architecture, lifestyle, and culture of mainstream society. This paper proposes a Smart Vernacular Architecture (SVA) framework and applies it to the indigenous dwellings of the Todas, a tribal community living in India's Nilgiri mountains. The architecture and indoor space quality of these dwellings are investigated in this paper based on specific thermal comfort parameters like the interior temperature and airflow. The indoor temperatures of Toda dwellings are simulated for the predicted climate of 2100 to evaluate their climate resilience. Finally, the paper illustrates the usage of virtual reality (VR) to explore and visualise the Toda settlement and the interior environment parameters of a Toda dwelling to effectively convey and preserve the knowledge of vernacular architecture. In a nutshell, this paper elucidates the advantages of vernacular architecture and construction methods to arouse the interest of academics, policymakers, students, and professionals in preserving architecture, culture, and indigenous knowledge.

Slimmable Multi-Task Image Compression for Human and Machine Vision

In the Internet of Things (IoT) communications, visual data is frequently processed among intelligent devices using artificial intelligence algorithms, replacing humans for analyzing and decision-making while only occasionally requiring human’s scrutiny. However, due to high redundancy of compressive encoders, existing image coding solutions for machine vision are not efficient at runtime. To balance the rate-accuracy performance and efficiency of image compression for machine vision while attaining high-quality reconstructed images for human vision, this paper introduces a novel slimmable multi-task compression framework for human and machine vision in visual IoT applications. Firstly, the image compression for human and machine vision under the constraint of bandwidth, latency, computational resources are modelled as a multi-task optimization problem. Secondly, slimmable encoders are employed to multiple human and machine vision tasks in which the parameters of the sub-encoder for machine vision tasks are shared among all tasks and jointly learned. Thirdly, to solve the feature match between latent representation and intermediate features of deep vision networks, feature transformation networks are introduced as decoders of machine vision feature compression. Finally, the proposed framework is successfully applied to human and machine vision tasks’ scenarios, e.g., object detection and image reconstruction. Experimental results show that the proposed method outperforms baselines and other image compression approaches on machine vision tasks with higher efficiency (shorter latency) in two vision tasks’ scenarios while retaining comparable quality on image reconstruction.

Task-Driven Video Compression for Humans and Machines: Framework Design and Optimization

Learned video compression has developed rapidly and achieved impressive progress in recent years. Despite efficient compression performance, existing signal fidelity oriented or semantic fidelity oriented video compression methods limit the capability to meet the requirements of both machine and human vision. To address this problem, a task-driven video compression framework is proposed to flexibly support vision tasks for both human vision and machine vision. Specifically, to improve the compression performance, the backbone of the video compression framework is optimized by using three novel modules, including multi-scale motion estimation, multi-frame feature fusion, and reference based in-loop filters. Then, based on the proposed efficient compression backbone, a task-driven optimization approach is designed to achieve the trade-off between signal fidelity oriented compression and semantic fidelity oriented compression. Moreover, a post-filter module is employed for the framework to further improve the performance of the human vision branch. Finally, rate-distortion performance, rate-accuracy performance, and subjective quality are employed as the evaluation metrics, and experimental results show the superiority of the proposed framework for both human vision and machine vision. The source code of this work can be found in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://mic.tongji.edu.cn</uri> .

An Artificial Neural SLAM Framework for Event-Based Vision

An Artificial Neural SLAM Framework for Event-Based Vision

A Real-Time Automated Visual Inspection System for Printed Circuit Boards Missing Footprints Detection

Visual inspection systems (VIS) are vital for recognizing and assessing parts in mass-produced products at the fabricating lines. In the past, item review was carried out physically, which made finding imperfections repetitive, moderate, and prone to error. VIS may be a strategy to abbreviate preparing times, boost item quality, and increment fabricating competitiveness. For the reason of reviewing lost components on uncovered printed circuit sheets, a visual inspection framework is required. The assessment assignment has become more challenging to accomplish the specified quality due to the more compact and complex surface of structured electronic components. This study proposes a real-time visual inspection system to assess lost impressions on Printed Circuit Boards (PCB). This system is composed of hardware and software frameworks. The main contribution of this study is the proposed software framework. The software framework consists of components region analysis and missing detection using image processing, cross-correlation, and production rules. Experimental results show the viability and achievability of the proposed system for PCB missing component detection.

TimeTuner: Diagnosing Time Representations for Time-Series Forecasting with Counterfactual Explanations.

Deep learning (DL) approaches are being increasingly used for time-series forecasting, with many efforts devoted to designing complex DL models. Recent studies have shown that the DL success is often attributed to effective data representations, fostering the fields of feature engineering and representation learning. However, automated approaches for feature learning are typically limited with respect to incorporating prior knowledge, identifying interactions among variables, and choosing evaluation metrics to ensure that the models are reliable. To improve on these limitations, this paper contributes a novel visual analytics framework, namely TimeTuner, designed to help analysts understand how model behaviors are associated with localized correlations, stationarity, and granularity of time-series representations. The system mainly consists of the following two-stage technique: We first leverage counterfactual explanations to connect the relationships among time-series representations, multivariate features and model predictions. Next, we design multiple coordinated views including a partition-based correlation matrix and juxtaposed bivariate stripes, and provide a set of interactions that allow users to step into the transformation selection process, navigate through the feature space, and reason the model performance. We instantiate TimeTuner with two transformation methods of smoothing and sampling, and demonstrate its applicability on real-world time-series forecasting of univariate sunspots and multivariate air pollutants. Feedback from domain experts indicates that our system can help characterize time-series representations and guide the feature engineering processes.

Visual Probing: Cognitive Framework for Explaining Self-Supervised Image Representations

Recently introduced self-supervised methods for image representation learning provide on par or superior results to their fully supervised competitors, yet the corresponding efforts to explain the self-supervised approaches lag behind. Motivated by this observation, we introduce a novel visual probing framework for explaining the self-supervised models by leveraging probing tasks employed previously in natural language processing. The probing tasks require knowledge about semantic relationships between image parts. Hence, we propose a systematic approach to obtain analogs of natural language in vision, such as visual words, context, and taxonomy. Our proposal is grounded in Marr's computational theory of vision and concerns features like textures, shapes, and lines. We show the effectiveness and applicability of those analogs in the context of explaining self-supervised representations. Our key findings emphasize that relations between language and vision can serve as an effective yet intuitive tool for discovering how machine learning models work, independently of data modality. Our work opens a plethora of research pathways towards more explainable and transparent AI.

OW-Adapter: Human-Assisted Open-World Object Detection with a Few Examples.

Open-world object detection (OWOD) is an emerging computer vision problem that involves not only the identification of predefined object classes, like what general object detectors do, but also detects new unknown objects simultaneously. Recently, several end-to-end deep learning models have been proposed to address the OWOD problem. However, these approaches face several challenges: a) significant changes in both network architecture and training procedure are required; b) they are trained from scratch, which can not leverage existing pre-trained general detectors; c) costly annotations for all unknown classes are needed. To overcome these challenges, we present a visual analytic framework called OW-Adapter. It acts as an adaptor to enable pre-trained general object detectors to handle the OWOD problem. Specifically, OW-Adapter is designed to identify, summarize, and annotate unknown examples with minimal human effort. Moreover, we introduce a lightweight classifier to learn newly annotated unknown classes and plug the classifier into pre-trained general detectors to detect unknown objects. We demonstrate the effectiveness of our framework through two case studies of different domains, including common object recognition and autonomous driving. The studies show that a simple yet powerful adaptor can extend the capability of pre-trained general detectors to detect unknown objects and improve the performance on known classes simultaneously.

EuclidNet: Deep Visual Reasoning for Constructible Problems in Geometry

In this paper, we present a visual reasoning framework driven by deep learning for solving constructible problems in geometry that is useful for automated geometry theorem proving. Constructible problems in geometry often ask for the sequence of straightedge-and-compass constructions to construct a given goal given some initial setup. Our EuclidNet framework leverages the neural network architecture Mask R-CNN to extract the visual features from the initial setup and goal configuration with extra points of intersection, and then generate possible construction steps as intermediary data models that are used as feedback in the training process for further refinement of the construction step sequence. This process is repeated recursively until either a solution is found, in which case we backtrack the path for a step-by-step construction guide, or the problem is identified as unsolvable. Our EuclidNet framework is validated on the problem set of Euclidea with an average of 75% accuracy without prior knowledge and complex Japanese Sangaku geometry problems, demonstrating its capacity to leverage backtracking for deep visual reasoning of challenging problems.