Visual Framework Research Articles

Enhancing algal bloom forecasting: A novel framework for machine learning performance evaluation during periods of special temporal patterns

The evaluation of algal bloom forecasting models typically relies on error metrics that quantify the forecasting performance over the whole test set as a single number. Furthermore, the comparison with simple baseline methods is often omitted. To address this, we introduce a novel framework for Model performance Analysis and Visualization of time series forecasting (MAVts). MAVts incorporates novel algorithms for the automatic identification and visualization of time series periods of interest where the forecasting models are evaluated and compared with simple baseline methods. The application of MAVts on evaluating algal bloom forecasting models composed of sophisticated machine learning (ML) methods, reveals that in 85% of experiments a single error metric is not enough and only in 12.5% of experiments a ML model outperforms all baselines on all metrics and periods of interest. Thus, MAVts emerges as a valuable tool for analyzing and comparing ML models, advancing environmental management and protection.

Social Semiotic Analysis of a Pet Protection Post on Instagram

This study applies Kress and Van Leeuwen's Visual Grammar framework to analyze an Instagram post advocating pet protection. The research focuses on representational, interactive, and compositional metafunctions to understand how visual and textual elements combine to convey urgency and empathy. The representational analysis highlights the dog's isolated portrayal and the use of the pronoun "me" to foster an emotional connection. The interactive analysis examines indirect gaze, close-up shots, and low angles to engage viewers and emphasize their role in pet protection. The compositional analysis reveals the strategic placement of verbal and visual elements, with bold text and contrasting colors enhancing the message's salience. This multimodal approach demonstrates the synergy between visual and textual elements, creating a cohesive and persuasive narrative. The study's findings align with existing literature on social semiotics and Visual Grammar, emphasizing the importance of integrated visual strategies in social media advocacy. The research offers practical insights for designing effective pet protection campaigns, highlighting the role of Visual Grammar in promoting responsible pet ownership and animal welfare. Future research should explore the long-term impact of these visual strategies on viewer behavior and the effectiveness of different semiotic modes across various social media platforms.

Auxiliary audio–textual modalities for better action recognition on vision-specific annotated videos

Most current audio–visual datasets are class-relevant, where audio and visual modalities are annotated. Thus, current audio–visual recognition methods apply cross-modality attention or modality fusion. However, leveraging the audio modality effectively in vision-specific videos for human activity recognition is of particular challenge. We address this challenge by proposing a novel audio–visual recognition framework that effectively leverages audio modality in any vision-specific annotated dataset. The proposed framework employs language models (e.g., GPT-3, CPT-text, BERT) for building a semantic audio–video label dictionary (SAVLD) that serves as a bridge between audio and video datasets by mapping each video label to its most K-relevant audio labels. Then, SAVLD along with a pre-trained audio multi-label model are used to estimate the audio–visual modality relevance. Accordingly, we propose a novel learnable irrelevant modality dropout (IMD) to completely drop the irrelevant audio modality and fuse only the relevant modalities. Finally, for the efficiency of the proposed multimodal framework, we present an efficient two-stream video Transformer to process the visual modalities (i.e., RGB frames and optical flow). The final predictions are re-ranked with GPT-3 recommendations of the human activity classes. GPT-3 provides high-level recommendations using the labels of the detected visual objects and the audio predictions of the input video. Our framework demonstrated a remarkable performance on the vision-specific annotated datasets Kinetics400 and UCF-101 by outperforming most relevant human activity recognition methods.

Airside Surveillance by Computer Vision in Low-Visibility and Low-Fidelity Environment

Low visibility at airports can significantly impact airside capacity, leading to ground delays and runway/taxiway incursions. Digital tower technology, enabled by live camera feeds, leverages computer vision to enhance airside surveillance and operational efficiency. However, technical challenges in digital camera systems can introduce low-fidelity transmission effects such as blurring, pixelation, or JPEG compression. Additionally, adverse weather conditions like rain and fog can further reduce visibility for tower controllers, whether from digital video or out-of-tower views. This paper proposes a computer vision framework and deep learning algorithms to detect and track aircraft in low-visibility (due to bad weather) and low-fidelity (due to technical issues) environments to enhance visibility using digital video input. The framework employs a convolutional neural network for aircraft detection and Kalman filters for tracking, especially in low-visibility conditions. Performance enhancements come from pre- and postprocessing algorithms like object filtering, corrupted image detection, and image enhancement. It proves effective on an airport video dataset from Houston Airport, enhancing visibility under adverse weather conditions.

A Method for Real-Time Recognition of Safflower Filaments in Unstructured Environments Using the YOLO-SaFi Model.

The identification of safflower filament targets and the precise localization of picking points are fundamental prerequisites for achieving automated filament retrieval. In light of challenges such as severe occlusion of targets, low recognition accuracy, and the considerable size of models in unstructured environments, this paper introduces a novel lightweight YOLO-SaFi model. The architectural design of this model features a Backbone layer incorporating the StarNet network; a Neck layer introducing a novel ELC convolution module to refine the C2f module; and a Head layer implementing a new lightweight shared convolution detection head, Detect_EL. Furthermore, the loss function is enhanced by upgrading CIoU to PIoUv2. These enhancements significantly augment the model's capability to perceive spatial information and facilitate multi-feature fusion, consequently enhancing detection performance and rendering the model more lightweight. Performance evaluations conducted via comparative experiments with the baseline model reveal that YOLO-SaFi achieved a reduction of parameters, computational load, and weight files by 50.0%, 40.7%, and 48.2%, respectively, compared to the YOLOv8 baseline model. Moreover, YOLO-SaFi demonstrated improvements in recall, mean average precision, and detection speed by 1.9%, 0.3%, and 88.4 frames per second, respectively. Finally, the deployment of the YOLO-SaFi model on the Jetson Orin Nano device corroborates the superior performance of the enhanced model, thereby establishing a robust visual detection framework for the advancement of intelligent safflower filament retrieval robots in unstructured environments.

On comparative analysis of a two dimensional star gold structure via regression models

In this research, the star gold structure with beta graphene is thoroughly examined. We mainly focus on computing degree-based topological indices, which provide information about the network’s connectivity and complexity as well as structural features. In addition, we compute an entropy measure to represent the uncertainty, information richness, and degree of unpredictability in the network. Furthermore, this study explores the relationships between topological descriptors and entropy using regression models that are logarithmic, linear, and quadratic. By merging these regression models, we uncover hidden patterns and understand the underlying ideas governing the network’s behaviour. Our findings shed light on the connection between topological indices and entropy. This work improves our understanding of star gold structure dynamics and provides a visual framework for interpreting their behaviour.

A Fusion Positioning System Based on Camera and LiDAR for Unmanned Rollers in Tunnel Construction.

As an important vehicle in road construction, the unmanned roller is rapidly advancing in its autonomous compaction capabilities. To overcome the challenges of GNSS positioning failure during tunnel construction and diminished visual positioning accuracy under different illumination levels, we propose a feature-layer fusion positioning system based on a camera and LiDAR. This system integrates loop closure detection and LiDAR odometry into the visual odometry framework. Furthermore, recognizing the prevalence of similar scenes in tunnels, we innovatively combine loop closure detection with the compaction process of rollers in fixed areas, proposing a selection method for loop closure candidate frames based on the compaction process. Through on-site experiments, it is shown that this method not only enhances the accuracy of loop closure detection in similar environments but also reduces the runtime. Compared with visual systems, in static positioning tests, the longitudinal and lateral accuracy of the fusion system are improved by 12 mm and 11 mm, respectively. In straight-line compaction tests under different illumination levels, the average lateral error increases by 34.1% and 32.8%, respectively. In lane-changing compaction tests, this system enhances the positioning accuracy by 33% in dim environments, demonstrating the superior positioning accuracy of the fusion positioning system amid illumination changes in tunnels.

A MULTIMODAL DISCOURSE ANALYSIS OF A MEDICAL TEXTBOOK IN VIETNAM - PRELIMINARY FINDINGS

Visual images in textbooks can stimulate students’ interest and encourage them to learn language in more critical and creative ways (Canning-Wilson, 1999). As a natural requirement of the field, medical textbooks contain a variety of visual images, but they seem to have received little attention from textbook developers in Vietnam possibly due to both a lack of awareness and research. Therefore, the study attempts to explore the visual elements in a medical textbook in use in the country, analyze their functions as well as their appropriateness for pedagogical purposes. This paper presents preliminary findings from the qualitative analysis of the yielded results couched with Kress and Van Leeuwen’s (2006) Visual Grammar framework to primarily identify three metafunctions of visual elements in the medical textbook. The study is believed to give pedagogical benefits to both teachers and students, and suggest practical improvement to critical aspects of the visual images, which is ultimately useful to developers of medical textbooks in particular, and other textbooks in general.

Australian property market explorer – a front-end native tool for visualizing property sales data

Online interactive 3D visualizations can enhance understanding of the built environment, including non-physical aspects such as property value and price. With access to large-scale open datasets, a key challenge in building such interactive 3D mapping interfaces is the efficiency of visualization methods. Modern GPU-accelerated front-end mapping libraries present a solution to this problem. This research proposes a ‘front-end’ native approach to create a web-based visual analytics framework for large scale spatiotemporal data. The Australian Property Market Explorer exemplifies this framework by visualizing a large dataset of property sales in New South Wales, Australia. This tool allows users to explore and analyse property market patterns at various geographic scales. The proposed framework offers advantages over traditional methods such as scalability and reliability while promising use-cases such as aiding planners in infrastructure and transportation decisions and assisting property valuation professionals in analysing property price trends.

Innovative cloud quantification: deep learning classification and finite-sector clustering for ground-based all-sky imaging

Abstract. Accurate cloud quantification is essential in climate change research. In this work, we construct an automated computer vision framework by synergistically incorporating deep neural networks and finite-sector clustering to achieve robust whole-sky image-based cloud classification, adaptive segmentation and recognition under intricate illumination dynamics. A bespoke YOLOv8 (You Only Look Once 8) architecture attains over 95 % categorical precision across four archetypal cloud varieties curated from extensive annual observations (2020) at a Tibetan highland station. Tailor-made segmentation strategies adapted to distinct cloud configurations, allied with illumination-invariant image enhancement algorithms, effectively eliminate solar interference and substantially boost quantitative performance even in illumination-adverse analysis scenarios. Compared with the traditional threshold analysis method, the cloud quantification accuracy calculated within the framework of this paper is significantly improved. Collectively, the methodological innovations provide an advanced solution to markedly escalate cloud quantification precision levels imperative for climate change research while offering a paradigm for cloud analytics transferable to various meteorological stations.

VR map construction for orchard robot teleoperation based on dual-source positioning and sparse point cloud segmentation

The unstructured nature of orchard environments presents significant challenges for autonomous navigation of orchard robots. Teleoperation, combined with virtual reality (VR), has emerged as a promising solution to overcome the limitations of on-board autonomous navigation capabilities in general-purpose robots. However, constructing accurate and semantically meaningful VR maps for orchard environments remains a challenge. This study proposes a novel VR map construction framework for orchard robot teleoperation visualization. It comprises two key components: dual-source combined positioning and semantic segmentation based on sparse point clouds. First, a novel sliding window-based data fusion approach is proposed to address the positioning challenges in semi-obstructed orchard environments. This approach combines GNSS data and laser matching to achieve robust and accurate positioning for orchard robots. Second, the framework capitalizes on the installation characteristics of mobile robot radars to achieve real-time classification of sparse point clouds. This allows for the removal of unstable elements, such as leaves, which would hinder effective teleoperation. By integrating these components, a VR map suitable for VR teleoperation of orchard robots is achieved. To validate the effectiveness of the proposed method, a VR teleoperation platform was constructed. The experimental results demonstrate that the method successfully fulfills the fundamental requirements for map visualization during VR remote operation of orchard robots. Furthermore, this study provides a valuable reference for the application of digital twins in agricultural robotics.

Visual analysis of fitness landscapes in architectural design optimization

In architectural design optimization, fitness landscapes are used to visualize design space parameters in relation to one or more objective functions for which they are being optimized. In our design study with domain experts, we developed a visual analytics framework for exploring and analyzing fitness landscapes spanning data, projection, and visualization layers. Within the data layer, we employ two surrogate models and three sampling strategies to efficiently generate a wide array of landscapes. On the projection layer, we use star coordinates and UMAP as two alternative methods for obtaining a 2D embedding of the design space. Our interactive user interface can visualize fitness landscapes as a continuous density map or a discrete glyph-based map. We investigate the influence of surrogate models and sampling strategies on the resulting fitness landscapes in a parameter study. Additionally, we present findings from a user study (N = 12), revealing how experts’ preferences regarding projection methods and visual representations may be influenced by their level of expertise, characteristics of the techniques, and the specific task at hand. Furthermore, we demonstrate the usability and usefulness of our framework by a case study from the architecture domain, involving one domain expert.

Copper-based fluorinated MOFs for activatable multicolor 19F MRI and regulated cell death-promoting therapy of tumor

Fluorine-19 magnetic resonance imaging (MRI) is emerging as a promising multiplexed molecular imaging technique that is complement to traditional anatomical 1H MRI. Unfortunately, the further development and popularization of 19F MRI, especially multicolor 19F MRI, is substantially restricted by the limited types of imaging probes. In this study, we constructed a stimuli-responsive nanoagent (Cu-DFL) based on copper-based fluorinated metal-organic frameworks (MOFs) for tumor visualization via dual-color 19F MRI and cancer therapy by promoting regulated cell death (RCD). The degradation of this nanoagent in the presence excess glutathione (GSH) in tumor microenvironment (TME) results in the release of two fluorinated organic ligands with distinct 19F chemical shifts, which substantially strengthens 19F signals for dual-color 19F MRI of tumor. Meanwhile, Cu2+ ions are discharged and soon reduced to Cu+ ions by GSH, which causes significant cuproptosis by inducing the aggregation of dihydrolipoyl transacetylase (DLAT). The presence of Cu+ ions also facilitates the intracellular Fenton reaction, which leads to considerable oxidative stress and subsequent mitochondria-mediated apoptosis. The downregulation of GPX4 due to the depletion of GSH and the enhanced oxidative stress contribute to excess lipid peroxidation (LPO), resulting in significant ferroptosis. These different types of RCD triggered by Cu-DFL contribute to the annihilation of cancer cells and the ablation of tumor. These results illustrated the promising potential of our nanoagent for accurate tumor visualization via multi-color 19F MRI and effective cancer therapy by stimulating multiple types of RCD, further demonstrating the potentiality of fluorinated MOFs for biomedical applications.

Remote sensing and machine learning approach for zoning of wastewater drainage system

This study aimed to optimize drainage in Tiruchirappalli city using Geographic Information Systems (GIS), machine learning, and remote sensing. The integration of these methods involved multiple steps: remote sensing data was used to gather up-to-date information on land use and land cover (LULC), GIS provided a spatial framework for data integration and visualization, and machine learning, specifically a Random Forest (RF) model, was utilized for detailed LULC classification. Hydrological analysis using a Digital Elevation Model (DEM) helped delineate the watershed and drainage network, determining suitable zones for wastewater systems. Key factors such as ground slope, land use, and proximity to water bodies were considered. Buffer analysis was employed to transform and overlay these data layers. For the site suitability analysis, the Analytic Hierarchy Process (AHP) method was employed within the GIS to integrate various weighted factors, enabling a comprehensive assessment of potential locations for wastewater treatment facilities. Wastewater drainage zones were determined to be appropriate for zones 1, 8, 9, 11, 15, and 20, which correspond to areas of 40.40, 1.11, 1.00, 2.04, 1.69, and 2.46 Sq.km, respectively. This comprehensive approach enabled the identification of optimal wastewater drainage zones and suitable locations for treatment facilities, enhancing urban wastewater management.

ZH-ECochG Bode Plot: A Novel Approach to Visualize Electrocochleographic Data in Cochlear Implant Users.

Background: Various representations exist in the literature to visualize electrocochleography (ECochG) recordings along the basilar membrane (BM). This lack of generalization complicates comparisons within and between cochlear implant (CI) users, as well as between publications. This study synthesized the visual representations available in the literature via a systematic review and provides a novel approach to visualize ECochG data in CI users. Methods: A systematic review was conducted within PubMed and EMBASE to evaluate studies investigating ECochG and CI. Figures that visualized ECochG responses were selected and analyzed. A novel visualization of individual ECochG data, the ZH-ECochG Bode plot (ZH = Zurich), was devised, and the recordings from three CI recipients were used to demonstrate and assess the new framework. Results: Within the database search, 74 articles with a total of 115 figures met the inclusion criteria. Analysis revealed various types of representations using different axes; their advantages were incorporated into the novel visualization framework. The ZH-ECochG Bode plot visualizes the amplitude and phase of the ECochG recordings along the different tonotopic regions and angular insertion depths of the recording sites. The graph includes the pre- and postoperative audiograms to enable a comparison of ECochG responses with the audiometric profile, and allows different measurements to be shown in the same graph. Conclusions: The ZH-ECochG Bode plot provides a generalized visual representation of ECochG data, using well-defined axes. This will facilitate the investigation of the complex ECochG potentials generated along the BM and allows for better comparisons of ECochG recordings within and among CI users and publications. The scripts used to construct the ZH-ECochG Bode plot are provided by the authors.

BrainTACO: an explorable multi-scale multi-modal brain transcriptomic and connectivity data resource

Exploring the relationships between genes and brain circuitry can be accelerated by joint analysis of heterogeneous datasets from 3D imaging data, anatomical data, as well as brain networks at varying scales, resolutions, and modalities. Generating an integrated view, beyond the individual resources’ original purpose, requires the fusion of these data to a common space, and a visualization that bridges the gap across scales. However, despite ever expanding datasets, few platforms for integration and exploration of this heterogeneous data exist. To this end, we present the BrainTACO (Brain Transcriptomic And Connectivity Data) resource, a selection of heterogeneous, and multi-scale neurobiological data spatially mapped onto a common, hierarchical reference space, combined via a holistic data integration scheme. To access BrainTACO, we extended BrainTrawler, a web-based visual analytics framework for spatial neurobiological data, with comparative visualizations of multiple resources. This enables gene expression dissection of brain networks with, to the best of our knowledge, an unprecedented coverage and allows for the identification of potential genetic drivers of connectivity in both mice and humans that may contribute to the discovery of dysconnectivity phenotypes. Hence, BrainTACO reduces the need for time-consuming manual data aggregation often required for computational analyses in script-based toolboxes, and supports neuroscientists by directly leveraging the data instead of preparing it.

Real-time spatial-temporal mapping and visualization of thermal comfort and HVAC control by integrating immersive augmented reality technologies and IoT-enabled wireless sensor networks: Towards immersive human-building interactions

Existing methods to assess and visualize thermal comfort mainly focused on either individual users (e.g., through wearables) or for specific locations where sensors are installed, rather than continuously for the entire indoor space. Also, most of the existing thermal comfort visualization methods and HVAC system control approaches rely on phones and mobile devices and lack a fully immersive/interactive experience and human-building interaction. Thus, this paper develops an immersive human-building interaction framework for real-time spatial-temporal mapping and visualization of indoor thermal comfort by integrating immersive augmented reality (AR) technologies and IoT-enabled smart wireless sensor networks (WSNs) (i.e., temperature and relative humidity data) to provide thermal comfort assessment for the entire space (rather than at localized/individual locations/points) and to enable users to interact with and adjust the HVAC systems from a distance using fully immersive and realistic environments. First, a smart WSN comprised of four IoT-enabled sensing stations was developed to collect real-time thermal comfort-related information (i.e., temperature and relative humidity). Second, an AR environment was created using the Unity engine, where virtual objects were overlayed into, and aligned with, the real-world environment. Third, the real-time data from the IoT-enabled WSN was integrated into the AR environment using various programming scripts that add multiple functionalities to the users, including controlling the HVAC system and visualizing the thermal comfort along the entire indoor space or room of interest through a spatial-temporal map. Fourth, the developed framework was validated and evaluated by 118 participants using a survey questionnaire. The findings showed that the graphical satisfaction achieved an overall average score of 4.75 out of 5, the user's sense of spatial presence was rated at an average score of 4.68 out of 5, the involvement aspects of the developed approach received an overall average score of 4.80 out of 5, and experienced realism was rated at an average score of 4.58 out of 5. Ultimately, this paper contributes to enhanced and intelligent human-building interaction and visualization through the use of emerging technologies by allowing the building users to control and interact with the HVAC systems using AR capabilities to improve the health and well-being of the building occupants.

Visual Ship Image Synthesis and Classification Framework Based on Attention-DCGAN

To improving ship image generation and classification tasks, a deep convolution generative adversarial network based on attention mechanism (ADCGAN) model was constructed. The rectified linear unit (ReLU) activation function was adopted, and three Deconv layers and Conv layers were added to both the generator and discriminator. Subsequently, an attention mechanism was added to the generator, while spectral normalization (SN) was added to the discriminator. Mean squared error (MSE) was used as loss function to stabilize the training process. Furthermore, ship classification tasks were performed using the generated ship images by end-to-end training of the classification network, enabling ship data augmentation and co-learning with other tasks. Experimental results on the Ship700 and Seaship7000 datasets demonstrate that the ADCGAN model can generate clear and robust ship images, with PSNR, LIPIPS, MS-SSIM values of 20.279 and 27.523, 0.596 and 0.096, 0.781 and 0.947, respectively. The effectiveness of the proposed method in ship image classification tasks was also verified, providing a data foundation for other collaborative tasks.

Topological Characterization and Uncertainty Visualization of Atmospheric Rivers

AbstractAtmospheric rivers (ARs) are long, narrow regions of water vapor in the Earth's atmosphere that transport heat and moisture from the tropics to the mid‐latitudes. ARs are often associated with extreme weather events in North America and contribute significantly to water supply and flood risk. However, characterizing ARs has been a major challenge due to the lack of a universal definition and their structural variations. Existing AR detection tools (ARDTs) produce distinct AR boundaries for the same event, making the risk assessment of ARs a difficult task. Understanding these uncertainties is crucial to improving the predictability of AR impacts, including their landfall areas and associated precipitation, which could cause catastrophic flooding and landslides over the coastal regions. In this work, we develop an uncertainty visualization framework that captures boundary and interior uncertainties, i.e., structural variations, of an ensemble of ARs that arise from a set of ARDTs. We first provide a statistical overview of the AR boundaries using the contour boxplots of Whitaker et al. that highlight the structural variations of AR boundaries based on their nesting relationships. We then introduce the topological skeletons of ARs based on Morse complexes that characterize the interior variation of an ensemble of ARs. We propose an uncertainty visualization of these topological skeletons, inspired by MetroSets of Jacobson et al. that emphasizes the agreements and disagreements across the ensemble members. Through case studies and expert feedback, we demonstrate that the two approaches complement each other, and together they could facilitate an effective comparative analysis process and provide a more confident outlook on an AR's shape, area, and onshore impact.

From quantitative metrics to clinical success: assessing the utility of deep learning for tumor segmentation in breast surgery.

Preventing positive margins is essential for ensuring favorable patient outcomes following breast-conserving surgery (BCS). Deep learning has the potential to enable this by automatically contouring the tumor and guiding resection in real time. However, evaluation of such models with respect to pathology outcomes is necessary for their successful translation into clinical practice. Sixteen deep learning models based on established architectures in the literature are trained on 7318 ultrasound images from 33 patients. Models are ranked by an expert based on their contours generated from images in our test set. Generated contours from each model are also analyzed using recorded cautery trajectories of five navigated BCS cases to predict margin status. Predicted margins are compared with pathology reports. The best-performing model using both quantitative evaluation and our visual ranking framework achieved a mean Dice score of 0.959. Quantitative metrics are positively associated with expert visual rankings. However, the predictive value of generated contours was limited with a sensitivity of 0.750 and a specificity of 0.433 when tested against pathology reports. We present a clinical evaluation of deep learning models trained for intraoperative tumor segmentation in breast-conserving surgery. We demonstrate that automatic contouring is limited in predicting pathology margins despite achieving high performance on quantitative metrics.