Visual Framework Research Articles

Automatic Screw Detection and Tool Recommendation System for Robotic Disassembly

Abstract Disassembly is an essential process for the recovery of end-of-life (EOL) electronics in remanufacturing sites. Nevertheless, the process remains labor-intensive due to EOL electronics’ high degree of uncertainty and complexity. The robotic technology can assist in improving disassembly efficiency; however, the characteristics of EOL electronics pose difficulties for robot operation, such as removing small components. For such tasks, detecting small objects is critical for robotic disassembly systems. Screws are widely used as fasteners in ordinary electronic products while having small sizes and varying shapes in a scene. To enable robotic systems to disassemble screws, the location information and the required tools need to be predicted. This paper proposes a computer vision framework for detecting screws and recommending related tools for disassembly. First, a YOLOv4 algorithm is used to detect screw targets in EOL electronic devices and a screw image extraction mechanism is executed based on the position coordinates predicted by YOLOv4. Second, after obtaining the screw images, the EfficientNetv2 algorithm is applied for screw shape classification. In addition to proposing a framework for automatic small-object detection, we explore how to modify the object detection algorithm to improve its performance and discuss the sensitivity of tool recommendations to the detection predictions. A case study of three different types of screws in EOL electronics is used to evaluate the performance of the proposed framework.

A novel visual guidance framework for robotic welding based on binocular cooperation

• An initial welding position guidance framework based on uncalibrated visual servoing and binocular cooperation is studied. The proposed method combines the advantages of passive and active vision sensors, and improves the flexibility of the vision system. • The global camera first controls the robot to approach the workpiece through the proposed EKF based uncalibrated visual servoing. Then, the derived cooperative motion equation will lead the robot towards the IWP. • 2 The features extraction algorithms for realizing UVS are studied. The method to estimate the desired position of welding torch is discussed. And a compression strategy for Yolo-v4 is designed to realize real-time welding torch detection, which can be an optional solution for target detection tasks on computational-resource-limited devices. • The simulation experiments demonstrate the feasibility and stability of the proposed guidance framework. The comparison experiments indicate that the framework can increase the flexibility of the vision system while maintaining satisfactory accuracy. Visual sensor based welding guidance technology has played an essential part in intelligentized robotic welding. However, scanning trajectory dependence of the laser vision sensor (LVS) and low accuracy of the passive vision sensor limit the application of single visual sensor based guidance methods in practical welding guidance. Aiming at this problem, a global camera is introduced to LVS system, and a novel initial welding point (IWP) guidance framework based on binocular cooperation is put forward. The robot is first controlled by the global camera to approach the workpiece through the proposed extended Kalman filter based uncalibrated visual servoing (UVS). Then, the derived cooperative equation of the LVS and the global camera will lead the robot toward the IWP. Subsequently, the feature extraction algorithms for UVS are designed. A compression strategy for Yolo-v4 is present to realize real-time welding torch detection in small computing platforms, and the estimation method of desired features is analyzed. Finally, the feasibility of the proposed algorithm is verified through experiments. The proposed framework combines the advantages of rich information for global camera and high precision for LVS, and could be a feasible solution to realize fully autonomous welding guidance.

TCMFVis: A visual analytics system toward bridging together traditional Chinese medicine and modern medicine

Although traditional Chinese medicine (TCM) and modern medicine (MM) have considerably different treatment philosophies, they both make important contributions to human health care. TCM physicians usually treat diseases using TCM formula (TCMF), which is a combination of specific herbs, based on the holistic philosophy of TCM, whereas MM physicians treat diseases using chemical drugs that interact with specific biological molecules. The difference between the holistic view of TCM and the atomistic view of MM hinders their combination. Tools that are able to bridge together TCM and MM are essential for promoting the combination of these disciplines. In this paper, we present TCMFVis, a visual analytics system that would help domain experts explore the potential use of TCMFs in MM at the molecular level. TCMFVis deals with two significant challenges, namely, (i) intuitively obtaining valuable insights from heterogeneous data involved in TCMFs and (ii) efficiently identifying the common features among a cluster of TCMFs. In this study, a four-level (herb-ingredient-target-disease) visual analytics framework was designed to facilitate the analysis of heterogeneous data in a proper workflow. Several set visualization techniques were first introduced into the system to facilitate the identification of common features among TCMFs. Case studies on two groups of TCMFs clustered by function were conducted by domain experts to evaluate TCMFVis. The results of these case studies demonstrate the usability and scalability of the system.

SOCRAT: a Dynamic Web Toolbox for Interactive Data Processing, Analysis and Visualization.

Many systems for exploratory and visual data analytics require platform-dependent software installation, coding skills, and analytical expertise. The rapid advances in data-acquisition, web-based information, and communication and computation technologies promoted the explosive growth of online services and tools implementing novel solutions for interactive data exploration and visualization. However, web-based solutions for visual analytics remain scattered and relatively problem-specific. This leads to per-case re-implementations of common components, system architectures, and user interfaces, rather than focusing on innovation and building sophisticated applications for visual analytics. In this paper, we present the Statistics Online Computational Resource Analytical Toolbox (SOCRAT), a dynamic, flexible, and extensible web-based visual analytics framework. The SOCRAT platform is designed and implemented using multi-level modularity and declarative specifications. This enables easy integration of a number of components for data management, analysis, and visualization. SOCRAT benefits from the diverse landscape of existing in-browser solutions by combining them with flexible template modules into a unique, powerful, and feature-rich visual analytics toolbox. The platform integrates a number of independently developed tools for data import, display, storage, interactive visualization, statistical analysis, and machine learning. Various use cases demonstrate the unique features of SOCRAT for visual and statistical analysis of heterogeneous types of data.

Multi-field interpretation of internal short circuit and thermal runaway behavior for lithium-ion batteries under mechanical abuse

Mechanical abuse-induced hazardous of lithium-ion batteries (LIBs), in which internal short circuits, thermal runaway, and mechanical failure can coincide and interact with each other, has become a critical issue that hinders the further application of LIBs. This study clarifies the bridging process from short circuit to thermal runaway for LIBs in complex mechanical abuse environment using a three-dimensional two-way coupled mechanical–electrochemical–thermal model. The developed model is verified by the two most common engineering scenarios including mechanical crushing and nail penetration. The complex structural damage–induced internal short circuit and thermal runaway behavior of LIBs are discussed. Moreover, the triggering mechanisms from internal short circuit to thermal runaway and the detailed exothermic reaction are revealed through model predictions. In particular, this study constructed a visual analysis framework of the coupled mechanical-electrochemical-thermal failure process for LIBs subjected to mechanical abuse. A model-based discussion of uneven lithium ions diffusion phenomenon is presented to interpret the electrochemical behavior of the battery caused by internal short circuit. The research results and the developed modeling method provide a robust tool for the mechanical abuse-safe design and evaluation of LIBs from a multi-disciplinary perspective.

Ship collision risk analysis: Modeling, visualization and prediction

Ship collisions are the primary threat to traffic safety in the sea, which can seriously threaten human lives, the environment and material assets. Therefore, the detection and analyze of ship collision risks have important theoretical significance and application value. To improve maritime safety and efficiency, we propose a modeling, visualization and prediction framework to analyze ship collision risk. In particular, to fully consider the maneuverability of the ship, we introduce the quaternion ship domain (QSD) into the vessel conflict ranking operator (VCRO). In addition, to further analyze and better understand collision risk, the kernel density estimation (KDE) model is employed to visualize the ship collision risk. The ship collision risk usually contains underlying patterns and laws. Thus, we proposed a convolutional long short-term memory network (ConvLSTM) model, which can extract spatial–temporal features and predict spatial–temporal risk. Finally, to verify the reliability and robustness of the framework, we conducted extensive experiments on the automatic identification system (AIS) data of Chengshantou water. The results show that the framework demonstrates superiority in risk calculation, visualization and prediction. Theoretically, the framework proposed in this paper can serve maritime intelligent transportation system well.

Active Pattern Classification for Automatic Visual Exploration of Multi-Dimensional Data

The practice of applying a classifier (called a pattern classifier and abbreviated as PC below) in a visual analysis system to identify patterns from interactively generated visualizations is gradually emerging. Demonstrated cases in existing works focus on ideal scenarios where the analyst can determine all the pattern types in advance without adjusting the classifier settings during the exploration process. However, in most real-world scenarios, analysts know nothing about data patterns before exploring the dataset and inevitably find novel patterns during the exploration. This difference makes the traditional classifier training and application mode less suitable. Analysts have to artificially determine whether each generated visualization contains new data patterns to adjust the classifier setting, thus affecting the automation of the data exploration. This paper proposes a novel PC-based data exploration approach. The core of the approach is an active-learning indicator for automatically identifying visualizations involving new pattern classes. Analysts thus can apply PCs to explore data while dynamically adjusting the PCs using these visualizations. We further propose a PC-based visualization framework that takes full advantage of the PC in terms of efficiency by allowing analysts to explore an exploring space, rather than a single visualization at a time. The results of the quantitative experiment and the performance of participants in the user study demonstrate the effectiveness and usability of the method.

Influencer marketing effectiveness: the mechanisms that matter

Purpose The continued evolution of influencer marketing has created a need to better understand influencer marketing effectiveness. With brands increasingly partnering with influencers, research is yet to provide an integrated perspective examining the critical role of both parties. This study aims to draw on the source credibility model and signaling theory to explain the mechanisms that matter in influencer marketing effectiveness. Design/methodology/approach The proposed model of influencer marketing effectiveness is analyzed using partial least squares with data from 281 followers of social media influencers. Findings The authors establish influencer characteristics of popularity and attractiveness as heuristic cues that inform judgments of influencer efficacy. Further, category involvement and altruistic motives for collaboration are shown to moderate followers’ reliance on these heuristic cues. Then, a sequential mediating effect demonstrates the critical roles of the influencer and partner brand in three desired outcomes: enhanced perception of brand authenticity, enhanced brand engagement and positive attitudes toward influencer posts. Research limitations/implications Future research should consider other heuristic cues that could inform influencer efficacy judgments and switch the focus toward the partner brand’s impact on such judgments. Practical implications A step-by-step visual framework is presented to help marketers and influencers translate these findings into key responsibilities for developing more effective and collaborative partnerships. Originality/value Besides presenting an integrated perspective, signaling theory provides an original lens for explaining influencer marketing effectiveness, addressing the need to expand the theoretical boundaries of influencer marketing research.

Developments in 3D Visualisation of the Rail Tunnel Subsurface for Inspection and Monitoring

Railway Tunnel SubSurface Inspection (RTSSI) is essential for targeted structural maintenance. ‘Effective’ detection, localisation and characterisation of fully concealed features (i.e., assets, defects) is the primary challenge faced by RTSSI engineers, particularly in historic masonry tunnels. Clear conveyance and communication of gathered information to end-users poses the less frequently considered secondary challenge. The purpose of this review is to establish the current state of the art in RTSSI data acquisition and information conveyance schemes, in turn formalising exactly what constitutes an ‘effective’ RTSSI visualisation framework. From this knowledge gaps, trends in leading RTSSI research and opportunities for future development are explored. Literary analysis of over 300 resources (identified using the 360-degree search method) informs data acquisition system operation principles, common strengths and limitations, alongside leading studies and commercial tools. Similar rigor is adopted to appraise leading information conveyance schemes. This provides a comprehensive whilst critical review of present research and future development opportunities within the field. This review highlights common shortcomings shared by multiple methods for RTSSI, which are used to formulate robust criteria for a contextually ‘effective’ visualisation framework. Although no current process is deemed fully effective; a feasible hybridised framework capable of meeting all stipulated criteria is proposed based on identified future research avenues. Scope for novel analysis of helical point cloud subsurface datasets obtained by a new rotating ground penetrating radar antenna is of notable interest.

Visual Aesthetic Quality Assessment of Urban Forests: A Conceptual Framework

Visual aesthetic quality is the visual pleasure level that attracts people and makes them prefer certain areas. Visual aesthetic quality is valued and considered for urban forests but remains challenging. This could be due to a lack of understanding of visual aesthetic quality assessment variables based on visual aesthetic theories. This study supports an integrated conceptual framework based on the result of a systematic literature review study to describe and measure aesthetics that incorporates objective and subjective factors through urban forest visual character and urban forest visual quality. The results include defining and understanding a description of visual aesthetic factors and variables as well as a thorough explanation of visual aesthetic theories to comprehend how to assess the visual aesthetic quality of urban forests. This study agrees with and supports the visual aesthetic theoretical framework, and we believe that due to our shared evolutionary history, humans have a standard set of urban forest visual aesthetic features with preferences that change according to cultural and personal variances. Furthermore, this research provides a foundation of visual aesthetic variables of urban forests that will assist urban forest researchers, urban forest managers, and decision-makers in managing and protecting the visual aesthetics of urban forests.

A monocular visual SLAM system augmented by lightweight deep local feature extractor using in-house and low-cost LIDAR-camera integrated device

ABSTRACT Simultaneous Localization and Mapping (SLAM) has been widely used in emergency response, self-driving and city-scale 3D mapping and navigation. Recent deep-learning based feature point extractors have demonstrated superior performance in dealing with the complex environmental challenges (e.g. extreme lighting) while the traditional extractors are struggling. In this paper, we have successfully improved the robustness and accuracy of a monocular visual SLAM system under various complex scenes by adding a deep learning based visual localization thread as an augmentation to the visual SLAM framework. In this thread, our feature extractor with an efficient lightweight deep neural network is used for absolute pose and scale estimation in real time using the highly accurate georeferenced prior map database at 20cm geometric accuracy created by our in-house and low-cost LiDAR and camera integrated device. The closed-loop error provided by our SLAM system with and without this enhancement is 1.03m and 18.28m respectively. The scale estimation of the monocular visual SLAM is also significantly improved (0.01 versus 0.98). In addition, a novel camera-LiDAR calibration workflow is also provided for large-scale 3D mapping. This paper demonstrates the application and research potential of deep-learning based vision SLAM with image and LiDAR sensors.

Anomaly Matters: An Anomaly-Oriented Model for Medical Visual Question Answering.

Medical images contain various abnormal regions, most of which are closely related to the lesions or diseases. The abnormality or lesion is one of the major concerns during clinical practice and therefore becomes the key in answering questions about medical images. However, the recent efforts still focus on constructing a generic Visual Question Answering framework for medical-domain tasks, which is not adequate for practical medical requirements and applications. In this paper, we present two novel medical-specific modules named multiplication anomaly sensitive module and residual anomaly sensitive module to utilize weakly supervised anomaly localization information in medical Visual Question Answering. Firstly, the proposed multiplication anomaly sensitive module designed for anomaly-related questions can mask the feature of the whole image according to the anomaly location map. Secondly, the residual anomaly sensitive module could learn a flexible anomaly feature while preserving the information of the original questioned image, which is more helpful in answering anomaly-unrelated questions. Thirdly, the transformer decoder and multi-task learning strategy are combined to further enhance the question-reasoning ability and the model generalization performance. Finally, qualitative and quantitative experiments on a variety of medical datasets exhibit the superiority of the proposed approaches compared to the state-of-the-art methods.

Investigations on visualization and interaction of ship structure multidisciplinary finite element analysis data for virtual environment

Although the extraction and sharing of important data in the finite element model (FEM) and analysis data is particularly significant in the ship structure collaborative design process, the efficiency of sharing finite element analysis (FEA) data using a standardized neutral format is still relatively low, which is attributed to the format that has to accommodate various data types generated by different commercial computer aided engineering (CAE) software and multidisciplinary analysis applications. Aiming to solve the problem of efficient visualization and sharing of heterogeneous FEA data in ship structure collaborative design and review, we proposed a new FEA data visualization framework for virtual environments (VE) to realize multidisciplinary data sharing in the collaborative design process, where only essential finite element (FE) information is stored to reduce the weight of the model and analysis data. In addition, the visualizing results can be updated in response to the modification of the model geometry and simulation variables. Furthermore, we implemented a general system based on our data framework. To maximize the efficiency of managing multidisciplinary data (statics, dynamics, ultimate strength), the system adopts a client/server (C/S) architecture and hierarchical management of data structures. The virtual visualization of these complex FEA data effectively enhances the efficiency of scientific communication, model evaluation and validation, and interdisciplinary discussion of research findings. Eventually, examples from commercial CAE software are provided to demonstrate the effectiveness and versatility of the system.

Disentangling classification and regression in Siamese‐based network for visual tracking

SummarySiamese‐based trackers have made great progress in visual tracking community, however, the shared structure of network between classification and regression tasks limits the ability of the trackers to obtain more robust classification prediction and more accurate regression prediction. In this paper, we propose an effective visual tracking framework (named Siamese Disentangled Tracking‐Head, SiamDTH), which disentangles classification and regression in Siamese‐based network for visual tracking from two aspects: feature decoupling and differentiated tracking‐head. First of all, we gather the features of receptive fields with different scales and ratios, and decouple the correlation features through two different styles of feature fusion mode for classification and regression respectively. Moreover, we design the differentiated tracking‐head structure in the sibling head for discriminately handling the parallel classification and regression tasks on visual tracking. Extensive experiments on visual tracking benchmarks including VOT2018, VOT2019 and OTB100 demonstrate that our proposed SiamDTH achieves state‐of‐the‐art performance with a considerable real‐time speed. Our source code is available at:https://github.com/xl0312/SiamDTH.

Visual comfort generative design framework based on parametric network in underground space

AbstractWith the growing demand for a high‐quality life, visual comfort (VC) is becoming increasingly important for improving the quality of underground spaces. The underground space landscape features can be defined by the spatial and material parameters of the components. This study proposes a novel parametric generative network (StepGN) for the 3D generative design of VC. It combines parametric modeling, VC evaluation, and a novel reinforcement learning (RL) model called encoded soft actor critic (ESAC) and simplifies the optimization of complex VC scenes into a parameter‐generating optimization process. Among them, parametric modeling is used to generate a 3D underground space scene with components and material properties through parameterization, and the optimization of parameters depends on the evaluation and RL model. The evaluation model provides reward value, and a new ESAC algorithm is developed. It combines the soft actor‐critic (SAC) algorithm with the encoder process and by setting the reward with a confidence threshold. In addition, the Swin cosine distance (SCD) is used to measure the diversity of the generated scenes. A comparison of the policy types and range conversion methods proves that the stochastic policy and Sigmoid function are more suitable for the generative design of VC. By comparing StepGN with a generative adversarial network‐based generative network (VCGN) and other RL processes, it shows that StepGN can generate discrete distributions of the VC levels and can realize a high‐comfort level scene, and the training speed and stability are considerably improved. Finally, StepGN is applied for the optimization of the Wujiaochang subway station scene in Shanghai, and it is proved that the VC of the generated results can provide a high comfort level.

The Impact of Visual Stimuli and Properties on Restorative Effect and Human Stress: A Literature Review

Restorative environments are known as places where human stress can be decreased through restoration of depleted psychological resources. Since the efficiency of natural environments in restoration is supported by the literature, designing a restorative built environment can be obtained by integrating natural objects into built environments. However, various factors may lead to the failure of design intentions in real restorative environments such as visual stimuli and their properties. In addition, previous literature has widely used images and immersive virtual environments (IVEs) to deliver restorative environments in the design stage, and the impact of the delivery methods on visual properties needs to be considered. Therefore, the key to this study is to investigate the impact of visual stimuli and their properties on restoration along with the type of delivery method. To achieve this objective, the authors performed a literature review in two main steps. During the first step, the authors reviewed restoration and visual studies separately to understand the restoration and visual processes. Then, the second step of the review was conducted based on the results from the first step to review studies investigating the impact of different visual stimuli and properties on restoration. The literature review was conducted by combining scientometric analysis with a systematic review. In total, 1608 publications were retrieved from the Web of Science for scientometric analysis after applying search criteria. Then, the authors explored cluster reports generated by scientometric analysis to find publications for a systematic review based on inclusion and exclusion criteria. According to the results from a systematic review of 65 publications, the authors developed a restoration pathway and a visual processing framework for the first step, and a framework of visual stimuli, visual properties, and restoration for the second step.

Monocular Camera Viewpoint-Invariant Vehicular Traffic Segmentation and Classification Utilizing Small Datasets.

The work presented here develops a computer vision framework that is view angle independent for vehicle segmentation and classification from roadway traffic systems installed by the Virginia Department of Transportation (VDOT). An automated technique for extracting a region of interest is discussed to speed up the processing. The VDOT traffic videos are analyzed for vehicle segmentation using an improved robust low-rank matrix decomposition technique. It presents a new and effective thresholding method that improves segmentation accuracy and simultaneously speeds up the segmentation processing. Size and shape physical descriptors from morphological properties and textural features from the Histogram of Oriented Gradients (HOG) are extracted from the segmented traffic. Furthermore, a multi-class support vector machine classifier is employed to categorize different traffic vehicle types, including passenger cars, passenger trucks, motorcycles, buses, and small and large utility trucks. It handles multiple vehicle detections through an iterative k-means clustering over-segmentation process. The proposed algorithm reduced the processed data by an average of 40%. Compared to recent techniques, it showed an average improvement of 15% in segmentation accuracy, and it is 55% faster than the compared segmentation techniques on average. Moreover, a comparative analysis of 23 different deep learning architectures is presented. The resulting algorithm outperformed the compared deep learning algorithms for the quality of vehicle classification accuracy. Furthermore, the timing analysis showed that it could operate in real-time scenarios.

Mu2e event visualization development using TEve and Eve-7

The Mu2e experiment at Fermilab will search for the Charged Lepton Flavor Violating (CLFV) neutrinoless μ−N→e−N conversion in the field of an aluminum nucleus. A custom offline event display has been developed for Mu2e using TEve, a ROOT based 3-D event visualization framework. Event displays are crucial for monitoring and debugging during live data taking as well as for public outreach.

Computer vision framework for crack detection of civil infrastructure—A review

Civil infrastructure (e.g., buildings, roads, underground tunnels) could lose its expected physical and functional conditions after years of operation. Timely and accurate inspection and assessment of such infrastructures are essential to ensure safety and serviceability, e.g., by preventing unsafe working conditions and hazards. Cracks, which are one of the most common distress, can indicate severe structural integrity issues that threaten the safety of the structure and people in the environment. As such, accurate, fast, and automatic detection of cracks on structure surfaces is a major issue for a variety of civil engineering applications. Due to advances in hardware data acquisition systems, significant progress has been made in the automatic detection and quantification of cracks in recent decades. This paper provides a comprehensive review of the research progress and prospects in computer vision frameworks for crack detection of civil infrastructures from multiple materials, including asphalt, concrete, and metal-like materials. The review encompasses major components of typical frameworks, i.e., data acquisition techniques, publicly available datasets, detection algorithms, and evaluation metrics. In particular, we provide a taxonomy of detection algorithms with a detailed discussion of the advantages, limitations, and application scenarios of the methods in each category, as well as the relationships between methods of different categories. We also discuss unsolved issues and key challenges in crack detection that could drive future research directions.

SIMCard: Toward better connected electronic health record visualization

SummaryRecently, several healthcare organizations use decision‐making systems based on electronic health record (EHR) data in order to guarantee patient's safety and improve the quality of healthcare. In essence, the evolutions of Internet of Things (IoT) technologies have been of great help for implementing an integrated and interoperable decision‐making system based on EHR and medical devices (MDs). Those IoT‐based systems allow Clinicians collecting real‐time health data and provide accurate patient's monitoring. Nevertheless, several studies have shown that it is hard to improve the quality of healthcare using the current EHR IoT‐based systems since they do not allow to easily express clinician needs. Interactive visualization tools have been proposed to improve the efficacy and utility of these EHR based systems. However, there is no framework that provides a visual summary of patient data to clinician for planning specific clinical tasks, subsequently evaluating clinician responses, visually exploring EHR data and MDs data, gaining insights, supporting dynamic coordination processes care, and forming and validating hypotheses and risks. This article addresses this problem and introduces SIMCard, an aggregation‐based connected EHR visualization framework for patient monitoring, interpreting and predicting with MDs. The proposed framework aims to synthesize patient's clinical data into a single aggregating model for both EHR and MD conforming to health standard and terminologies. It also allows to link the aggregating model to the relevant medical knowledge in order to provide a connected and dynamic care and preventive plan. Last but not least, it provides an aggregated visualization model capable of displaying graphically a patient's personal data from databases, healthcare devices and sensors to reduce cognitive barriers related to the complexity of medical information and interpretation of health data. To demonstrate the refinement and design of our system and to observe user's actual practice of visualizing and analyzing real‐world dataset, we evaluated our system and compare to existing ones.