Deep Vision Research Articles

DCV2I$\text{DCV}^2\text{I}$: Leveraging deep vision models to support geographers' visual interpretation in dune segmentation

AbstractVisual interpretation is extremely important in human geography as the primary technique for geographers to use photograph data in identifying, classifying, and quantifying geographic and topological objects or regions. However, it is also time‐consuming and requires overwhelming manual effort from professional geographers. This paper describes our interdisciplinary team's efforts in integrating computer vision models with geographers' visual image interpretation process to reduce their workload in interpreting images. Focusing on the dune segmentation task, we proposed an approach called featuring a deep dune segmentation model to identify dunes and label their ranges in an automated way. By developing a tool to connect our model with ArcGIS—one of the most popular workbenches for visual interpretation, geographers can further refine the automatically generated dune segmentation on images without learning any CV or deep learning techniques. Our approach thus realized a noninvasive change to geographers' visual interpretation routines, reducing their manual efforts while incurring minimal interruptions to their work routines and tools they are familiar with. Deployment with a leading Chinese geography research institution demonstrated the potential of in supporting geographers in researching and solving drylands desertification.

Accurate prediction of disease-risk factors from volumetric medical scans by a deep vision model pre-trained with 2D scans.

The application of machine learning to tasks involving volumetric biomedical imaging is constrained by the limited availability of annotated datasets of three-dimensional (3D) scans for model training. Here we report a deep-learning model pre-trained on 2D scans (for which annotated data are relatively abundant) that accurately predicts disease-risk factors from 3D medical-scan modalities. The model, which we named SLIViT (for 'slice integration by vision transformer'), preprocesses a given volumetric scan into 2D images, extracts their feature map and integrates it into a single prediction. We evaluated the model in eight different learning tasks, including classification and regression for six datasets involving four volumetric imaging modalities (computed tomography, magnetic resonance imaging, optical coherence tomography and ultrasound). SLIViT consistently outperformed domain-specific state-of-the-art models and was typically as accurate as clinical specialists who had spent considerable time manually annotating the analysed scans. Automating diagnosis tasks involving volumetric scans may save valuable clinician hours, reduce data acquisition costs and duration, and help expedite medical research and clinical applications.

ATTACK-COSM: attacking the camouflaged object segmentation model through digital world adversarial examples

The camouflaged object segmentation model (COSM) has recently gained substantial attention due to its remarkable ability to detect camouflaged objects. Nevertheless, deep vision models are widely acknowledged to be susceptible to adversarial examples, which can mislead models, causing them to make incorrect predictions through imperceptible perturbations. The vulnerability to adversarial attacks raises significant concerns when deploying COSM in security-sensitive applications. Consequently, it is crucial to determine whether the foundational vision model COSM is also susceptible to such attacks. To our knowledge, our work represents the first exploration of strategies for targeting COSM with adversarial examples in the digital world. With the primary objective of reversing the predictions for both masked objects and backgrounds, we explore the adversarial robustness of COSM in full white-box and black-box settings. In addition to the primary objective of reversing the predictions for masked objects and backgrounds, our investigation reveals the potential to generate any desired mask through adversarial attacks. The experimental results indicate that COSM demonstrates weak robustness, rendering it vulnerable to adversarial example attacks. In the realm of COS, the projected gradient descent (PGD) attack method exhibits superior attack capabilities compared to the fast gradient sign (FGSM) method in both white-box and black-box settings. These findings reduce the security risks in the application of COSM and pave the way for multiple applications of COSM.

DCV2I: A Practical Approach for Supporting Geographers’ Visual Interpretation in Dune Segmentation with Deep Vision Models

Visual interpretation is extremely important in human geography as the primary technique for geographers to use photograph data in identifying, classifying, and quantifying geographic and topological objects or regions. However, it is also time-consuming and requires overwhelming manual effort from professional geographers. This paper describes our interdisciplinary team's efforts in integrating computer vision models with geographers' visual image interpretation process to reduce their workload in interpreting images. Focusing on the dune segmentation task, we proposed an approach featuring a deep dune segmentation model to identify dunes and label their ranges in an automated way. By developing a tool to connect our model with ArcGIS, one of the most popular workbenches for visual interpretation, geographers can further refine the automatically-generated dune segmentation on images without learning any CV or deep learning techniques. Our approach thus realized a non-invasive change to geographers' visual interpretation routines, reducing their manual efforts while incurring minimal interruptions to their work routines and tools they are familiar with. Deployment with a leading Chinese geography research institution demonstrated the potential of our approach in supporting geographers in researching and solving drylands desertification.

Automated Natural Language Explanation of Deep Visual Neurons with Large Models (Student Abstract)

Interpreting deep neural networks through examining neurons offers distinct advantages when it comes to exploring the inner workings of Deep Neural Networks. Previous research has indicated that specific neurons within deep vision networks possess semantic meaning and play pivotal roles in model performance. Nonetheless, the current methods for generating neuron semantics heavily rely on human intervention, which hampers their scalability and applicability. To address this limitation, this paper proposes a novel post-hoc framework for generating semantic explanations of neurons with large foundation models, without requiring human intervention or prior knowledge. Experiments are conducted with both qualitative and quantitative analysis to verify the effectiveness of our proposed approach.

A Deep Vision Sensing-Based Smart Control Method for Apple-Picking Robots Under the Context of Agricultural E-Commerce

The growing development of the agricultural e-commerce industry is promoting the Digital transformation of agricultural production and logistics. Aiming at the problem of labor shortage in apple picking, this paper proposes a performance control method for robot apple picking based on a nonlinear wide convolutional neural network (NLWCNN) and multi-objective cooperative distance transformation algorithm (MOCDTA). This method utilizes deep learning algorithms and visual sensing technology to enable robots to accurately recognize apples and perform autonomous picking operations based on their maturity and position. We have designed a machine learning-based algorithm that enables robots to accurately recognize and distinguish apples of different maturity levels through training on a large amount of sample data. At the same time, we also combined the motion control algorithm of the robot to enable efficient picking operations based on the position and maturity information of the apples. The experimental results indicate that the NLWCNN method can significantly improve the efficiency and accuracy of apple picking, ensure the safety and traceability of agricultural products, increase consumer trust, and promote agricultural product transactions.

MGMT promoter methylation prediction based on multiparametric MRI via vision graph neural network.

Glioblastoma (GBM) is aggressive and malignant. The methylation status of the -methylguanine-DNA methyltransferase (MGMT) promoter in GBM tissue is considered an important biomarker for developing the most effective treatment plan. Although the standard method for assessing the MGMT promoter methylation status is via bisulfite modification and deoxyribonucleic acid (DNA) sequencing of biopsy or surgical specimens, a secondary automated method based on medical imaging may improve the efficiency and accuracy of those tests. We propose a deep vision graph neural network (ViG) using multiparametric magnetic resonance imaging (MRI) to predict the MGMT promoter methylation status noninvasively. Our model was compared to the RSNA radiogenomic classification winners. The dataset includes 583 usable patient cases. Combinations of MRI sequences were compared. Our multi-sequence fusion strategy was compared with those using single MR sequences. Our best model [Fluid Attenuated Inversion Recovery (FLAIR), T1-weighted pre-contrast (T1w), T2-weighted (T2)] outperformed the winning models with a test area under the curve (AUC) of 0.628, an accuracy of 0.632, a precision of 0.646, a recall of 0.677, a specificity of 0.581, and an F1 score of 0.661. Compared to the winning models with single MR sequences, our ViG utilizing fused-MRI showed a significant improvement statistically in AUC scores, which are FLAIR (), T1w (), T1wCE (), and T2 (). Our model is superior to challenge champions. A graph representation of the medical images enabled good handling of complexity and irregularity. Our work provides an automatic secondary check pipeline to ensure the correctness of MGMT methylation status prediction.

Distributed Split Learning for Map-Based Signal Strength Prediction Empowered by Deep Vision Transformer

Distributed Split Learning for Map-Based Signal Strength Prediction Empowered by Deep Vision Transformer

How Can Human Hierarchical and Active Visual Perceptual Behavior Influence Scenic Image Categorization? A Deep Vision Model for Architectural Scenery Toward Education

How Can Human Hierarchical and Active Visual Perceptual Behavior Influence Scenic Image Categorization? A Deep Vision Model for Architectural Scenery Toward Education

Changen2: Multi-Temporal Remote Sensing Generative Change Foundation Model.

Our understanding of the temporal dynamics of the Earth's surface has been significantly advanced by deep vision models, which often require a massive amount of labeled multi-temporal images for training. However, collecting, preprocessing, and annotating multi-temporal remote sensing images at scale is non-trivial since it is expensive and knowledge-intensive. In this paper, we present scalable multi-temporal change data generators based on generative models, which are cheap and automatic, alleviating these data problems. Our main idea is to simulate a stochastic change process over time. We describe the stochastic change process as a probabilistic graphical model, namely the generative probabilistic change model (GPCM), which factorizes the complex simulation problem into two more tractable sub-problems, i.e., condition-level change event simulation and image-level semantic change synthesis. To solve these two problems, we present Changen2, a GPCM implemented with a resolution-scalable diffusion transformer which can generate time series of remote sensing images and corresponding semantic and change labels from labeled and even unlabeled single-temporal images. Changen2 is a "generative change foundation model" that can be trained at scale via self-supervision, and is capable of producing change supervisory signals from unlabeled single-temporal images. Unlike existing "foundation models", our generative change foundation model synthesizes change data to train task-specific foundation models for change detection. The resulting model possesses inherent zero-shot change detection capabilities and excellent transferability. Comprehensive experiments suggest Changen2 has superior spatiotemporal scalability in data generation, e.g., Changen2 model trained on 256 2 pixel single-temporal images can yield time series of any length and resolutions of 1,024 2 pixels. Changen2 pre-trained models exhibit superior zero-shot performance (narrowing the performance gap to 3% on LEVIR-CD and approximately 10% on both S2Looking and SECOND, compared to fully supervised counterpart) and transferability across multiple types of change tasks, including ordinary and off-nadir building change, land-use/land-cover change, and disaster assessment. The model and datasets are available at https://github.com/Z-Zheng/pytorch-change-models.

Privacy Aware Feature Level Domain Adaptation for Training Deep Vision Models With Private Medical Stethoscope Dataset

Privacy Aware Feature Level Domain Adaptation for Training Deep Vision Models With Private Medical Stethoscope Dataset

Object Recognition and Grasping for Collaborative Robots Based on Vision.

This study introduces a parallel YOLO-GG deep learning network for collaborative robot target recognition and grasping to enhance the efficiency and precision of visual classification and grasping for collaborative robots. First, the paper outlines the target classification and detection task, the grasping system of the robotic arm, and the dataset preprocessing method. The real-time recognition and grasping network can identify a diverse spectrum of unidentified objects and determine the target type and appropriate capture box. Secondly, we propose a parallel YOLO-GG deep vision network based on YOLO and GG-CNN. Thirdly, the YOLOv3 network, pre-trained with the COCO dataset, identifies the object category and position, while the GG-CNN network, trained using the Cornell Grasping dataset, predicts the grasping pose and scale. This study presents the processes for generating a target's grasping frame and recognition type using GG-CNN and YOLO networks, respectively. This completes the investigation of parallel networks for target recognition and grasping in collaborative robots. Finally, the experimental results are evaluated on the self-constructed NEU-COCO dataset for target recognition and positional grasping. The speed of detection has improved by 14.1%, with an accuracy of 94%. This accuracy is 4.0% greater than that of YOLOv3. Experimental proof was obtained through a robot grasping actual objects.

A Deep Vision Learning-Based Intelligent Recognition Method for Dynamic Sports Gestures

Nowadays, artificial intelligence-assisted automatic technological analyses of sports matches have received a more general demand. Among them, the effective recognition of sports actions from videos or images acts as the most fundamental issue to be solved. As sports gestures are characterized by remarkable specialty and instantaneity, it is required to develop specific effective recognition algorithms for this purpose. As a result, this work takes the case of Taekwondo sports as an example, and introduces deep vision learning to develop a specific intelligent recognition method for dynamic sports gestures. After sampling the key frames from dynamic videos, vital joint points of limbs are first obtained as the basic bottom features. Then, gesture expertise of this domain is formed to construct a multiview feature extraction work flow to better analyze complex characteristics of each gesture. Finally, the extracted feature representation is input into a recurrent neural network structure to output the discriminative results. At last, experiments are conducted for assessment, whose results show that the proposal can reach an accuracy of 95.6% and can be well suitable for the investigated problem scenario.

Dual-Stream Fusion Network with ConvNeXtV2 for Pig Weight Estimation Using RGB-D Data in Aisles.

In the field of livestock management, noncontact pig weight estimation has advanced considerably with the integration of computer vision and sensor technologies. However, real-world agricultural settings present substantial challenges for these estimation techniques, including the impacts of variable lighting and the complexities of measuring pigs in constant motion. To address these issues, we have developed an innovative algorithm, the moving pig weight estimate algorithm based on deep vision (MPWEADV). This algorithm effectively utilizes RGB and depth images to accurately estimate the weight of pigs on the move. The MPWEADV employs the advanced ConvNeXtV2 network for robust feature extraction and integrates a cutting-edge feature fusion module. Supported by a confidence map estimator, this module effectively merges information from both RGB and depth modalities, enhancing the algorithm's accuracy in determining pig weight. To demonstrate its efficacy, the MPWEADV achieved a root-mean-square error (RMSE) of 4.082 kg and a mean absolute percentage error (MAPE) of 2.383% in our test set. Comparative analyses with models replicating the latest research show the potential of the MPWEADV in unconstrained pig weight estimation practices. Our approach enables real-time assessment of pig conditions, offering valuable data support for grading and adjusting breeding plans, and holds broad prospects for application.

Diffusion Convolution Neural Network-Based Multiview Gesture Recognition for Athletes in Dynamic Scenes

This paper focuses on deep vision sensing-assisted gesture recognition for athletes in dynamic scenes. Although many research attention had been devoted to this field in recent years, most of existing works failed to fully take characteristics of dynamic scenes into consideration. To deal with this challenge, this paper proposes a diffusion convolution neural network-based multiview gesture recognition approach in dynamic scenes. For one thing, the dynamic spatiotemporal slice position selection based on the body mask heatmap is adopted to calculate positions of horizontal and vertical slices. Thus, the dynamic selection of slice positions in two directions can be realized, and then the extraction of bi-directional spatiotemporal slice images can be completed. For another, action sequences through the 3D residual neural network are learned, and the spatiotemporal information among frames are mined through recurrent networks. Through their combination, a multi-view gesture recognition approach for athletes is constructed. In the experiments, two standard datasets UCF101 and HMDB51 are utilized to establish simulation environment. The proposed method can reach the accuracy beyond 95% on the two datasets. Compared with several typical recognition methods, the proposed method shows higher accuracy.

Deep PLS: A Lightweight Deep Learning Model for Interpretable and Efficient Data Analytics.

The salient progress of deep learning is accompanied by nonnegligible deficiencies, such as: 1) interpretability problem; 2) requirement for large data amounts; 3) hard to design and tune parameters; and 4) heavy computation complexity. Despite the remarkable achievements of neural networks-based deep models in many fields, the practical applications of deep learning are still limited by these shortcomings. This article proposes a new concept called the lightweight deep model (LDM). LDM absorbs the useful ideas of deep learning and overcomes their shortcomings to a certain extent. We explore the idea of LDM from the perspective of partial least squares (PLS) by constructing a deep PLS (DPLS) model. The feasibility and merits of DPLS are proved theoretically, after that, DPLS is further generalized to a more common form (GDPLS) by adding a nonlinear mapping layer between two cascaded PLS layers in the model structure. The superiority of DPLS and GDPLS is demonstrated through four practical cases involving two regression problems and two classification tasks, in which our model not only achieves competitive performance compared with existing neural networks-based deep models but also is proven to be a more interpretable and efficient method, and we know exactly how it improves performance, how it gives correct results. Note that our proposed model can only be regarded as an alternative to fully connected neural networks at present and cannot completely replace the mature deep vision or language models.

Practical single-line diagram recognition based on digital image processing and deep vision models

The Single-Line Diagram (SLD) is a standardized graphical representation of power systems. Power engineering software typically requires users to model power systems by drawing SLDs. Although existing computer vision-based SLD recognition approaches facilitate system modeling to some extent, their performance is usually limited by the low resolution of images, the imbalance in size and number of electrical symbols, and the interference between adjacent graphical elements on practical SLDs. This paper proposes a novel SLD recognition framework to address these challenges. First, a text and symbol separation module based on digital image processing is developed to enable independent analysis of text labels and electrical symbols to avoid mutual interference. To accurately recognize electrical symbols of different sizes, the proposed framework integrates the Swin Transformer into the Faster Region-based Convolutional Neural Network (Faster R-CNN), which ultimately achieves a minimum precision of 98.9% and a minimum recall of 98.12%. In addition, the framework modifies the Text Super-Resolution Network (TSRN) to improve text recognition performance on low-resolution images. Combined with the Attentional Scene Text Recognizer with Flexible Rectification (ASTER) model, the proposed text recognition model results in an average accuracy increase of 1% on the TextZoom dataset and 97.1% accuracy in recognizing practical SLD text labels. A synthetic SLD generation method is also proposed to augment the actual SLD datasets to overcome the class imbalance of electrical symbols. The proposed framework significantly improves the performance of existing engineering drawing recognition frameworks. Relevant source code is available online at https://github.com/Lattle-y/work.git.

Deep Vision in Analysis and Recognition of Radar Data: Achievements, Advancements, and Challenges

Deep Vision in Analysis and Recognition of Radar Data: Achievements, Advancements, and Challenges

Unified Architecture Adaptation for Compressed Domain Semantic Inference.

Advances in both lossy image compression and semantic content understanding have been greatly fueled by deep learning techniques, yet these two tasks have been developed separately for the past decades. In this work, we address the problem of directly executing semantic inference from quantized latent features in the deep compressed domain without pixel reconstruction. Although different methods have been proposed for this problem setting, they either are restrictive to a specific architecture, or are sub-optimal in terms of compressed domain task accuracy. In contrast, we propose a lightweight, plug-and-play solution which is generally compliant with popular learned image coders and deep vision models, making it attractive to vast applications. Our method adapts prevalent pixel domain neural models that are deployed for various vision tasks to directly accept quantized latent features (other than pixels). We further suggest training the compressed domain model by transferring knowledge from its corresponding pixel domain counterpart. Experiments show that our method is compliant with popular learned image coders and vision task models. Under fair comparison, our approach outperforms a baseline method by a) more than 3% top-1 accuracy for compressed domain classification, and b) more than 7% mIoU for compressed domain semantic segmentation, at various data rates.

UAV Dynamic Object Tracking with Lightweight Deep Vision Reinforcement Learning

Several approaches have applied Deep Reinforcement Learning (DRL) to Unmanned Aerial Vehicles (UAVs) to do autonomous object tracking. These methods, however, are resource intensive and require prior knowledge of the environment, making them difficult to use in real-world applications. In this paper, we propose a Lightweight Deep Vision Reinforcement Learning (LDVRL) framework for dynamic object tracking that uses the camera as the only input source. Our framework employs several techniques such as stacks of frames, segmentation maps from the simulation, and depth images to reduce the overall computational cost. We conducted the experiment with a non-sparse Deep Q-Network (DQN) (value-based) and a Deep Deterministic Policy Gradient (DDPG) (actor-critic) to test the adaptability of our framework with different methods and identify which DRL method is the most suitable for this task. In the end, a DQN is chosen for several reasons. Firstly, a DQN has fewer networks than a DDPG, hence reducing the computational resources on physical UAVs. Secondly, it is surprising that although a DQN is smaller in model size than a DDPG, it still performs better in this specific task. Finally, a DQN is very practical for this task due to the ability to operate in continuous state space. Using a high-fidelity simulation environment, our proposed approach is verified to be effective.