Modulation Of Perception Research Articles

Enhanced YOLOv8 Ship Detection Empower Unmanned Surface Vehicles for Advanced Maritime Surveillance

The evolution of maritime surveillance is significantly marked by the incorporation of Artificial Intelligence and machine learning into Unmanned Surface Vehicles (USVs). This paper presents an AI approach for detecting and tracking unmanned surface vehicles, specifically leveraging an enhanced version of YOLOv8, fine-tuned for maritime surveillance needs. Deployed on the NVIDIA Jetson TX2 platform, the system features an innovative architecture and perception module optimized for real-time operations and energy efficiency. Demonstrating superior detection accuracy with a mean Average Precision (mAP) of 0.99 and achieving an operational speed of 17.99 FPS, all while maintaining energy consumption at just 5.61 joules. The remarkable balance between accuracy, processing speed, and energy efficiency underscores the potential of this system to significantly advance maritime safety, security, and environmental monitoring.

Causal inference predicts the transition from integration to segmentation in motion perception

Motion provides a powerful sensory cue for segmenting a visual scene into objects and inferring the causal relationships between objects. Fundamental mechanisms involved in this process are the integration and segmentation of local motion signals. However, the computations that govern whether local motion signals are perceptually integrated or segmented remain unclear. Hierarchical Bayesian causal inference has recently been proposed as a model for these computations, yet a hallmark prediction of the model – its dependency on sensory uncertainty – has remained untested. We used a recently developed hierarchical stimulus configuration to measure how human subjects integrate or segment local motion signals while manipulating motion coherence to control sensory uncertainty. We found that (a) the perceptual transition from motion integration to segmentation shifts with sensory uncertainty, and (b) that perceptual variability is maximal around this transition point. Both findings were predicted by the model and challenge conventional interpretations of motion repulsion effects.

Integration of Artificial Vision and Image Processing into a Pick and Place Collaborative Robotic System

In the field of robotics, pick and place applications are becoming increasingly popular due to their ability to automate repetitive tasks that can create temporary or permanent injuries. To enhance the efficiency of these applications, object recognition using a fixed camera or one mounted on a robotic hand has been employed. This paper explores the possibilities of implementing a low-cost camera into a collaborative robotic system. A software architecture has been developed, including modules for perception, pick and place, and part transfer. A comprehensive overview of various intuitive drag-and-drop image processing technologies and their suitability for object recognition in a robotic context is provided. The challenges related to lighting and the effect of shadows in object recognition are discussed. A critical assessment is made of the architecture development platform as well as the study and the results are performed, and the effectiveness of the proposed solution based on the Niop architecture is verified.

AI+AR based Framework for Guided Visual Diagnosis of Equipment

Automated solutions for effective support services, such as failure diagnosis and repair, are crucial to keep customer satisfaction and loyalty. However, providing consistent, high quality, and timely support is a difficult task. In practice, customer support usually requires technicians to perform onsite diagnosis, but service quality is often adversely affected by limited expert technicians, high turnover, and minimal automated tools. To address these challenges, we present a novel solution framework for aiding technicians in performing visual equipment diagnosis. We envision a workflow where the technician reports a failure and prompts the system to automatically generate a diagnostic plan that includes parts, areas of interest, and necessary tasks. The plan is used to guide the technician with augmented reality (AR), while a perception module analyzes and tracks the technician’s actions to recommend next steps. Our framework consists of three components: planning, tracking, and guiding. The planning component automates the creation of a diagnostic plan by querying a knowledge graph (KG). We propose to leverage Large Language Models (LLMs) for the construction of the KG to accelerate the extraction process of parts, tasks, and relations from manuals. The tracking component enhances 3D detections by using perception sensors with a 2D nested object detection model. Finally, the guiding component reduces process complexity for technicians by combining 2D models and AR interactions. To validate the framework, we performed multiple studies to:1) determine an effective prompt method for the LLM to construct the KG; 2) demonstrate benefits of our 2D nested object model combined with AR model.

Advancing complex urban traffic forecasting: A fully attentional spatial-temporal network enhanced by graph representation

Accurate urban traffic forecasting is essential for intelligent transportation systems (ITS). However, the majority of existing forecasting methodologies predominantly concentrate on point-based forecasts (e.g., traffic detector forecasts). A limited number of them pay attention to the urban bidirectional road segments and the complex road network topology. To advance accurate traffic forecasting in complex urban scenarios, this paper proposes a Graph Representation enhanced Fully Attentional Spatial-Temporal network (GR-FAST). First, we construct a refined bidirectional road network graph (BRG) to depict the urban road network topology more accurately, particularly focusing on the turning patterns at intersections. Then, we adopt the graph representation methodology and introduce spatial information encoding (SIE) to explicitly characterize the significance of roads and network structure from multiple perspectives. Enhanced by SIE, spatial attention can capture spatial dependencies from both road network topologies and traffic pattern similarities, thereby forming a unified urban spatial cognition. Finally, a multi-scale residual perception (MRP) module is designed to balance the interplay of short-term temporal variability and long-term periodicity. Experiments on a real-world urban dataset from Wuhan, China, demonstrate that GR-FAST outperforms the state-of-the-art deep learning methods, achieving an improvement of 9.19%. Furthermore, ablation studies suggest that the explicit incorporation of complex road spatial topologies can significantly enhance forecasting accuracy.

Two-stage Rule-induction visual reasoning on RPMs with an application to video prediction

Raven’s Progressive Matrices (RPMs) are frequently used in evaluating human’s visual reasoning ability. Researchers have made considerable efforts in developing systems to automatically solve the RPM problem, often through a black-box end-to-end convolutional neural network for both visual recognition and logical reasoning tasks. Based on the intrinsic natures of RPM problem, we propose a Two-stage Rule-Induction Visual Reasoner (TRIVR), which consists of a perception module and a reasoning module, to tackle the challenges of real-world visual recognition and subsequent logical reasoning tasks, respectively. For the reasoning module, we further propose a “2+1” formulation that models human’s thinking in solving RPMs and significantly reduces the model complexity. It derives a reasoning rule from each RPM sample, which is not feasible for existing methods. As a result, the proposed reasoning module is capable of yielding a set of reasoning rules modeling human in solving the RPM problems. To validate the proposed method on real-world applications, an RPM-like Video Prediction (RVP) dataset is constructed, where visual reasoning is conducted on RPMs constructed using real-world video frames. Experimental results on various RPM-like datasets demonstrate that the proposed TRIVR achieves a significant and consistent performance gain compared with state-of-the-art models.

PFEI-Net: A profound feature exploration and interaction network for ceramic substrate surface defect detection

Ceramic substrates serve as the foundational material for numerous electronic devices, and their surface quality directly affects performance and longevity. Therefore, surface defect detection on ceramic substrates is an indispensable task during the manufacturing process. Ceramic substrate surface defects exhibit low contrast, along with intraclass difference and interclass similarity, posing substantial challenges for automated visual inspection. Existing deep learning-based methods utilize carefully crafted backbone networks and feature pyramid structures that perceive multilevel information to deal with the above challenges. However, these methods still suffer from insufficient extraction of discriminative features and low effectiveness in feature fusion, thereby hindering the detection performance. To address these issues, we propose the profound feature exploration and interaction network (PFEI-Net). Initially, we propose the discriminative feature mining backbone (DFM-backbone) to progressively explore the effective discriminative features. In the DFM-backbone, the global information grated perception module is devised to construct long-range feature dependencies and aggregate rich features, whereas the contextual semantic flexible extraction module is designed to meticulously perceive contextual semantic information in a targeted manner, enhancing the comprehensive representation of the discriminative features. Then, we propose the multipath semantic interaction guided-feature pyramid network (MSIG-FPN) to facilitate the fusion and interaction of the valuable information. In the MSIG-FPN, the hierarchical refocus–reaggregation module is constructed to refocus on task-relevant features and provide reliable guidance for the network, whereas the cross-stage semantic deep fusion module is designed to integrate multi-level contextual information deeply across stages and is employed to construct the semantic complementary paths, thereby preventing the loss of crucial features. Finally, built upon the backbone of DFM-backbone and MSIG-FPN, we establish the PFEI-Net, achieving accurate detection of surface defects on ceramic substrates. Experimental results highlight that the PFEI-Net achieves excellent performance with a remarkable 89.3 % mean average precision (mAP) and an inference time of 26.1 ms, which offers a promising and viable solution for automated surface defect detection of ceramic substrate.

Spatial Prior-Guided Bi-Directional Cross-Attention Transformers for Tooth Instance Segmentation.

Tooth instance segmentation of dental panoramic X-ray images is of significant clinical importance. Teeth exhibit symmetry within the upper and lower jawbones and are arranged in a specific order. However, previous studies frequently overlook this crucial spatial prior information, resulting in the misidentifications of tooth categories, especially for adjacent or similarly shaped teeth. In this paper, we propose SPGTNet, a spatial prior-guided transformer method, designed to both the extracted tooth positional features from CNNs and the long-range contextual information from vision transformers, specifically for dental panoramic X-ray image segmentation. Initially, a center-based spatial prior perception module is employed to identify the centroid of each tooth, thereby enhancing the spatial prior information for the CNN sequence features. Subsequently, a bi-directional cross-attention module is designed to facilitate the interaction between the spatial prior information of the CNN sequence features and the long-range contextual features of the vision transformer sequence features. Finally, an instance identification head is employed to generate the tooth segmentation results. Extensive experiments on three public benchmark datasets demonstrate the effectiveness and superiority of our proposed method compared to other state-of-the-art approaches. The proposed method accurately identifies and analyzes tooth structures, thereby providing crucial information for dental diagnosis, treatment planning, and research.

Cascade contour-enhanced panoptic segmentation for robotic vision perception.

Panoptic segmentation plays a crucial role in enabling robots to comprehend their surroundings, providing fine-grained scene understanding information for robots' intelligent tasks. Although existing methods have made some progress, they are prone to fail in areas with weak textures, small objects, etc. Inspired by biological vision research, we propose a cascaded contour-enhanced panoptic segmentation network called CCPSNet, attempting to enhance the discriminability of instances through structural knowledge. To acquire the scene structure, a cascade contour detection stream is designed, which extracts comprehensive scene contours using channel regulation structural perception module and coarse-to-fine cascade strategy. Furthermore, the contour-guided multi-scale feature enhancement stream is developed to boost the discrimination ability for small objects and weak textures. The stream integrates contour information and multi-scale context features through structural-aware feature modulation module and inverse aggregation technique. Experimental results show that our method improves accuracy on the Cityscapes (61.2 PQ) and COCO (43.5 PQ) datasets while also demonstrating robustness in challenging simulated real-world complex scenarios faced by robots, such as dirty cameras and rainy conditions. The proposed network promises to help the robot perceive the real scene. In future work, an unsupervised training strategy for the network could be explored to reduce the training cost.

An Automatic Mapping Method of Navigation Map for Outdoor Road Scenes

Abstract. In the process of autonomous navigation of outdoor mobile robots, four modules are involved: perception, localization, planning, and controlling. The perception module utilizes sensors such as cameras and radars based on various principles to analyze the robot's surroundings in real-time. The localization module uses GPS (Global Positioning System), IMU (Inertial Measurement Unit), and prior maps for real-time positioning analysis. The planning module plans optimal path based on the outputs of the first two modules, and the controlling module directs the robot's chassis to move along the planned optimal path. For the localization module, the accuracy of GPS positioning results heavily depends on weather conditions and GPS signal receptions. Even if the positioning results of imu are integrated, the positioning accuracy still cannot meet the needs of robot navigation. Therefore, using prior maps for repositioning can compensate for this accuracy deficiency. The planning module also requires path planning based on prior maps. If the a priori map storage is large, it will lead to difficulties in usage, maintenance, and updates. Therefore, it is crucial to research lightweight navigation map mapping methods. In this paper, an automatically mapping method of lightweight navigation maps is proposed, combining cameras and LiDAR (Light Detection and Ranging), including semantic informations necessary for outdoor navigation positioning, such as pole-like objects and traffic signs for robot longitudinal positioning, and lane line elements for robot lateral positioning. This method automatic generates robot navigation maps in Lanelet2 format, providing support for subsequent positioning and path planning modules.

A platform-agnostic deep reinforcement learning framework for effective Sim2Real transfer towards autonomous driving

Autonomous driving presents unique challenges, particularly in transferring agents trained in simulation to real-world environments due to the discrepancies between the two. To address this issue, here we propose a robust Deep Reinforcement Learning (DRL) framework that incorporates platform-dependent perception modules to extract task-relevant information, enabling the training of a lane-following and overtaking agent in simulation. This framework facilitates the efficient transfer of the DRL agent to new simulated environments and the real world with minimal adjustments. We assess the performance of the agent across various driving scenarios in both simulation and the real world, comparing it to human drivers and a proportional-integral-derivative (PID) baseline in simulation. Additionally, we contrast it with other DRL baselines to clarify the rationale behind choosing this framework. Our proposed approach helps bridge the gaps between different platforms and the Simulation to Reality (Sim2Real) gap, allowing the trained agent to perform consistently in both simulation and real-world scenarios, effectively driving the vehicle.

3D Head Pose Estimation via Normal Maps: A Generalized Solution for Depth Image, Point Cloud, and Mesh

Head pose estimation plays a crucial role in various applications, including human–machine interaction, autonomous driving systems, and 3D reconstruction. Current methods address the problem primarily from a 2D perspective, which limits the efficient utilization of 3D information. Herein, a novel approach, called pose orientation‐aware network (POANet), which leverages normal maps for orientation information embedding, providing abundant and robust head pose information, is introduced. POANet incorporates the axial signal perception module and the rotation matrix perception module, these lightweight modules make the approach achieve state‐of‐the‐art (SOTA) performance with few computational costs. This method can directly analyze various topological 3D data without extensive preprocessing. For depth images, POANet outperforms existing methods on the Biwi Kinect head pose dataset, reducing the mean absolute error (MAE) by ≈30% compared to the SOTA methods. POANet is the first method to perform rigid head registration in a landmark‐free manner. It also incorporates few‐shot learning capabilities and achieves an MAE of about on the Headspace dataset. These features make POANet a superior alternative to traditional generalized Procrustes analysis for mesh data processing, offering enhanced convenience for human phenotype studies.

MFH‐Net: A Hybrid CNN‐Transformer Network Based Multi‐Scale Fusion for Medical Image Segmentation

ABSTRACTIn recent years, U‐Net and its variants have gained widespread use in medical image segmentation. One key aspect of U‐Net's design is the skip connection, facilitating the retention of detailed information and leading to finer segmentation results. However, existing research often concentrates on enhancing either the encoder or decoder, neglecting the semantic gap between them, and resulting in suboptimal model performance. In response, we introduce Multi‐Scale Fusion module aimed at enhancing the original skip connections and addressing the semantic gap. Our approach fully incorporates the correlation between outputs from adjacent encoder layers and facilitates bidirectional information exchange across multiple layers. Additionally, we introduce Channel Relation Perception module to guide the fused multi‐scale information for efficient connection with decoder features. These two modules collectively bridge the semantic gap by capturing spatial and channel dependencies in the features, contributing to accurate medical image segmentation. Building upon these innovations, we propose a novel network called MFH‐Net. On three publicly available datasets, ISIC2016, ISIC2017, and Kvasir‐SEG, we perform a comprehensive evaluation of the network. The experimental results show that MFH‐Net exhibits higher segmentation accuracy in comparison with other competing methods. Importantly, the modules we have devised can be seamlessly incorporated into various networks, such as U‐Net and its variants, offering a potential avenue for further improving model performance.

The design and implementation of a smart greenhouse control system

Abstract Against the dual backdrop of continuously improving agricultural production levels and the global scarcity of water resources, countries worldwide are actively exploring the topic of water conservation. The development of an intelligent irrigation control system could make a significant contribution to water conservation efforts, while also helping to increase crop yields. This project adopts a modular design approach, utilizing sensors to collect real-time environmental data surrounding the crops. If the data falls outside the predefined optimal range for crop growth, the system automatically initiates irrigation, cooling, and supplemental lighting. The hardware design primarily consists of a data perception module and a control terminal module. After testing, the system runs stably and can accurately obtain information such as soil moisture, light intensity and ambient temperature, and basically realize the digital intelligent control of crop growing environment.

MHW-GAN: Multidiscriminator Hierarchical Wavelet Generative Adversarial Network for Multimodal Image Fusion.

Image fusion technology aims to obtain a comprehensive image containing a specific target or detailed information by fusing data of different modalities. However, many deep learning-based algorithms consider edge texture information through loss functions instead of specifically constructing network modules. The influence of the middle layer features is ignored, which leads to the loss of detailed information between layers. In this article, we propose a multidiscriminator hierarchical wavelet generative adversarial network (MHW-GAN) for multimodal image fusion. First, we construct a hierarchical wavelet fusion (HWF) module as the generator of MHW-GAN to fuse feature information at different levels and scales, which avoids information loss in the middle layers of different modalities. Second, we design an edge perception module (EPM) to integrate edge information from different modalities to avoid the loss of edge information. Third, we leverage the adversarial learning relationship between the generator and three discriminators for constraining the generation of fusion images. The generator aims to generate a fusion image to fool the three discriminators, while the three discriminators aim to distinguish the fusion image and edge fusion image from two source images and the joint edge image, respectively. The final fusion image contains both intensity information and structure information via adversarial learning. Experiments on public and self-collected four types of multimodal image datasets show that the proposed algorithm is superior to the previous algorithms in terms of both subjective and objective evaluation.

Ancient Painting Inpainting with Regional Attention-Style Transfer and Global Context Perception

Ancient paintings, as a vital component of cultural heritage, encapsulate a profound depth of cultural significance. Over time, they often suffer from different degradation conditions, leading to damage. Existing ancient painting inpainting methods struggle with semantic discontinuities, blurred textures, and details in missing areas. To address these issues, this paper proposes a generative adversarial network (GAN)-based ancient painting inpainting method named RG-GAN. Firstly, to address the inconsistency between the styles of missing and non-missing areas, this paper proposes a Regional Attention-Style Transfer Module (RASTM) to achieve complex style transfer while maintaining the authenticity of the content. Meanwhile, a multi-scale fusion generator (MFG) is proposed to use the multi-scale residual downsampling module to reduce the size of the feature map and effectively extract and integrate the features of different scales. Secondly, a multi-scale fusion mechanism leverages the Multi-scale Cross-layer Perception Module (MCPM) to enhance feature representation of filled areas to solve the semantic incoherence of the missing region of the image. Finally, the Global Context Perception Discriminator (GCPD) is proposed for the deficiencies in capturing detailed information, which enhances the information interaction across dimensions and improves the discriminator’s ability to identify specific spatial areas and extract critical detail information. Experiments on the ancient painting and ancient Huaniao++ datasets demonstrate that our method achieves the highest PSNR values of 34.62 and 23.46 and the lowest LPIPS values of 0.0507 and 0.0938, respectively.

SparseMorph: A weakly-supervised lightweight sparse transformer for mono- and multi-modal deformable image registration

PurposeDeformable image registration (DIR) is crucial for improving the precision of clinical diagnosis. Recent Transformer-based DIR methods have shown promising performance by capturing long-range dependencies. Nevertheless, these methods still grapple with high computational complexity. This work aims to enhance the performance of DIR in both computational efficiency and registration accuracy. MethodsWe proposed a weakly-supervised lightweight Transformer model, named SparseMorph. To reduce computational complexity without compromising the representative feature capture ability, we designed a sparse multi-head self-attention (SMHA) mechanism. To accumulate representative features while preserving high computational efficiency, we constructed a multi-branch multi-layer perception (MMLP) module. Additionally, we developed an anatomically-constrained weakly-supervised strategy to guide the alignment of regions-of-interest in mono- and multi-modal images. ResultsWe assessed SparseMorph in terms of registration accuracy and computational complexity.Within the mono-modal brain datasets IXI and OASIS, our SparseMorph outperforms the state-of-the-art method TransMatch with improvements of 3.2 % and 2.9 % in DSC scores for MRI-to-CT registration tasks, respectively. Moreover, in the multi-modal cardiac dataset MMWHS, our SparseMorph shows DSC score improvements of 9.7 % and 11.4 % compared to TransMatch in MRI-to-CT and CT-to-MRI registration tasks, respectively. Notably, SparseMorph attains these performance advantages while utilizing 33.33 % of the parameters of TransMatch. ConclusionsThe proposed weakly-supervised deformable image registration model, SparseMorph, demonstrates efficiency in both mono- and multi-modal registration tasks, exhibiting superior performance compared to state-of-the-art algorithms, and establishing an effective DIR method for clinical applications.

OAR-UNet: Enhancing Long-Distance Dependencies for Head and Neck OAR Segmentation

Accurate segmentation of organs at risk (OARs) is a crucial step in the precise planning of radiotherapy for head and neck tumors. However, manual segmentation methods using CT images, which are still predominantly applied in clinical settings, are inefficient and expensive. Additionally, existing segmentation methods struggle with small organs and have difficulty managing the complex interdependencies between organs. To address these issues, this study proposed an OAR-UNet segmentation method based on a U-shaped architecture with two key designs. To tackle the challenge of segmenting small organs, a Local Feature Perception Module (LFPM) is developed to enhance the sensitivity of the method to subtle structures. Furthermore, a Cross-shaped Transformer Block (CSTB) with a cross-shaped attention mechanism is introduced to improve the ability of the model to capture and process long-distance dependency information. To accelerate the convergence of the Transformer, we designed a Local Encoding Module (LEM) based on depthwise separable convolutions. In our experimental evaluation, we utilized two publicly available datasets, SegRap2023 and PDDCA, achieving Dice coefficients of 78.22% and 89.42%, respectively. These results demonstrate that our method outperforms both previous classic methods and state-of-the-art (SOTA) methods.

MSTAN: multi-scale spatiotemporal attention network with adaptive relationship mining for remaining useful life prediction in complex systems

Abstract In the era of smart manufacturing and advanced industrial systems, the high degree of integration and intelligence of equipment demands higher reliability and safety from systems. Existing methods often rely on historical data for Remaining Useful Life (RUL) prediction to achieve Prognostic and Health Management (PHM). However, the internal units of complex equipment exhibit significant spatial correlation and temporal diversity, making PHM for complex equipment a multidimensional challenge involving both temporal and spatial information, thereby severely limits the effectiveness of RUL prediction for complex systems. Addressing these challenges, this study introduces a multi-scale spatiotemporal attention network with adaptive relationship mining, specifically designed for the remaining useful life (RUL) prediction of such equipment. The core of the proposed method lies in the multi-scale feature perception module, which adeptly extracts varied scale features from multidimensional sensor data. Following this, an innovative adaptive relationship mining module is integrated to uncover multi-order coupling relationships between diverse sensors, enhancing the model’s predictive accuracy. Furthermore, a spatiotemporal attention module is employed to discern and emphasize crucial spatiotemporal correlations. To validate the effectiveness and superiority of the proposed method, the Commercial Modular Aero-propulsion System Simulation (C-MAPSS) dataset is employed for comprehensive performance evaluation, the IEEE 2012 PHM bearing dataset is also adopted to demonstrate the generalization and robustness of the proposed method. The results not only show a notable improvement over existing methods but also offer a more intuitive understanding through visual representations, marking a significant stride in enhancing the safety and efficiency of complex systems.

Research on Obstacle Avoidance Strategy of Automated Heavy Vehicle Platoon in High-Speed Scenarios

Amidst the rapid progression of the road transport industry, the safety and efficiency of heavy-vehicle platoons have garnered significant attention. The study tackles the challenge of obstacle avoidance presented by vehicles owing to their considerable mass, delayed response times, and line-of-sight impediments, by introducing a cooperative obstacle avoidance system for heavy-vehicle platoons based on deep reinforcement learning. The system comprises three primary modules: perception, decision-making, and control. Initially, the perception module acquires real-time environmental data. Subsequently, the decision-making module formulates obstacle avoidance decisions based on the acquired data. Specifically, it implements a two-stage braking obstacle avoidance strategy under low collision risk scenarios, while employing a fifth-degree polynomial for planning and tracking obstacle avoidance paths under high collision risk conditions suitable for steering maneuvers. The control module utilizes the local multi-agent deep deterministic policy gradient (LADDPG) algorithm to train the heavy-vehicle platoon agents, ensuring the formation’s maintenance while mitigating collisions with other vehicles and obstacles. The effectiveness of the proposed system is substantiated through simulation experiments, demonstrating its adaptability to various traffic conditions, selection of suitable obstacle avoidance strategies, and significant enhancement of obstacle avoidance performance and heavy-vehicle platoon stability.