Perceptual Capabilities Research Articles

TCFAP-Net: Transformer-based Cross-feature Fusion and Adaptive Perception Network for large-scale point cloud semantic segmentation

Point cloud semantic segmentation is an ingredient in understanding real-world scenes. Most existing approaches perform poorly on scene boundaries and struggle with recognizing objects of different scales. In this paper, we propose a novel framework that incorporates Transformer into the U-Net architecture for inferring pointwise semantics. Specifically, the Transformer-based cross-feature fusion module is designed first to employ geometric and semantic information to learn feature offsets to overcome the border ambiguity of segmentation results, and then it utilizes the Transformer to learn cross-feature enhanced and fused encoder features. Additionally, to facilitate the overall network’s structure-to-detail perception capabilities, the adaptive perception module is designed, which employs cross-attention to adaptively allocate weights to encoder features at varying resolutions, establishing long-range contextual dependencies. Ablation studies validate the individual contributions of our module design choices. Compared with the existing competitive methods, our approach achieves state-of-the-art performance and exhibits superior results on benchmarks. Code is available at https://github.com/xiluo-cug/TCFAP-Net.

Global Semantic-Sense Aggregation Network for Salient Object Detection in Remote Sensing Images.

Salient object detection (SOD) aims to accurately identify significant geographical objects in remote sensing images (RSI), providing reliable support and guidance for extensive geographical information analyses and decisions. However, SOD in RSI faces numerous challenges, including shadow interference, inter-class feature confusion, as well as unclear target edge contours. Therefore, we designed an effective Global Semantic-aware Aggregation Network (GSANet) to aggregate salient information in RSI. GSANet computes the information entropy of different regions, prioritizing areas with high information entropy as potential target regions, thereby achieving precise localization and semantic understanding of salient objects in remote sensing imagery. Specifically, we proposed a Semantic Detail Embedding Module (SDEM), which explores the potential connections among multi-level features, adaptively fusing shallow texture details with deep semantic features, efficiently aggregating the information entropy of salient regions, enhancing information content of salient targets. Additionally, we proposed a Semantic Perception Fusion Module (SPFM) to analyze map relationships between contextual information and local details, enhancing the perceptual capability for salient objects while suppressing irrelevant information entropy, thereby addressing the semantic dilution issue of salient objects during the up-sampling process. The experimental results on two publicly available datasets, ORSSD and EORSSD, demonstrated the outstanding performance of our method. The method achieved 93.91% Sα, 98.36% Eξ, and 89.37% Fβ on the EORSSD dataset.

Optimized Design of EdgeBoard Intelligent Vehicle Based on PP-YOLOE.

Advances in deep learning and computer vision have overcome many challenges inherent in the field of autonomous intelligent vehicles. To improve the detection accuracy and efficiency of EdgeBoard intelligent vehicles, we proposed an optimized design of EdgeBoard based on our PP-YOLOE+ model. This model innovatively introduces a composite backbone network, incorporating deep residual networks, feature pyramid networks, and RepResBlock structures to enrich environmental perception capabilities through the advanced analysis of sensor data. The incorporation of an efficient task-aligned head (ET-head) in the PP-YOLOE+ framework marks a pivotal innovation for precise interpretation of sensor information, addressing the interplay between classification and localization tasks with high effectiveness. Subsequent refinement of target regions by detection head units significantly sharpens the system's ability to navigate and adapt to diverse driving scenarios. Our innovative hardware design, featuring a custom-designed mainboard and drive board, is specifically tailored to enhance the computational speed and data processing capabilities of intelligent vehicles. Furthermore, the optimization of our Pos-PID control algorithm allows the system to dynamically adjust to complex driving scenarios, significantly enhancing vehicle safety and reliability. Besides, our methodology leverages the latest technologies in edge computing and dynamic label assignment, enhancing intelligent vehicles' operations through seamless sensor integration. Our custom dataset, specifically designed for this study, includes 4777 images captured by intelligent vehicles under a variety of environmental and lighting conditions. The dataset features diverse scenarios and objects pertinent to autonomous driving, such as pedestrian crossings and traffic signs, ensuring a comprehensive evaluation of the model's performance. We conducted extensive testing of our model on this dataset to thoroughly assess sensor performance. Evaluated against metrics including accuracy, error rate, precision, recall, mean average precision (mAP), and F1-score, our findings reveal that the model achieves a remarkable accuracy rate of 99.113%, an mAP of 54.9%, and a real-time detection frame rate of 192 FPS, all within a compact parameter footprint of just 81 MB. These results demonstrate the superior capability of our PP-YOLOE+ model to integrate sensor data, achieving an optimal balance between detection accuracy and computational speed compared with existing algorithms.

Working with machines in mathematics

Machine learning is making significant contributions to many fields but how can it be used as a tool for mathematicians? This article explores the emerging role of machine learning in mathematical research, highlighting how its perceptual capabilities can augment human intuition and lead to new discoveries.

MoS2/ZnO-heterostructured optoelectronic synapse for multiwavelength optical information-based sensing, memory, and processing

The rapid development of artificial intelligence and humanoid robots requires hardware-level revolution of synapses-based bionic vision system. However, current optoelectronic synapses featuring multiwavelength sensing and multifunctional processing necessitate intricate integrations of multi-materials and floating gate configurations, which induces complex fabrication process and limited integration density. Here, we propose a novel two-terminal optoelectronic synaptic device based on MoS2/ZnO heterostructure which integrates sense and memory functions and realizes multifunctional optical information processing while circumvents the need of complex configurations. By simultaneously exploiting the photo-adsorption oxygen dissociation property of ZnO and the persistent photoconductivity effect of MoS2, the device demonstrates synaptic capability of four-color perception with a single synaptic event consuming a mere 2.55×10−9 J. Different from the traditional power-density sensitive nociceptor, for the first time photon energy-sensitive nociceptors is developed, which can mimic the process of UV irradiation to the retina leading to fear of visible light. Diverse information processing ability, such as photonic information encryption, associative learning, and logic calculation, was demonstrated in a single device, indicating a practical strategy for facile design for future high integration intensity neuromorphic devices.

Object Detection in Hazy Environments, Based on an All-in-One Dehazing Network and the YOLOv5 Algorithm

This study introduces an advanced algorithm for intelligent vehicle target detection in hazy conditions, aiming to bolster the environmental perception capabilities of autonomous vehicles. The proposed approach integrates a hybrid convolutional module (HDC) into an all-in-one dehazing network, AOD-Net, to expand the perceptual domain for image feature extraction and refine the clarity of dehazed images. To accelerate model convergence and enhance generalization, the loss function has been optimized. For practical deployment in intelligent vehicle systems, the ShuffleNetv2 lightweight network module is incorporated into the YOLOv5s network backbone, and the feature pyramid network (FPN) within the neck network has been refined. Additionally, the network employs a global shuffle convolution (GSconv) to balance accuracy with parameter count. To further focus on the target, a convolutional block attention module (CBAM) is introduced, which helps in reducing the network’s parameter count without compromising accuracy. A comparative experiment was conducted, and the results indicated that our algorithm achieved an impressive mean average precision (mAP) of 76.8% at an intersection-over-union (IoU) threshold of 0.5 in hazy conditions, outperforming YOLOv5 by 7.4 percentage points.

Ontology based autonomous robot task processing framework.

In recent years, the perceptual capabilities of robots have been significantly enhanced. However, the task execution of the robots still lacks adaptive capabilities in unstructured and dynamic environments. In this paper, we propose an ontology based autonomous robot task processing framework (ARTProF), to improve the robot's adaptability within unstructured and dynamic environments. ARTProF unifies ontological knowledge representation, reasoning, and autonomous task planning and execution into a single framework. The interface between the knowledge base and neural network-based object detection is first introduced in ARTProF to improve the robot's perception capabilities. A knowledge-driven manipulation operator based on Robot Operating System (ROS) is then designed to facilitate the interaction between the knowledge base and the robot's primitive actions. Additionally, an operation similarity model is proposed to endow the robot with the ability to generalize to novel objects. Finally, a dynamic task planning algorithm, leveraging ontological knowledge, equips the robot with adaptability to execute tasks in unstructured and dynamic environments. Experimental results on real-world scenarios and simulations demonstrate the effectiveness and efficiency of the proposed ARTProF framework. In future work, we will focus on refining the ARTProF framework by integrating neurosymbolic inference.

Electromagnetic Field & ToF Sensor Fusion for Advanced Perceptual Capability of Robots

Electromagnetic Field & ToF Sensor Fusion for Advanced Perceptual Capability of Robots

YOLOv8-RMDA: Lightweight YOLOv8 Network for Early Detection of Small Target Diseases in Tea.

In order to efficiently identify early tea diseases, an improved YOLOv8 lesion detection method is proposed to address the challenges posed by the complex background of tea diseases, difficulty in detecting small lesions, and low recognition rate of similar phenotypic symptoms. This method focuses on detecting tea leaf blight, tea white spot, tea sooty leaf disease, and tea ring spot as the research objects. This paper presents an enhancement to the YOLOv8 network framework by introducing the Receptive Field Concentration-Based Attention Module (RFCBAM) into the backbone network to replace C2f, thereby improving feature extraction capabilities. Additionally, a mixed pooling module (Mixed Pooling SPPF, MixSPPF) is proposed to enhance information blending between features at different levels. In the neck network, the RepGFPN module replaces the C2f module to further enhance feature extraction. The Dynamic Head module is embedded in the detection head part, applying multiple attention mechanisms to improve multi-scale spatial location and multi-task perception capabilities. The inner-IoU loss function is used to replace the original CIoU, improving learning ability for small lesion samples. Furthermore, the AKConv block replaces the traditional convolution Conv block to allow for the arbitrary sampling of targets of various sizes, reducing model parameters and enhancing disease detection. the experimental results using a self-built dataset demonstrate that the enhanced YOLOv8-RMDA exhibits superior detection capabilities in detecting small target disease areas, achieving an average accuracy of 93.04% in identifying early tea lesions. When compared to Faster R-CNN, MobileNetV2, and SSD, the average precision rates of YOLOv5, YOLOv7, and YOLOv8 have shown improvements of 20.41%, 17.92%, 12.18%, 12.18%, 10.85%, 7.32%, and 5.97%, respectively. Additionally, the recall rate (R) has increased by 15.25% compared to the lowest-performing Faster R-CNN model and by 8.15% compared to the top-performing YOLOv8 model. With an FPS of 132, YOLOv8-RMDA meets the requirements for real-time detection, enabling the swift and accurate identification of early tea diseases. This advancement presents a valuable approach for enhancing the ecological tea industry in Yunnan, ensuring its healthy development.

Enhanced Road Safety through Integrated Speed Breaker, Pothole, and Lane Detection System

Abstract: Amidst the rapid evolution of autonomous vehicle technologies, ensuring road safety remains a paramount challenge. Effective detection of lanes and potential hazards, including speed breakers and potholes, is critical for safe autonomous driving. In this study, we introduce an innovative Lane, Speed Breaker, and Pothole Detection System (LSPDS) utilizing YOLOv4 Tiny, a state-of-the-art object detection algorithm and computer vision techniques. Our system integrates computer vision and machine learning techniques for analysis of road conditions. By employing camera sensors, we capture the road scene and apply image processing algorithms to identify lanes, speed breakers, and potholes. Moreover, the system incorporates Firebase for user authentication and SMS services for real-time alerts. YOLOv4 Tiny is employed for accurate detection and classification of these features within the captured images, thereby enhancing the perception capabilities of autonomous vehicles.

Reconfigurable memlogic long wave infrared sensing with superconductors

Optical sensors with in-cell logic and memory capabilities offer new horizons in realizing machine vision beyond von Neumann architectures and have been attempted with two-dimensional materials, memristive oxides, phase-changing materials etc. Noting the unparalleled performance of superconductors with both quantum-limited optical sensitivities and ultra-wide spectrum coverage, here we report a superconducting memlogic long-wave infrared sensor based on the bistability in hysteretic superconductor-normal phase transition. Driven cooperatively by electrical and optical pulses, the device offers deterministic in-sensor switching between resistive and superconducting (hence dissipationless) states with persistence > 105 s. This results in a resilient reconfigurable memlogic system applicable for, e.g., encrypted communications. Besides, a high infrared sensitivity at 12.2 μm is achieved through its in-situ metamaterial perfect absorber design. Our work opens the avenue to realize all-in-one superconducting memlogic sensors, surpassing biological retina capabilities in both sensitivity and wavelength, and presents a groundbreaking opportunity to integrate visional perception capabilities into superconductor-based intelligent quantum machines.

Active Visual Perception Enhancement Method Based on Deep Reinforcement Learning

Traditional object detection methods using static cameras are constrained by their limited perspectives, hampering the effective detection of low-confidence targets. To address this challenge, this study introduces a deep reinforcement learning-based visual perception enhancement technique. This approach leverages pan–tilt–zoom (PTZ) cameras to achieve active vision, enabling them to autonomously make decisions and actions tailored to the current scene and object detection outcomes. This optimization enhances both the object detection process and information acquisition, significantly boosting the intelligent perception capabilities of PTZ cameras. Experimental findings demonstrate the robust generalization capabilities of this method across various object detection algorithms, resulting in an average confidence level improvement of 23.80%.

Advancements in Deep Learning for Autonomous Driving in Indian Road Conditions

Recent advancements in deep learning techniques tailored for autonomous driving in Indian road conditions are crucial for revolutionizing transportation systems. Indian roads present unique challenges, including unpredictable traffic patterns, diverse road infrastructures, and challenging weather conditions. Deep learning is pivotal in addressing these challenges, focusing on perception, decision-making, and control. Various architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Deep Reinforcement Learning (DRL), are analyzed for their efficacy in autonomous driving. Sensor fusion techniques enhance perception capabilities, such as lidar, radar, and camera data integration. Additionally, advancements in semantic segmentation and object detection algorithms improve scene understanding and obstacle recognition. Collaborative efforts between academia, industry, and government agencies are essential for accelerating the deployment of autonomous driving technology in Indian road conditions. In conclusion, leveraging cutting-edge deep learning methodologies enables safe and efficient navigation of autonomous vehicles, ushering in a transformative future in transportation systems.

Automatic extrinsic calibration for structured light camera and repetitive LiDARs

AbstractThe integration of camera and LiDAR technologies has the potential to significantly enhance construction robots’ perception capabilities by providing complementary construction information. Structured light cameras (SLCs) are a desirable alternative as they provide comprehensive information on construction defects. However, fusing these two types of information depends largely on the sensors’ relative positions, which can only be established through extrinsic calibration. This paper introduces a novel calibration algorithm considering a customized board for SLCs and repetitive LiDARs, which are designed to facilitate the automation of construction robots. The calibration board is equipped with four symmetrically distributed hemispheres, whose centers are obtained by fitting the spheres and adoption with the geometric constraints. Subsequently, the spherical centers serve as reference features to estimate the relationship between the sensors. These distinctive features enable our proposed method to only require one calibration board pose and minimize human intervention. We conducted both simulation and real-world experiments to assess the performance of our algorithm. And the results demonstrate that our method exhibits enhanced accuracy and robustness.

2D Ca2Nb3O10 Optoelectronic Neuromorphic Device for Ultrasensitive UV‐C Vision and Encrypted Communication

AbstractNeuromorphic device that percepts ultraviolet (UV) radiations is essential for supplementing human visual perception. However, UV‐responsive synapses with high sensitivity and wavelength selectivity have not yet been developed so far. Herein, 2D optoelectronic synapses that are able to distinguish between UV‐A, UV‐,B and UV‐C radiations using a single material are demonstrated. The synapses use Ca2Nb3O10 nanosheets with wide bandgap and persistent photoconductivity effect (PPC) as UV‐responsive layer. The devices can detect UV‐C rays at ultralow intensity of 70 nW cm−2, which is the lowest record among 2D UV synaptic devices. Moreover, such a device demonstrates UV image vision, recognition, and memorization functions based on its diverse synaptic behaviors. The atomically thin UV synapses provide implications for enhancing human visual perception capabilities with spectral selectivity, ultrahigh sensitivity, and low‐energy consumption.

Detection of Military Targets on Ground and Sea by UAVs with Low-Altitude Oblique Perspective

Small-scale low-altitude unmanned aerial vehicles (UAVs) equipped with perception capability for military targets will become increasingly essential for strategic reconnaissance and stationary patrols in the future. To respond to challenges such as complex terrain and weather variations, as well as the deception and camouflage of military targets, this paper proposes a hybrid detection model that combines Convolutional Neural Network (CNN) and Transformer architecture in a decoupled manner. The proposed detector consists of the C-branch and the T-branch. In the C-branch, Multi-gradient Path Network (MgpNet) is introduced, inspired by the multi-gradient flow strategy, excelling in capturing the local feature information of an image. In the T-branch, RPFormer, a Region–Pixel two-stage attention mechanism, is proposed to aggregate the global feature information of the whole image. A feature fusion strategy is proposed to merge the feature layers of the two branches, further improving the detection accuracy. Furthermore, to better simulate real UAVs’ reconnaissance environments, we construct a dataset of military targets in complex environments captured from an oblique perspective to evaluate the proposed detector. In ablation experiments, different fusion methods are validated, and the results demonstrate the effectiveness of the proposed fusion strategy. In comparative experiments, the proposed detector outperforms most advanced general detectors.

Bioinspired sensors and applications in intelligent robots: a review

PurposeVision, audition, olfactory, tactile and taste are five important senses that human uses to interact with the real world. As facing more and more complex environments, a sensing system is essential for intelligent robots with various types of sensors. To mimic human-like abilities, sensors similar to human perception capabilities are indispensable. However, most research only concentrated on analyzing literature on single-modal sensors and their robotics application.Design/methodology/approachThis study presents a systematic review of five bioinspired senses, especially considering a brief introduction of multimodal sensing applications and predicting current trends and future directions of this field, which may have continuous enlightenments.FindingsThis review shows that bioinspired sensors can enable robots to better understand the environment, and multiple sensor combinations can support the robot’s ability to behave intelligently.Originality/valueThe review starts with a brief survey of the biological sensing mechanisms of the five senses, which are followed by their bioinspired electronic counterparts. Their applications in the robots are then reviewed as another emphasis, covering the main application scopes of localization and navigation, objection identification, dexterous manipulation, compliant interaction and so on. Finally, the trends, difficulties and challenges of this research were discussed to help guide future research on intelligent robot sensors.

Vision-Locomotion Coordination Control for a Powered Lower-Limb Prosthesis Using Fuzzy-Based Dynamic Movement Primitives

An amputee cannot directly use the perceptual visual information to control the movements and gait patterns of his worn prosthesis. In order to help an amputee walk and cross over obstacles smoothly, the paper proposes a vision-locomotion coordination control method for a powered lower-limb prosthesis (PLLP), in which a vision system is proposed to detect obstacles, and a complete vision-locomotion loop is then constructed. With deep learning techniques, the vision system can recognize common obstacles (e.g., garbage cans, bricks and boxes) and obtain the features of obstacles (e.g., distance from obstacles to the depth camera and height of obstacles). Through integrating the vision system into the locomotion control system, the PLLP can make obstacle avoidance decisions and use dynamic movement primitives with type-2 fuzzy models (T2FDMPs) to help amputees cross over obstacles simultaneously. Utilizing the type-2 fuzzy models, smooth trajectories for crossing over obstacles can be obtained. The experimental results show that the PLLP with visual information can switch an amputee’s gaits between level walking and obstacle avoidance adaptively, which demonstrates the effectiveness of the vision-locomotion coordination control system. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The paper is motivated by the challenge of obstacle avoidance of prostheses. Traditional prostheses cannot achieve autonomous obstacle avoidance because they lack of environmental perception capability. In addition, most prostheses always utilize use the human-robot interaction between amputees and prostheses to recognize the environments. However, due to noise and individual differences, recognition results are not accurate enough. We found that the integration of vision can improve the accuracy and efficiency of environmental recognition. Therefore, it is necessary to construct a complete vision and locomotion closed-loop. In the paper, a vision-locomotion coordination control is proposed, and to make the PLLP cross over obstacles smoothly, a novel trajectory shaping with fuzzy-based dynamic movement primitives is developed. The coordination control is partitioned into the obstacle detection, trajectory shaping and joint control, which can help the PLLP fulfill several obstacle avoidance tasks.

Wearable apparatus for correction of visual alignment under torsional strabismus.

A wearable optical apparatus that compensates for eye misalignment (strabismus) to correct for double vision (diplopia) is proposed. In contrast to prism lenses, commonly used to compensate for horizontal and/or vertical misalignment, the proposed approach is able to compensate for any combination of horizontal, vertical, and torsional misalignment. If the action of the extraocular muscles is compromised (e.g., by nerve damage), a patient may lose their ability to maintain visual alignment, negatively affecting their binocular fusion and stereo depth perception capability. Torsional misalignment cannot be mitigated by standard Fresnel prism lenses. Surgical procedures intended to correct torsional misalignment may be unpredictable. A wearable device able to rectify visual alignment and restore stereo depth perception without surgical intervention could potentially be of great value to people with strabismus. We propose a novel lightweight wearable optical device for visual alignment correction. The device comprises two mirrors and a Fresnel prism, arranged in such a way that together they rotationally shift the view seen by the affected eye horizontally, vertically, and torsionally. The extent of the alignment correction on each axis can be arbitrarily adjusted according to the patient's particular misalignment characteristics. The proposed approach was tested by computer simulation, and a prototype device was manufactured. The prototype device was tested by a strabismus patient exhibiting horizontal and torsional misalignment. In these tests, the device was found to function as intended, allowing the patient to enjoy binocular fusion and stereo depth perception while wearing the device for daily activities over a period of several months. The proposed device is effective in correcting arbitrary horizontal, vertical, and torsional misalignment of the eyes. The results of the initial testing performed are highly encouraging. Future study is warranted to formally assess the effectiveness of the device on multiple test patients.

HOW CEO TECHNICAL BACKGROUND CHARACTERISTICS AFFECT THE EFFECTIVENESS OF DIGITAL TRANSFORMATION OF MANUFACTURING ENTERPRISES - THE MEDIATING ROLE OF DYNAMIC CAPABILITIES

The digital transformation of manufacturing enterprises has an important impact on both the economy and society and the enterprise itself, and the CEO has an important impact on the effect of the digital transformation of manufacturing enterprises. The 20-Shenzhen A-share listed manufacturing companies from 2016 to 2021 are used as research subjects. The effect of the CEO's technical background on the digital transformation of these companies is studied empirically, and the dynamic capability is broken down into external perception capability and internal research and development capability to test their mediating effect. The research found that the technical background characteristics of the CEO significantly affect the effectiveness of digital transformation in manufacturing enterprises; the technical background characteristics of the CEO can influence the strategic decision and implementation of manufacturing enterprises; and the research conclusion enriches the related research on digital transformation and high-order theory in manufacturing enterprises and also provides theoretical reference for the training of the CEO of Chinese manufacturing enterprises.