Perceptual Capabilities Research Articles

Hazard and Safety Analysis of Machine-Learning-Based Perception Capabilities in Autonomous Vehicles

Hazard and Safety Analysis of Machine-Learning-Based Perception Capabilities in Autonomous Vehicles

The impact of servitization transformation on the ESG performance of manufacturing firms

In the new era of green and low-carbon economic transformation, servitization transformation enhances the production efficiency and perceptual capabilities of manufacturing firms and offers a novel approach to fulfilling environmental, social, and governance (ESG) responsibilities. This study employs a sample of Chinese-listed manufacturing firms from 2009 to 2021 to provide a theoretical analysis of the intrinsic mechanisms by which servitization transformation impacts ESG performance, including the roles played by total factor productivity, human capital, and firm goodwill. We then empirically examine the impact and mechanisms by which servitization transformation impacts the ESG performance of firms, finding the following. First, servitization transformation significantly improves ESG performance and this conclusion still holds following a series of robustness tests. Second, servitization transformation enhances ESG performance through the following three channels: improving total factor productivity, optimizing the structure of human capital, and accumulating firm goodwill. Third, the ESG performance of state-owned, high-tech, and large firms is more sensitive to servitization transformation. This study enriches and expands the application of servitization transformation research to manufacturing, providing policy insights that would allow manufacturing firms to better fulfill their ESG responsibilities and promote green and sustainable economic development.

BGF-YOLOv10: Small Object Detection Algorithm from Unmanned Aerial Vehicle Perspective Based on Improved YOLOv10

With the rapid development of deep learning, unmanned aerial vehicles (UAVs) have acquired intelligent perception capabilities, demonstrating efficient data collection across various fields. In UAV perspective scenarios, captured images often contain small and unevenly distributed objects, and are typically high-resolution. This makes object detection in UAV imagery more challenging compared to conventional detection tasks. To address this issue, we propose a lightweight object detection algorithm, BGF-YOLOv10, specifically designed for small object detection, based on an improved version of YOLOv10n. First, we introduce a novel YOLOv10 architecture tailored for small objects, incorporating BoTNet, variants of C2f and C3 in the backbone, along with an additional small object detection head, to enhance detection performance for small objects. Second, we embed GhostConv into both the backbone and head, effectively reducing the number of parameters by nearly half. Finally, we insert a Patch Expanding Layer module in the neck to restore the feature spatial resolution. Experimental results on the VisDrone-DET2019 and UAVDT datasets demonstrate that our method significantly improves detection accuracy compared to YOLO series networks. Moreover, when compared to other state-of-the-art networks, our approach achieves a substantial reduction in the number of parameters.

CSSDet: small object detection via cross-scale feature enhancement on drone-view images

ABSTRACT Object detection on drone-view images is vital for applications like intelligent transportation, abnormal behavior detection, and urban surveillance. However, the diverse perspectives and altitudes from which Unmanned Aerial Vehicle (UAV) capture scenes pose challenges due to multi-scale differences and small object sizes. To address this, we propose CSSDet, emphasizing multi-scale object detection and small object enhancement. CSSDet integrates an additional detection head for small objects and employs a bidirectional weighted feature pyramid network for effective cross-scale feature fusion, enhancing global perceptual capability. Additionally, we introduce coordinate attention and Involution modules to mitigate information loss. Experiments are conducted on two publicly available datasets, VisDrone and UAVDT. Quantitative results from the experiments indicate that the proposed CSSDet achieves optimal performance across a wide range of evaluation metrics, with particular strengths demonstrated in multi-scale object detection and small-size object detection. Qualitative results showcase its remarkable effectiveness in various aspects of object detection, including small object detection, objects at different scales and categories, multi-perspective scenarios, and complex scenes. The substantial experimental findings confirm that CSSDet attains comprehensive object detection performance on drone-view images. The source code is available at https://github.com/Gui-Cheng/CSSDet.

Cascaded hybrid convolutional autoencoder network for spectral-spatial nonlinear hyperspectral unmixing

ABSTRACT Due to the complex interaction of photons between materials, linear hyperspectral unmixing (HU) is limited in real scenarios, making nonlinear unmixing a promising alternative. With the advancement of deep learning (DL), nonlinear unmixing methods based on the convolutional autoencoder (CAE) have gained considerable traction in HU. However, these unmixing methods struggle to integrate spectral and spatial information while reducing the loss of material details during the unmixing process. Therefore, we propose a cascaded hybrid CAE nonlinear unmixing network, called CHCANet, which effectively leverages convolutional combinations to deeply explore the spectral-spatial information from hyperspectral data and preserve the material details through self-perception. Specifically, each CAE in CHCANet combines 1-D and 2-D convolutions, fully utilizing the flexibility and simplicity of 1-D convolutions to capture spectral features and the spatial correlation handling capability of the 2-D convolution. Moreover, we apply the self-perception mechanism to the nonlinear HU task, which can establish the cycle consistency of the network, strengthen mutual connections between encoders, and effectively preserve high-level semantic information. Following this, the optimized self-perception loss further enhances CHCANet’s perception capability of nonlinear components and strengthens the connection between the decoder directly associated with image reconstruction. Extensive experiments on synthetic and real datasets demonstrate the effectiveness of CHCANet and show excellent competitiveness compared to state-of-the-art unmixing methods.

MM-IRSTD: Conv Self-Attention-Based Multi-Modal Small and Dim Target Detection in Infrared Dual-Band Images

Infrared multi-band small and dim target detection is an important research direction in the fields of modern remote sensing and military surveillance. However, achieving high-precision detection remains challenging due to the small scale, low contrast of small and dim targets, and their susceptibility to complex background interference. This paper innovatively proposes a dual-band infrared small and dim target detection method (MM-IRSTD). In this framework, we integrate a convolutional self-attention mechanism module and a self-distillation mechanism to achieve end-to-end dual-band infrared small and dim target detection. The Conv-Based Self-Attention module consists of a convolutional self-attention mechanism and a multilayer perceptron, effectively extracting and integrating input features, thereby enhancing the performance and expressive capability of the model. Additionally, this module incorporates a dynamic weight mechanism to achieve adaptive feature fusion, significantly reducing computational complexity and enhancing the model’s global perception capability. During model training, we use a spatial and channel similarity self-distillation mechanism to drive model updates, addressing the similarity discrepancy between long-wave and mid-wave image features extracted through deep learning, thus improving the model’s performance and generalization capability. Furthermore, to better learn and detect edge features in images, this paper designs an edge extraction method based on Sobel. Finally, comparative experiments and ablation studies validate the advancement and effectiveness of our proposed method.

Bioinspired Photoluminescent "Spider Web" as Ultrafast and Ultrasensitive Airflow-Acoustic Bimodal Sensor for Human-Computer Interaction and Intelligent Recognition.

Nature provides massive biomimetic design inspiration for constructing structural materials with desired performances. Spider webs can perceive vibrations generated by airflow and acoustic waves from prey and transfer the corresponding information to spiders. Herein, by mimicking the perception capability and structure of a spider web, an ultrafast and ultrasensitive airflow-acoustic bimodal sensor (HOF-TCPB@SF) is developed based on the postfunctionalization of hydrogen-bonded organic framework (HOF-TCPB) on silk film (SF) through hydrogen bonds. The "spider web-like" HOF-TCPB@SF possesses light weight and high elasticity, endowing this airflow sensor with superior properties including an ultralow detection limit (DL, 0.0076 m s-1), and excellent repeatability (480 cycles). As an acoustic sensor, HOF-TCPB@SF exhibits ultrahigh sensitivity (105140.77 cps Pa-1 cm-2) and ultralow DL (0.2980 dB), with the greatest response frequency of 375 Hz and the ability to identify multiple sounds. Moreover, both airflow and acoustic sensing processes show an ultrafast response speed (40 ms) and multiangle recognition response (0-180°). The perception mechanisms of airflow and acoustic stimuli are analyzed through finite element simulation. This bimodal sensor also achieves real-time airflow monitoring, speech recognition, and airflow-acoustic interoperability based on human-computer interaction, which holds great promise for applications in health care, tunnel engineering, weather forecasting, and intelligent textiles.

An Enhanced Fault Localization Technique for Distribution Networks Utilizing Cost-Sensitive Graph Neural Networks

Accurate and timely fault diagnosis is of great significance for the stable operation of a distribution network. Traditional artificial intelligence-based localization methods rely heavily on large-scale labeled datasets, making them prone to being affected by distributed generators. To address this issue, this study proposes a fault location method for distribution systems using a cost-sensitive graph attention network model. In this approach, the physical structure of the distribution network is considered a crucial constraint for model training, thereby enhancing its information perception capabilities. Specifically, the electrical nodes and lines of the distribution network are mapped to the vertices and edges within the graph attention network, with the attention weights determined by the correlation of adjacent node fault characteristics, thus improving the sensitivity to grounding faults. Additionally, a cost-sensitive matrix is used to balance class distribution, enhancing the robustness and generalization ability of the model. Fault localization experiments were conducted on the IEEE-33 bus distribution system to validate the effectiveness of the proposed fault localization method. Factors such as data disturbance, varying fault grounding resistances, and distributed power supply access were employed to assess the model’s anti-interference performance. The experimental results demonstrate that the fault location method exhibits high positioning accuracy and excellent robustness.

Hybrid Extreme Learning for Reliable Short-Term Traffic Flow Forecasting

Reliable forecasting of short-term traffic flow is an essential component of modern intelligent transport systems. However, existing methods fail to deal with the non-linear nature of short-term traffic flow, often making the forecasting unreliable. Herein, we propose a reliable short-term traffic flow forecasting method, termed hybrid extreme learning, that effectively learns the non-linear representation of traffic flow, boosting forecasting reliability. This new algorithm probes the non-linear nature of short-term traffic data by exploiting the artificial bee colony that selects the best-implied layer deviation and input weight matrix to enhance the multi-structural information perception capability. It speeds up the forecasting time by calculating the output weight matrix, which guarantees the real usage of the forecasting method, boosting the time reliability. We extensively evaluate the proposed hybrid extreme learning method on well-known short-term traffic flow forecasting datasets. The experimental results show that our method outperforms existing methods by a large margin in both forecasting accuracy and time, effectively demonstrating the reliability improvement of the proposed method. This reliable method may open the avenue of deep learning techniques in short-term traffic flow forecasting in real scenarios.

Unsupervised anomaly detection in shearers via autoencoder networks and multi-scale correlation matrix reconstruction

As the main equipment of coal mining production, the anomaly detection of shearer is important to ensure production efficiency and coal mine safety. One key challenge lies in the limited or even absence of labeled monitoring data for the equipment, coupled with the high costs associated with manual annotation. Another challenge stems from the complex structure of the mining machines, making it difficult to reflect the overall operational state through local anomaly detection. Consequently, the application of decoupled local anomaly detection for mining machines in practical production remains challenging. This paper presents an unsupervised learning-based method for detecting anomalies in shearer. The method includes a module for constructing a Multi-scale Correlation Matrix (MSCM) of mining machine operating conditions, as well as the CNN-ConvLSTM Autoencoder (C-CLA) network. The module for constructing an MSCM enhances the representation of interrelationships between various features of the equipment from different perspectives using multiple correlation analysis methods. The C-CLA network integrates convolutional and convolutional recurrent neural networks, with the convolutional structure extracting local spatial features and the ConvLSTM structure further capturing information from different time scales and feature scales, thereby enhancing the model’s perceptual capabilities towards changes in equipment status. Finally, shearer anomaly detection is achieved through the analysis of reconstructed residual matrices. The rationality and practicality of the proposed method have been validated on our dataset, and the model’s generalization capability has been verified through repeated experiments in similar scenarios. However, due to variations in the working environment of different mining faces and differences in equipment models, implementing detection on other mining faces often requires retraining the model with new data. Furthermore, we compared our method with other anomaly detection techniques, and our detection efficiency was superior by approximately 3%. This method effectively detects anomalies in the shearer.

Self‐Sensing Artificial‐Muscle‐Empowered Humanlike Perception, Interaction, and Positioning

Bioinspired soft and hybrid robots provide a promising solution to developing robots that can interact and collaborate with humans safely and effectively. Bellow‐like pneumatic artificial muscles can produce biological muscle‐like contraction with comparable response time and force output, but closed‐loop control has been a challenge without an effective length sensing solution. Herein, a self‐sensing artificial muscle (SSAM), which can sense its own length regardless of the external loadings and driving pressures, is proposed. Empowered by a seamlessly integrated mutual‐inductance‐based length sensor (MILS), the SSAM can sense its length change as small as 0.01 mm (0.012% of the initial liength 80 mm) in a wide range. The working principle of the MILS is analyzed theoretically to provide a design guideline. An SSAM‐based earthworm‐like locomotion robot is developed with the capability of knowing its real‐time body length change at different gaits. An artificial arm driven by one SSAM is also developed and demonstrated with humanlike capabilities of loading perception, interactive and responsive movements, and accurate positioning. This work provides a promising solution to develop muscle‐driven hybrid robotic systems with embodied intelligence for skilled manipulation and sophisticated human–machine interactions.

Optical Micro/Nanofiber Enabled Multiaxial Force Sensor for Tactile Visualization and Human-Machine Interface.

Tactile sensors with capability of multiaxial force perception play a vital role in robotics and human-machine interfaces. Flexible optical waveguide sensors have been an emerging paradigm in tactile sensing due to their high sensitivity, fast response, and antielectromagnetic interference. Herein, a flexible multiaxial force sensor enabled by U-shaped optical micro/nanofibers (MNFs) is reported. The MNF is embedded within an elastomer film topped with a dome-shaped protrusion. When the protrusion is subjected to vector forces, the embedded MNF undergoes anisotropic deformations, yielding time-resolved variations in light transmission. Detection of both normal and shear forces is achieved with sensitivities reaching 50.7dB N-1 (14% kPa-1) and 82.2dB N-1 (21% kPa-1), respectively. Notably, the structural asymmetry of the MNF induces asymmetrical optical modes, granting the sensor directional responses to four-directional shear forces. As proof-of-concept applications, tactile visualizations for texture and relief pattern recognition are realized with a spatial resolution of 160µm. Moreover, a dual U-shaped MNF configuration is demonstrated as a human-machine interface for cursor manipulation. This work represents a step towards advanced multiaxial tactile sensing.

Enhanced neural phase locking through audio-tactile stimulation.

Numerous studies have underscored the close relationship between the auditory and vibrotactile modality. For instance, in the peripheral structures of both modalities, afferent nerve fibers synchronize their activity to the external sensory stimulus, thereby providing a temporal code linked to pitch processing. The Frequency Following Response is a neurological measure that captures this phase locking activity in response to auditory stimuli. In our study, we investigated whether this neural signal is influenced by the simultaneous presentation of a vibrotactile stimulus. Accordingly, our findings revealed a significant increase in phase locking to the fundamental frequency of a speech stimulus, while no such effects were observed at harmonic frequencies. Since phase locking to the fundamental frequency has been associated with pitch perceptual capabilities, our results suggests that audio-tactile stimulation might improve pitch perception in human subjects.

Artistic-style text detector and a new Movie-Poster dataset

Although current text detection algorithms demonstrate effectiveness in general scenarios, their performance declines when confronted with artistic-style text featuring complex structures. This paper proposes a method that utilizes the Criss-Cross Attention and the residual dense block to address the incomplete and misdiagnosis of artistic-style text detection by current algorithms. Specifically, our method mainly consists of a feature extraction backbone, a Recycle Criss-Cross Attention module, a Residual Feature Pyramid Network, and a Boundary Discrimination Module. The Recycle Criss-Cross Attention module significantly enhances the model’s perceptual capabilities in complex environments by fusing horizontal and vertical contextual information, allowing it to capture detailed features overlooked in artistic-style text. We incorporate the residual dense block into the feature pyramid network to suppress the effect of background noise during feature fusion. Aiming to omit the complex post-processing, we explore a Boundary Discrimination Module that guides the correct generation of boundary proposals. Furthermore, given that movie poster titles often use stylized art fonts, we collected a Movie-Poster dataset to address the scarcity of artistic-style text data. Extensive experiments demonstrate that our proposed method performs superiorly on the Movie-Poster dataset and produces excellent results on multiple benchmark datasets. The code and the Movie-Poster dataset will be available at: https://github.com/AXNing/Artistic-style-text-detection.

A cooperative perception based adaptive signal control under early deployment of connected and automated vehicles

Connected vehicle-based adaptive traffic signal control requires certain market penetration rates (MPRs) to be effective, usually exceeding 10%. Cooperative perception based on connected and automated vehicles (CAVs) can effectively improve overall data collection efficiency and reduce required MPR. However, the distribution of observed vehicles under cooperative perception is highly skewed and imbalanced, especially under very low CAV MPRs (e.g., 1%). To address this challenge, this paper proposes a novel deep reinforcement learning-based adaptive traffic signal control (RL-TSC) method that integrates a traffic flow model, known as the cell transmission model (CTM), denoted as CAVLight. Traffic states estimated from the CTM are integrated with the data collected from the cooperative perception environment to update the states in the CAVLight model. The design of reward function aims for reducing total vehicle delays and stabilizing agent behaviors. Extensive numerical experiments under a real-world intersection with varying traffic demand levels and CAV MPRs are conducted to compare the performance of CAVLight and other benchmark algorithms, including a fixed-time controller, an actuated controller, the max pressure model, and an optimization-based adaptive TSC. Results demonstrate the superiority of CAVLight in performance and generalizability over benchmarks, especially under 1% CAV MPR scenario with high traffic demands. The influence of CTM integration on CAVLight is further explored through RL agent policy visualization and sensitivity analysis in CTM parameters and CAV perception capabilities (i.e., detection range and detection accuracy).

Minimal perception: enabling autonomy in resource-constrained robots.

The rapidly increasing capabilities of autonomous mobile robots promise to make them ubiquitous in the coming decade. These robots will continue to enhance efficiency and safety in novel applications such as disaster management, environmental monitoring, bridge inspection, and agricultural inspection. To operate autonomously without constant human intervention, even in remote or hazardous areas, robots must sense, process, and interpret environmental data using only onboard sensing and computation. This capability is made possible by advancements in perception algorithms, allowing these robots to rely primarily on their perception capabilities for navigation tasks. However, tiny robot autonomy is hindered mainly by sensors, memory, and computing due to size, area, weight, and power constraints. The bottleneck in these robots lies in the real-time perception in resource-constrained robots. To enable autonomy in robots of sizes that are less than 100 mm in body length, we draw inspiration from tiny organisms such as insects and hummingbirds, known for their sophisticated perception, navigation, and survival abilities despite their minimal sensor and neural system. This work aims to provide insights into designing a compact and efficient minimal perception framework for tiny autonomous robots from higher cognitive to lower sensor levels.

Super-resolution water body mapping with a feature collaborative CNN model by fusing Sentinel-1 and Sentinel-2 images

Mapping water bodies from remotely sensed imagery is crucial for understanding hydrological and biogeochemical processes. The identification of water extent is mainly dependent on optical and synthetic aperture radar (SAR) images. However, the use of remote sensing for water body mapping is often undermined by the mixed pixel dilemma inherent to traditional hard classification approaches. At the same time, the presence of clouds in optical imagery and speckle noise in SAR imagery, coupled with the difficulty in differentiating between water-like surfaces and actual water bodies, significantly compromise the accuracy of water body identification. This paper proposes a DEEP feature collaborative convolutional neural network (CNN) for Water Super-Resolution Mapping based on Optical and SAR images (DeepOSWSRM), which collaboratively leverages Sentinel-1 and Sentinel-2 imagery to address the challenges of missing data and mixed pixels. The Sentinel-1 image provides complementary water distribution information for the cloudy areas of the Sentinel-2 image, while the Sentinel-2 image enhances the perception capabilities for small water bodies in the Sentinel-1 image. Using PlanetScope imagery as the true reference data, the effectiveness of the proposed method was assessed through two experimental scenarios: one utilizing synthetic coarse-resolution imagery degraded from Sentinel-1 and Sentinel-2 data and another using actual Sentinel-1 and Sentinel-2 data, encompassing both simulated and real cloud conditions. A comparative analysis was conducted against three state-of-the-art CNN-based water mapping methods and two CNN SRM methods. The findings demonstrate that the proposed DeepOSWSRM method successfully produces accurate, fine-resolution water body maps, with its performance mainly benefiting from the fusion of SAR and optical images.

Enhancing Inter-AUV Perception: Adaptive 6-DOF Pose Estimation with Synthetic Images for AUV Swarm Sensing

The capabilities of AUV mutual perception and localization are crucial for the development of AUV swarm systems. We propose the AUV6D model, a synthetic image-based approach to enhance inter-AUV perception through 6D pose estimation. Due to the challenge of acquiring accurate 6D pose data, a dataset of simulated underwater images with precise pose labels was generated using Unity3D. Mask-CycleGAN technology was introduced to transform these simulated images into realistic synthetic images, addressing the scarcity of available underwater data. Furthermore, the Color Intermediate Domain Mapping strategy is proposed to ensure alignment across different image styles at pixel and feature levels, enhancing the adaptability of the pose estimation model. Additionally, the Salient Keypoint Vector Voting Mechanism was developed to improve the accuracy and robustness of underwater pose estimation, enabling precise localization even in the presence of occlusions. The experimental results demonstrated that our AUV6D model achieved millimeter-level localization precision and pose estimation errors within five degrees, showing exceptional performance in complex underwater environments. Navigation experiments with two AUVs further verified the model’s reliability for mutual 6D pose estimation. This research provides substantial technical support for more complex and precise collaborative operations for AUV swarms in the future.

Soft Magnetoelastic Tactile Multi-Sensors with Energy-Absorbing Properties for Self-Powered Human-Machine Interfaces.

Tactile sensors play a key role in human-machine interfaces (HMIs) for augmented and virtual reality, point-of-care devices, and human-robot collaboration, which show the promise of revolutionizing our ways of life. Here, we present a sensor (EMTS) that utilizes the magnetoelastic effect in a soft metamaterial to convert mechanical pressure into electrical signals. With this unique mechanism, the proposed EMTS simultaneously possesses self-powering, waterproof, and compliant features. The soft metamaterial is essentially a porous magnetoelastomer structure designed based on the Fourier series expansion, which allows for programmable mechanical response and sensing performance of the EMTS. Fabricated by simple 3D-printed molds, the EMTS also holds potential for low-cost production. Particularly, the porous magnetoelastomer structure comes with selectable buckling instabilities that can significantly enhance biomechanical-to-electrical energy conversion. Also, with the embedded magnetic microparticles, the energy-absorbing performance of the sensor is greatly improved, which is highly beneficial to HMIs. To pursue practical applications, the EMTSs are further integrated with two systems as control and perception modules. It is demonstrated that the EMTS is able to identify different hand gestures to control a lighting system even in a high-humidity environment. Also, the EMTS stands out for its superior capability of simultaneous impact perception and energy absorption in drop tests. Overall, with its compelling array of features, the presented EMTS gives impetus to multi-sensing technology and practically enables a variety of HMI applications.

VLAI: Exploration and Exploitation based on Visual-Language Aligned Information for Robotic Object Goal Navigation

Object Goal Navigation(ObjectNav) is the task that an agent need navigate to an instance of a specific category in an unseen environment through visual observations within limited time steps. This work plays a significant role in enhancing the efficiency of locating specific items in indoor spaces and assisting individuals in completing various tasks, as well as providing support for people with disabilities. To achieve efficient ObjectNav in unfamiliar environments, global perception capabilities, understanding the regularities of space and semantics in the environment layout are significant. In this work, we propose an explicit-prediction method called VLAI that utilizes visual-language alignment information to guide the agent's exploration, unlike previous navigation methods based on frontier potential prediction or egocentric map completion, which only leverage visual observations to construct semantic maps, thus failing to help the agent develop a better global perception. Specifically, when predicting long-term goals, we retrieve previously saved visual observations to obtain visual information around the frontiers based on their position on the incrementally built incomplete semantic map. Then, we apply our designed Chat Describer to this visual information to obtain detailed frontier object descriptions. The Chat Describer, a novel automatic-questioning approach deployed in Visual-to-Language, is composed of Large Language Model(LLM) and the visual-to-language model(VLM), which has visual question-answering functionality. In addition, we also obtain the semantic similarity of target object and frontier object categories. Ultimately, by combining the semantic similarity and the boundary descriptions, the agent can predict the long-term goals more accurately. Our experiments on the Gibson and HM3D datasets reveal that our VLAI approach yields significantly better results compared to earlier methods. The code is released athttps://github.com/31539lab/VLAI.