Scene Information Research Articles

Wakeup-Darkness: When Multimodal Meets Unsupervised Low-light Image Enhancement

Low-light image enhancement is a crucial visual task, and many unsupervised methods overlook the degradation of visible information in low-light scenes, adversely affecting the fusion of complementary information and hindering the generation of satisfactory results. To address this, we introduce Wakeup-Darkness, a multimodal enhancement framework that innovatively enriches user interaction through voice and textual commands. This approach signifies a technical leap and represents a paradigm shift in user engagement. We introduce a cross-modal feature fusion (CMFF) that synergizes semantic and depth context with low-light enhancement operations. Moreover, We propose a gated residual block (GRB) and a channel-aware look-up table (LUT) to adjust the intensity distribution of each channel. Crucially, the proposed Wakeup-Darkness scheme demonstrates remarkable generalization in unsupervised scenarios. The source code can be accessed from https://github.com/zhangbaijin/Wakeup-Dakness .

Scene-Dependent Prediction in Latent Space for Video Anomaly Detection and Anticipation.

Video anomaly detection (VAD) plays a crucial role in intelligent surveillance. However, an essential type of anomaly named scene-dependent anomaly is overlooked. Moreover, the task of video anomaly anticipation (VAA) also deserves attention. To fill these gaps, we build a comprehensive dataset named NWPU Campus, which is the largest semi-supervised VAD dataset and the first dataset for scene-dependent VAD and VAA. Meanwhile, we introduce a novel forward-backward framework for scene-dependent VAD and VAA, in which the forward network individually solves the VAD and jointly solves the VAA with the backward network. Particularly, we propose a scene-dependent generative model in latent space for the forward and backward networks. First, we propose a hierarchical variational auto-encoder to extract scene-generic features. Next, we design a score-based diffusion model in latent space to refine these features more compact for the task and generate scene-dependent features with a scene information auto-encoder, modeling the relationships between video events and scenes. Finally, we develop a temporal loss from key frames to constrain the motion consistency of video clips. Extensive experiments demonstrate that our method can handle both scene-dependent anomaly detection and anticipation well, achieving state-of-the-art performance on ShanghaiTech, CUHK Avenue, and the proposed NWPU Campus datasets.

Realisation of an Application Specific Multispectral Snapshot-Imaging System Based on Multi-Aperture-Technology and Multispectral Machine Learning Loops.

Multispectral imaging (MSI) enables the acquisition of spatial and spectral image-based information in one process. Spectral scene information can be used to determine the characteristics of materials based on reflection or absorption and thus their material compositions. This work focuses on so-called multi aperture imaging, which enables a simultaneous capture (snapshot) of spectrally selective and spatially resolved scene information. There are some limiting factors for the spectral resolution when implementing this imaging principle, e.g., usable sensor resolutions and area, and required spatial scene resolution or optical complexity. Careful analysis is therefore needed for the specification of the multispectral system properties and its realisation. In this work we present a systematic approach for the application-related implementation of this kind of MSI. We focus on spectral system modeling, data analysis, and machine learning to build a universally usable multispectral loop to find the best sensor configuration. The approach presented is demonstrated and tested on the classification of waste, a typical application for multispectral imaging.

360 Layout Estimation via Orthogonal Planes Disentanglement and Multi-View Geometric Consistency Perception.

Existing panoramic layout estimation solutions tend to recover room boundaries from a vertically compressed sequence, yielding imprecise results as the compression process often muddles the semantics between various planes. Besides, these data-driven approaches impose an urgent demand for massive data annotations, which are laborious and time-consuming. For the first problem, we propose an orthogonal plane disentanglement network (termed DOPNet) to distinguish ambiguous semantics. DOPNet consists of three modules that are integrated to deliver distortion-free, semantics-clean, and detail-sharp disentangled representations, which benefit the subsequent layout recovery. For the second problem, we present an unsupervised adaptation technique tailored for horizon-depth and ratio representations. Concretely, we introduce an optimization strategy for decision-level layout analysis and a 1D cost volume construction method for feature-level multi-view aggregation, both of which are designed to fully exploit the geometric consistency across multiple perspectives. The optimizer provides a reliable set of pseudo-labels for network training, while the 1D cost volume enriches each view with comprehensive scene information derived from other perspectives. Extensive experiments demonstrate that our solution outperforms other SoTA models on both monocular layout estimation and multi-view layout estimation tasks.

Intrinsically anisotropic 1D NbTe4 for self-powered polarization-sensitive photodetection

Polarization-sensitive photodetection enhances scene information capture, crucial for modern optoelectronic devices. One-dimensional (1D) materials with intrinsic anisotropy, capable of directly sensing polarized light, are promising for such photodetectors. NbTe4, a quasi-1D transition metal tetra-chalcogenide, offers significant benefits for polarization-sensitive photodetection due to its structural anisotropy. Nonetheless, to date, the anisotropic properties of 1D NbTe4 have not been reported. Herein, NbTe4 nanobelts were synthesized via mechanical exfoliation from needle-like bulk crystals, and their anisotropic and optoelectronic properties were comprehensively studied. Angle-resolved polarized Raman spectroscopy, in conjunction with azimuth-dependent reflectance difference microscopy, confirmed that 1D NbTe₄ exhibits intrinsic structural and in-plane optical anisotropy. The 1D NbTe4 device demonstrated characteristic anisotropic photodetection behavior, achieving dichroic ratios of 1.16 at 671 nm and 1.25 at 1064 nm. Meanwhile, the device exhibits a pronounced photothermoelectric effect, conferring a broad spectral photoresponse ranging from visible to near-infrared wavelengths (532-1064 nm), with a rapid response time of 158 ms. This study demonstrates that NbTe4 inherently possesses in-plane anisotropy, making it a promising candidate for polarization-sensitive photodetection applications.

MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting

Crafting a single, versatile physics-based controller that can breathe life into interactive characters across a wide spectrum of scenarios represents an exciting frontier in character animation. An ideal controller should support diverse control modalities, such as sparse target keyframes, text instructions, and scene information. While previous works have proposed physically simulated, scene-aware control models, these systems have predominantly focused on developing controllers that each specializes in a narrow set of tasks and control modalities. This work presents MaskedMimic, a novel approach that formulates physics-based character control as a general motion inpainting problem. Our key insight is to train a single unified model to synthesize motions from partial (masked) motion descriptions, such as masked keyframes, objects, text descriptions, or any combination thereof. This is achieved by leveraging motion tracking data and designing a scalable training method that can effectively utilize diverse motion descriptions to produce coherent animations. Through this process, our approach learns a physics-based controller that provides an intuitive control interface without requiring tedious reward engineering for all behaviors of interest. The resulting controller supports a wide range of control modalities and enables seamless transitions between disparate tasks. By unifying character control through motion inpainting, MaskedMimic creates versatile virtual characters. These characters can dynamically adapt to complex scenes and compose diverse motions on demand, enabling more interactive and immersive experiences.

Scene-cGAN: A GAN for underwater restoration and scene depth estimation

Despite their wide scope of application, the development of underwater models for image restoration and scene depth estimation is not a straightforward task due to the limited size and quality of underwater datasets, as well as variations in water colours resulting from attenuation, absorption and scattering phenomena in the water column. To address these challenges, we present an all-in-one conditional generative adversarial network (cGAN) called Scene-cGAN. Our cGAN is a physics-based multi-domain model designed for image dewatering, restoration and depth estimation. It comprises three generators and one discriminator. To train our Scene-cGAN, we use a multi-term loss function based on uni-directional cycle-consistency and a novel dataset. This dataset is constructed from RGB-D in-air images using spectral data and concentrations of water constituents obtained from real-world water quality surveys. This approach allows us to produce imagery consistent with the radiance and veiling light corresponding to representative water types. Additionally, we compare Scene-cGAN with current state-of-the-art methods using various datasets. Results demonstrate its competitiveness in terms of colour restoration and its effectiveness in estimating the depth information for complex underwater scenes.

Robotic Grasping Detection Algorithm Based on 3D Vision Dual-Stream Encoding Strategy

The automatic generation of stable robotic grasping postures is crucial for the application of computer vision algorithms in real-world settings. This task becomes especially challenging in complex environments, where accurately identifying the geometric shapes and spatial relationships between objects is essential. To enhance the capture of object pose information in 3D visual scenes, we propose a planar robotic grasping detection algorithm named SU-Grasp, which simultaneously focuses on local regions and long-distance relationships. Built upon a U-shaped network, SU-Grasp introduces a novel dual-stream encoding strategy using the Swin Transformer combined with spatial semantic enhancement. Compared to existing baseline methods, our algorithm achieves superior performance across public datasets, simulation tests, and real-world scenarios, highlighting its robust understanding of complex spatial environments.

Adaptive Multi-Function Radar Temporal Behavior Analysis

The performance of radar mode recognition has been significantly enhanced by the various architectures of deep learning networks. However, these approaches often rely on supervised learning and are susceptible to overfitting on the same dataset. As a transitional phase towards Cognitive Multi-Functional Radar (CMFR), Adaptive Multi-Function Radar (AMFR) possesses the capability to emit identical waveform signals across different working modes and states for task completion, with dynamically adjustable waveform parameters that adapt based on scene information. From a reconnaissance perspective, the valid signals received exhibit sparsity and localization in the time series. To address this challenge, we have redefined the reconnaissance-focused research priorities for radar systems to emphasize behavior analysis instead of pattern recognition. Based on our initial comprehensive digital system simulation model of a radar, we conducted reconnaissance and analysis from the perspective of the reconnaissance side, integrating both radar and reconnaissance aspects into environmental simulations to analyze radar behavior under realistic scenarios. Within the system, waveform parameters on the radar side vary according to unified rules, while resource management and task scheduling switch based on operational mechanisms. The target in the reconnaissance side maneuvers following authentic behavioral patterns while adjusting the electromagnetic space complexity in the environmental aspect as required. The simulation results indicate that temporal annotations in signal flow data play a crucial role in behavioral analysis from a reconnaissance perspective. This provides valuable insights for future radar behavior analysis incorporating temporal correlations and sequential dependencies.

Integrating synthetic datasets with CLIP semantic insights for single image localization advancements

Accurate localization of pedestrians and mobile robots is critical for navigation, emergency response, and autonomous driving. Traditional localization methods, such as satellite signals, often prove ineffective in certain environments, and acquiring sufficient positional data can be challenging. Single image localization techniques have been developed to address these issues. However, current deep learning frameworks for single image localization that rely on domain adaptation fail to effectively utilize semantically rich high-level features obtained from large-scale pretraining. This paper introduces a novel framework that leverages the Contrastive Language-Image Pre-training model and prompts to enhance feature extraction and domain adaptation through semantic information. The proposed framework generates an integrated score map from scene-specific prompts to guide feature extraction and employs adversarial components to facilitate domain adaptation. Furthermore, a reslink component is incorporated to mitigate the precision loss in high-level features compared to the original data. Experimental results demonstrate that the use of prompts reduces localization errors by 26.4 % in indoor environments and 24.3 % in outdoor settings. The model achieves localization errors as low as 0.75 m and 8.09 degrees indoors, and 4.56 m and 7.68 degrees outdoors. Analysis of prompts from labeled datasets confirms the model’s capability to effectively interpret scene information. The weights of the integrated score map enhance the model’s transparency, thereby improving interpretability. This study underscores the efficacy of integrating semantic information into image localization tasks.

Object Tracking Algorithm Based on Integrated Multi-Scale Templates Guided by Judgment Mechanism

The object tracking algorithm TransT, based on Transformer, achieves significant improvements in accuracy and success rate by fusing the extracted features of convolutional neural networks with the structure of Transformer. However, when dealing with the deformation of the object’s appearance, the algorithm exhibits issues such as insufficient tracking accuracy and drift, which directly affect the stability of the algorithm. In order to overcome this problem, this paper demonstrates how to expand scene information at the scale level during the fusion process and, on this basis, achieve accurate recognition and positioning. The predicted results are promptly fed back to the subsequent tracking process, from which temporal templates are embedded. Starting from both location and time can effectively improve the adaptive ability of the tracking model. In the final experimental comparison results, the algorithm proposed in this paper can adapt well to the situation of object deformation, and the overall performance of the tracking model has also been improved.

Retinotopic biases in contextual feedback signals to V1 for object and scene processing

Identifying the objects embedded in natural scenes relies on recurrent processing between lower and higher visual areas. How is cortical feedback information related to objects and scenes organised in lower visual areas? The spatial organisation of cortical feedback converging in early visual cortex during object and scene processing could be retinotopically specific as it is coded in V1, or object centred as coded in higher areas, or both. Here, we characterise object and scene-related feedback information to V1. Participants identified foreground objects or background scenes in images with occluded central and peripheral subsections, allowing us to isolate feedback activity to foveal and peripheral regions of V1. Using fMRI and multivoxel pattern classification, we found that background scene information is projected to both foveal and peripheral V1 but can be disrupted in the fovea by a sufficiently demanding object discrimination task, during which we found evidence of foveal object decoding when using naturalistic stimuli. We suggest that the feedback connections during scene perception project back to earlier visual areas an automatic sketch of occluded information to the predicted retinotopic location. In the case of a cognitive task however, feedback pathways project content to foveal retinotopic space, potentially for introspection, functioning as a cognitive active blackboard and not necessarily predicting the object’s location. This feedback architecture could reflect the internal mapping in V1 of the brain’s endogenous models of the visual environment that are used to predict perceptual inputs.

Feature fusion and context interaction for RGB-D indoor semantic segmentation

Depth information in indoor scenes is crucial to understanding the spatial and occluding relationships of objects. Depth information contains rich geometric details and is unaffected by lighting, texture, and color, significantly enhancing traditional color images. RGB images and depth images have been widely applied in various image analysis. However, effectively leveraging the complementary information from both modalities remains a challenge. To address this issue, we propose a unified and effective feature fusion and context interaction network (FCINet). It includes three modules. Firstly, we construct a multi-modal feature fusion module (MFFM), which can achieve the aggregation of two modalities along spatial and channel dimensions. Secondly, we construct a global and local information interaction context module (GLCM) to encode rich semantic information. Finally, we construct a feature alignment and fusion module (FAFM), which integrates upsampled low-level features with high-level features to mitigate spatial information loss. Experimental results indicate that the model achieved favorable outcomes on both the NYU Depth v2 dataset and the more complex SUN RGB-D dataset.

Effect of attention on ensemble perception: Comparison between exogenous attention, endogenous attention, and depth.

Ensemble perception is an important ability of human beings that allows one to extract summary information for scenes and environments that contain information that far exceeds the processing limit of the visual system. Although attention has been shown to bias ensemble perception, two important questions remain unclear: (1) whether direct manipulations on different types of spatial attention could produce similar effects on ensembles and (2) whether factors potentially influencing the attention distribution, such as depth perception, could evoke an indirect effect of attention on ensemble representation. This study aims to address these questions. In Experiments 1 and 2, two types of precues were used to evoke exogenous and endogenous attention, respectively, and the ensemble color perceptions were examined. We found that both exogenous and endogenous attention biased ensemble representation towards the attended items, and the latter produced a greater effect. In Experiments 3 and 4, we examined whether depth perception could affect color ensembles by indirectly influencing attention allocation in 3D space. The items were separated in two depth planes, and no explicit cues were applied. The results showed that color ensemble was biased to closer items when depth information was task relevant. This suggests that ensemble perception is naturally biased in 3D space, probably through the mechanism of attention. Computational modeling consistently showed that attention exerted a direct shift on the ensemble statistics rather than averaging the feature values over the cued and noncued items, providing evidence against an averaging process of individual perception.

Configurable multiple virtual lenses conjugated with singlet physical lens for achromatic extended depth-of-field imaging.

An achromatic extended depth-of-field (EDOF) system can obtain clear scene information that is crucial for target recognition, dynamic monitoring, and other applications. However, the imaging performance of most optical systems is depth-variant and wavelength-variant, which leads to the generation of chromatic aberrations. Traditional optical design and image post-processing algorithms cannot effectively eliminate these chromatic aberrations. Here, we propose a deep configurable multiple virtual lenses optimization method that embeds four virtual lenses in parallel conjugated with a real lens. Combined with a lens fusion recovery network (LFRNet), it compensates for chromatic aberrations at different depths to achieve achromatic EDOF imaging. Trainable virtual optics can eliminate chromatic aberrations and overcome the limitations of traditional optics. The proposed framework reduces the optical design complexity and improves the imaging quality of a simple optical system. We validate our method using a singlet lens, and the experimental results show that the reconstructed images have an average peak signal-to-noise ratio (PSNR) improvement of 12.1447 dB and an average structural similarity index measure (SSIM) improvement of 0.2465. The proposed method opens a new avenue for ultra-compact, high-freedom, high-efficiency, and wholly configurable deep optics design, and empowers various advanced applications, such as portable photography and other complex vision tasks.

Deep and shallow feature fusion framework for remote sensing open pit coal mine scene recognition

Understanding land use and damage in open-pit coal mining areas is crucial for effective scientific oversight and management. Current recognition methods exhibit limitations: traditional approaches depend on manually designed features, which offer limited expressiveness, whereas deep learning techniques are heavily reliant on sample data. In order to overcome the aforementioned limitations, a three-branch feature extraction framework was proposed in the present study. The proposed framework effectively fuses deep features (DF) and shallow features (SF), and can accomplish scene recognition tasks with high accuracy and fewer samples. Deep features are enhanced through a neighbouring feature attention module and a Graph Convolutional Network (GCN) module, which capture both neighbouring features and the correlation between local scene information. Shallow features are extracted using the Gray-Level Co-occurrence Matrix (GLCM) and Gabor filters, which respectively capture local and overall texture variations. Evaluation results on the AID and RSSCN7 datasets demonstrate that the proposed deep feature extraction model achieved classification accuracies of 97.53% and 96.73%, respectively, indicating superior performance in deep feature extraction tasks. Finally, the two kinds of features were fused and input into the particle swarm algorithm optimised support vector machine (PSO-SVM) to classify the scenes of remote sensing images, and the classification accuracy reached 92.78%, outperforming four other classification methods.

Enhanced outdoor visual localization using Py-Net voting segmentation approach.

Camera relocalization determines the position and orientation of a camera in a 3D space. Althouh methods based on scene coordinate regression yield highly accurate results in indoor scenes, they exhibit poor performance in outdoor scenarios due to their large scale and increased complexity. A visual localization method, Py-Net, is therefore proposed herein. Py-Net is based on voting segmentation and comprises a main encoder containing Py-layer and two branch decoders. The Py-layer comprises pyramid convolution and 1 × 1 convolution kernels for feature extraction across multiple levels, with fewer parameters to enhance the model's ability to extract scene information. Coordinate attention was added at the end of the encoder for feature correction, which improved the model robustness to interference. To prevent the feature loss caused by repetitive structures and low-texture images in the scene, deep over-parameterized convolution modules were incorporated into the seg and vote decoders. Landmark segmentation and voting maps were used to establish the relation between images and landmarks in 3D space, reducing anomalies and achieving high precision with a small number of landmarks. The experimental results show that, in multiple outdoor scenes, Py-Net achieves lower distance and angle errors compared to existing methods. Additionally, compared to VS-Net, which also uses a voting segmentation structure, Py-Net reduces the number of parameters by 31.85% and decreases the model size from 236MB to 170 MB.

Fast Motion State Estimation Based on Point Cloud by Combing Deep Learning and Spatio-Temporal Constraints

Moving objects in the environment have a higher priority and more challenges in growing domains like unmanned vehicles and intelligent robotics. Estimating the motion state of objects based on point clouds in outdoor scenarios is currently a challenging area of research. This is due to factors such as limited temporal information, large volumes of data, extended network processing times, and the ego-motion. The number of points in a point cloud frame is typically 60,000–120,000 points, but most current motion state estimation methods for point clouds only downsample to a few thousand points for fast processing. The downsampling step will lead to the loss of scene information, which means these methods are far from being used in practical applications. Thus, this paper proposes a motion state estimation method that combines spatio-temporal constraints and deep learning. It starts by estimating and compensating the ego-motion of multi-frame point cloud data and mapping multi-frame data to a unified coordinate system; then the point cloud motion segmentation model on the multi-frame point cloud is proposed for motion object segmentation. Finally, spatio-temporal constraints are utilized to correlate the moving object at different moments and estimate the motion vectors. Experiments on KITTI, nuScenes, and real captured data show that the proposed method has good results, with an average vector deviation of only 0.036 m and 0.043 m in KITTI and nuScenes under a processing time of about 80 ms. The EPE3D error under the KITTI data is only 0.076 m, which proves the effectiveness of the method.

Dual-agent intelligent fire detection method for large commercial spaces based on numerical databases and artificial intelligence

Fires in commercial buildings can lead to significant casualties and property damage. Thus, rapidly and accurately identifying the source of a fire and its intensity is crucial for guiding rescue operations. This study aims to explore the use of distributed fiber optic temperature sensing systems, combined with deep learning algorithms, to predict key information about fires in commercial buildings, including the location of the fire source, the intensity of fire development, and the critical planar distribution of carbon monoxide. Based on the Fire Dynamic Simulation (FDS) data, a dual-agent model was developed, and with a sensor sampling interval of 3 s, 30 s of historical data achieved prediction accuracies of 94.23 % for fire intensity and 99.28 % for fire location. Upon reducing the intervals for fire location, the accuracy reached 93.56 %. Additionally, the Random Forest algorithm for mapping carbon monoxide distribution demonstrated an overall Mean Absolute Percentage Error (MAPE) of 5.87 %. Within the 0–150 ppm range, the MAPE was 2.46 %, far below the error levels of existing CO sensors. The database configurations and the sensors' spatiotemporal arrangement were also optimized to rapid and reliable fire prediction. This research demonstrates the feasibility of using artificial intelligence for critical fire scene information detection.

Research and application of 3D spatial safety electronic fence technology based on beidou high-precision positioning in substation construction projects

Abstract In recent years, the Beidou Navigation Satellite System (BDS) has developed rapidly, achieving global coverage and being widely used in various fields. The high-precision positioning capability of the Beidou system provides strong technical support for a variety of application scenarios, especially in fields requiring high-accuracy location information, such as safety monitoring in substation engineering. Substations are critical components of the power system, and their operational safety is vital for the stable operation of the entire grid. However, due to the complexity of the equipment and harsh working environments in substation engineering, utilizing Beidou’s high-precision positioning capabilities can significantly enhance the safety management level of substation construction projects and reduce the occurrence of accidents. This paper primarily explores and investigates high-precision positioning-based three-dimensional (3D) spatial safety electronic fence technology for substation construction projects. The process begins with using laser radar to capture 3D scene information and extract key parts of the point cloud model. Comprehensive positioning sensors, integrating the Beidou Navigation Satellite System (BDS) and inertial navigation, are then installed on crane booms to achieve real-time monitoring of the boom’s posture. Finally, a software platform is established to calculate the real-time 3D spatial distance between the crane boom and surrounding sensitive facilities, ensuring the safety of large-scale construction machinery. Field test results validate the effectiveness of the proposed method.