Target Scene Research Articles

Research on Character Trajectory Prediction Method with Fused Perspective Information in Virtual Scene

Character trajectory prediction in virtual scene is conducive to improving the preloading of scene resources. Most state-of-the-art trajectory prediction methods are devoted to the pedestrian trajectory in the real world. These methods cannot work well in virtual scene when encountering simple role interaction, more random motion and limited types of accessible data. In this paper, we proposed a role trajectory prediction method based on viewpoint information (RIPA algorithm, Role Intent Prediction Algorithm), which includes role interest calculation module based on viewpoint information and a role trajectory prediction module based on information of intention. Firstly, the role interest calculation module based on view information is used to predict the role’s interest in the virtual object. Secondly, the role interest is integrated into the trajectory prediction model, and the attention mechanism is introduced to improve the prediction performance. The experimental results show that, compared with the traditional LSTM model, the role trajectory prediction method proposed in this paper can improve the prediction accuracy by about 2%, and can predict the future target scene and role target position at least 1s in advance.

Disentangling diagnostic object properties for human scene categorization

It usually only takes a single glance to categorize our environment into different scene categories (e.g. a kitchen or a highway). Object information has been suggested to play a crucial role in this process, and some proposals even claim that the recognition of a single object can be sufficient to categorize the scene around it. Here, we tested this claim in four behavioural experiments by having participants categorize real-world scene photographs that were reduced to a single, cut-out object. We show that single objects can indeed be sufficient for correct scene categorization and that scene category information can be extracted within 50 ms of object presentation. Furthermore, we identified object frequency and specificity for the target scene category as the most important object properties for human scene categorization. Interestingly, despite the statistical definition of specificity and frequency, human ratings of these properties were better predictors of scene categorization behaviour than more objective statistics derived from databases of labelled real-world images. Taken together, our findings support a central role of object information during human scene categorization, showing that single objects can be indicative of a scene category if they are assumed to frequently and exclusively occur in a certain environment.

Specular highlight removal and depth estimation based on polarization characteristics of light field

In this paper, a method based on polarization characteristics of the light field is proposed to remove specular highlight and accurately estimate the depth information for highly reflective surfaces. This method innovatively combines the polarization characteristics and the advantages of the light field imaging to effectively suppress the specular highlight and realize the visual imaging as well as depth estimation. Firstly, a focused light field camera which spatial and angular resolutions can be flexibly adjusted as needed is designed and used to capture the light field images of the target scenes, and the image correction algorithm is performed on the target images to correct image distortion. Then, the relationship between the image grayscale and the polarization angle is established to remove the specular highlight according to the Stokes parameters of light field, and the color transfer algorithm is also used to restore output images. Finally, a method based on entropy mean gradient ratio is designed to quantitatively evaluate the effectiveness of our method. In addition, a depth estimation system of focused light field is built independently to reconstruct the depth information for different highly reflective surfaces by using the proposed method. The experimental results show that our proposed method is suitable for local specular highlight removal under diffuse light source, which provides a new imaging method for highly reflective surfaces.

Mobilenetv2_CA Lightweight Object Detection Network in Autonomous Driving

A lightweight network target detection algorithm was proposed, based on MobileNetv2_CA, focusing on the problem of high complexity, a large number of parameters, and the missed detection of small targets in the target detection network based on candidate regions and regression methods in autonomous driving scenarios. First, Mosaic image enhancement technology is used in the data pre-processing stage to enhance the feature extraction of small target scenes and complex scenes; second, the Coordinate Attention (CA) mechanism is embedded into the Mobilenetv2 backbone feature extraction network, combined with the PANet and Yolo detection heads for multi-scale feature fusion; finally, a Lightweight Object Detection Network is built. The experimental test results show that the designed network obtained the highest average detection accuracy of 81.43% on the Voc2007 + 2012 dataset, and obtained the highest average detection accuracy of 85.07% and a detection speed of 31.84 FPS on the KITTI dataset. The total amount of network parameters is only 39.5 M. This is beneficial to the engineering application of MobileNetv2 network in automatic driving.

Vehicle target detection algorithm based on yolov5

With the rapid advancement of automobile manufacturing technology and the intensification of competition among automobile brands, the development of autonomous driving has been pushed to the forefront of automobile development, which causes approximately $650 billion in losses due to traffic accidents worldwide every year. For in the complex vehicle target scene, because of its many vehicle targets, dense targets often exist between the occlusion, overlap, a variety of different weather reasons, rain, fog, sunny days, etc. resulting in obstruction of the field of view, and the vehicle camera often intercepted images mostly exist blurred, ghosting and other problems, making the vehicle target detection for detailed feature extraction requirements are high, the detection accuracy is often difficult to meet the requirements, proposed based on YOLOv5s vehicle target detection algorithm is proposed. ACmix attention mechanism is introduced to make the model achieve the purpose of expanding the target perception field, adaptively focusing on different target regions, and capturing more information features, etc. The test shows that the accuracy of the model increases by 1.1% on a subset of BDD100K data set after improvement. The accuracy of the improved model increased by 1.1% on the subset of BDD100K dataset and by 1.2% on the PASCAL VOC2007 general dataset.

Distributed Inference in Resource-Constrained IoT for Real-Time Video Surveillance

Advances in communication technologies and computational capabilities of Internet of Things (IoT) devices enable a range of complex applications that require ever increasing processing of sensors' data. An illustrative example is real-time video surveillance that captures videos of target scenes and process them to detect anomalies using deep learning (DL). Running deep learning models requires huge processing and incurs high computation delay and energy consumption on resource-constraint IoT devices. In this article, we introduce methods for distributed inference over IoT devices and edge server. Two distinct algorithms are proposed to split the deep neural network layers computation between IoT device and an edge server; the early split strategy (ESS) for battery powered IoT devices and the late split strategy (LSS) for IoT devices connected to regular power source. The evaluation shows that both the ESS and LSS schemes achieve the target inference delay deadline when tested over VGG16 and MobileNet_V2 CNN models. In terms of computational load, the ESS scheme achieves nearly 15–20% reduction whereas LSS scheme achieves up to 60% reduction. The gains in energy saving of IoT devices for both the ESS and LSS schemes are nearly 18% and 52%, respectively.

Monocular metasurface camera for passive single-shot 4D imaging

It is a grand challenge for an imaging system to simultaneously obtain multi-dimensional light field information, such as depth and polarization, of a scene for the accurate perception of the physical world. However, such a task would conventionally require bulky optical components, time-domain multiplexing, and active laser illumination. Here, we experimentally demonstrate a compact monocular camera equipped with a single-layer metalens that can capture a 4D image, including 2D all-in-focus intensity, depth, and polarization of a target scene in a single shot under ambient illumination conditions. The metalens is optimized to have a conjugate pair of polarization-decoupled rotating single-helix point-spread functions that are strongly dependent on the depth of the target object. Combined with a straightforward, physically interpretable image retrieval algorithm, the camera can simultaneously perform high-accuracy depth sensing and high-fidelity polarization imaging over an extended depth of field for both static and dynamic scenes in both indoor and outdoor environments. Such a compact multi-dimensional imaging system could enable new applications in diverse areas ranging from machine vision to microscopy.

Optimized sinusoidal patterns for high-performance computational ghost imaging.

Computational ghost imaging (CGI) can reconstruct scene images by two-order correlation between sampling patterns and detected intensities from a bucket detector. By increasing the sampling rates (SRs), imaging quality of CGI can be improved, but it will result in an increasing imaging time. Herein, in order to achieve high-quality CGI under an insufficient SR, we propose two types of novel sampling methods for CGI, to the best of our knowledge, cyclic sinusoidal-pattern-based CGI (CSP-CGI) and half-cyclic sinusoidal-pattern-based CGI (HCSP-CGI), in which CSP-CGI is realized by optimizing the ordered sinusoidal patterns through "cyclic sampling patterns," and HCSP-CGI just uses half of the sinusoidal pattern types of CSP-CGI. Target information mainly exists in the low-frequency region, and high-quality target scenes can be recovered even at an extreme SR of 5%. The proposed methods can significantly reduce the sampling number and real-time ghost imaging possible. The experiments demonstrate the superiority of our method over state-of-the-art methods both qualitatively and quantitatively.

A Target-Based Non-Uniformity Self-Correction Method for Infrared Push-Broom Hyperspectral Sensors

Non-uniformity in the response of spectral image elements is an inevitable phenomenon in hyperspectral imaging, which mainly manifests itself as the presence of band noise in the acquired hyperspectral data. This problem is prominent in the infrared band owing to the detector material, operating environment, and other factors. Non-uniformity is an important factor that can affect the quality of the hyperspectral data, which has a serious impact on both data analysis and applications and requires corrections via technical means wherever possible. This paper proposes a novel target-based non-uniformity self-correction method for infrared push-broom hyperspectral images. The Mars Mineralogical Spectrometer (MMS) onboard the Tianwen-1 orbiter was used as the research and application object. The model is constructed and applied to the target scene characteristics and detection patterns of Mars remote sensing exploration, which are combined with the causes of noise generation in the infrared spectral image bands. The design of the MMS dual-channel Visible-Near-Infrared (V-NIR) and Near-Mid-Infrared (N-MIR) co-field of view co-target detection and laboratory calibration data for the V-NIR spectral band can achieve non-uniformity corrections (NUCs). Therefore, for the MMS in-orbit Mars exploration mission, the method selected spectral data (920–1055 nm) characterized by a reduced atmospheric influence to iteratively obtain the homogeneous region, which was used to calculate the non-uniformity correction factor for the N-MIR spectral band. This method was compared, validated, and evaluated with other conventional methods using both laboratory and in-orbit hyperspectral data. The results showed that the experimental data corrections were comparable to laboratory calibrations, with a maximum relative deviation of <2.6%. These results prove that our method not only provides an excellent non-uniformity correction, but also ensures spectral fidelity. It can thus be used as a non-uniformity correction process for the MMS and similar hyperspectral imagers.

PR-Alignment: Multidimensional Adaptive Registration Algorithm Based on Practical Application Scenarios

In the present day, 3D point clouds are considered to be an important form of representing the 3D world. In computer vision, mobile robotics, and computer graphics, point cloud registration is a basic task, and it is widely used in 3D reconstruction, reverse engineering, among other applications. However, the mainstream method of point cloud registration is subject to the problems of a long registration time as well as a poor modeling effect, and these two factors cannot be balanced. To address this issue, we propose an adaptive registration mechanism based on a multi-dimensional analysis of practical application scenarios. Through the use of laser point clouds and RGB images, we are able to obtain geometric and photometric information, thus improving the data dimension. By adding target scene classification information to the RANSAC algorithm, combined with geometric matching and photometric matching, we are able to complete the adaptive estimation of the transformation matrix. We demonstrate via extensive experiments that our method achieves a state-of-the-art performance in terms of point cloud registration accuracy and time compared with other mainstream algorithms, striking a balance between expected performance and time cost.

Super-resolution computed tomography imaging spectrometry

Computed tomography imaging spectrometry (CTIS) is a snapshot spectral imaging technique that relies on a limited number of projections of the target data cube (2D spatial and 1D spectral), which can be reconstructed via a delicate tomographic reconstruction algorithm. However, the restricted angle difference between the projections and the space division multiplexing of the projections make the reconstruction suffer from severe artifacts as well as a low spatial resolution. In this paper, we demonstrate super-resolution computed tomography imaging spectrometry (SRCTIS) by assimilating the information obtained by a conventional CTIS system and a regular RGB camera, which has a higher pixel resolution. To improve the reconstruction accuracy of CTIS, the unique information provided by the zero-order diffraction of the target scene is used as a guidance image for filtering to better preserve the edges and reduce artifacts. The recovered multispectral image is then mapped onto the RGB image according to camera calibration. Finally, based on the spectral and the spatial continuities of the target scene, the multispectral information obtained from CTIS is propagated to each pixel of the RGB image to enhance its spectral resolution, resulting in SRCTIS. Both stimulative studies and proof-of-concept experiments were then conducted, and the results quantified by key metrics, such as structural similarity index measurement and spectral angle mapping have suggested that the developed method cannot only suppress the reconstruction artifacts, but also simultaneously achieve high spatial and spectral resolutions.

UAV's Status Is Worth Considering: A Fusion Representations Matching Method for Geo-Localization.

Visual geo-localization plays a crucial role in positioning and navigation for unmanned aerial vehicles, whose goal is to match the same geographic target from different views. This is a challenging task due to the drastic variations in different viewpoints and appearances. Previous methods have been focused on mining features inside the images. However, they underestimated the influence of external elements and the interaction of various representations. Inspired by multimodal and bilinear pooling, we proposed a pioneering feature fusion network (MBF) to address these inherent differences between drone and satellite views. We observe that UAV's status, such as flight height, leads to changes in the size of image field of view. In addition, local parts of the target scene act a role of importance in extracting discriminative features. Therefore, we present two approaches to exploit those priors. The first module is to add status information to network by transforming them into word embeddings. Note that they concatenate with image embeddings in Transformer block to learn status-aware features. Then, global and local part feature maps from the same viewpoint are correlated and reinforced by hierarchical bilinear pooling (HBP) to improve the robustness of feature representation. By the above approaches, we achieve more discriminative deep representations facilitating the geo-localization more effectively. Our experiments on existing benchmark datasets show significant performance boosting, reaching the new state-of-the-art result. Remarkably, the recall@1 accuracy achieves 89.05% in drone localization task and 93.15% in drone navigation task in University-1652, and shows strong robustness at different flight heights in the SUES-200 dataset.

As-built BIM reconstruction of piping systems using PipeNet

Building Information Modeling (BIM) is gradually recognized and promoted as the new standard practice in the construction industry as well as the built environment. Mechanical, electrical, and plumbing system, as a system requiring regular maintenance, takes an important place in the building operation and maintenance; its failure can cause significant impacts operationally, economically, and even environmentally. As-built BIM is needed for efficient BIM-enabled facility management of buildings that were built before the existence of BIM or with outdated BIM. Modeling of the as-built BIM is currently practiced in a very manual and tedious way, requiring a considerable amount of time and effort even for a skilled modeler. This paper proposes a solution that reconstructs the as-built BIM of piping systems in buildings from LiDAR scanned point cloud data automatically. Compared to the existing works, it requires no additional data other than the unstructured point cloud with XYZ fields, no data preprocessing, and no prior knowledge of the pipe directions or dimensions. The solution comprises two stages. The first is a novel deep learning network, PipeNet, that is able to detect pipes regardless of the size of the input data and the scale of the target scene, and predict the pipe centerline points together with other pipe parameters. In the second stage, the pipe model is reconstructed through line fitting, refinement, and graph-based connectivity analysis constrained by domain knowledge that maximizes the coherence of the piping system model. The final output is converted to the Industry Foundation Classes format which is neutrally acceptable in the BIM industry. The solution is validated on both synthetic and actual scan data, and the results demonstrate its robustness, fast speed, and high recognition rate and precision. The results are discussed in detail and further improvements such as improving the precision of recognition are suggested for future works.

Bi-Classifier Adversarial Network for Cross-Scene Hyperspectral Image Classification

Labeling hyperspectral images (HSIs) is time-consuming and labor-intensive for researchers, so the deficiency of adequate labeling samples is a giant obstacle to conducting HSI classification. Especially, such issue is exacerbated when there are no available labeled samples in the target scene. For the sake of resolving aforesaid issue, we put forward a novel cross-scene HSI classification method namely bi-classifier adversarial augmentation network (BCAN) so as to transfer knowledge from a similar but different source domain to an unlabeled target domain. First, the source and target domain distributions are aligned by maximizing and minimizing the decision discrepancy between two classifiers, respectively. Then, more accurate samples corresponding to pseudo-labels are selected as reliable samples and added to the training set. Finally, the spectral band random zeroing (SBRZ) method is proposed to expand the training samples for reliable samples, which handles the problem of insufficient network training resulted from insufficient samples in the source domain. By using multi-classifiers for domain adaptation and data augmentation, the accuracy of the network for cross-scene HSI classification tasks are improved. BCAN can extract the source domain’s helpful information to complete the target domain classification task. Experiments conducted on ten HSI data pairs show that BCAN outperforms many state-of-the-art baselines.

A survey of hyperspectral image super-resolution method

A survey of hyperspectral image super-resolution method

Target-independent dynamic wavefront sensing method based on distorted grating and deep learning

A real-time wavefront sensing method for arbitrary targets is proposed, which provides an effective way for diversified wavefront sensing application scenarios. By using a distorted grating, the positive and negative defocus images are simultaneously acquired on a single detector. A fine feature, which is independent of the target itself but corresponding to the wavefront aberration, is defined. A lightweight and efficient network combined with an attention mechanism (AM-EffNet) is proposed to establish an accurate mapping between the features and the incident wavefronts. Comparison results show that the proposed method has superior performance compared to other methods and can achieve high-accuracy wavefront sensing in varied target scenes only by using the point target dataset to train the network well.

Hyperspectral Image Super-Resolution via Learning an Undercomplete Dictionary and Intra-Algorithmic Postprocessing

This study proposes a novel hyperspectral image super-resolution (HSI-SR) method, whose fundamental requirement to set up is only a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">single</i> low-resolution HSI (LR-HSI) of the target scene. To fully exploit the inherent HSIs’ features, including spectral correlation, local sparsity, nonlocal self-similarity, and piecewise smoothness in the spatial domain, we first break down the HSI-SR problem into several identical sub-cube SR (SC-SR) problems. Then, we design an SC-SR modular model involving two parts: 1- Injecting spatio-spectral information from co-structural sub-cube-patches (SCPs) into each other’s sparse representation via learning an undercomplete dictionary, and 2- Intra-algorithmic post-processing. The second part, which makes the most of the spatial piecewise smoothness characteristic, consists of two modules, wavelet-based spatial smoothing (SpaS) and adaptive filtering (AF). After merging all the super-resolved SCs and obtaining a high-resolution HSI, we employ spectral smoothing (SpeS) to remove the possible remaining distortions in the spectral vector of each pixel. Experiments on simulated- and real-LR datasets demonstrate the outperformance of our method.

Adaptive Coarse-to-Fine Single Pixel Imaging With Generative Adversarial Network Based Reconstruction

Single Pixel Imaging (SPI) that only uses one light intensity sensor has been researched extensively as an alternative imaging method. It is proven to work at low light conditions and unconventional wavelength bands where the classical pixel array sensors are limited. However, the major issues of SPI remain as the image quality and computational efficiency. Thus, this paper proposes an adaptive coarse-to-fine (C2F) sampling method to replace the typical uniform sampling method to achieve image reconstruction with better quality. This scalable sampling mechanism is adaptive to the target scene as it will progressively sample according to the image complexity and quality indicator. Subsequently, a deep Generative Adversarial Network (GAN) model is also proposed to improve the time efficiency of the multi-scale image reconstruction. The results show that C2F sampling consistently outperforms uniform sampling in terms of image quality (21% in SSIM, 8% in PSNR and 17% in RMSE). Besides, improvement in the efficiency is also achieved by the proposed GAN reconstruction, whereby the total time taken is only 0.025% of the time taken for the conventional L1 reconstruction method (≈ 4000 times faster). In conclusion, the proposed adaptive C2F SPI using GAN reconstruction method can serve as an optimised solution to improve both the image quality and computational efficiency in SPI.

Rain Removal from Light Field Images with 4D Convolution and Multi-scale Gaussian Process.

Existing deraining methods mainly focus on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera, which has emerged as a popular device in the computer vision and graphics research communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel network, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. In order to make full use of the LFI, we adopt 4D convolutional layers to build the proposed rain steak removal network to simultaneously process all sub-views of the LFI. In the proposed network, the rain detection model, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via calculating pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI. Extensive quantitative and qualitative evaluations conducted on both synthetic LFIs and real-world LFIs demonstrate the effectiveness of our proposed method.

Spatial Non-Stationary Near-Field Channel Modeling and Validation for Massive MIMO Systems

Existing deraining methods focus mainly on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera. LFIs are becoming popular in the computer vision and graphics communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel method, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. To make full use of the LFI, it adopts 4D convolutional layers to simultaneously process all sub-views of the LFI. In the pipeline, the rain detection network, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via computing pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI.