Background Objects Research Articles

Tensorial Multiview Representation for Saliency Detection via Nonconvex Approach.

In the study of salient object detection, multiview features play an important role in identifying various underlying salient objects. As to current common patch-based methods, all different features are handled directly by stacking them into a high-dimensional vector to represent related image patches. These approaches ignore the correlations inhering in the original spatial structure, which may lead to the loss of certain underlying characterization such as view interaction. In this article, different from currently available approaches, a tensorial feature representation framework is developed for the salient object detection in order to better explore the complementary information of multiview features. Under the tensor framework, a tensor low-rank constraint is applied to the background to capture its intrinsic structure, a tensor group sparsity regularization is posed on the salient part, and a tensorial sliced Laplacian regularization is then introduced to enlarge the gap between the subspaces of the background and salient object. Moreover, a nonconvex tensor Log-determinant function, instead of the tensor nuclear norm, is adopted to approximate the tensor rank for effectively suppressing the confusing information resulted from underlying complex backgrounds. Further, we have deduced the closed-form solution of this nonconvex minimization problem and established a feasible algorithm whose convergence is mathematically proven. Experiments on five well-known public datasets are provided and the simulations demonstrate that our method outperforms the latest unsupervised handcrafted features-based methods in the literature. Furthermore, our model is flexible with various deep features and is competitive with the state-of-the-art approaches.

Structure-aware Video Style Transfer with Map Art

Changing the style of an image/video while preserving its content is a crucial criterion to access a new neural style transfer algorithm. However, it is very challenging to transfer a new map art style to a certain video in which “content” comprises a map background and animation objects. In this article, we present a novel comprehensive system that solves the problems in transferring map art style in such video. Our system takes as input an arbitrary video, a map image, and an off-the-shelf map art image. It then generates an artistic video without damaging the functionality of the map and the consistency in details. To solve this challenge, we propose a novel network, Map Art Video Network (MAViNet), the tailored objective functions, and a rich training set with rich animation contents and different map structures. We have evaluated our method on various challenging cases and many comparisons with those of the related works. Our method substantially outperforms state-of-the-art methods in terms of visual quality and meets the mentioned criteria in this research domain.

Night-Time Detection of Subpixel Emitters with VIIRS Mid-Wave Infrared Bands M12–M13

In this paper, we present a new approach to subpixel infrared (IR) emitter detection in VIIRS mid-wave (MWIR) infrared bands M12–M13 at night, based on the presence of a tightly clustered background diagonal present in full granule scattergrams of M12 versus M13 radiances. This diagonal is found universally in night-time VIIRS data collected worldwide. The diagonal feature is absent during the day due to solar reflectance. The existence of the diagonal is attributed to close spacing in the bandpass centers of the VIIRS’ two MWIR bands. Apparently, the M12 and M13 emissivities are highly correlated to background objects, such clouds, ocean and land surfaces. The VIIRS Nightfire (VNF) algorithm detects pixels containing IR emitters based on their departure from the background diagonal. This paper outlines the method and compares VNF results with those from MODIS and VIIRS hotspot detection.

The Light of Earendel – The Most Distant Star Yet Observed

The gravity of massive objects can magnify background objects, making them appear larger and brighter. Precise alignment between a background star and a foreground lens leads to extreme magnifications, allowing individual stars to be detected at great distances. This technique has recently revealed the most distant star yet observed, which existed when the universe was only 900 million years old.

Robust visual tracking using very deep generative model

Deep learning algorithms provide visual tracking robustness at an unprecedented level, but realizing an acceptable performance is still challenging because of the natural continuous changes in the features of foreground and background objects over videos. One of the factors that most affects the robustness of tracking algorithms is the choice of network architecture parameters, especially the depth. A robust visual tracking model using a very deep generator (RTDG) was proposed in this study. We constructed our model on an ordinary convolutional neural network (CNN), which consists of feature extraction and binary classifier networks. We integrated a generative adversarial network (GAN) into the CNN to enhance the tracking results through an adversarial learning process performed during the training phase. We used the discriminator as a classifier and the generator as a store that produces unlabeled feature-level data with different appearances by applying masks to the extracted features. In this study, we investigated the role of increasing the number of fully connected (FC) layers in adversarial generative networks and their impact on robustness. We used a very deep FC network with 22 layers as a high-performance generator for the first time. This generator is used via adversarial learning to augment the positive samples to reduce the gap between the hungry deep learning algorithm and the available training data to achieve robust visual tracking. The experiments showed that the proposed framework performed well against state-of-the-art trackers on OTB-100, VOT2019, LaSOT and UAVDT benchmark datasets.

Automatic Shadow Generation via Exposure Fusion

Shadow generation aims to generate a plausible shadow for the inserted foreground object in a composite image. Besides the composite image and the associated mask of the inserted foreground object, existing methods also require a mask of all background objects as well as their shadows as an auxiliary input, which is laborious in practical applications. Meanwhile, most existing methods use a linear illumination transformation to darken the shadow region, which is prone to produce unrealistic shadows especially when background illumination is complex. To address these problems, this paper proposes an automatic shadow generation method, which avoids the laborious acquisition of the background object masks while harmonizing the shadow region to achieve plausible shadow effects. Specifically, to implicitly exploit background illumination to infer the shadow shape of the inserted foreground object, we first propose a Hierarchy Attention U-Net (HAU-Net) to sequentially build global interactions between the foreground object and background across spatial and channel dimensions. Since the spatial-variant property of the shadow, we formulate shadow harmonization as an exposure fusion problem and propose an Illumination-Aware Fusion Network (IFNet), which uses an improved illumination model with a double linear transformation to produce multiple under-exposure images of the shadow region. IFNet then learns pixel-wise fusion kernels that consider the local smoothness of the shadow to fuse the composite image with these under-exposure images to generate the realistic shadow of the foreground object. Extensive experiments on the DESOBA and Shadow-AR datasets demonstrate that our method achieves state-of-the-art performance for shadow generation on both the BOS and BOS-free test images.

METHODS OF COLLECTING AND PROCESSING IMAGES OBTAINED USING UAVs TO DETECT SPECIFIED OBJECTS

This study introduces a novel approach for identifying and quantifying specific structural objects, using fruit as an example, through images captured by drones in real time. The approach involves a series of steps: dynamically capturing images of the specified objects in 3D space using a fleet of drones, synchronizing video feeds from different drones, and finally, detecting and counting the objects. During the detection phase, a new version of the YOLOv5 model, YOLOv5-v1, was trained on a custom dataset of apple images to learn the features that differentiate apples from their surroundings. The initial size of the source network’s binding block was adjusted to avoid misidentifying small objects in the image background, thereby improving counting accuracy. The authors conducted computational experiments on a dataset of apple images they created to assess the approach’s effectiveness. The synchronization of video streams was assessed using the SSIM index, which ranged from 0.79 to 0.92, with an average value of 0.87, and the PSNR index, which ranged from 22 to 39. These values indicate the high efficiency of the developed synchronization method and the high quality of the resulting combined images. The results on the test dataset showed that the improved model could effectively identify fruits captured by the drone camera, with recall, precision, mAP, and F1-score of 92.13%, 84.59%, 87.94%, and 89.02%, respectively. The proposed approach was also compared with several other cutting-edge models, YOLOv3, YOLOv4, and YOLOv5, and was found to be superior in accuracy and speed. Our model’s average recognition speed was 0.015 seconds per video frame (66.7 frames/s), which was 1.13 and 3.53 times faster than the YOLOv4, and YOLOv3 networks, respectively. The average reliability index for detecting and counting fruit was 86.75%. These results demonstrate the effectiveness of using drone-captured images to detect specified objects. Future research could also explore including additional features, such as fruit shape, by training the model on a larger dataset.

SPV-SSD: An Anchor-Free 3D Single-Stage Detector with Supervised-PointRendering and Visibility Representation

Recently, 3D object detection based on multi-modal sensor fusion has been increasingly adopted in automated driving and robotics. For example, the semantic information provided by cameras and the geometric information provided by light detection and ranging (LiDAR) are fused to perceive 3D objects, as single modal sensors are unable to capture enough information from the environment. Many state-of-the-art methods fuse the signals sequentially for simplicity. By sequentially, we mean using the image semantic signals as auxiliary input for LiDAR-based object detectors would make the overall performance heavily rely on the semantic signals. Moreover, the error introduced by these signals may lead to detection errors. To remedy this dilemma, we propose an approach coined supervised-PointRendering to correct the potential errors in the image semantic segmentation results by training auxiliary tasks with fused features of the laser point geometry feature, the image semantic feature and a novel laser visibility feature. The laser visibility feature is obtained through the raycasting algorithm and is adopted to constrain the spatial distribution of fore- and background objects. Furthermore, we build an efficient anchor-free Single Stage Detector (SSD) powered by an advanced global-optimal label assignment to achieve a better time–accuracy balance. The new detection framework is evaluated on the extensively used KITTI and nuScenes datasets, manifesting the highest inference speed and at the same time outperforming most of the existing single-stage detectors with respect to the average precision.

Real‐time video anomaly detection for smart surveillance

AbstractHuman monitoring of surveillance cameras for anomaly detection may be a monotonous task as it requires constant attention to judge if the captured activities are anomalous or suspicious. This paper exploits background subtraction (BS), convolutional autoencoder, and object detection for a fully automated surveillance system. BS was performed by modelling each pixel as a mixture of Gaussians (MoG) to concatenate only the higher‐order learning in the foreground. Next, the foreground objects are fed to the convolutional autoencoders to filter out abnormal events from normal ones and automatically identify signs of threat and violence in real time. Then, object detection is introduced on the entire scene and the region of interest is highlighted with a bounding box to minimize human intervention in video stream processing. At recognition time, the network generates an alarm for the presence of an anomaly to notify of the identification of potentially suspicious actions. Finally, the complete system is validated upon several benchmark datasets and proved to be robust for complex video anomaly detection. The (AUC) average area under the curve for the frame‐level evaluation for all benchmarks is 94.94%. The best improvement ratio of AUC between the proposed system and state‐of‐the‐art methods is 7.7%.

Photometric Confirmation and Characterization of the Ennomos Collisional Family in the Jupiter Trojans

Collisional families offer a unique window into the interior composition of asteroid populations. Previous dynamical studies of the Jupiter Trojans have uncovered a handful of potential collisional families, two of which have been subsequently confirmed through spectral characterization. In this paper, we present new multiband photometric observations of the proposed Ennomos family and derive precise g − i colors of 75 candidate family members. While the majority of the targets have visible colors that are indistinguishable from background objects, we identify 13 objects with closely grouped dynamical properties that have significantly bluer colors. We determine that the true Ennomos collisional family is tightly confined to and , and the majority of its confirmed members have near-solar spectral slopes, including some of the bluest objects hitherto discovered in the Trojan population. The property of distinctly neutral colors that is shared by both the Ennomos family and the previously characterized Eurybates family indicates that the spectral properties of freshly exposed surfaces in the Jupiter region are markedly different than the surfaces of uncollided Trojans. This implies that the processes of ice sublimation and space weathering at 5.2 au yield a distinct regolith chemistry from the primordial environment within which the Trojans were initially accreted. It also suggests that the Trojans were emplaced in their present-day location from elsewhere sometime after the initial population formed, which is a key prediction of recent dynamical instability models of solar system evolution.

Coal Mine Belt Conveyor Foreign Objects Recognition Method of Improved YOLOv5 Algorithm with Defogging and Deblurring

The belt conveyor is the main equipment for underground coal transportation. Its coal flow is mixed with large coal, gangue, anchor rods, wooden strips, and other foreign objects, which easily causes failure of the conveyor belt, such as scratching, tearing, and even broken belts. Aiming at the problem that it was difficult to accurately identify the foreign objects of underground belt conveyors due to the influence of fog, high-speed operation, and obscuration, the coal mine belt conveyor foreign object recognition method of improved YOLOv5 algorithm with defogging and deblurring was proposed. In order to improve the clarity of the monitoring video of the belt conveyor, the dark channel priori defogging algorithm is applied to reduce the impact of fog on the clarity of the monitoring video, and the image is sharpened by user-defined convolution method to reduce the blurring effect on the image in high-speed operation condition. In order to improve the precision of foreign object identification, the convolution block attention module is used to improve the feature expression ability of the foreign object in the complex background. Through adaptive spatial feature fusion, the multi-layer feature information of the foreign object image is more fully fused so as to achieve the goal of accurate recognition of foreign objects. In order to verify the recognition effect of the improved YOLOv5 algorithm, a comparative test is conducted with self-built data set and a public data set. The results show that the performance of the improved YOLOv5 algorithm is better than SSD, YOLOv3, and YOLOv5. The belt conveyor monitoring video of resolution for 1920 × 1080 in Huangling Coal Mine is used for identification verification, the recognition accuracy can reach 95.09%, and the recognition frame rate is 56.50 FPS. The improved YOLOv5 algorithm can provide a reference for the accurate recognition of targets in a complex underground environment.

An Approach to Accurate Ship Image Recognition in a Complex Maritime Transportation Environment

In order to monitor traffic in congested waters, permanent video stations are now commonly used on interior riverbank bases. It is frequently challenging to identify ships properly and effectively in such images because of the intricate backdrop scenery and overlap between ships brought on by the fixed camera location. This work proposes Ship R-CNN(SR-CNN), a Faster R-CNN-based ship target identification algorithm with improved feature fusion and non-maximum suppression (NMS). The SR-CNN approach can produce more accurate target prediction frames for prediction frames with distance intersection over union (DIOU) larger than a specific threshold in the same class weighted by confidence scores, which can enhance the model’s detection ability in ship-dense conditions. The SR-CNN approach in NMS replaces the intersection over union (IOU) filtering criterion, which solely takes into account the overlap of prediction frames, while DIOU, also takes into account the centroid distance. The screening procedure in NMS, which is based on a greedy method, is then improved by the SR-CNN technique by including a confidence decay function. In order to generate more precise target prediction frames and enhance the model’s detection performance in ship-dense scenarios, the proposed SR-CNN technique weights prediction frames in the same class with DIOU greater than a predetermined threshold by the confidence score. Additionally, the SR-CNN methodology uses two feature weighting methods based on the channel domain attention mechanism and regularized weights to provide a more appropriate feature fusion for the issue of a difficult ship from background differentiation in busy waters. By gathering images of ship monitoring, a ship dataset is created to conduct comparative testing. The experimental results demonstrate that, when compared to the three traditional two-stage target detection algorithms Faster R-CNN, Cascade R-CNN, and Libra R-CNN, this paper’s algorithm Ship R-CNN can effectively identify ship targets in the complex background of far-shore scenes where the distinction between the complex background and the ship targets is low. The suggested approach can enhance detection and decrease misses for small ship targets where it is challenging to distinguish between ship targets and complex background objects in a far-shore setting.

Color Contrast Enhanced Rendering for Optical See-Through Head-Mounted Displays.

Most commercially available optical see-through head-mounted displays (OST-HMDs) utilize optical combiners to simultaneously visualize the physical background and virtual objects. The displayed images perceived by users are a blend of rendered pixels and background colors. Enabling high fidelity color perception in mixed reality (MR) scenarios using OST-HMDs is an important but challenging task. We propose a real-time rendering scheme to enhance the color contrast between virtual objects and the surrounding background for OST-HMDs. Inspired by the discovery of color perception in psychophysics, we first formulate the color contrast enhancement as a constrained optimization problem. We then design an end-to-end algorithm to search the optimal complementary shift in both chromaticity and luminance of the displayed color. This aims at enhancing the contrast between virtual objects and the real background as well as keeping the consistency with the original displayed color. We assess the performance of our approach using a simulated OST-HMD environment and an off-the-shelf OST-HMD. Experimental results from objective evaluations and subjective user studies demonstrate that the proposed approach makes rendered virtual objects more distinguishable from the surrounding background, thereby bringing a better visual experience.

Objects Detection With 3-D Roadside LIDAR Under Snowy Weather

Rain and snowfall will increase noise, change the resolution of objects in the point cloud and present great challenges to the accurate recognition of traffic objects. Accordingly, this article proposes an efficient real-time method for roadside 3-D light detection and ranging (LIDAR) background point cloud extraction and object segmentation under snowy weather. We first use a historical point cloud sequence to quickly construct a background model, extract the background point cloud from the current frame by using a background difference method and update the background model in real-time. Then, the noise caused by snowfall in the non-background point cloud is filtered based on the beam density difference of the object point clouds. Finally, the remaining object point cloud is accurately segmented based on the proposed hierarchical object clustering method. We use an intelligent roadside system equipped with 3-D roadside LIDAR to collect point cloud data in a snowfall environment and evaluate the proposed method qualitatively and quantitatively. Experimental results show that the proposed method effectively avoids the problem of under-segmentation and over-segmentation of object point clouds under snowy conditions, and the precision and recall rate of traffic object segmentation reached 96.41% and 95.02%, respectively, thereby indicating a significant improvement in the accuracy and reliability of traffic object detection using roadside LIDAR under snowy conditions.

MSFTrans: a multi-task frequency-spatial learning transformer for building extraction from high spatial resolution remote sensing images

ABSTRACT Building extraction is significant in urban planning, economic evaluation, and driverless technology development. However, automatic building extraction from high spatial resolution remote sensing images has been a challenging task due to the various building shapes and colors, imaging conditions, and complex background objects. Current methods in building extraction are generally based on deep convolution networks, and they mostly use an encoder-decoder architecture, wherein detailed building features and small buildings are easily omitted in continuous convolution operations. Moreover, buildings with blurred boundaries are only completely extracted with difficulty. To meet these challenges, we propose a multi-task architecture of frequency-spatial learning Transformer to extract buildings from high spatial resolution remote sensing images. Different from current architecture, we designed a frequency-spatial learning module in the framework of multi-task to synthesize the multi-scale spatial features and frequency decomposition features of high-resolution image. Spiking convolution is proposed in this study to enhance the frequency features of buildings by mimicking the neural transmission in human brains. In this way, multi-scale building features can be better preserved and distinguished from background objects. Moreover, a masked-attention Transformer is adopted to improve multi-scale building mask prediction accuracy by synthesizing successive pixel-wise up-sampled feature maps. We also propose a strategy to evaluate the practical transferability of the proposed method by mimicking practical application cases through training and evaluating images with different spatial resolutions from different study areas and datasets. Experiments using five public building datasets (WHU-Building Satellite Dataset I, WHU-Building Satellite Dataset II, Massachusetts Buildings Dataset, Inria Aerial Image Dataset, xBD Building Dataset) demonstrate the strong potential applicability of our proposed method for practical application cases. Our method outperforms five recently proposed state-of-the-art semantic segmentation methods with 36.60% accuracy improvement on extracted buildings and approximately 53.55% recall progress in extracting small building instances. The implementation code will be released after the paper is published.

A Network for Landslide Detection Using Large-Area Remote Sensing Images with Multiple Spatial Resolutions

Accurately detecting landslides over a large area with complex background objects is a challenging task. Research in the area suffers from three drawbacks in general. First, the models are mostly modified from typical networks, and are not designed specifically for landslide detection. Second, the images used to construct and evaluate models of landslide detection are limited to one spatial resolution, which struggles to meet the requirements of such relevant applications as emergency response. Third, assessments are primarily carried out by using the training data on different parts of the same study area. This makes it difficult to objectively evaluate the transferability of the model, because ground objects in the same area are distributed with similar spectral characteristics. To respond to the challenges above, this study proposes DeenNet, specifically designed for landslide detection. Different from the widely used encoder–decoder networks, DeenNet maintains multi-scale landslide features by decoding the input feature maps to a large scale before encoding a module. The decoding operation is conducted by deconvolution of the input feature maps, while encoding is conducted by convolution. Our model is trained on two earthquake-triggered landslide datasets, constructed using images with different spatial resolutions from different sensor platforms. Two other landslide datasets of different study areas with different spatial resolutions were used to evaluate the trained model. The experimental results demonstrated an at least 6.17% F1-measure improvement by DeenNet compared with three widely used typical encoder–decoder-based networks. The decoder–encoder network structure of DeenNet proves to be effective in maintaining landslide features, regardless of the size of the landslides in different evaluation images. It further validated the capacity of DeenNet in maintaining landslide features, which provides a strong applicability in the context of applications.

Object Video Tracking using a Pan-Tilt-Zoom System

This paper implements object video tracking system that represents object location in subsequent video frames. A closed-circuit television (CCTV) camera mounted on a rotating system is used for capturing video while the object of interest always kept at the centre of the screen. Three tracking algorithms were selected and implemented: template matching, contour matching and optical flow. Measures of their accuracy and speed were taken for comparison. The software was implemented in a personal computer with C# programming language, with the aid of EmguCV which is a wrapper for OpenCV, a famous image processing library. The system implemented for this study is able to successfully track a rigid body, discernible from the background objects with size up to 400×300 pixel for the Phase Alternate Line (PAL) system of 720×576 pixel frame size. Tracking was stable even with the existence of rotation and scaling. Some faults were observed when occlusion was present or when the target was moving with a speed faster than that of the rotation system of 30 degrees/s horizontal and 15 degrees/s vertical.

Scene-relative object motion biases depth percepts

An important function of the visual system is to represent 3D scene structure from a sequence of 2D images projected onto the retinae. During observer translation, the relative image motion of stationary objects at different distances (motion parallax) provides potent depth information. However, if an object moves relative to the scene, this complicates the computation of depth from motion parallax since there will be an additional component of image motion related to scene-relative object motion. To correctly compute depth from motion parallax, only the component of image motion caused by self-motion should be used by the brain. Previous experimental and theoretical work on perception of depth from motion parallax has assumed that objects are stationary in the world. Thus, it is unknown whether perceived depth based on motion parallax is biased by object motion relative to the scene. Naïve human subjects viewed a virtual 3D scene consisting of a ground plane and stationary background objects, while lateral self-motion was simulated by optic flow. A target object could be either stationary or moving laterally at different velocities, and subjects were asked to judge the depth of the object relative to the plane of fixation. Subjects showed a far bias when object and observer moved in the same direction, and a near bias when object and observer moved in opposite directions. This pattern of biases is expected if subjects confound image motion due to self-motion with that due to scene-relative object motion. These biases were large when the object was viewed monocularly, and were greatly reduced, but not eliminated, when binocular disparity cues were provided. Our findings establish that scene-relative object motion can confound perceptual judgements of depth during self-motion.

DyStSLAM: an efficient stereo vision SLAM system in dynamic environment

Simultaneous localization and mapping (SLAM) is the basis for many robotic applications. Most SLAM algorithms are based on the assumption that the scene is static. In real-world applications, moving objects are inevitable, which will greatly impact the ego-pose estimation accuracy. This paper presents DyStSLAM, a visual SLAM system with a stereo configuration that can efficiently identify moving objects and accomplish dynamic data association. First of all, DyStSLAM extracts feature points, estimates the disparity map, and performs instance segmentation simultaneously. Then, the obtained results are combined to estimate the motion confidence and discriminate between moving objects and static ones. A confidence based matching algorithm is proposed to associate dynamic objects and estimate the pose of each moving object. At the same time, static objects are used to estimate the pose of the camera. Finally, after nonlinear optimization, a sparse point cloud map of both static background and dynamic objects is constructed. Compared with ORB-SLAM2, the proposed method outperforms by 31% for absolute trajectory error on the KITTI dataset.

Hue-Preserving Color Transforms for LED Wall Virtual Production Workflows

Virtual production using light-emitting diode (LED) wall display technology is gaining popularity in the entertainment industry to produce motion pictures, episodic television, live broadcast, and esports content. This new paradigm typically uses one or more large LED displays that contain millions of individual LEDs that are used to display a virtual background and/or foreground objects from a virtual scene that is simultaneously captured by the digital photographic camera on set, resulting in the so-called “in-camera visual effects.” In a virtual production workflow, digital cameras capture the actors and objects on the set in the foreground while simultaneously capturing the image shown on the LED wall behind the actors. Realtime video game rendering engines update the image shown on the LED wall to compensate for changes in camera location, camera pose, and lens focal length. Modern digital camera workflows typically include color transforms that were not designed to accurately render the large areas of saturated colors that are possible when capturing the image shown on an LED wall. Some examples of a hue shift that can occur in typical workflows are blue to cyan, red to pink, red to orange, and green to yellow. We found that these hue shifts can occur dynamically while racking focus to and from the LED wall and also statically when the LED wall is kept out of focus, which is a common technique that is used to minimize moiré artifacts. This article explores a simple modification of these common color transforms to preserve the hue of the scene while also creating a similar look to the existing color transforms.