Scene Change Detection Research Articles

Mechanical Assembly Monitoring Method Based on Depth Image Multiview Change Detection

In the process of product assembly, parts need to be assembled in a given assembly sequence. Failure to detect the correctness of the newly assembled parts in time can affect the quality and assembly efficiency of products. For effective detection of newly assembled parts in the assembly process from different viewing angles, this study applies scene change detection to mechanical assembly monitoring for the first time and proposes a mechanical assembly monitoring method based on depth image multiview change detection. This method includes a semantic fusion network and an attention-based feature extraction network (AFE Net) for multiview change detection. To make this method suitable for the change detection of the mechanical assembly, this study involves the following innovations. 1) Considering the assembly parts with a single color, symmetry, and no texture, this study employs depth images as the input of the semantic fusion network. Subsequently, the semantic segmentation network is used to segment the parts on the depth images for the generation of color semantic segmentation images. Then, the semantic segmentation images and depth images are merged and input into the multiview change detection network. 2) In the multiview change detection network, an attention-based feature extraction module is designed to rapidly focus on the key information of the current task from a large amount of input information and improve the processing efficiency and accuracy of the task. Furthermore, through up-sampling, the size of the feature map is unified, and high-dimensional semantic information and low-dimensional spatial information are merged, which effectively increases the amount of feature information. 3) To verify the effectiveness of the multiview change detection of the assembly, this study establishes a multiview assembly process dataset and evaluates the proposed method using this dataset. The results show that, compared with other change detection networks, the comprehensive index F1 of the method based on the abovementioned dataset reaches an optimal value of 96.9% while consuming less time and with clearer boundary processing. Overall, the proposed network structure is suitable for the change detection of the mechanical assembly and can also be applied to multiview monitoring in product assembly.

Multi-Temporal Scene Classification and Scene Change Detection With Correlation Based Fusion.

Classifying multi-temporal scene land-use categories and detecting their semantic scene-level changes for remote sensing imagery covering urban regions could straightly reflect the land-use transitions. Existing methods for scene change detection rarely focus on the temporal correlation of bi-temporal features, and are mainly evaluated on small scale scene change detection datasets. In this work, we proposed a CorrFusion module that fuses the highly correlated components in bi-temporal feature embeddings. We first extract the deep representations of the bi-temporal inputs with deep convolutional networks. Then the extracted features will be projected into a lower-dimensional space to extract the most correlated components and compute the instance-level correlation. The cross-temporal fusion will be performed based on the computed correlation in CorrFusion module. The final scene classification results are obtained with softmax layers. In the objective function, we introduced a new formulation to calculate the temporal correlation more efficiently and stably. The detailed derivation of backpropagation gradients for the proposed module is also given. Besides, we presented a much larger scale scene change detection dataset with more semantic categories and conducted extensive experiments on this dataset. The experimental results demonstrated that our proposed CorrFusion module could remarkably improve the multi-temporal scene classification and scene change detection results.

Hierarchical Paired Channel Fusion Network for Street Scene Change Detection.

Street Scene Change Detection (SSCD) aims to locate the changed regions between a given street-view image pair captured at different times, which is an important yet challenging task in the computer vision community. The intuitive way to solve the SSCD task is to fuse the extracted image feature pairs, and then directly measure the dissimilarity parts for producing a change map. Therefore, the key for the SSCD task is to design an effective feature fusion method that can improve the accuracy of the corresponding change maps. To this end, we present a novel Hierarchical Paired Channel Fusion Network (HPCFNet), which utilizes the adaptive fusion of paired feature channels. Specifically, the features of a given image pair are jointly extracted by a Siamese Convolutional Neural Network (SCNN) and hierarchically combined by exploring the fusion of channel pairs at multiple feature levels. In addition, based on the observation that the distribution of scene changes is diverse, we further propose a Multi-Part Feature Learning (MPFL) strategy to detect diverse changes. Based on the MPFL strategy, our framework achieves a novel approach to adapt to the scale and location diversities of the scene change regions. Extensive experiments on three public datasets (i.e., PCD, VL-CMU-CD and CDnet2014) demonstrate that the proposed framework achieves superior performance which outperforms other state-of-the-art methods with a considerable margin.

Indoor Scene Change Captioning Based on Multimodality Data

This study proposes a framework for describing a scene change using natural language text based on indoor scene observations conducted before and after a scene change. The recognition of scene changes plays an essential role in a variety of real-world applications, such as scene anomaly detection. Most scene understanding research has focused on static scenes. Most existing scene change captioning methods detect scene changes from single-view RGB images, neglecting the underlying three-dimensional structures. Previous three-dimensional scene change captioning methods use simulated scenes consisting of geometry primitives, making it unsuitable for real-world applications. To solve these problems, we automatically generated large-scale indoor scene change caption datasets. We propose an end-to-end framework for describing scene changes from various input modalities, namely, RGB images, depth images, and point cloud data, which are available in most robot applications. We conducted experiments with various input modalities and models and evaluated model performance using datasets with various levels of complexity. Experimental results show that the models that combine RGB images and point cloud data as input achieve high performance in sentence generation and caption correctness and are robust for change type understanding for datasets with high complexity. The developed datasets and models contribute to the study of indoor scene change understanding.

Dynamic Scene Change Detection in Video Coding

Video compression has become a source of different research studies. It is necessary in order to address channel bandwidth limitations and growing video demand, including digital libraries and streaming media delivery via the Internet. A video is a number of frames captured by a camera while a scene is a series of consecutive frames captured from a specific narrative viewpoint. To compress a video, firstly, the intra frames are separated from inter frames using scene change detection methods. Then, block-based motion estimation algorithms are used to eliminate the temporal redundancy between successive frames. This paper describes some scene change detection methods for use on the uncompressed video to detect scene types such as cut, dissolve, wipe, etc. Absolute Frame Difference (AFD), Mean Absolute Frame Differences (MAFD), Mean Histogram Absolute Frame Difference (MHAFD), and Maximum Gradient Value (MGV) techniques are adaptively tested on different video types to identify accurate scene change in both low and high object motion scenes. Test results show that the proposed approach (MHAFD) obtains a better accuracy, F1-scor measure of 100%, especially for cuts and gradual transitions (wipe) video types. Dissolve scene change is detected with a high precision of 100% (i.e., no false detection) with the (MAFD) detector. Besides, in terms of time complexity for analyzing all the video samples, the proposed method (MHAFD ) provides the best result compared to the selected detectors.

A Novel Siamese-Based Approach for Scene Change Detection With Applications to Obstructed Routes in Hazardous Environments

The demand for automatic scene change detection has massively increased in the last decades due to its importance regarding safety and security issues. Although deep learning techniques have provided significant enhancements in the field, such methods must learn which object belongs to the foreground or background beforehand. In this article, we propose an approach that employs siamese U-Nets to address the task of change detection, such that the model learns to perform semantic segmentation using background reference frames only. Therefore, any object that comes up into the scene defines a change. The experimental results show the robustness of the proposed model over the well-known public dataset CDNet2014. Additionally, we also consider a private dataset called “PetrobrasROUTES,” which comprises obstruction or abandoned objects in escape routes in hazardous environments. Moreover, the experiments show that the proposed approach is more robust to noise and illumination changes.

Genetic Algorithm Based Rate Control for AV1

Rate control persists as a core problem in video coding area. This paper proposes a robust rate control framework for the recently released AV1 standard, which is a royalty-free video codec specification generated by AOM. Firstly, an exponential rate control model as well as its initialization process and update strategy is proposed to characterize the correspondences between rate and quantization parameter. Next, this paper proposes RC-EMD to measure the similarity between two video frames. Then based on the RC-EMD, a dynamic bit allocation method using genetic algorithm is proposed for AV1 encoder, which can simultaneously detect scene changes. Unlike the existing work, the proposed method explores a larger solution space and automatically adapts to actual input videos, which results in improved performance.

Robust, imperceptible and blind video steganography using RGB secret, maximum likelihood estimation and Fibonacci encryption

According to the International Telecommunication Union, currently 51% of the world's population has internet access, out of which over 80% prefers video sharing. Henceforth for secure communication, secret data hiding in video can be a viable option. Here a blind, key-based video steganography has been proposed which can hide RGB image within a colour video. The novelty of this approach lies in the security imposed by various factors including scene change detection by maximum likelihood estimation; pre-processing of secret image by Fibonacci encryption and discrete wavelet transform based keyed embedding technique, which can hide colour images without any perceivable differences and does not require cover video frame during extraction. Quality of the method has been analysed by peak signal to noise ratio, structural similarity index, normalised absolute error and correlation coefficient; robustness has been tested against random cropping, rotating, resizing and histogram equalization; comparison with the literature demonstrates its superiority.

Complexity reduction method for High Efficiency Video Coding encoding based on scene-change detection and image texture information

Recently, as demand for high-quality video and realistic media has increased, High Efficiency Video Coding has been standardized. However, High Efficiency Video Coding requires heavy cost in terms of computational complexity to achieve high coding efficiency, which causes problems in fast coding processing and real-time processing. In particular, High Efficiency Video Coding inter-coding has heavy computational complexity, and the High Efficiency Video Coding inter prediction uses reference pictures to improve coding efficiency. The reference pictures are typically signaled in two independent lists according to the display order, to be used for forward and backward prediction. If an event occurs in the input video, such as a scene change, the inter prediction performs unnecessary computations. Therefore, the reference picture list should be reconfigured to improve the inter prediction performance and reduce computational complexity. To address this problem, this article proposes a method to reduce computational complexity for fast High Efficiency Video Coding encoding using information such as scene changes obtained from the input video through preprocessing. Furthermore, reference picture lists are reconstructed by sorting the reference pictures by similarity to the current coded picture using Angular Second Moment, Contrast, Entropy, and Correlation, which are image texture parameters from the input video. Simulations are used to show that both the encoding time and coding efficiency could be improved simultaneously by applying the proposed algorithms.

A Method of Scene Boundary Detection for Indexing Video Data Efficiently

In a cell animation, a background scene is presented with one cell. When animation scenes are changed, a relatively large scene change occurs because their backgrounds are also changed. Unlike a camera-based real movie, a cell animation is made in the way of manual drawing. For this reason, such animation does not use many colors. In order to apply the characteristics of animations as most as possible and detect a scene change of a cell animation effectively, this study proposes a new scene change detection technique with the stepwise use of the color and block-based histograms. The proposed algorithm receives continuous animation images as input, changes the RGB color space into the HSI color space, executes the difference operation of color values of the two images, and thereby primarily determines whether the neighboring images are scene change candidates. If they are judged to be scene change candidates, the color histogram for each sub-region is made, and then a weight value is applied to finally determine whether a scene change occurs. To compare the proposed scene change detection method with conventional scene change detection methods, the block-based scene change detection method and the histogram-based scene change detection method as conventional ones were implemented. In this study, to qualitatively evaluate the performance of the proposed method of detecting scene boundaries, the accuracy measure was defined. The two conventional algorithms are not complicated relatively and are widely used to extract a scene change. The experiment of this study reveals that the proposed method more accurately detects an animation scene change than other conventional methods. The proposed animation scene boundary detection method is expected to be applied usefully to the fields of digital video data indexing and retrieval, dynamic motion analysis, and other related areas.

Online-Learning-Based Bayesian Decision Rule for Fast Intra Mode and CU Partitioning Algorithm in HEVC Screen Content Coding

Screen content coding (SCC) is an extension of high efficiency video coding by adopting new coding modes to improve the coding efficiency of SCC at the expense of increased complexity. This paper proposes an online-learning approach for fast mode decision and coding unit (CU) size decision in SCC. To make a fast mode decision, the corner point is first extracted as a unique feature in screen content, which is an essential pre-processing step to guide Bayesian decision modeling. Second, the distinct color number in a CU is derived as another unique feature in screen content to build the precise model using online-learning for skipping unnecessary modes. Third, the correlation of the modes among spatial neighboring CUs is analyzed to further eliminate unnecessary mode candidates. Finally, the Bayesian decision rule using online-learning is applied again to make a fast CU size decision. To ensure the accuracy of the Bayesian decision models, new scene change detection is designed to update the models. Results show that the proposed algorithm achieves 36.69% encoding time reduction with 1.08% Bjøntegaard delta bitrate (BDBR) increment under all intra configuration. By integrating into the existing fast SCC approach, the proposed algorithm reduces 48.83% encoding time with a 1.78% increase in BDBR.

Digital Movies Tracking Using Scene-Based Watermarking System

Digital watermarking can be used to prevent the illegal piracy of copyrighted digital video content. However to be successful as one of the application in the digital cinema, it must be imperceptibly embedded, reliably retrieved, secure against illicit removal. This paper proposes a novel watermarking technique for digital motion content in light of scene change detection using the discrete wavelet transform (DWT). To reduce the computational time, a watermark has been embedded in scene-changed frame. The scene-changed frames are detected by successive estimation of statistical measure technique, where the correlation is utilized as a measure. The performance of the proposed system has been analyzed using the mean square error, normalized correlation, bit error rate, and the similarity index. Empirical results reveal that the proposed method is greatly improved in comparison with the customary techniques in terms of the imperceptibility, recovery after the show capture with camcorder, speed and robustness under various attacks, such as the noise addition, Gaussian low pass filtering, geometrical shift, sharpening, and video attacks in the application of digital cinema. Since an illegal copy of the movie obtained by camcorder affects the geometrical features, in particular, the system has been tested and found to be resilient against geometrical attacks under various conditions.

Hadamard Transform-Based Optimized HEVC Video Coding

The High Efficiency Video Coding (HEVC/H.265) standard achieves great improvement in compression efficiency over the widely used H.264/AVC standard at a cost of much higher complexity. When encoding videos using HEVC, the selection of the quantization parameter (QP) can significantly affect the coding efficiency. Typical algorithms for adaptive quantization employ fixed bitrate budgeting or fixed QP offsets for different frames and different blocks, without considering detailed input video characteristics, some of which might be captured using computer vision methods. The problem of determining adaptive settings of HEVC coding parameters has not been satisfactorily solved. In this paper, we proposed a Hadamard (“HAD”) energy-based optimized HEVC video encoder, in which HAD energy is used to measure the amount of residual information to be encoded in a block so as to determine the QP value for each block for a better coding efficiency. HAD energy is also used to expedite the time-consuming mode decision process and for a precise scene change detection to avoid flicker artifacts. Experiment using the widely used open source HEVC encoder $\times$ 265-v1.8 showed that the proposed algorithm was able to achieve an average of 14% saving in Bjontegaard-delta-rate and an average of 20% saving in encoding time as compared the “medium” preset of $\times$ 265, while the proposed algorithm also produced an improvement of 3.3% in coding efficiency for the HEVC reference software HM-16.6.

Effective scene change detection in complex environments

One of the fundamental operations in computer vision applications is change detection, in which moving foreground objects are segmented from a static background. A common approach for change detection is the comparison of an image frame with the stored background model using a matching algorithm, a process known as background subtraction. However, such techniques fail in environments with dynamic backgrounds, illumination changes, or shadow and camera jitters. This study focuses on effectively detecting scene changes in complex environments. To this end, we proposed a new colour descriptor named local colour difference pattern (LCDP) that is insusceptible to shadow and is able to capture both colour and texture features at a pixel location. Furthermore, a scene change detection framework was proposed to handle dynamic scenes based on sample consensus that integrates LCDP and a novel spatial model fusion mechanism. Experiments using the CDnet benchmark dataset demonstrated the effectiveness of the proposed approach to change detection in complex environments.

Histogram Shape-Based Scene-Change Detection Algorithm

This paper proposes histogram shape-based scene-change detection to automatically recognize the changing content in a moving image for frame rate up-conversion; in the frame rate up-conversion, it is important to detect both local and global scene changes. In addition, a threshold value should be fixed regardless of the image characteristics that requires low computation. The proposed algorithm simply extracts the shape of a two-dimensional histogram using point-based distance calculation. It also uses block merging and blocks smoothing to further improve the accuracy of the scene change detection. In the experimental results, the proposed algorithm enhanced the average F1 score to 0.607 (a 189.63% improvement) as compared with the benchmark methods. The average computation time of the proposed algorithm also decreased to 1.696 ms (a 65.81% reduction) compared with the benchmark algorithms.

Effective scene change detection in complex environments

One of the fundamental operations in computer vision applications is change detection, in which moving foreground objects are segmented from a static background. A common approach for change detection is the comparison of an image frame with the stored background model using a matching algorithm, a process known as background subtraction. However, such techniques fail in environments with dynamic backgrounds, illumination changes, or shadow and camera jitters. This study focuses on effectively detecting scene changes in complex environments. To this end, we proposed a new colour descriptor named local colour difference pattern (LCDP) that is insusceptible to shadow and is able to capture both colour and texture features at a pixel location. Furthermore, a scene change detection framework was proposed to handle dynamic scenes based on sample consensus that integrates LCDP and a novel spatial model fusion mechanism. Experiments using the CDnet benchmark dataset demonstrated the effectiveness of the proposed approach to change detection in complex environments.

Unsupervised Scene Change Detection via Latent Dirichlet Allocation and Multivariate Alteration Detection

Scene change detection is the process of identifying the differences between the multitemporal image scenes, which has significant potential in the application of urban development and land management at the semantic level. Traditional scene change detection methods are based on the supervised scene classification, and then directly compare the independent classification results without considering the temporal correlation between the unchanged regions. However, few studies have focused on detecting the semantic changes of multitemporal image scenes with unsupervised methods. In this paper, we propose a novel unsupervised scene change detection method by using latent Dirichlet allocation (LDA) and multivariate alteration detection (MAD). First, the scene is represented by the bag-of-visual-words model, and adopt the union dictionary to ensure the consistency of dictionary space. Then, LDA is used to achieve the middle-level feature dimension reduction, and generate the common topic space of the two multitemporal image scene datasets. And finally, the MAD method was applied to detect the semantic changes of corresponding multitemporal image scenes. Two experiments with high-resolution remote sensing image scene datasets demonstrated that our proposed approach can get a better performance in unsupervised scene change detection without prior knowledge.

Change detection in complex auditory scenes is predicted by auditory memory, pitch perception, and years of musical training

Our world is a sonically busy place and we use both acoustic information and experience-based knowledge to make sense of the sounds arriving at our ears. The knowledge we gain through experience has the potential to shape what sounds are prioritized in a complex scene. There are many examples of how visual expertise influences how we perceive objects in visual scenes, but few studies examine how auditory expertise is associated with attentional biases toward familiar real-world sounds in complex scenes. In the current study, we investigated whether musical expertise is associated with the ability to detect changes to real-world sounds in complex auditory scenes, and whether any such benefit is specific to musical instrument sounds. We also examined whether change detection is better for human-generated sounds in general or only communicative human sounds. We found that musicians had less change deafness overall. All listeners were better at detecting human communicative sounds compared to human non-communicative sounds, but this benefit was driven by speech sounds and sounds that were vocally generated. Musical listening skill, speech-in-noise, and executive function abilities were used to predict rates of change deafness. Auditory memory, musical training, fine-grained pitch processing, and an interaction between training and pitch processing accounted for 45.8% of the variance in change deafness. To better understand perceptual and cognitive expertise, it may be more important to measure various auditory skills and relate them to each other, as opposed to comparing experts to non-experts.

Fast Bayesian decision based block partitioning algorithm for HEVC

The newly published High Efficiency Video Coding (HEVC) Standard has greatly enhanced the coding performance in comparison to its predecessors. However, HEVC also has high computational complexity, which limits its application. In this paper, we propose a fast Bayesian Decision based Block Partitioning (BDBP) algorithm for HEVC encoder. Firstly, the scene change detection based on average grey difference is used to divide the video sequence into the online learning phase and the fast partitioning phase. Secondly, in the online learning phase, the statistical parameters are extracted from coding units (CUs) in every depth to establish the Gaussian mixture models which are resolved by expectation maximization algorithm; in the fast partitioning phase, the conditional probabilities for CU to decide partitioning and non-partitioning are calculated. Finally, the minimum risk Bayesian decision rule is used to choose the decision with smaller risk, and the decision is regarded as the judgment of the current CU. Experimental results show that the proposed algorithm reduces the computational complexity of HM13.0 to 54.1% in encoding time with 0.92% increase in the BD-Rate and 0.05dB decrease in the BD-PSNR. Moreover, the proposed algorithm also demonstrates better performance over other state-of-the-art work.

Context-dependent role of selective attention for change detection in multi-speaker scenes.

Disappearance of a voice or other sound source may often go unnoticed when the auditory scene is crowded. We explored the role of selective attention for this change deafness with magnetoencephalography in multi-speaker scenes. Each scene was presented two times in direct succession, and one target speaker was frequently omitted in Scene 2. When listeners were previously cued to the target speaker, activity in auditory cortex time locked to the target speaker's sound envelope was selectively enhanced in Scene 1, as was determined by a cross-correlation analysis. Moreover, the response was stronger for hit trials than for miss trials, confirming that selective attention played a role for subsequent change detection. If selective attention to the streams where the change occurred was generally required for successful change detection, neural enhancement of this stream would also be expected without cue in hit compared to miss trials. However, when listeners were not previously cued to the target, no enhanced activity for the target speaker was observed for hit trials, and there was no significant difference between hit and miss trials. These results, first, confirm a role for attention in change detection for situations where the target source is known. Second, they suggest that the omission of a speaker, or more generally an auditory stream, can alternatively be detected without selective attentional enhancement of the target stream. Several models and strategies could be envisaged for change detection in this case, including global comparison of the subsequent scenes.