Inter-frame Difference Research Articles

Research on Abnormal State Detection of CZ Silicon Single Crystal Based on Multimodal Fusion

The Czochralski method is the primary technique for single-crystal silicon production. However, anomalous states such as crystal loss, twisting, swinging, and squareness frequently occur during crystal growth, adversely affecting product quality and production efficiency. To address this challenge, we propose an enhanced multimodal fusion classification model for detecting and categorizing these four anomalous states. Our model initially transforms one-dimensional signals (diameter, temperature, and pulling speed) into time–frequency domain images via continuous wavelet transform. These images are then processed using a Dense-ECA-SwinTransformer network for feature extraction. Concurrently, meniscus images and inter-frame difference images are obtained from the growth system’s meniscus video feed. These visual inputs are fused at the channel level and subsequently processed through a ConvNeXt network for feature extraction. Finally, the time–frequency domain features are combined with the meniscus image features and fed into fully connected layers for multi-class classification. The experimental results show that the method can effectively detect various abnormal states, help the staff to make a more accurate judgment, and formulate a personalized treatment plan for the abnormal state, which can improve the production efficiency, save production resources, and protect the extraction equipment.

The effect on gastrointestinal peristalsis for magnetic resonance cholangiopancreatography during breath-holding methods.

The breath-hold (BH) 3D magnetic resonance cholangiopancreatography method has been reported to suppress "respiratory artifacts"; however, the influence of gastrointestinal peristalsis around the target organs has not been discussed. In contrast, the autonomic nervous system has been reported to affect gastrointestinal peristalsis and BH imaging has been reported to influence venous blood flow signal (BFS) through its involvement with the autonomic nervous system. We examined the impact of BH imaging on gastrointestinal peristalsis. Seven healthy volunteers participated. Three respiratory patterns-free breathing (FB), BH at maximum inspiration (Insp-BH), and BH at maximum expiration (Exp-BH)-were used. Gastrointestinal peristalsis was measured using cine MRI. Cine MRI data were analyzed using the normalized interframe difference method, focusing on the duodenum and gastric body. Hemodynamic changes resulting from BH methods were evaluated using 2D phase contrast, targeting the inferior vena cava (IVC). The BFS was examined for all phases of each respiratory pattern. Peristalsis variation in the duodenum showed no significant differences among FB, Exp-BH, and Insp-BH. In the gastric body, no significant differences were observed between FB and Exp-BH or between Exp-BH and Insp-BH. However, a significant difference emerged between FB and Insp-BH. Regarding BFS, in the IVC, significant differences were observed between Exp-BH and Insp-BH and between FB and Insp-BH (both, p < 0.01), with no significant difference between FB and Exp-BH. Insp-BH reduces venous blood flow and suppresses the influence of peristalsis variation.

Extremely compact video representation for efficient near-duplicates detection

With the influx of online video uploads, a robust method for detecting video copies under various distortions is essential for copyright protection and search efficiency. We propose a compact video representation that uses a Siamese Neural Network for near-duplicate detection. This approach achieves up to 0.998 recall and 0.853 precision, even with distorted 56x56px miniatures. Consequently, we derive extremely compact video descriptors, facilitating video fragment retrieval in large datasets. Our method pre-selects frames from a video by identifying local maximums from the interframe differences curve, preserving characteristic sequence patterns even after extreme compression. Utilizing a simpler convolution-based model in the Siamese Neural Network, we improve results by up to 8% over VGG-16, with a 1.7-fold reduction in inference time. With compact descriptors of 1.875 kB, our models outperform others by more than two times. Additionally, we introduce a new dataset with dynamically changing scenes, enhancing its suitability for near-duplicate video detection.

Development of a body movement detection system to avoid re-exposure during radiography.

During the radiographic examination of the chest and bones in hospitals, communicating and maintaining posture is difficult for some patients, and movement before or during X-ray irradiation may necessitate re-exposure owing to body wobbling movements or breathing movements. To prevent the need for re-exposure during radiography and to determine the exposure timing, a body movement detection system that considers breathing movements was developed in this study. The posture of a patient was monitored using an RGB camera. The acquired video data was analyzed to detect body movement using either an inter-frame difference method or an optical flow estimation method. The performance of the system was evaluated by detecting the body and breathing movements during positioning. Consequently, the inter-frame difference method detected 179.8-1222.2pixels during body movements, and the optical flow estimation method confirmed that the feature points moved by 5.5-26.6mm (4.2-20.3pixels). When detecting breathing movements, 82-585pixels were detected by the inter-frame difference method, and the optical flow estimation method showed that the feature points moved by 5.2mm (2-4pixels). Therefore, the proposed method can detect body movements during radiography to prevent re-exposure due to body wobble and breathing movements. For healthcare providers, it will lead to reduce not only concerns about patient exposure but also unnecessary radiographic workload.

Multi-Person Action Recognition Based on Millimeter-Wave Radar Point Cloud

Human action recognition has many application prospects in human-computer interactions, innovative furniture, healthcare, and other fields. The traditional human motion recognition methods have limitations in privacy protection, complex environments, and multi-person scenarios. Millimeter-wave radar has attracted attention due to its ultra-high resolution and all-weather operation. Many existing studies have discussed the application of millimeter-wave radar in single-person scenarios, but only some have addressed the problem of action recognition in multi-person scenarios. This paper uses a commercial millimeter-wave radar device for human action recognition in multi-person scenarios. In order to solve the problems of severe interference and complex target segmentation in multiplayer scenarios, we propose a filtering method based on millimeter-wave inter-frame differences to filter the collected human point cloud data. We then use the DBSCAN algorithm and the Hungarian algorithm to segment the target, and finally input the data into a neural network for classification. The classification accuracy of the system proposed in this paper reaches 92.2% in multi-person scenarios through experimental tests with the five actions we set.

Research on the perception method of tiny objects in low-light and wide-field video

At present, many trackers exhibit commendable performance in well-illuminated scenarios but overlook target tracking in low-light environments. As night falls, the tracker's accuracy drops dramatically. Challenges such as high image resolution, intricate backgrounds, uneven illumination, and the resemblance between targets and backgrounds in Hawk-Eye surveillance videos make tracking small objects in low-light and wide-field scenarios exceedingly difficult for previous trackers. To address these challenges, this paper introduces an innovative approach by integrating the difference constraint method into the CF (correlation filters) tracker, which generates a change-aware mask using inter-frame difference information. In addition, a dual regression model and inter-frame difference constraint term are introduced to restrict each other for dual filter learning. In this paper, we construct a new benchmark comprising 41 night surveillance sequences captured by Hawk-Eye cameras. Exhaustive experiments are conducted on this benchmark. The results show that the proposed method maintains superior accuracy, surpasses state-of-the-art trackers in this dataset, and achieves a real-time performance of 27 fps on a single CPU, substantially advancing tiny object tracking on Hawk-Eye surveillance videos in low light and in night scenes.

Moving Object Detection Method Based on the Fusion of Online Moving Window Robust Principal Component Analysis and Frame Difference Method

The accuracy of moving object detection has a great impact on the accuracy of extracting the shape center coordinates of moving workpieces. The classic inter-frame difference method has "cavity" and "double shadow" issues for workpieces with comparable internal colors. In order to solve this problem, a moving object detection algorithm combining the three-frame difference method and Online Moving Window Robust Principal Component Analysis (OMWRPCA) is proposed. By using the OMWRPCA to extract the background image in the current frame and comparing it to the previous and current frames, the "cavity" and "double shadow" problems are avoided as well as the effects of background pixels. This paper presents a case study of a visual sorting experiment bench in an "intelligent manufacturing production demonstration line". The experiments show that the workpiece shape center coordinates obtained by the improved moving object detection algorithm are closer to the actual value than those obtained by the traditional algorithm, and the F-measure scores are above 0.8, which are more accurate than the other two algorithms. It is compared with the traditional algorithm of the frame difference method and the Online Mixture of Gaussian Matrix Factorization (OMoGMF).

Market Sentiment Analysis Based on Image Processing With Put-Call Volatility Gap Surface

Analyzing the market sentiment and forecasting movements in asset prices is extremely important and has been attempted by researchers and market practitioners. Asset volatilities, regardless of historical ones or implied from the option prices, are crucial barometers of the market. And this research proposes a new approach that combines image processing and machine learning to capture the relationships between sentiment-related features and asset movement. The proposed research is based on the tick-level SPY options transactions, and the dataset contains around 1.5 million trading records. Specially, we obtained the gap between the call surface and the put surface as the second-level implied volatility surface (IVS). After adopting the traditional convolutional neural network (CNN) to compare the predictive effects among implied volatility (IV) call surface, IV put surface, and IV gap surface, the results indicate that the IV gap provides the most significant predictability. Besides, our project creatively proposes the interframe difference approach to optimized CNN and a recurrent CNN (RCNN) to fully utilize spatial and temporal features of the IV surface data. According to experiment results, the directional accuracy of prediction ranges from 67.20% to 72.05% for asset movement forecasting at the millisecond level.

A Empirical Research on AI-Powered Athletic Posture Detection in Sports Training

Abstract: The current investigation delineates the efficacy of AI-facilitated detection of athletic postures within the realm of sports training. Employing a synthesis of literature review and empirical methodologies, data were amassed and scrutinized, affirming the study’s validity. The salient outcomes are manifold: (1) The frame difference algorithm efficaciously discerns inter-frame variances, evidencing pronounced adaptability and robustness, thereby enabling the recognition of weightlifting postures. (2) Confronting the challenge of negligible inter-frame disparities inherent in the frame difference algorithm, the research introduces a novel detection technique predicated on the cumulative inter-frame differences, which precisely pinpoints regions of posture alteration in weightlifting athletes. (3) Leveraging the dynamic space model of optical flow, the study ascertains the directional channel predicated on optical flow trajectory analyses, facilitating the identification of three distinct weightlifting postures: squatting, descending, and standing. (4) In alignment with the distinctive postural attributes of weightlifting athletes, a human posture paradigm was formulated, and a BP neural network classifier was deployed for both training and evaluative purposes, culminating in the successful differentiation of athlete from non-athlete entities within the training milieu. (5) The application of AI in posture recognition was extended to the scrutiny of pivotal postures and motions in weightlifting athletes, with experimental findings revealing a 98.21% accuracy rate in the recognition of force-exertion postures via the inter-frame difference method, and a flawless 100% accuracy in the identification of the apex and squatting postures. The enumeration of detected postures—encompassing knee extension, knee flexion, force application, squatting, and standing—through the poselet keyframe extraction approach, corresponded with the video count. Prospectively, AI’s role in athletic posture detection promises to augment coaches’ and athletes’ comprehension of their proficiencies and deficiencies, thereby steering training refinement and bolstering both the efficacy of training and the athletes’ caliber.

A Fish Detection and Tracking Method Based on Improved Interframe Difference and YOLO-CTS

A Fish Detection and Tracking Method Based on Improved Interframe Difference and YOLO-CTS

ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detection.

Recent camouflaged object detection (COD) attempts to segment objects visually blended into their surroundings, which is extremely complex and difficult in real-world scenarios. Apart from the high intrinsic similarity between camouflaged objects and their background, objects are usually diverse in scale, fuzzy in appearance, and even severely occluded. To this end, we propose an effective unified collaborative pyramid network that mimics human behavior when observing vague images and videos, i.e. zooming in and out. Specifically, our approach employs the zooming strategy to learn discriminative mixed-scale semantics by the multi-head scale integration and rich granularity perception units, which are designed to fully explore imperceptible clues between candidate objects and background surroundings. The former's intrinsic multi-head aggregation provides more diverse visual patterns. The latter's routing mechanism can effectively propagate inter-frame differences in spatiotemporal scenarios and be adaptively deactivated and output all-zero results for static representations. They provide a solid foundation for realizing a unified architecture for static and dynamic COD. Moreover, considering the uncertainty and ambiguity derived from indistinguishable textures, we construct a simple yet effective regularization, uncertainty awareness loss, to encourage predictions with higher confidence in candidate regions. Our highly task-friendly framework consistently outperforms existing state-of-the-art methods in image and video COD benchmarks.

Characterization of Immediate Pressing Tactics in Soccer in the Age of Artificial Intelligence

Abstract This paper focuses on the use of feature extraction techniques as well as parameter estimation to analyze the immediate pressing tactics in soccer games. The motion target detection method is used to capture the movements of the soccer player. By setting the rotation angle of the point cloud, the soccer movement action is represented in the form of a coordinate system. By combining the inter-frame difference method and setting the motion image threshold, the motion target can be obtained. Utilize Hu moments to extract the features of soccer motion. Combine the center of mass and velocity of soccer motion to reduce the error rate of motion feature extraction. Pairwise quaternions are utilized to represent soccer motion parameters to improve motion estimation. The results show that the soccer team has the greatest success rate of practicing immediate pressing tactics in 3s-4s, and the success rate of applying immediate pressing tactics after 4s is significantly lower. Team C has the highest success rate of huddling with defensive immediate pressing tactics, which reaches 56.1%. The success rate of huddling is closest to that of team A and team B, which are 43.54% and 43.97%, respectively.

An Enhanced Spatio-Temporal Human Detected Keyframe Extraction

Due to the immense availability of Closed-Circuit Television surveillance, it is quite difficult for crime investigation due to its huge storage and complex background. Content-based video retrieval is an excellent method to identify the best Keyframes from these surveillance videos. As the crime surveillance reports numerous action scenes, the existing keyframe extraction is not exemplary. At this point, the Spatio-temporal Histogram of Oriented Gradients - Support Vector Machine feature method with the combination of Background Subtraction is appended over the recovered crime video to highlight the human presence in surveillance frames. Additionally, the Visual Geometry Group trains these frames for the classification report of human-detected frames. These detected frames are processed to extract the keyframe by manipulating an inter-frame difference with its threshold value to favor the requisite human-detected keyframes. Thus, the experimental results of HOG-SVM illustrate a compression ratio of 98.54%, which is preferable to the proposed work's compression ratio of 98.71%, which supports the criminal investigation.

Fine-gained Motion Enhancement for action recognition: Focusing on action-related regions

Fine-grained motion information is crucial to improve the performance of video action recognition. Great strides have been made, with direct subtraction at adjacent original frames or frame features to learn motion representations. However, existing methods lack the ability to focus on the motion details about the action-related regions and to model the video events at multi-scale. To solve the above problems, this paper proposes a Fine-grained Motion Enhancement Network (FMENet), which comprises an Inter-frame Difference Motion Enhancement (IDME) module and a Multi-fiber Multi-level SpatioTemporal (MMST) module. In specific, the IDME module combines inter-frame local difference attention with a foreground encoder to differentiate foreground motion-related and static-related features, in conjunction with inter-frame feature subtraction operation to learn fine-grained motion features. Complementary to the motion features provided by IDME module, the MMST module introduces multi-level receptive fields in both spatial and temporal dimensions to further encode the semantic and appearance information of events. Finally, the IDME and MMST modules are embedded in a standard 2D framework to construct the FMENet with introducing limited computational cost. Extensive experiments on the Mini-Kinetics-200, Something–Something V1, HMDB-51 and UCF-101 datasets have been conducted to demonstrate the superiority of our method over other methods.

A Feedback-Driven DNN Inference Acceleration System for Edge-Assisted Video Analytics

With the proposal of edge computing, lots of intelligence applications have made significant progress. For enormous video analysis, how to further accelerate the process is still a major challenge. To overcome the challenge, researchers propose various video frame filtering systems to reduce the data transmission. In this work, we propose a Feedback-Driven DNN Inference Acceleration system (FDDIA). FDDIA is committed to further reducing the latency of DNN inference and the transmission according to the feedback information. Specifically, on the device side, FDDIA first uses the detection results of prior frames as feedback to determine the key candidate regions and uses the inter-frame difference to determine the new object regions. Those regions containing large objects are down-sampled to further reduce the frame. On the edge side, FDDIA first integrates all candidate regions into a smaller new image. Then it remaps the detection result for the new image back to the original frame and returns the result to the device. We evaluate FDDIA on different video benchmarks for three object detection tasks. The results show FDDIA improves the average end-to-end latency by 44% and average bandwidth usage by 41% than the existing advanced method while maintaining a high accuracy.

Detection of attack behaviour of pig based on deep learning

Attack behaviour detection of the pig is a valid method to protect the health of pig. Due to the farm conditions and the illumination changes of the piggery, the images of the pig in the videos are often being overlapped, which lead to difficulties in recognizing pig attack behaviour. We propose an improved YOLOX target detection model to overcome these difficulties. The improvements of the proposed model are: (1) the normalization attention mechanism is adopted to gain global information in the last block of the neck network and (2) the loss function IoU in YOLOX is replaced by DIoU to improve the detection accuracy. The pig attack behaviour considered in this paper includes the ear biting, the tail biting, the head to head collision and the head to body collision. The dataset is builded from the artificially observed attack video segments by using the inter-frame difference method. In the pig attack behaviour detection experiments, the improved YOLOX model achieves 93.21% precision which is 5.30% higher than the YOLOX model. The experiment results show that the improved YOLOX can realize pig attack behaviour detection with high precision.

Surprise-based JND estimation for perceptual quantization in H.265/HEVC codecs

Just noticeable distortion (JND), reflecting the perceptual redundancy directly, has been widely used in image and video compression. However, the human visual system (HVS) is extremely complex and the visual signal processing has not been fully understood, which result in existing JND models are not accurate enough and the bitrate saving of JND-based perceptual compression schemes is limited. This paper presents a novel pixel-based JND model for videos and a JND-based perceptual quantization scheme for HEVC codecs. In particular, positive and negative perception effects of the inter-frame difference and the motion information are analyzed and measured with an information-theoretic approach. Then, a surprise-based JND model is developed for perceptual video coding (PVC). In our PVC scheme, the frame-level perceptual quantization parameter (QP) is derived on the premise that the coding distortion is infinitely close to the estimated JND threshold. On the basis of the frame-level perceptual QP, we determine the perceptual QP for each coding unit through a perceptual adjustment function to achieve better perceptual quality. Experimental results indicate that the proposed JND model outperforms existing models significantly, the proposed perceptual quantization scheme improves video compression efficiency with better perceptual quality and lower coding complexity.

Flight Parameter—Wind Vortex Characteristic Control Model of a Four-Multirotor Unmanned Aerial Vehicle Operating in Pesticide Spraying of Rice

The downwash airflow generated by the rotors can enhance the penetration of pesticide droplets, allowing them to penetrate deeper into the canopy and more comprehensively control pests. Downwash air will produce a “wind vortex” in plant canopy, and the parameters of a “wind vortex” represent the effect of pesticide deposition to a certain extent. To obtain the corresponding relationship between wind vortices and flight parameters and control the effect of pesticide spraying, this paper carried out unmanned aerial vehicle (UAV) flight experiments in the field. Wind vortices were generated in rice canopy by downwash airflow, and the parameters of wind vortices were obtained by identifying wind-vortex images using the inter-frame difference method. The wind-vortex parameter control model was established, which can calculate the altitude and speed of the UAV when applying pesticide according to the target wind-vortex parameter. The deviations in the altitude were determined to be 0.67 and 0.43 m, and the deviations in the speed were 0.29 and 0.35 m/s during downwind and headwind UAV operations, respectively. The model relation functions were established, and their accuracies were found to be 97.1%, 92.3%, 69%, and 58% (downwind), and 97%, 78.4%, 62%, and 57% (upwind), respectively, indicating that downwind UAV operation leads to a clear relation between the wind-vortex parameters and the UAV-flight parameters. The model establishes the corresponding relationship between the wind-vortex parameters and flight parameters, which provides a theoretical basis for studying the precise application control method of UAV.

Vision-based estimation of MDS-UPDRS scores for quantifying Parkinson's disease tremor severity.

Parkinson's disease (PD) is a common neurodegenerative movement disorder among older individuals. As one of the typical symptoms of PD, tremor is a critical reference in the PD assessment. A widely accepted clinical approach to assessing tremors in PD is based on part III of the Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS). However, expert assessment of tremor is a time-consuming and laborious process that poses considerable challenges to the medical evaluation of PD. In this paper, we proposed a novel model, Global Temporal-difference Shift Network (GTSN), to estimate the MDS-UPDRS score of PD tremors based on video. The PD tremor videos were scored according to the majority vote of multiple raters. We used Eulerian Video Magnification (EVM) pre-processing to enhance the representations of subtle PD tremors in the videos. To make the model better focus on the tremors in the video, we proposed a special temporal difference module, which stacks the current optical flow to the result of inter-frame difference. The prediction scores were obtained from the Residual Networks (ResNet) embedded with a novel module, the Global Shift Module (GSM), which allowed the features of the current segment to include the global segment features. We carried out independent experiments using PD tremor videos of different body parts based on the scoring content of the MDS-UPDRS. On a fairly large dataset, our method achieved an accuracy of 90.6% for hands with rest tremors, 85.9% for tremors in the leg, and 89.0% for the jaw. An accuracy of 84.9% was obtained for postural tremors. Our study demonstrated the effectiveness of computer-assisted assessment for PD tremors based on video analysis. The latest version of the code is available at https://github.com/199507284711/PD-GTSN.

Visual attention framework for identifying semantic information from construction monitoring video

Construction safety management has been extensively investigated. Construction cameras have been widely adopted to monitor people’s performance in construction on-site. However, manually analyzing large quantities of video or image data is time-consuming and labor-intensive. Existing studies mostly focus on single element identification in videos or images, while the deeper semantic understanding of construction scenes with the whole scene is limited. Drawing on the attention mechanism, a framework is proposed to address this problem and identify semantic information such as multiple objects, relationships, and attributes from construction videos. This framework comprises the following two-step modeling approach: (1) a frame extraction model with an interframe difference mechanism is proposed to extract frames/images from construction videos and (2) an image scene understanding model that integrates a ResNet101 “encoder” and an LSTM + Attention “decoder” is put forward to identify semantic information/natural language descriptions from frames/images. Finally, the proposed framework is validated by multiple experiments with offline image datasets of construction scenes. The contributions of this research are twofold: (1) The proposed visual attention framework represents a significant and data-driven advancement in the cross-modal processing of construction video-image-natural language descriptions; (2) The automatic generation of video semantic information facilitates construction safety management such as workers’ safety state estimation and monitoring video/image retrieval and storage.