Single Image Frame Research Articles

An Improved Sparrow Search Algorithm for Optimizing Support Vector Machines

Support Vector Machine (SVM) is often used in regression and classification problems. However, SVM needs to find proper kernel function to solve high-dimensional problems. We propose an improved Sparrow Search Algorithm Support Vector Machine (ISSA-SVM) algorithm to optimize the SVM kernel parameters. First, the problem of slow convergence due to the lack of ergodicity and poor diversity of the initial population is effectively overcome by using Sine chaotic map. Second, adaptive dynamic weight factors are induced not only to balance the global and local search capabilities, but also accelerate the convergence speed of the sparrow search algorithm. The simulation results of 11 benchmark test functions show that ISSA has faster convergence, more accurate search capability, and easier to jump out of local extremes than the SSA, Gray Wolf Optimization (GWO) and Whale Optimization Algorithm (WOA). That indicates ISSA has better convergence, better robustness, and stronger competitiveness. The experimental results on the coal gangue dataset show that the classification accuracy of ISSA-SVM algorithm is improved by 7.09% and 4.25% compared with SVM and SSA-SVM, respectively. Meanwhile, the classification time for a single image frame of ISSA-SVM algorithm is reduced by 20.15% and 13.74% compared with SVM and SSA-SVM, respectively.

UIU-Net: U-Net in U-Net for Infrared Small Object Detection.

Learning-based infrared small object detection methods currently rely heavily on the classification backbone network. This tends to result in tiny object loss and feature distinguishability limitations as the network depth increases. Furthermore, small objects in infrared images are frequently emerged bright and dark, posing severe demands for obtaining precise object contrast information. For this reason, we in this paper propose a simple and effective "U-Net in U-Net" framework, UIU-Net for short, and detect small objects in infrared images. As the name suggests, UIU-Net embeds a tiny U-Net into a larger U-Net backbone, enabling the multi-level and multi-scale representation learning of objects. Moreover, UIU-Net can be trained from scratch, and the learned features can enhance global and local contrast information effectively. More specifically, the UIU-Net model is divided into two modules: the resolution-maintenance deep supervision (RM-DS) module and the interactive-cross attention (IC-A) module. RM-DS integrates Residual U-blocks into a deep supervision network to generate deep multi-scale resolution-maintenance features while learning global context information. Further, IC-A encodes the local context information between the low-level details and high-level semantic features. Extensive experiments conducted on two infrared single-frame image datasets, i.e., SIRST and Synthetic datasets, show the effectiveness and superiority of the proposed UIU-Net in comparison with several state-of-the-art infrared small object detection methods. The proposed UIU-Net also produces powerful generalization performance for video sequence infrared small object datasets, e.g., ATR ground/air video sequence dataset. The codes of this work are available openly at https://github.com/danfenghong/IEEE_TIP_UIU-Net.

Orientation Keypoints for 6D Human Pose Estimation.

Most realtime human pose estimation approaches are based on detecting joint positions. Using the detected joint positions, the yaw and pitch of the limbs can be computed. However, the roll along the limb, which is critical for application such as sports analysis and computer animation, cannot be computed as this axis of rotation remains unobserved. In this paper we therefore introduce orientation keypoints, a novel approach for estimating the full position and rotation of skeletal joints, using only single-frame RGB images. Inspired by how motion-capture systems use a set of point markers to estimate full bone rotations, our method uses virtual markers to generate sufficient information to accurately infer rotations with simple post processing. The rotation predictions improve upon the best reported mean error for joint angles by 48% and achieves 93% accuracy across 15 bone rotations. The method also improves the current state-of-the-art results for joint positions by 14% as measured by MPJPE on the principle dataset, and generalizes well to in-the-wild datasets.

Detecting Maritime Infrared Targets in Harsh Environment by Improved Visual Attention Model Preselector and Anti-Jitter Spatiotemporal Filter Discriminator

Without any prior knowledge, it has always been a serious challenge to accurately detect infrared targets under a maritime harsh environment (MHE). To solve this problem, the main contribution of this paper is to use the improved visual attention model (VAM) preselector and anti-jitter spatiotemporal filter (ASF) discriminator to detect infrared targets in the MHE. The proposed method consists of image preprocessing, the single frame detection of suspected targets and a multi-frame judgment of real targets. First, in the process of single-frame image processing, a combination of the Gaussian difference filtering and local minimum filtering is applied to overcome the uneven background brightness distribution and improve the saliency of the target. Second, an intensity standard deviation method is designed to determine the unevenness of the background. According to the difference in background smoothness, an appropriate “center-surround difference” operation is selected to suppress sea wave interference, and the single frame suspected target candidate region is obtained. Third, in order to “align” the multi-frame image sequence, a method for correcting the position of the inter-frame jitter is proposed. The inter-frame jitter is measured and compensated by the inter-frame block matching results. Finally, according to the three assumptions of multi-frame spatiotemporal filtering, single-frame false targets are filtered out and combined with the OTSU method to segment the real target area. To evaluate the performance index of the proposed method by comparing it with the four other state-of-the-art methods for dealing with the MHE. The experimental results show that the algorithm achieves the maximum detection rate (DR) on the premise of being far lower than the false alarm rate (FAR) of the comparison method. The final experimental results also confirmed that the proposed algorithm is more suitable for infrared target detection in diverse MHEs.

A Generic Approach towards Amharic Sign Language Recognition

In the day-to-day life of communities, good communication channels are crucial for mutual understanding. The hearing-impaired community uses sign language, which is a visual and gestural language. In terms of orientation and expression, it is separate from written and spoken languages. Despite the fact that sign language is an excellent platform for communication among hearing-impaired persons, it has created a communication barrier between hearing-impaired and non-disabled people. To address this issue, researchers have proposed sign language to text translation systems for English and other European languages as a solution. The goal of this research is to design and develop an Amharic digital text converter system using Ethiopian sign language. The proposed system was created with the help of two key deep learning algorithms: a pretrained deep learning model and a Long Short-Term Memory (LSTM). The LSTM was used to extract sequence information from a sequence of image frames of a specific sign language, while the pretrained deep learning model was used to extract features from single frame images. The dataset used to train the algorithms was gathered in video format from Addis Ababa University. Prior to feeding the obtained dataset to the deep learning models, data preprocessing activities such as cleaning and video to image frame segmentation were conducted. The system was trained, validated, and tested using 80%, 10%, and 10% of the 2475 images created during the preprocessing step. Two pretrained deep learning models, EfficientNetB0 and ResNet50, were used in this investigation, and they attained an accuracy of 72.79%. In terms of precision and f1-score, ResNet50 outperformed EfficientNetB0. For the proposed system, a graphical user interface prototype was created, and the best performing model was chosen and implemented. The proposed system can be utilized as a starting point for other researchers to improve upon, based on the outcomes of the experiment. More high-quality training datasets and high-performance training machines, such as GPU-enabled computers, can be added to the system to improve it.

Faint Space Debris Detection Algorithm Based on Small Aperture Telescope Detection System

Ground-based optical observation has unique advantages in space target observation. However, due to the weak light-gathering ability of small-aperture optoelectronic observation telescopes, the space debris in the image is weak and easily drowned in noise. In order to solve the above problems, we use digital image processing technology to extract faint space debris. We propose a high detection rate space debris automatic extraction algorithm, aiming to automatically detect space debris. We first establish a new space target description model. Our algorithm is mainly divided into two stages. The purpose of the first stage is to reduce the influence of a large number of stars. We perform wavelet transform and guided filtering for three consecutive frames, and the reconstructed wavelet that takes the median value can achieve the effect of eliminating stars. In the second stage, we adopt the method of robust principal component analysis and attribute the problem of target detection to the problem of separating the target and background of a single frame of image. After a large number of experimental results analysis, it is proved that the algorithm can effectively detect faint debris in the monitoring system of small aperture telescope, and has high precision and low computational complexity.

PE Classroom Teaching Innovation Based on Deep Learning from the Perspective of New Curriculum Standard

In recent years, due to the influence of various factors, most of the physical quality of primary and secondary school students in China are in a state of subhealth. According to relevant studies, nearly 70 percent of Chinese students lack daily physical exercise. Physical quality is the primary factor and prerequisite of study and work, so how to improve the physical quality of primary and middle school students has become the top priority of physical education. Based on the requirements and guidance of China’s physical education in the new curriculum standard, this paper innovates China’s physical education classroom teaching to a certain extent based on deep learning algorithm. The final results show that the students who choose PE class based on deep learning algorithm account for about 85% of the total number of students, which exceeds the students who choose traditional PE class by 70%. Therefore, we believe that the estimation and recognition of students’ movement posture in PE class can not only greatly improve students’ enthusiasm for physical exercise but also avoid sports injuries caused by inaccurate operation in the process of exercise. Human posture estimation is to detect the position of each part of the human body from the image and calculate its direction and scale information. The advent of the era of big data is based on the relationship between multiple frames of images, while human posture recognition is based on the processing of single-frame static images. Correctly recognizing the pose information of multiple frames of continuous images still makes it possible to realize correct behavior analysis and understanding.

Self-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions

This work presents the vision pipeline for our in-house developed autonomous reconfigurable pavement sweeping robot named Panthera. As the goal of Panthera is to be an autonomous self-reconfigurable robot, it has to understand the type of pavement it is moving in so that it can adapt smoothly to changing pavement width and perform cleaning operations more efficiently and safely. deep learning (DL) based vision pipeline is proposed for the Panthera robot to recognize pavement features, including pavement type identification, pavement surface condition prediction, and pavement width estimation. The DeepLabv3+ semantic segmentation algorithm was customized to identify the pavement type classification, an eight-layer CNN was proposed for pavement surface condition prediction. Furthermore, pavement width estimation was computed by fusing the segmented pavement region on the depth map. In the end, the fuzzy inference system was implemented by taking input as the pavement width and its conditions detected and output as the safe operational speed. The vision pipeline was trained using the DL provided with the custom pavement images dataset. The performance was evaluated using offline test and real-time field trial images captured through the reconfigurable robot Panthera stereo vision sensor. In the experimental analysis, the DL-based vision pipeline components scored 88.02% and 93.22% accuracy for pavement segmentation and pavement surface condition assessment, respectively, and took approximately 10 ms computation time to process the single image frame from the vision sensor using the onboard computer.

Cabbage and Weed Identification Based on Machine Learning and Target Spraying System Design.

The complexity of natural elements seriously affects the accuracy and stability of field target identification, and the speed of an identification algorithm essentially limits the practical application of field pesticide spraying. In this study, a cabbage identification and pesticide spraying control system based on an artificial light source was developed. With the image skeleton point-to-line ratio and ring structure features of support vector machine classification and identification, a contrast test of different feature combinations of a support vector machine was carried out, and the optimal feature combination of the support vector machine and its parameters were determined. In addition, a targeted pesticide spraying control system based on an active light source and a targeted spraying delay model were designed, and a communication protocol for the targeted spraying control system based on electronic control unit was developed to realize the controlled pesticide spraying of targets. According to the results of the support vector machine classification test, the feature vector comprised of the point-to-line ratio, maximum inscribed circle radius, and fitted curve coefficient had the highest identification accuracy of 95.7%, with a processing time of 33 ms for a single-frame image. Additionally, according to the results of a practical field application test, the average identification accuracies of cabbage were 95.0%, average identification accuracies of weed were 93.5%, and the results of target spraying at three operating speeds of 0.52 m/s, 0.69 m/s and 0.93 m/s show that the average invalid spraying rate, average missed spraying rate, and average effective spraying rate were 2.4, 4.7, and 92.9%, respectively. Moreover, it was also found from the results that with increasing speeds, the offset of the centre of the mass of the target increased and reached a maximum value of 28.6 mm when the speed was 0.93 m/s. The void rate and pesticide saving rate were 65 and 33.8% under continuous planting conditions and 76.6 and 53.3% under natural seeding deficiency conditions, respectively.

Lane Detection Based on Two-Stage Noise Features Filtering and Clustering

Lane detection is essential for autonomous vehicles, and vision-based lane detection is widely used in the field of intelligent driving cars because of its low cost. Aiming to solve the problem of false detection caused by surrounding vehicles during lane detection, a new multi-lane detection method with strong anti-interference ability is proposed. The main contribution of this work is the utilization of object detection neural network which is applied to remove vehicles from road images as much as possible. Compared with the traditional feature-based detection algorithms, this method can preserve lane line features more accurately and effectively. Firstly, binarization of global optimal threshold method is utilized to extract the basic features of lanes. In order to further remove non-lane line noises and extract lane features as clear as possible, YOLOv4-tiny object detection network is introduced to detect and filter vehicles on the road. Then the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm with denoising function is applied to cluster the different lane features. Finally, an improved RANSAC (Random Sample Consensus) method is used to perform quadratic curve fitting on the clustered features. The experimental results on dataset show that consumed time of a single frame image is only 0.119 second and the precision of the proposed method can reach 94.4%. Compared with other detection algorithms, the proposed algorithm has good real-time performance, stability and robustness in a variety of complex scenarios.

Novel streak-resolving algorithm for particle streak velocimetry

In this paper, a novel streak-resolving algorithm is proposed for particle streak velocimetry (PSV) to resolve velocity fields for a single image frame with multiple particle streaks. A model streak function, which was based on the temporal integral of the particle image intensity moving along its trajectory during the exposure period, was approximated using a multivariable least-square fit procedure to reconstruct the streak information and the corresponding particle trajectory. Lagrangian tracking was achieved, and the velocities were evaluated by differentiating the resolved particle trajectory with respect to time. Two types of flows, accelerating parallel flow and the rotational flow of Hill’s vortex, were used to generate synthetic streaks for the performance tests. Three types of relative error were defined and used to evaluate the performance of the algorithm in terms of statistical mean and standard deviation (SD) errors. The accuracy of the fitted streak parameters, such as particle image intensity and diameter, were also evaluated and compared. The results reveal that the error and SD were low if the image noise is below 1.0%; for noise levels of 5.0%, the error was up to 10% with an SD of up to 12%. The error and SD of the particle image intensity and particle image diameter for both flow types were also 0%–7% for clean and up to 12% for noisy images. The processing results for experimental streak images of flow past a cylinder reveal that these images can be resolved using the proposed algorithm with a residual mean error of 4.38 and an SD of 9.48. These results suggest that the proposed novel approach can be used to resolve velocity fields with only a single image frame and without expensive hardware for high-speed imaging and thus is suitable for diverse applications.

IRSTFormer: A Hierarchical Vision Transformer for Infrared Small Target Detection

Infrared small target detection occupies an important position in the infrared search and track system. The most common size of infrared images has developed to 640×512. The field-of-view (FOV) also increases significantly. As the result, there is more interference that hinders the detection of small targets in the image. However, the traditional model-driven methods do not have the capability of feature learning, resulting in poor adaptability to various scenes. Owing to the locality of convolution kernels, recent convolutional neural networks (CNN) cannot model the long-range dependency in the image to suppress false alarms. In this paper, we propose a hierarchical vision transformer-based method for infrared small target detection in larger size and FOV images of 640×512. Specifically, we design a hierarchical overlapped small patch transformer (HOSPT), instead of the CNN, to encode multi-scale features from the single-frame image. For the decoder, a top-down feature aggregation module (TFAM) is adopted to fuse features from adjacent scales. Furthermore, after analyzing existing loss functions, a simple yet effective combination is exploited to optimize the network convergence. Compared to other state-of-the-art methods, the normalized intersection-over-union (nIoU) on our IRST640 dataset and public SIRST dataset reaches 0.856 and 0.758. The detailed ablation experiments are conducted to validate the effectiveness and reasonability of each component in the method.

Study on the Influence of Random Vibration of Space-Based Payload on Area-Array Camera Frame-by-Frame Imaging

Satellite vibration is an important factor that can seriously reduce the image quality of remote sensing imaging. In this paper, the influence of the random vibration of the payload on the frame-by-frame imaging quality of the area-array CMOS camera is studied. Firstly, the mode of the camera’s dynamic frame-by-frame imaging is established; secondly, the curvature mapping function between the ground and camera focal planes is derived based on the invariance of the field of view (FOV). The frequency domain-filtered template of random vibration is constructed. Then, the distortion and degradation images, of single-frame images at different attitude angles, are inversed; finally, the influence of attitude angle, exposure time, and the error of velocity, caused by random vibrations on dynamic frame-by-frame imaging, are analyzed. The simulation results show that the degree of image edge distortion gets larger when the attitude angle increases from 0° to 50°. At the same time, the MTF decreases rapidly with the increase of the error of velocity and the attitude angle. Meanwhile, when the output-image SSIM decreases by 0.1, the MSE increases by 18.5. The experimental results show that the field of view (FOV) of dynamic imaging should be reasonably set, and the error of velocity should be effectively reduced to obtain high-quality remote sensing images.

Translating Hand Gestures Using 3D Convolutional Neural Network

Hand gestures are one of the mediums that many people use to communicate with each other. The use of gesture recognition applications has become increasingly popular in recent years especially in computer vision areas. Typically, gestures can easily be recognized from a single image frame (i.e. alphabet from sign language), however the ability to recognize complex gestures with subtle differences between movement requires more works and larger datasets. In this work, we introduce a simple gesture recognition system that translates 5 different hand gestures, namely “doing other things”, “swiping down”, “swiping left”, “zooming out with two fingers” and “drumming fingers”. We used datasets obtained from Jester dataset. The inputs were processed in ‘RGB’ format during the pre-processing phase and a spatiotemporal filter were used as a feature extraction method, which were also the main building block in this system. Next, we trained the features using 3D Convolution Neural Network (3D-CNN). Further, we used real-time video to test the developed recognition system with 5 different actors. Findings show that the developed model can translate hand gestures with accuracy of 85.70% and 0.4% losses.

A Lightweight Semantic Segmentation Model of Wucai Seedlings Based on Attention Mechanism

Accurate wucai seedling segmentation is of great significance for growth detection, seedling location, and phenotype detection. To segment wucai seedlings accurately in a natural environment, this paper presents a lightweight segmentation model of wucai seedlings, where U-Net is used as the backbone network. Specifically, to improve the feature extraction ability of the model for wucai seedlings of different sizes, a multi-branch convolution block based on inception structure is proposed and used to design the encoder. In addition, the expectation “maximizationexpectation” maximization attention module is added to enhance the attention of the model to the segmentation object. In addition, because of the problem that a large number of parameters easily increase the difficulty of network training and computational cost, the depth-wise separable convolution is applied to replace the original convolution in the decoding stage to lighten the model. The experimental results show that the precision, recall, MIOU, and F1-score of the proposed model on the self-built wucai seedling dataset are 0.992, 0.973, 0.961, and 0.982, respectively, and the average recognition time of single frame image is 0.0066 s. Compared with several state-of-the-art models, the proposed model achieves better segmentation performance and has the characteristics of smaller-parameter scale and higher real-time performance. Therefore, the proposed model can achieve good segmentation effect for wucai seedlings in natural environment, which can provide important basis for target spraying, growth recognition, and other applications.

Film Effect Optimization by Deep Learning and Virtual Reality Technology in New Media Environment.

Today, new media technology has widely penetrated art forms such as film and television, which has changed the way of visual expression in the new media environment. To better solve the problems of weak immersion, poor interaction, and low degree of simulation, the present work uses deep learning technology and virtual reality (VR) technology to optimize the film playing effect. Firstly, the optimized extremum median filter algorithm is used to optimize the “burr” phenomenon and a low compression ratio of the single video image. Secondly, the Generative Adversarial Network (GAN) in deep learning technology is used to enhance the data of the single video image. Finally, the decision tree algorithm and hierarchical clustering algorithm are used for the color enhancement of VR images. The experimental results show that the contrast of a single-frame image optimized by this system is 4.21, the entropy is 8.66, and the noise ratio is 145.1, which shows that this method can effectively adjust the contrast parameters to prevent the loss of details and reduce the dazzling intensity. The quality and diversity of the specific types of images generated by the proposed GAN are improved compared with the current mainstream GAN method with supervision, which is in line with the subjective evaluation results of human beings. The Frechet Inception Distance value is also significantly improved compared with Self-Attention Generative Adversarial Network. It shows that the sample generated by the proposed method has precise details and rich texture features. The proposed scheme provides a reference for optimizing the interactivity, immersion, and simulation of VR film.

Single-shot multi-color fluorescence microscopy via a colorimetry camera.

Multi-color fluorescence microscopy presents highly detailed biological samples interactively. However, current multi-color methods suffer from an intricate optical setup, complicated image analysis, or a long acquisition time. To address these issues, here we develop a simple multi-color method based on a customized colorimetry camera to enable the detection of multiple structures from single-shot acquisition. The unfiltered channel (W pixels) and color channels (R, G, B, and NIR pixels) in this customized camera simultaneously provide a broad detection wavelength range and high detection sensitivity. We built a simple optical setup by replacing the monochrome camera in a basic fluorescence microscopy system with a colorimetry camera, and developed effective image analysis procedures to reconstruct a multi-color image from a single frame of a raw image. We demonstrated single-shot four-color wide-field fluorescence imaging on fixed cos-7 cells with < 5% cross talk, which is comparable to the best reported values. Our method greatly simplifies both the optical system and image analysis in the widely used method of multi-color fluorescence microscopy, thus offering an effective and easy way to study multiple objects at the same time.

A Fusion Algorithm of Object Detection and Tracking for Unmanned Surface Vehicles.

To provide reliable input for obstacle avoidance and decision-making, unmanned surface vehicles (USV) need to have the function of sensing the position of other USV targets in the process of cooperation and confrontation. Due to the small size of the target and the interference of the water and sky background, the current algorithms are prone to missed detection and drift problems when detecting and tracking USV. Therefore, in this paper, we propose a fusion algorithm of detection and tracking for USV targets. To solve the problem of vague features in the single-frame image, high-resolution and deep semantic information are obtained through a cross-stage partial network, and the anchor and convolution structure in the network has been improved given the characteristics of USV; besides, to meet the real-time requirements, the detected target is quickly tracked through correlation filtering, and the correlation characteristics of multi-frame images are obtained; then, the correlation characteristics are used to significantly reduce missed detection, and the tracking drift problems are corrected, combined with high-resolution semantic features of a single frame. Finally, the fusion algorithm is designed. In this paper, we constructed a picture dataset and a video dataset to test the effect of detection, tracking, and fusion algorithm separately, which proves the superiority of the fusion algorithm in this paper. The results show that, compared with a single detection algorithm and tracking algorithm, the fusion one can increase the success rate by more than 10%.

Deep iterative residual back-projection networks for single-image super-resolution

The process of single-image super-resolution (SR) has certain limitations, such as an insufficient utilization of high-frequency information in images and a network structure that is insufficiently flexible to reconstruct the feature information of different complexities. Therefore, deep iterative residual back-projection networks are proposed. Residual learning was used to ease the difficulty in training and fully discover the feature information of the image, and a back-projection method was applied to study the interdependence between high- and low-resolution images. In addition, the network structure reconstructs smooth-feature and high-frequency information of the image separately and transmits only the residual features among all residual blocks of the network structure. The experiment results show that compared with most single-frame image SR methods, the proposed approach not only achieves a significant improvement in objective indicators, but it also provides richer texture information in the reconstructed predicted image.

Analysis of Aerobics Auxiliary Training Based on Deep Learning

With the in-depth integration of information technology and subject teaching, it is also an inevitable trend to apply modern information technology to aerobics teaching. In this paper, the N-best algorithm is used in the video and real-time camera in aerobics, so that the human posture parameters in a single-frame image can be estimated. By using the relative position and motion direction of each part of the human body to describe the characteristics of aerobics, the Laplace scoring method is used to reduce the dimension of the data, and the discriminant human motion feature vector with a strong local topological structure is obtained. Finally, the iterative self-organizing data analysis technology (ISODATA algorithm) is used to dynamically determine the keyframe. In the aerobics video keyframe extraction experiments, the ST-FMP model improves the recognition accuracy of nondeterministic body parts of the flexible hybrid articulated human model (FMP) by about 15 percentage points and achieves 81% keyframe extraction accuracy, which is better than the keyframe algorithms of KFE and motion block. The proposed algorithm is sensitive to human motion features and human pose and is suitable for motion video annotation review.