Sequence Of Image Frames Research Articles

On robust optical flow estimation on image sequences with differently exposed frames using primal–dual optimization

Optical flow methods are used to estimate pixelwise motion information based on consecutive frames in image sequences. The image sequences traditionally contain frames that are similarly exposed. However, many real-world scenes contain high dynamic range content that cannot be captured well with a single exposure setting. Such scenes result in certain image regions being over- or underexposed, which can negatively impact the quality of motion estimates in those regions. Motivated by this, we propose to capture high dynamic range scenes using different exposure settings every other frame. A framework for OF estimation on such image sequences is presented, that can straightforwardly integrate techniques from the state-of-the-art in conventional OF methods. Different aspects of robustness of OF methods are discussed, including estimation of large displacements and robustness to natural illumination changes that occur between the frames, and we demonstrate experimentally how to handle such challenging flow estimation scenarios. The flow estimation is formulated as an optimization problem whose solution is obtained using an efficient primal–dual method.

From handcrafted to learned representations for human action recognition: A survey

Human action recognition is an important branch among the studies of both human perception and computer vision systems. Along with the development of artificial intelligence, deep learning techniques have gained remarkable reputation when dealing with image categorization tasks (e.g., object detection and classification). However, since human actions normally present in the form of sequential image frames, analyzing human action data requires significantly increased computational power than still images when deep learning techniques are employed. Such a challenge has been the bottleneck for the migration of learning-based image representation techniques to action sequences, so that the old fashioned handcrafted human action representations are still widely used for human action recognition tasks. On the other hand, since handcrafted representations are usually ad-hoc and overfit to specific data, they are incapable of being generalized to deal with various realistic scenarios. Consequently, resorting to deep learning action representations for human action recognition tasks is eventually a natural option. In this work, we provide a detailed overview of recent advancements in human action representations. As the first survey that covers both handcrafted and learning-based action representations, we explicitly discuss the superiorities and limitations of exiting techniques from both kinds. The ultimate goal of this survey is to provide comprehensive analysis and comparisons between learning-based and handcrafted action representations respectively, so as to inspire action recognition researchers towards the study of both kinds of representation techniques.

Optical flow estimation on image sequences with differently exposed frames

Optical flow (OF) methods are used to estimate dense motion information between consecutive frames in image sequences. In addition to the specific OF estimation method itself, the quality of the input image sequence is of crucial importance to the quality of the resulting flow estimates. For instance, lack of texture in image frames caused by saturation of the camera sensor during exposure can significantly deteriorate the performance. An approach to avoid this negative effect is to use different camera settings when capturing the individual frames. We provide a framework for OF estimation on such sequences that contain differently exposed frames. Information from multiple frames are combined into a total cost functional such that the lack of an active data term for saturated image areas is avoided. Experimental results demonstrate that using alternate camera settings to capture the full dynamic range of an underlying scene can clearly improve the quality of flow estimates. When saturation of image data is significant, the proposed methods show superior performance in terms of lower endpoint errors of the flow vectors compared to a set of baseline methods. Furthermore, we provide some qualitative examples of how and when our method should be used.

Embedded intention scripts representation and real-time interpretation metrics extraction methodology with gaze annotation on visual content

This paper proposes a system platform with a capability of analyzing and measuring the degree of similarities and differences of the gaze behaviors of the multiple users with the intention of the provider in real-time. In the proposed system platform, a video content incorporates the intention flow script, which describes the intended sequence and duration of the information embedded into the visual content. The video content also incorporates the intention weight script for assigning importance values to individual object information so that the significance of the gaze behavior can be assesses. In this paper, a set of evaluation metrics and their usages are defined, which can be used to evaluate the degree of gazing behavior closeness and deviation from the intention. The proposed method can be applied in many applications, such as in educational and commercial domains, where the information contents are usually designed with a specific intent. In the paper, we verified the proposed method by incorporation intention script into a sequence of educational image frames to analyze the gaze behavior of multiple users.

Phosphorylation of the exchange factor DENND3 by ULK in response to starvation activates Rab12 and induces autophagy.

Unc-51-like kinases (ULKs) are the most upstream kinases in the initiation of autophagy, yet the molecular mechanisms underlying their function are poorly understood. We report a new role for ULK in the induction of autophagy. ULK-mediated phosphorylation of the guanine nucleotide exchange factor DENND3 at serines 554 and 572 upregulates its GEF activity toward the small GTPase Rab12. Through binding to LC3 and associating with LC3-positive autophagosomes, active Rab12 facilitates autophagosome trafficking, thus establishing a crucial role for the ULK/DENND3/Rab12 axis in starvation-induced autophagy.

Water Hazard Detection for Intelligent Vehicle Based on Vision Information

Water hazard is one of the most dangerous obstacles for unmanned vehicle to travel in off-road environment. The vision information for water hazard body includes relatively high brightness, low saturation and smooth texture. In this paper, static image and sequence image are captured by CCD. Color and texture information extracted from image pixels would be utilized to form a feature matrix to train the SVM classifier. While, with the assistance of RBF kernel function, low-dimensional sample space is projected to high-dimensional space. Based on large amount of experiments, RBF kernel function parameters are optimized by using grid method. The optimized RBF kernel function parameters are proved satisfactory when detecting in the static water image. Notice that there is certain relationship between target position and its scale for the adjacent frame. SURF feature detection and matching method can be used to match feature point between adjacent frames in image sequences. The searching window size and position in new frame could be updated in time, which allows to detect water hazard in a relatively small region. Water obstacle tracking experiment with patents proved the satisfactory performance of the SURF method in this paper.

Echo decorrelation imaging for quantification of tissue structural changes during ultrasound ablation

Echo decorrelation imaging is a pulse-echo method that maps millisecond-scale changes in backscattered ultrasound signals, potentially providing real-time feedback during thermal ablation treatments. Decorrelation between echo signals from sequential image frames is spatially mapped and temporally averaged, resulting in images of cumulative, heat-induced tissue changes. Theoretical analysis indicates that the mapped echo decorrelation parameter is equivalent to a spatial decoherence spectrum of the tissue reflectivity, and also provides a method to compensate decorrelation artifacts caused by tissue motion and electronic noise. Results are presented from experiments employing 64-element linear arrays that perform bulk thermal ablation, focal ablation, and pulse-echo imaging using the same piezoelectric elements, ensuring co-registration of ablation and image planes. Decorrelation maps are shown to correlate with ablated tissue histology, including vital staining to map heat-induced cell death, for both ex vivo ablation of bovine liver tissue and in vivo ablation of rabbit liver with VX2 carcinoma. Receiver operating characteristic curve analysis shows that echo decorrelation predicts local ablation with greater success than integrated backscatter imaging. Using artifact-compensated echo decorrelation maps, heating-induced decoherence of tissue scattering media is assessed for ex vivo and in vivo ultrasound ablation by unfocused and focused beams.

Spatiotemporal lipid profiling during early embryo development of Xenopus laevis using dynamic ToF-SIMS imaging

Time-of-flight secondary ion mass spectrometry (ToF-SIMS) imaging has been used for the direct analysis of single intact Xenopus laevis embryo surfaces, locating multiple lipids during fertilization and the early embryo development stages with subcellular lateral resolution (∼4 μm). The method avoids the complicated sample preparation for lipid analysis of the embryos, which requires selective chemical extraction of a pool of samples and chromatographic separation, while preserving the spatial distribution of biological species. The results show ToF-SIMS is capable of profiling multiple components (e.g., glycerophosphocholine, SM, cholesterol, vitamin E, diacylglycerol, and triacylglycerol) in a single X. laevis embryo. We observe lipid remodeling during fertilization and early embryo development via time course sampling. The study also reveals the lipid distribution on the gamete fusion site. The methodology used in the study opens the possibility of studying developmental biology using high resolution imaging MS and of understanding the functional role of the biological molecules.

Vehicle Association and Tracking in Image Sequences Using Feature-Based Similarity Comparison

Object association and tracking have attracted great attention in the computer vision. In this paper, we present an object association and tracking method for monitoring multiple vehicles on the road based on objects' visual features and the similarity comparison between them. First, we identify vehicles using the difference operation between the current frame in CCTV image sequences and the referential images that are stored in a database, and then extract various features from the vehicles identified. Finally, we associate the objects in the current frame with those in the next frames using similarity comparison, and track multiple objects over a sequence of CCTV image frames. Empirical study using CCTV images shows that our method has achieved the considerable effectiveness in tracking vehicles on the road.

Estimation of Angular Acceleration By a Monocular Vision Sensor

Recently, monitoring of two-body ground vehicles carrying extremely hazardous materials has been considered as one of the most important national issues. This issue induces large cost in terms of national economy and social benefit. To monitor and counteract accidents promptly, an efficient methodology is required. For accident monitoring, GPS can be utilized in most cases. However, it is widely known that GPS cannot provide sufficient continuity in urban cannons and tunnels. To complement the weakness of GPS, this paper proposes an accident monitoring method based on a monocular vision sensor. The proposed method estimates angular acceleration from a sequence of image frames captured by a monocular vision sensor. The possibility of using angular acceleration is investigated to determine the occurrence of accidents such as jackknifing and rollover. By an experiment based on actual measurements, the feasibility of the proposed method is evaluated.

Detecting and measuring moving objects with airborne repeat station imaging in rapid succession mode

This letter demonstrates the potential of a repeat station imaging and registration approach applied to rapid sequence aerial images captured along the same flightline for detecting moving objects and measuring their velocities. Sequential images share similar view geometries and thus can be precisely registered with simple warping transformations based on just a few matching points. Moving objects were detected as brightness anomalies between sequential images. Precise registration and low-pass filtering minimized false detections associated with misregistration. Velocities were estimated by measuring displacements of moving objects depicted in registered sequential image frames and dividing by the time interval between exposures.

Robust Video-Based Barcode Recognition via Online Sequential Filtering

We consider the visual barcode recognition problem in a noisy video data setup. Unlike most existing single-frame recognizers that require considerable user effort to acquire clean, motionless and blur-free barcode signals, we eliminate such extra human efforts by proposing a robust video-based barcode recognition algorithm. We deal with a sequence of noisy blurred barcode image frames by posing it as an online filtering problem. In the proposed dynamic recognition model, at each frame we infer the blur level of the frame as well as the digit class label. In contrast to a frame-by-frame based approach with heuristic majority voting scheme, the class labels and frame-wise noise levels are propagated along the frame sequences in our model, and hence we exploit all cues from noisy frames that are potentially useful for predicting the barcode label in a probabilistically reasonable sense. We also suggest a visual barcode tracking approach that efficiently localizes barcode areas in video frames. The effectiveness of the proposed approaches is demonstrated empirically on both synthetic and real data setup.

Features classification using geometrical deformation feature vector of support vector machine and active appearance algorithm for automatic facial expression recognition

This paper proposes a method for facial expression recognition in image sequences. Face is detected from the scene and then facial features are detected using image normalization, and thresholding techniques. Using an optimization algorithm the Candide wire frame model is adapted properly on the first frame of face image sequence. In the subsequent frames of image sequence facial features are tracked using active appearance algorithm. Once the model fits on the first frame, animation parameters of model are set to zero, to obtain the shape of model for the neutral facial expression of the same face. The last frame of the image sequence corresponds to greatest facial expression intensity. The geometrical displacement of the Candide wire frame nodes, between the neutral expression frame and the last frame, is used as an input to the multiclass support vector machine, which classifies facial expression into one of the class such as happy, surprise, sadness, anger, disgust, fear and neutral. This method is applicable for frontal as well as tilted faces with angle $$\pm 30\,^{\circ }, \pm 45\,^{\circ }, \pm 60\,^{\circ }$$ ± 30 ? , ± 45 ? , ± 60 ? with respect to y axis.

Adaptive scanning methodology for object‐oriented visual information processing

The raster scan method in visual information processing is the rectangular pattern of image capture and reconstruction which systematically covers the area progressively, one line at a time starting from the top left corner. This traditional scanning method might work well for still image processing but for moving images, a scanning method such as diagonal snake scan can provide a better object correlation in a sequence of image frames. A novel adaptive scanning method for object-oriented visual information processing is proposed. Here, image pixels are collected starting from the centre of the image area and progressively extended outwards similar to the water drop ripple effect. Simulation results have shown that the proposed work uses up to 33% less processing time to cover the region-of-interest in an image compared to the raster scan method. Preliminary simulation results have also shown that with the proposed adaptive scanning methodology, there is a significant improvement in the performance of the object-oriented visual compression.

A Novel Traffic Tracking System Based on division of Video into Frames and Processing

The objective of this paper is to visualize and analyze video.Videos are sequence of image frames. In this work, algorithm will be developed to analyze a frame and the same will be applied to all frames in a video. It is expected see unwanted objects in video frame, which can be removed by converting colour frames into a gray scale and implement thresh holding algorithm on an image. Threshold can be set depending on the object to be detected. Gray scale image will be converted to binary during thresh holding process. To reduce noise, to improve the robustness of the system, and to reduce the error rate in detection and tracking process, morphological image processing method for binary images is used. Morphological processing will be applied on binary image to remove small unwanted objects that are presented in a frame. A developed blob analysis technique for extracted binary image facilitates pedestrian and car detection. Processing blob’s information of relative size and location leads to distinguishing between pedestrian and car. The threshold, morphological and blobs process is applied to all frames in a video and finally original video with tagged cars will be displayed.

Remote sensing of wave runup over breakwater slope in field using optical imagery

Yoo, J., Choi, J.-Y., Yoon, J.-J., Jun, K.-C., Shim, J.-S. Kim, S. and Min, I.-K., 2013. Measurements of wave runup over breakwater observed in field using optical imagery.In this paper, wave run-up on the seaward slope of a breakwater in field was measured remotely using optical video imagery and investigated through comparisons with an empirical formula. In order to extract properties of wave run-up and over-topping on the breakwater slope from field videos, individual image frames were sequentially sampled at 3 Hz from a video burst first. Then, the sampled sequential image frames were rectified into real world coordinates. A crossshore image timestack was generated by collecting an image pixel array, which was defined along the cross-shore transect on the seaward breakwater face of interest. Trajectory of the run-up on the seaward face evolving in time was extracted from the image timestack using a line detection method. The properties of the run-up estimated from field video bursts were combined to the measurements of incident waves, in order to derive an empirical relation between incident waves and run-up. The video-based relative wave run-up height (i.e. Ru,2% / Hs) at a typical armored breakwater with tetrapods was found to be about 2.15. This value is about in the range of the general formula, which was derived from laboratory experiments in earlier studies.

Design of an efficient real-time algorithm using reduced feature dimension for recognition of speed limit signs.

We propose a real-time algorithm for recognition of speed limit signs from a moving vehicle. Linear Discriminant Analysis (LDA) required for classification is performed by using Discrete Cosine Transform (DCT) coefficients. To reduce feature dimension in LDA, DCT coefficients are selected by a devised discriminant function derived from information obtained by training. Binarization and thinning are performed on a Region of Interest (ROI) obtained by preprocessing a detected ROI prior to DCT for further reduction of computation time in DCT. This process is performed on a sequence of image frames to increase the hit rate of recognition. Experimental results show that arithmetic operations are reduced by about 60%, while hit rates reach about 100% compared to previous works.

Single Molecule Fluorescence Detection and Tracking in Mammalian Cells: The State-of-the-Art and Future Perspectives

Insights from single-molecule tracking in mammalian cells have the potential to greatly contribute to our understanding of the dynamic behavior of many protein families and networks which are key therapeutic targets of the pharmaceutical industry. This is particularly so at the plasma membrane, where the method has begun to elucidate the mechanisms governing the molecular interactions that underpin many fundamental processes within the cell, including signal transduction, receptor recognition, cell-cell adhesion, etc. However, despite much progress, single-molecule tracking faces challenges in mammalian samples that hinder its general application in the biomedical sciences. Much work has recently focused on improving the methods for fluorescent tagging of target molecules, detection and localization of tagged molecules, which appear as diffraction-limited spots in charge-coupled device (CCD) images, and objectively establishing the correspondence between moving particles in a sequence of image frames to follow their diffusive behavior. In this review we outline the state-of-the-art in the field and discuss the advantages and limitations of the methods available in the context of specific applications, aiming at helping researchers unfamiliar with single molecules methods to plan out their experiments.

Features classification using support vector machine for a facial expression recognition system

A methodology for automatic facial expression recognition in image sequences is proposed, which makes use of the Candide wire frame model and an active appearance algorithm for tracking, and support vector machine (SVM) for classification. A face is detected automatically from the given image sequence and by adapting the Candide wire frame model properly on the first frame of face image sequence, facial features in the subsequent frames are tracked using an active appearance algorithm. The algorithm adapts the Candide wire frame model to the face in each of the frames and then automatically tracks the grid in consecutive video frames over time. We require that first frame of the image sequence corresponds to the neutral facial expression, while the last frame of the image sequence corresponds to greatest intensity of facial expression. The geometrical displacement of Candide wire frame nodes, defined as the difference of the node coordinates between the first and the greatest facial expression intensity frame, is used as an input to the SVM, which classify the facial expression into one of the classes viz happy, surprise, sadness, anger, disgust, and fear.

Effect of plasma nitriding on fatigue crack growth on AISI 4140 steel under variable amplitude loading

AISI 4140 steel was subjected to a plasma nitrification process using 50/50 gas mixture of N 2 and H 2 at 500 °C for 2 h. The structural and mechanical properties of the plasma-nitrided and -unnitrided specimens were investigated using microhardness test device, servo-hydraulic universal testing machine, scanning electron microscope, and optical imaging devices. The influence of the plasma nitrification on fatigue crack growth (FCG) was investigated by applying block overload cycles on test specimens. The rate of FCG was accurately determined from the sequence of image frames taken from the specimens during testing. Results of analyses indicated that the application of a 10-cycle tensile block overload results in the retardation of FCG for both unnitrided and nitrided specimens, however, the period of crack retardation is more prolonged for the unnitrided specimens. Besides, it was also found that in a constant amplitude fatigue test the plasma nitrification leads to an increase in total FCG life.