Input Frames Research Articles

Automatic 3D-display-friendly scene extraction from video sequences and optimal focusing distance identification

This paper proposes a method for an automatic detection of 3D-display-friendly scenes from video sequences. Manual selection of such scenes by a human user would be extremely time consuming and would require additional evaluation of the result on 3D display. The input videos can be intentionally captured or taken from other sources, such as films. First, the input video is analyzed and the camera trajectory is estimated. The optimal frame sequence that follows defined rules, based on optical attributes of the display, is then extracted. This ensures the best visual quality and viewing comfort. The following identification of a correct focusing distance is an important step to produce a sharp and artifact-free result on a 3D display. Two novel and equally efficient focus metrics for 3D displays are proposed and evaluated. Further scene enhancements are proposed to correct the unsuitably captured video. Multiple image analysis approaches used in the proposal are compared in terms of both quality and time performance. The proposal is experimentally evaluated on a state-of-the-art 3D display by Looking Glass Factory and is suitable even for other multi-view devices. The problem of optimal scene detection, which includes the input frames extraction, resampling, and focusing, was not addressed in any previous research. Separate stages of the proposal were compared with existing methods, but the results show that the proposed scheme is optimal and cannot be replaced by other state-of-the-art approaches.

A Light-Weight Autoregressive CNN-Based Frame Level Transducer Decoder for End-to-End ASR

A convolutional neural network (CNN) transducer decoder was proposed to reduce the decoding time of an end-to-end automatic speech recognition (ASR) system while maintaining accuracy. The CNN of 177 k parameters and a kernel size of 6 generates the probabilities of the current token at the token level, at the token transition of the output token sequence. Two probabilities of the current token, one from the encoder and the other from the CNN are added to the frame level to reduce the decoding step to the number of input frames. An encoder composed of an 18-layer conformer was combined with the proposed decoder for training with the Librispeech data set. The forward-backward algorithm was used for training. The space and re-appearance tokens are added to the 300-word piece tokens to represent the token string. A space token appears at a frame between two words. A comparison with the autoregressive decoders such as transformer and RNN-T decoders demonstrates that this work provides comparable WERs with much less decoding time. A comparison with non-autoregressive decoders such as CTC indicates that this work enhanced WERs.

Video Frame Interpolation With Many-to-Many Splatting and Spatial Selective Refinement.

In this work, we first propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently. Given a frame pair, we estimate multiple bidirectional flows to directly forward warp the pixels to the desired time step before fusing any overlapping pixels. In doing so, each source pixel renders multiple target pixels and each target pixel can be synthesized from a larger area of visual context, establishing a many-to-many splatting scheme with robustness to undesirable artifacts. For each input frame pair, M2M has a minuscule computational overhead when interpolating an arbitrary number of in-between frames, hence achieving fast multi-frame interpolation. However, directly warping and fusing pixels in the intensity domain is sensitive to the quality of motion estimation and may suffer from less effective representation capacity. To improve interpolation accuracy, we further extend an M2M++ framework by introducing a flexible Spatial Selective Refinement (SSR) component, which allows for trading computational efficiency for interpolation quality and vice versa. Instead of refining the entire interpolated frame, SSR only processes difficult regions selected under the guidance of an estimated error map, thereby avoiding redundant computation. Evaluation on multiple benchmark datasets shows that our method is able to improve the efficiency while maintaining competitive video interpolation quality, and it can be adjusted to use more or less compute as needed.

An Effective Video Transformer With Synchronized Spatiotemporal and Spatial Self-Attention for Action Recognition.

Convolutional neural networks (CNNs) have come to dominate vision-based deep neural network structures in both image and video models over the past decade. However, convolution-free vision Transformers (ViTs) have recently outperformed CNN-based models in image recognition. Despite this progress, building and designing video Transformers have not yet obtained the same attention in research as image-based Transformers. While there have been attempts to build video Transformers by adapting image-based Transformers for video understanding, these Transformers still lack efficiency due to the large gap between CNN-based models and Transformers regarding the number of parameters and the training settings. In this work, we propose three techniques to improve video understanding with video Transformers. First, to derive better spatiotemporal feature representation, we propose a new spatiotemporal attention scheme, termed synchronized spatiotemporal and spatial attention (SSTSA), which derives the spatiotemporal features with temporal and spatial multiheaded self-attention (MSA) modules. It also preserves the best spatial attention by another spatial self-attention module in parallel, thereby resulting in an effective Transformer encoder. Second, a motion spotlighting module is proposed to embed the short-term motion of the consecutive input frames to the regular RGB input, which is then processed with a single-stream video Transformer. Third, a simple intraclass frame interlacing method of the input clips is proposed that serves as an effective video augmentation method. Finally, our proposed techniques have been evaluated and validated with a set of extensive experiments in this study. Our video Transformer outperforms its previous counterparts on two well-known datasets, Kinetics400 and Something-Something-v2.

Evaluating SLAM Performance with Synthesized Datasets from Unreal-Based Emulators

The rapid advancement in visual-inertial simultaneous localization and mapping (SLAM) has opened numerous applications in computer vision. However, the scarcity of high quality, publicly accessible datasets hampers the evaluation of SLAM performance in varied and tailored environments. In this study, I employed the AirSim simulator and the Unreal Engine 4 to generate a trajectory resembling that of the TUM VI Room 1 ground truth dataset within the ArchViz indoor environment representing a well-lit, furnished room. I further modified the environment and trajectory through various expansions, addition of features, and data smoothing to ensure a more stable sequence of input frames into the SLAM architecture. I then examined the efficiency of visual ORB-SLAM3 by inputting images of resolution 256×144 and 512×288 at 30 frames per second (FPS), while also adjusting the feature threshold - the maximum number of feature points that ORB-SLAM3 tracks per frame. This investigation of the camera parameters within AirSim and ORB-SLAM3 has led to the essential finding that the resolution of the input images must coincide with the dimensions of the film. The subsequent runs under these variables reveal that higher resolution images lead to considerably better tracking, with an optimal feature threshold ranging between 3000~12000 feature points per frame. Moreover, ORB- SLAM3 demonstrated significantly enhanced robustness within dynamic environments containing moving objects when using higher resolution inputs, with a decreased error of close to 0cm compared to 23.19cm for lower resolutions (averaged over three runs). Finally, I conducted qualitative testing using real-life indoor environments recorded with an iPhone Xr camera, which produces results that highlight the challenges faced by ORB-SLAM3 due to factors such as glare and motion blur.

Arbitrary Timestep Video Frame Interpolation with Time-Dependent Decoding

Given an observed low frame rate video, video frame interpolation (VFI) aims to generate a high frame rate video, which has smooth video frames with higher frames per second (FPS). Most existing VFI methods often focus on generating one frame at a specific timestep, e.g., 0.5, between every two frames, thus lacking the flexibility to increase the video’s FPS by an arbitrary scale, e.g., 3. To better address this issue, in this paper, we propose an arbitrary timestep video frame interpolation (ATVFI) network with time-dependent decoding. Generally, the proposed ATVFI is an encoder–decoder architecture, where the interpolation timestep is an extra input added to the decoder network; this enables ATVFI to interpolate frames at arbitrary timesteps between input frames and to increase the video’s FPS at any given scale. Moreover, we propose a data augmentation method, i.e., multi-width window sampling, where video frames can be split into training samples with multiple window widths, to better leverage training frames for arbitrary timestep interpolation. Extensive experiments were conducted to demonstrate the superiority of our model over existing baseline models on several testing datasets. Specifically, our model trained on the GoPro training set achieved 32.50 on the PSNR metric on the commonly used Vimeo90k testing set.

Wanet: weight and attention network for video summarization

In this paper, we propose a deep learning-based model, called Weight and Attention Network (WANet), for video summarization. The network comprises a simple multi-head attention mechanism, followed by a feed-forward network to obtain the frame importance scores. Summary keyshots are obtained from the scores using a combination of kernel temporal segmentation and the knapsack algorithm. Contrary to past methods, we first enrich the input frames with similar information as opposed to letting the model learn all the features by itself. A novel weight assignment mechanism is introduced to assign weights to the input frames based on their similarity before passing the same to the model. Experimental results on the SumMe and TVSum datasets indicate the effectiveness of the present method when compared to state-of-the-art methods applied to the same datasets.

Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks.

Convolutional neural networks (CNNs) have become instrumental in advancing multi-frame image super-resolution (SR), a technique that merges multiple low-resolution images of the same scene into a high-resolution image. In this paper, a novel deep learning multi-frame SR algorithm is introduced. The proposed CNN model, named Exponential Fusion of Interpolated Frames Network (EFIF-Net), seamlessly integrates fusion and restoration within an end-to-end network. Key features of the new EFIF-Net include a custom exponentially weighted fusion (EWF) layer for image fusion and a modification of the Residual Channel Attention Network for restoration to deblur the fused image. Input frames are registered with subpixel accuracy using an affine motion model to capture the camera platform motion. The frames are externally upsampled using single-image interpolation. The interpolated frames are then fused with the custom EWF layer, employing subpixel registration information to give more weight to pixels with less interpolation error. Realistic image acquisition conditions are simulated to generate training and testing datasets with corresponding ground truths. The observation model captures optical degradation from diffraction and detector integration from the sensor. The experimental results demonstrate the efficacy of EFIF-Net using both simulated and real camera data. The real camera results use authentic, unaltered camera data without artificial downsampling or degradation.

Image Intrinsic-Based Unsupervised Monocular Depth Estimation in Endoscopy.

Unsupervised monocular depth estimation plays a vital role for endoscopy-based minimally invasive surgery (MIS). However, it remains challenging due to the distinctive imaging characteristics of endoscopy which disrupt the assumption of photometric consistency, a foundation relied upon by conventional methods. Distinct from recent approaches taking image pre-processing strategy, this paper introduces a pioneering solution through intrinsic image decomposition (IID) theory. Specifically, we propose a novel end-to-end intrinsic-based unsupervised monocular depth learning framework that is comprised of an image intrinsic decomposition module and a synthesis reconstruction module. This framework seamlessly integrates IID with unsupervised monocular depth estimation, and dedicated losses are meticulously designed to offer robust supervision for network training based on this novel integration. Noteworthy, we rely on the favorable property of the resulting albedo map of IID to circumvent the challenging images characteristics instead of pre-processing the input frames. The proposed method is extensively validated on SCARED and Hamlyn datasets, and better results are obtained than state-of-the-art techniques. Beside, its generalization ability and the effectiveness of the proposed components are also validated. This innovative method has the potential to elevate the quality of 3D reconstruction in monocular endoscopy, thereby enhancing the accuracy and robustness of augmented reality navigation technology in MIS. Our code will be available at: https://github.com/bobo909/IID-SfmLearner.

Spatiotemporal Deep Learning-Based Cine Loop Quality Filter for Handheld Point-of-Care Echocardiography.

The reliability of automated image interpretation of point-of-care (POC) echocardiography scans depends on the quality of the acquired ultrasound data. This work reports on the development and validation of spatiotemporal deep learning models to assess the suitability of input ultrasound cine loops collected using a handheld echocardiography device for processing by an automated quantification algorithm (e.g. ejection fraction estimation). POC echocardiograms (n=885 DICOM cine loops from 175 patients) from two sites were collected using a handheld ultrasound device and annotated for image quality at the frame-level. Attributes of high-quality frames for left ventricular (LV) quantification included a temporally-stable LV, reasonable coverage of LV borders, and good contrast between the borders and chamber. Attributes of low-quality frames included temporal instability of the LV and/or imaging artifacts (e.g., lack of contrast, haze, reverberation, acoustic shadowing). Three different neural network architectures were investigated - (a) frame-level convolutional neural network (CNN) which operates on individual echo frames (VectorCNN), (b) single-stream sequence-level CNN which operates on a sequence of echo frames (VectorCNN+LSTM) and (c) two-stream sequence-level CNNs which operate on a sequence of echo and optical flow frames (VectorCNN+LSTM+Average, VectorCNN+LSTM+MinMax, and VectorCNN+LSTM+ConvPool). Evaluation on a sequestered test dataset containing 76 DICOM cine loops with 16,914 frames showed that VectorCNN+LSTM can effectively utilize both spatial and temporal information to regress the quality of an input frame (accuracy: 0.925, sensitivity = 0.860, specificity = 0.952), compared to the frame-level VectorCNN that only utilizes spatial information in that frame (accuracy: 0.903, sensitivity = 0.791, specificity = 0.949). Furthermore, an independent sample t-test indicated that the cine loops classified to be of adequate quality by the VectorCNN+LSTM model had a statistically significant lower bias in the automatically estimated EF (mean bias = - 3.73 ± 7.46 %, versus a clinically obtained reference EF) compared to the loops classified as inadequate (mean bias = -15.92 ± 12.17 %) (p = 0.007). Thus, cine loop stratification using the proposed spatiotemporal CNN model improves the reliability of automated point-of-care echocardiography image interpretation.

Modelling spatio-temporal energy consumption from nighttime radiance satellite dataset

National electricity consumption increases in line with continuous population growth and other socio-economic factors. The national electric power capacity goal develops largely for industrial manufacture and new settlement. The electrification- ratio on the target; is based on the accessibility of electricity services. The spatial distribution of electricity services coverage over the Indonesian territory is insufficient, particularly over the remote area that is out of electric services. Modeling by spatial (location) and temporal (year) to estimate electricity or energy consumption is necessary to develop using a low-light nighttime satellite dataset, therefore spatial boundaries can be accomplished. The modeling procedure starts by preparing the data frame of the independent variable input (amount of radiance) and the dependent variable output (the consumption of electricity or energy). The modelling method uses the curve-fitting approach where the indicator results by evaluating the R-square and RMSE values. The output model function is used to convert radiances into electrical power consumption units with a certain degree of accuracy. The selection of the input-output variable was achieved after variable analysis with the highest R-square outcome. Results indicate that the model functions in a polynomial form and correlations between variables are not simple. The selection of various model functions did not change the degree of correlation. The accumulative of energy radiances as independent variable input provides the optimum correlation result. The energy consumption from street lighting, in general, offers appropriate information that can be seen from satellites. The model function can be applied to a narrower spatial scale by input variable constraints.

MmPose-NLP: A Natural Language Processing Approach to Precise Skeletal Pose Estimation Using mmWave Radars.

In this article, we presented mmPose-NLP, a novel natural language processing (NLP) inspired sequence-to-sequence (Seq2Seq) skeletal key-point estimator using millimeter-wave (mmWave) radar data. To the best of our knowledge, this is the first method to precisely estimate up to 25 skeletal key points using mmWave radar data alone. Skeletal pose estimation is critical in several applications ranging from autonomous vehicles, traffic monitoring, patient monitoring, and gait analysis, to defense security forensics, and aid both preventative and actionable decision making. The use of mmWave radars for this task, over traditionally employed optical sensors, provides several advantages, primarily its operational robustness to scene lighting and adverse weather conditions, where optical sensor performance degrade significantly. The mmWave radar point-cloud (PCL) data are first voxelized (analogous to tokenization in NLP) and N frames of the voxelized radar data (analogous to a text paragraph in NLP) is subjected to the proposed mmPose-NLP architecture, where the voxel indices of the 25 skeletal key points (analogous to keyword extraction in NLP) are predicted. The voxel indices are converted back to real-world 3-D coordinates using the voxel dictionary used during the tokenization process. Mean absolute error (MAE) metrics were used to measure the accuracy of the proposed system against the ground truth, with the proposed mmPose-NLP offering < 3 cm localization errors in the depth, horizontal, and vertical axes. The effect of the number of input frames versus performance/accuracy was also studied for N = {1,2, ... ,10} . A comprehensive methodology, results, discussions, and limitations are presented in this article. All the source codes and results are made available on GitHub for further research and development in this critical yet emerging domain of skeletal key-point estimation using mmWave radars.

Video Analysis of Small Bowel Capsule Endoscopy Using a Transformer Network.

Although wireless capsule endoscopy (WCE) detects small bowel diseases effectively, it has some limitations. For example, the reading process can be time consuming due to the numerous images generated per case and the lesion detection accuracy may rely on the operators' skills and experiences. Hence, many researchers have recently developed deep-learning-based methods to address these limitations. However, they tend to select only a portion of the images from a given WCE video and analyze each image individually. In this study, we note that more information can be extracted from the unused frames and the temporal relations of sequential frames. Specifically, to increase the accuracy of lesion detection without depending on experts' frame selection skills, we suggest using whole video frames as the input to the deep learning system. Thus, we propose a new Transformer-architecture-based neural encoder that takes the entire video as the input, exploiting the power of the Transformer architecture to extract long-term global correlation within and between the input frames. Subsequently, we can capture the temporal context of the input frames and the attentional features within a frame. Tests on benchmark datasets of four WCE videos showed 95.1% sensitivity and 83.4% specificity. These results may significantly advance automated lesion detection techniques for WCE images.

Power Requirements Evaluation of Embedded Devices for Real-Time Video Line Detection

In this paper, the comparison of the power requirements during real-time processing of video sequences in embedded systems was investigated. During the experimental tests, four modules were tested: Raspberry Pi 4B, NVIDIA Jetson Nano, NVIDIA Jetson Xavier AGX, and NVIDIA Jetson Orin AGX. The processing speed and energy consumption have been checked, depending on input frame size resolution and the particular power mode. Two vision algorithms for detecting lines located in airport areas were tested. The results show that the power modes of the NVIDIA Jetson modules have sufficient computing resources to effectively detect lines based on the camera image, such as Jetson Xavier in mode MAXN or Jetson Orin in mode MAXN, with a resolution of 1920 × 1080 pixels and a power consumption of about 19 W for 24 FPS for both algorithms tested.

Augmented reality with tracking based on linear equation

Augmented Reality (AR) is a human-computer interaction tool where the view of the real-world environment is enhanced by computer-generated elements or objects. This work is inserted in the context of AR, more specifically in systems with monitor based design, where image processing and computer vision tools are used, such as: spatial filtering, image segmentation, morphological processing, color image processing, Optical Flow and Homography. In this dissertation a new augmented reality algorithm with a markers tracking method based on linear equations is presented. This algorithm consists of segmenting the colored markers from an input frame, extracting characteristics of the markers that allow the estimation of lines for projection of a 2D image in a 3D plane of the scene. The obtained results are promising to demonstrate that the algorithm developed has excellent results when compared to the method proposed by Lucas and Kanade (1981), in relation to marker occlusion and consistency by maintaining an increase in reality at run time.

Deep multi-scale pyramidal features network for supervised video summarization

Video data are witnessing exponential growth, and extracting summarized information is challenging. It is always necessary to reduce the load of video traffic for the efficient video storage, transmission, and retrieval requirements. The aim of video summarization (VS) is to extract the most important contents from video repositories effectively. Recent attempts have used fewer representative features, which have been fed to recurrent networks to achieve VS. However, generating the desired summaries can become challenging due to the limited representativeness of extracted features and a lack of consideration for feature refinement. In this article, we introduce a vision transformer (ViT)-assisted deep pyramidal refinement network that can extract and refine multi-scale features and can predict an importance score for each frame. The proposed network comprises four main modules; initially, a dense prediction transformer with a ViT backbone is applied for the first time in this domain to acquire the optimal representations from the input frames. Then, feature maps are obtained from various layers separately and processed individually to support multi-scale progressive feature fusion and refinement before the data are passed to the ultimate prediction module. Next, a customized pyramidal refinement block is employed to refine the multi-level feature set before predicting the importance scores. Finally, video summaries are produced by selecting keyframes based on the predictions. To explore the performance of the proposed network, extensive experiments are conducted on the TVSum and SumMe datasets, and our network is found to achieve F1-scores of 62.4% and 51.9%, respectively, outperforming state-of-the-art alternatives by 0.9% and 0.5%.

Pose‐guided adversarial video prediction for image‐to‐video person re‐identification

AbstractThe image‐to‐video (I2V) person re‐identification (Re‐ID) is a cross‐modality pedestrian retrieval task, whose crux is to reduce the large modality discrepancy between images and videos. To this end, this paper proposes to predict the following video frames from a single image. Thus, the I2V person Re‐ID can be transformed to video‐to‐video (V2V) Re‐ID. Considering that predicting video frames from a single image is an ill‐posed problem, this paper proposes two strategies to improve the quality of the predicted videos. First, a pose‐guided video prediction pipeline is proposed. The given single image and pedestrian pose are encoded via image encoder and pose encoder, respectively; then, the image feature and pose feature are concatenated as the input of the video decoder. The authors minimize the difference between the predicted video and true video, and simultaneously minimize the difference between the true pose and predicted pose. Second, the conditional adversarial training strategy is employed to generate high‐quality video frames. Specifically, the discriminator takes the source image as condition and distinguishes whether the input frames are fake or true following frames of the source image. Experimental results demonstrate that the pose‐guided adversarial video prediction can effectively improve accuracy of I2V Re‐ID.

Scalable frame resolution for efficient continuous sign language recognition

In this paper, we explore the spatial redundancy in continuous sign language recognition (CSLR), aiming to improve its efficiency. Despite recent advances in accuracy in CSLR, state-of-the-art CSLR methods typically require large amounts of computations and memory occupation, which are not friendly towards fast inference under limited computation/memory budgets. Based on a simple observation that not all frames are equally important for CSLR, we propose AdaSize to handle this problem by modeling the frame resolution decision as an end-to-end learnable task to save unnecessary computations. Specifically, a lightweight 2D convolutional neural network (CNN) is first used to quickly browse input frames under a low resolution (e.g., 112 × 112). These extracted coarse and cheap features are sent into a recurrent policy network to dynamically determine the desired resolution for each frame. Once the optimal resolution for each frame is decided, frames with different resolutions are fed into the following backbones to extract representative features. Finally, these features pass through a sequence of temporal modules and a classifier to predict sentences. Extensive experiments on four large-scale datasets, including PHOENIX14, PHOENIX14-T, CSL-Daily and CSL, demonstrate the effectiveness of AdaSize. AdaSize could consistently achieve comparable accuracy with state-of-the-art CSLR methods, with only 0.38× computations, 0.41× memory usage and 1.25× throughput. Comparisons with commonly-used lightweight backbones and other efficient methods verify the superiority of AdaSize under similar computational/memory budgets. We finally plot the frame resolution decisions for AdaSize, hoping to provide insightful analysis of the inherent spatial redundancy in videos.

Reward shaping using convolutional neural network

In this paper, we propose Value Iteration Network for Reward Shaping (VIN-RS), a potential-based reward shaping mechanism using Convolutional Neural Network (CNN). The proposed VIN-RS embeds a CNN trained on computed labels using the message passing mechanism of the Hidden Markov Model. The CNN processes images or graphs of the environment to predict the shaping values. Recent work on reward shaping still has limitations towards training on a representation of the Markov Decision Process (MDP) and building an estimate of the transition matrix. The advantage of VIN-RS is to construct an effective potential function from an estimated MDP while automatically inferring the environment transition matrix. The proposed VIN-RS estimates the transition matrix through a self-learned convolution filter while extracting environment details from the input frames or sampled graphs. Due to (1) the previous success of using message passing for reward shaping; and (2) the CNN planning behavior, we use these messages to train the CNN of VIN-RS. Experiments are performed on tabular games, Atari 2600 and MuJoCo, for discrete and continuous action space. Our results illustrate promising improvements in the learning speed and maximum cumulative reward compared to the state-of-the-art. The improvement achieved by VIN-RS can only be observed for some of the games due to the underlying nature of some environments. In terms of the studied MuJoCo games, there is on average an increase of 30% in the maximum reward reached during early stages of learning.

People Counting and Tracking System in Real-Time using Deep Learning Techniques

Abstract: The aim of project is to automatically estimate the number of people at indoor and outdoor places. People counting systems can be used in retail environment such as determining conversion ratio, advertising and promotional evaluation. This system can be used for transportation management system and video surveillance. The number of customers is indispensable data for management and decision making in public places like large-scale markets, shopping centers, airports, stations, museums, laboratories, classrooms, cafeteria etc. In this system, firstly extract the frames from the video, then draw a desired reference line on the input frame, detect the people using MobileNet-SSD object detection model, mark the centroid on the detected person, track the movement of that marked centroid and calculate the direction of centroid movement whether it is moving upwards or downwards. If the centroid movement is downward direction, then we increment in counter, else if the centroid movement is upward direction, then we increment out counter. People counting is essential for retailers of any size, but it's especially important for small businesses that don’t have the benefit of assaying data from multitudinous locales when making pivotal opinions. When used intelligently, people counting can shape businesses in multitudinous ways other than just furnishing information on nethermost business.