In-between Frames Research Articles

FI-Net: A Lightweight Video Frame Interpolation Network Using Feature-Level Flow

Video frame interpolation is a classic computer vision task that aims to generate in-between frames given two consecutive frames. In this paper, a flow-based interpolation method (FI-Net) is proposed. FI-Net is a lightweight end-to-end neural network that takes two frames in arbitrary size as input and outputs the estimated intermediate frame. Novelly, it computes optical flow at feature level instead of image level. Such practice can increase the accuracy of estimated flow. Multi-scale technique is utilized to handle large motions. For training, a comprehensive loss function that contains a novel content loss (Sobolev loss) and a semantic loss is introduced. It forces the generated frame to be close to the ground truth one at both pixel level and semantic level. We compare FI-Net with previous methods and it achieves higher performance with less time consumption and much smaller model size.

Phase-based frame rate up-conversion for depth video

Due to the limits of frame rate sampling like time-of-flight and transmission bandwidth constraints of depth video, depth frame rate up-conversion is imperative. Most researches interpolate the depth frames using the motion information of the texture images, which require well-rectified images. They usually find dense correspondences to synthesize an in-between frame. Estimating the correspondences is sensitive to occlusion, disocclusion, and color or luminance changes. This paper proposes a phase-based method to interpolate depth frames without explicitly finding correspondences. Combining top-down phase difference preservation and bottom-up phase difference correction, varying amounts of motions are well modeled in a complex-steerable pyramid decomposition. The pixel amplitude is also shifted and interpolated based on the phase information. Finally, a depth value correction is employed to correct the inaccurate depth value. Experimental results show that the depth video interpolated by our proposed approach provides better subjective and objective qualities.

Application of vector graphics binaryzation and refining treatment in two-dimensional animation modeling

Two-dimensional animation production technology which tends to be more mature has played a significant role in all aspects of society. Study on two-dimensional animation production technology is usually divided into three aspects: two-dimensional bitmap image vectorization, automatic generation and coloring of in-between frames and three-dimensional assistance of two-dimensional image rendering. In this paper, the vectorization of two-dimensional bitmap image was emphatically studied, and the image preprocessing operation was introduced in detail. The realization of two-dimensional vector graphics is based on computer graphics, including image preprocessing and vector representation. Based on the theories and practice, a specific algorithm for the preprocessing of two-dimensional vector graphics was given, and the key frames of the animation were plotted and preprocessed.

Context-Aware Computer Aided Inbetweening.

This paper presents a context-aware computer aided inbetweening (CACAI) technique that interpolates planar strokes to generate inbetween frames from a given set of key frames. The inbetweening is context-aware in the sense that not only the stroke's shape but also the context (i.e., the neighborhood of a stroke) in which a stroke appears are taken into account for the stroke correspondence and interpolation. Given a pair of successive key frames, the CACAI automatically constructs the stroke correspondence between them by exploiting the context coherence between the corresponding strokes. Meanwhile, the construction algorithm is able to incorporate the user's interaction with ease and allows the user more effective control over the correspondence process than existing stroke matching techniques. With a one-to-one stroke correspondence, the CACAI interpolates the shape and context between the corresponding strokes for the generation of intermediate frames. In the interpolation sequence, both the shape of individual strokes and the spatial layout between them are well retained such that the feature characteristics and visual appearance of the objects in the key frames can be fully preserved even when complex motions are involved in these objects. We have developed a prototype system to demonstrate the ease of use and effectiveness of the CACAI.

Skeleton-based Adaptive In-between Generation for Hand-drawn Key Frames

For improving the efficiency of 2D animation production, this paper presents a method to create in-between frames based on hand-drawn key-frames. The outlines of characters or objects on two key-frames are used as inputs. First, a skeleton linkage of the target object is automatically constructed by rasterizing each of input key-frames and then applying a pixel-based skeleton extraction method. Secondary, a pair of skeleton linkages having corresponding structure between the current key-frame and the next key-frame is constructed by applying the stroke matching algorithm. After these processes, motion transitions between the skeleton linkages are generated based on our simulation model. When the in-between frames are created only in the 2D plane, the outlines at in-between frames can be generated by a 2D deformation. In case that the in-between transitions are containing a rotation around an axis which is no perpendicular to the drawing plane, however, a 3D structure is required. For achieving such in-between transitions, our method constructs a 3D structure by inflating 2D mesh based on the input outlines. Finally, the contours from the view-point for the created 3D structure are projected onto the 2D plane during in-between transitions. In our method, we adopt the Photic Extremum Lines (PEL) to extract the 2D contours from the obtained 3D shape. In this way, we achieve the in-between creation containing spatial rotation such as hand-flipping, which has not been achieved by general ways of in-between creation method.

Stroke Correspondence Construction Using Manifold Learning

AbstractStroke correspondence construction is a precondition for generating inbetween frames from a set of key frames. In our case, each stroke in a key frame is a vector represented as a Disk B‐Spline Curve (DBSC) which is a flexible and compact vector format. However, it is not easy to construct correspondences between multiple DBSC strokes effectively because of the following points: (1) with the use of shape descriptors, the dimensionality of the feature space is high; (2) the number of strokes in different key frames is usually large and different from each other and (3) the length of corresponding strokes can be very different. The first point makes matching difficult. The other two points imply ‘many to many’ and ‘part to whole’ correspondences between strokes. To solve these problems, this paper presents a DBSC stroke correspondence construction approach, which introduces a manifold learning technique to the matching process. Moreover, in order to handle the mapping between unequal numbers of strokes with different lengths, a stroke reconstruction algorithm is developed to convert the ‘many to many’ and ‘part to whole’ stroke correspondences to ‘one to one’ compound stroke correspondence.

Working Within and In-Between Frames: An Academic Tourist/ Midnight Robber and the Academy

Having assumed the role of academic tourist/midnight robber from the Caribbean island of Trinidad and Tobago, I discover that I have never fully dealt with the emotional aspect of using academic discourses that came out of a Eurocentric frame. Academic habitus, conventions, laws, shame, guilt, and fear of the repercussions because of my dual status inhibit me. In this autoethnographic text, I make a bold move to become vulnerable as I critically explore the social, political, and cultural contexts of a dissertation culture that I experienced. I use methodological tools provided by specifically chosen tour guides, Carnival midnight robber talk, and a “layered account” to move from the outside to the inside and back out again of the experience of using academic discourses in an ethnographic study of Trinidad Carnival.

Determination of venous blood flow velocity using digital videomicroscopy (A short methodical communication)

There is a limited number of methods to measure blood flow velocity in small veins. A cheap and simple new videomicroscopic method developed in our laboratories is described in the paper. A stretch of the saphenous vein of the rat was exposed by careful micropreparation on the thigh of anesthetized animals. Bolus amount (approx. 5 μl) of saline was infused into a small side branch through a microcannula to dilute flowing blood. Videomicroscopic picture of the vein was then taken of the exposed upstream stretch of the vein. Serial pictures were digitized and analyzed using macro functions of the Image J software. Sensitive areas of serial pictures were selected and fitted. Consecutive pictures were subtracted from each other to better characterize their alteration in-between frames. Greyscale intensity values measured at different points of the inner diameter were averaged for each point of the vessel axis. Cross-correlations along the axis were then computed for consecutive frames with delays of 40, 80, 120 and 160 msec. Pixel offsets producing cross-correlation maxima were determined and used to compute mean flow velocity. Combination of digital subtraction and cross-correlation computations yielded easily identifiable maximums. Mean flow velocities could be determined with limited uncertainty. The described technique gives a cheap, simple and reproducible mean to determine mean blood flow velocities in small veins in anesthetized animals, where other current techniques (ultrasonography, laser-Doppler, fluorescently labelled red cell movement) are either expensive or can be applied with difficulty only.

BetweenIT: An Interactive Tool for Tight Inbetweening

AbstractThe generation of inbetween frames that interpolate a given set of key frames is a major component in the production of a 2D feature animation. Our objective is to considerably reduce the cost of the inbetweening phase by offering an intuitive and effective interactive environment that automates inbetweening when possible while allowing the artist to guide, complement, or override the results. Tight inbetweens, which interpolate similar key frames, are particularly time‐consuming and tedious to draw. Therefore, we focus on automating these high‐precision and expensive portions of the process. We have designed a set of user‐guided semi‐automatic techniques that fit well with current practice and minimize the number of required artist‐gestures. We present a novel technique for stroke interpolation from only two keys which combines a stroke motion constructed from logarithmic spiral vertex trajectories with a stroke deformation based on curvature averaging and twisting warps. We discuss our system in the context of a feature animation production environment and evaluate our approach with real production data.

Highly stylised animation

In this paper we argue for our NPAR system as an effective 2D alternative to most NPR research, which is focused on frame coherent stylised rendering of 3D models. Our approach gives a highly stylised look to images without the support of 3D models. Nevertheless, they still behave as though they are animated by drawing, which they are. First, a stylised brush tool is used to freely draw extreme poses of characters. Each character is built of 2D drawn brush strokes which are manually grouped into layers. Each layer is assigned its place in a drawing hierarchy called a hierarchical display model (HDM). Next, multiple HDMs are created for the same character, each corresponding to a specific view. A collection of HDMs essentially reintroduces some correspondence information to the 2D drawings needed for inbetweening and, in effect, eliminates the need for a true 3D model. Once the models are composed the animator starts by defining keyframes from extreme poses in time. Next, brush stroke trajectories defined by the keyframe HDMs are inbetweened automatically across intermediate frames. Finally, each HDM of each generated inbetween frame is traversed and all elements are drawn one on another from back to front. Our techniques support highly rendered styles which are particularly difficult to animate by traditional means including the ‘airbrushed’, scraperboard, watercolour, Gouache, ‘ink-wash’, pastel, and the ‘crayon’ styles. In addition, we describe the data path to be followed to create highly stylised animations by incorporating real footage. We believe our system offers a new fresh perspective on computer-aided animation production and associated tools.

A new application of pre-normalized principal component analysis for improvement of image quality and clinical diagnosis in human brain PET studies—Clinical brain studies using [ 11C]-GR205171, [ 11C]- l-deuterium-deprenyl, [ 11C]-5-Hydroxy- l-Tryptophan, [ 11C]- l-DOPA and Pittsburgh Compound-B

Principal component analysis (PCA) is one of the most applied multivariate image analysis tool on dynamic Positron Emission Tomography (PET). Independent of used reconstruction methodologies, PET images contain correlation in-between pixels, correlations in-between frame and errors caused by the reconstruction algorithm including different corrections, which can affect the performance of the PCA. In this study, we have investigated a new approach of application of PCA on pre-normalized, dynamic human PET images. A range of different tracers have been used for this purpose to explore the performance of the new method as a way to improve detection and visualization of significant changes in tracer kinetics and to enhance the discrimination between pathological and healthy regions in the brain. We compare the new results with the results obtained using other methods. Images generated using the new approach contain more detailed anatomical information with higher quality, precision and visualization, compared with images generated using other methods.

Volume-Wise Application of Principal Component Analysis on Masked Dynamic PET Data in Sinogram Domain

Most of the methods used for analyzing PET data are applied in the spatial domain (image domain), in which reconstructed images contain all different types of effects and errors caused by the reconstruction algorithm such as correlation in-between pixels, correlations in-between frames, and streak-artifacts. In this paper, we have investigated a new, pixel wise, noise prenormalization method used for transformation of input data followed by volume-wise application of principal component analysis (PCA) on masked dynamic PET data in the sinogram domain. We are aiming to improve the performance of PCA and to provide images with improved quality and signal extraction. We compare the performance of PCA and the image quality obtained with the new method with previously published approaches. The results show improvement of performance of PCA with respect to image quality, signal extraction, precision, and visualization.

Digital Cubism, Part 2

The author has developed a technique which allows him to show disparate events in a single frame, imaged by different cameras in different places, yet all smoothly joined together visually. The technique is called the multicamera collage, or MCC. The MCC is a collection of five programs: two shaders and three standalone programs. The shaders capture data from the rendering program when making the initial images and supply new date for rerendering. The standalone programs prepare the images for collaging, process individual collages, and build in-between frames for animation.

Exploiting temporally coherent visibility for accelerated walkthroughs

This paper is concerned with the efficient generation of graphic images by exploiting temporally coherent visibility to minimize polygon flow. The basic idea is to manage the visible polygon set (VPS) somewhat loosely, thereby reducing the computational effort considerably while maintaining the quality of the rendered image at a reasonably high level. Our method executes the culling-prediction-interpolation-rendering loop repeatedly, in an intelligent fashion. It calculates the exact visible polygons for two frames: one for the current view and the other for the predicted view after some frames. The in-between frames can be rendered from those polygons that belong to the two extreme frames. Furthermore, our algorithm regulates the frame rate to the desired target frame rate by controlling the number of the in-between frames. Because our algorithm needs to calculate the visible polygons for the key frames, any other culling algorithms can be easily used with it in this part. In navigation applications including walkthrough, our algorithm causes greatly enhanced performance of the culling process.

Time-series animation techniques for visualizing urban growth

Time-series animation is a visually intuitive way to display urban growth. Animations of landuse change for the Baltimore-Washington region were generated by showing a series of images one after the other in sequential order. Before creating an animation, various issues which will affect the appearance of the animation should be considered, including the number of original data frames to use, the optimal animation display speed, the number of intermediate frames to create between the known frames, and the output media on which the animations will be displayed. To create new frames between the known years of data, the change in each theme (i.e. urban development, water bodies, transportation routes) must be characterized and an algorithm developed to create the in-between frames. Example time-series animations were created using a temporal GIS database of the Baltimore-Washington area. Creating the animations involved generating raster images of the urban development, water bodies, and principal transportation routes; overlaying the raster images on a background image; and importing the frames to a movie file. Three-dimensional perspective animations were created by draping each image over digital elevation data prior to importing the frames to a movie file.

Data Animator — Software that Visualizes Data as Computer-Generated Animation on Personal Computers: an Application to Hamilton Harbour

Abstract Data Animator, V1.0, is a scientific visualization package for microcomputers. Its main purpose is to generate two-dimensional animations from any data set collected over time. Geographical references such as a shore and/or bathymetry information, etc., may be added for additional clarity. Visualization of data as animations greatly simplifies the interpretation of field measurements. Data Animator is designed (but not restricted) to display data collected in aquatic environments, lakes, rivers, estuaries, oceans, etc., in a clear, concise way using colour to represent ranges of data values. Data sets can also be displayed as static images (keyframes). A graphic user interface allows the user to choose viewpoint, fonts, colour palette, data and keyframes. All Data Animator's options can be accessed through a graphical user interface (GUI). Point-and-click mouse operations allow the user to manipulate many features, with immediate on-screen feedback. Animations are generated by defining keyframes of known data, each located at a specific time. The program can then interpolate over time, between keyframes, to create smoothly animated transitions (in-between frames). Two types of graphs can be rendered with Data Animator. Plane-type graphs are horizontal slices at a depth specified by the user. Transect-type graphs are vertical slices along a straight line defined by the user. Data Animator can make use of both shore outline information and three-dimensional bathymetry information. This allows for the generation of realistic-looking graphs that follow the shape of the aquatic environment. Animations can be displayed on a computer monitor or transferred to video tape. pH data from Hamilton Harbour have been visualized and the results are discussed.

Feature matching and affine transformation for 2D cell animation

This paper presents an application for feature matching and affine transformation in computer graphics. The study case considered is 2D cell animation, which is still a labor-intensive process in current cartoon film production. One key problem in automating 2D cell animation in inbetween frame generation. The objective of our work is to investigate how to automatically generate inbetweening from pairs of hand-drawn 2D key frames. Our technique first establishes the correspondence of feature points and then estimates the affine transformation between each pair of 2D key frames. The inbetween frames are then automatically generated by interpolation.

Contour image sequence compression through motion analysis and hybrid coding method

This paper presents a motion analysis algorithm (MAA) and a hybrid coding method for contour image sequence compression. The contour image sequence consists of objects moving and rotating in a 3-D world with occlusion, shape, and size variations from frame to frame. The MAA separates the moving image sequence into several object-oriented subsequences (OOSs). In each OOS, the component is either stationary or moves smoothly, and the motion parameters can be easily estimated. The first and last frames of OOS are key frames, and the others are in-between frames. The key frames are unpredictable, and the entire frames need to be encoded. The in-between frames are compensable, and they are encoded by the motion parameter coding. The hybrid coder uses vectorgraph coding to remove spatial redundancy of the key frames and motion parameter coding to reduce the temporal redundancy of the OOSs. The motion parameters are encoded as combinations of 2-D translation, 2-D rotation, and scaling. There are many applications for contour image sequence compression. The cartoon image sequence (a sequence of line drawing sketches) and the high-frame-rate videophone for sign language transmission are good examples. Experiments show that our method encodes the contour image sequence at a very high compression ratio without losing intelligibility.

The Multiresolution Dissolve

This article describes the Multiresolution Dissolve, a new technique used to achieve dissolves from one digital image to another, exhibiting many varied special effects. These include cross-dissolves of edge details before coarse image features, cross-dissolves of scene illumination before other image details, and a new smooth dissolve that reduces the double-exposed look of standard cross-dissolves. This method produces new and interesting results when applied to fading between background and titles or images. The Multiresolution Dissolve generates a series of bandpass filtered images of start and end images and then produces in-between frames of the dissolve sequence by summing the bandpassed images with weights that vary over time and by frequency band. Computational time and space are greatly reduced by the use of multiresolution pyramid image processing techniques.

Computer-Generated Key-Frame Animation

Special animation languages have been developed for specifying an image sequence to be generated by a computer. Although this approach to computer animation has provided some form of organization and control of the specification of image sequences, the computer remains out of reach to most animators because of the communication difficulty. An animator's ideas involve mainly pictures and their motion. Thus it is appropriate that the communication of ideas between the animator and the computer should be largely through pictures. An interactive computer-controlled graphical system allows the animator to develop pictorial sequences directly on a cathode-ray tube display, without forcing the animator to become a computer programer. The implementation of computer-generated key-frame animation is based on techniques that have been developed for conventional cel animation. Key frames are created and stored on a suitable medium such as digital magnetic tape or disc. In addition to the picture, information about the kind of motion and the number of frames from the previous key frame is specified. During playback of this sequence, the in-between frames are computed by interpolating between key frames and are displayed at the cine rate on the cathode-ray display. Thus previewing is available in seconds.