Pixel Domain Research Articles

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition.

This paper presents a dual camera system for high spatiotemporal resolution (HSTR) video acquisition, where one camera shoots a video with high spatial resolution and low frame rate (HSR-LFR) and another one captures a low spatial resolution and high frame rate (LSR-HFR) video. Our main goal is to combine videos from LSR-HFR and HSR-LFR cameras to create an HSTR video. We propose an end-to-end learning framework, AWnet, mainly consisting of a FlowNet and a FusionNet that learn an adaptive weighting function in pixel domain to combine inputs in a frame recurrent fashion. To improve the reconstruction quality for cameras used in reality, we also introduce noise regularization under the same framework. Our method has demonstrated noticeable performance gains in terms of both objective PSNR measurement in simulation with different publicly available video and light-field datasets and subjective evaluation with real data captured by dual iPhone 7 and Grasshopper3 cameras. Ablation studies are further conducted to investigate and explore various aspects, such as reference structure, camera parallax, exposure time, etc) of our system to fully understand its capability for potential applications.

Spectral Imagery Tensor Decomposition for Semantic Segmentation of Remote Sensing Data through Fully Convolutional Networks

This work aims at addressing two issues simultaneously: data compression at input space and semantic segmentation. Semantic segmentation of remotely sensed multi- or hyperspectral images through deep learning (DL) artificial neural networks (ANN) delivers as output the corresponding matrix of pixels classified elementwise, achieving competitive performance metrics. With technological progress, current remote sensing (RS) sensors have more spectral bands and higher spatial resolution than before, which means a greater number of pixels in the same area. Nevertheless, the more spectral bands and the greater number of pixels, the higher the computational complexity and the longer the processing times. Therefore, without dimensionality reduction, the classification task is challenging, particularly if large areas have to be processed. To solve this problem, our approach maps an RS-image or third-order tensor into a core tensor, representative of our input image, with the same spatial domain but with a lower number of new tensor bands using a Tucker decomposition (TKD). Then, a new input space with reduced dimensionality is built. To find the core tensor, the higher-order orthogonal iteration (HOOI) algorithm is used. A fully convolutional network (FCN) is employed afterwards to classify at the pixel domain, each core tensor. The whole framework, called here HOOI-FCN, achieves high performance metrics competitive with some RS-multispectral images (MSI) semantic segmentation state-of-the-art methods, while significantly reducing computational complexity, and thereby, processing time. We used a Sentinel-2 image data set from Central Europe as a case study, for which our framework outperformed other methods (included the FCN itself) with average pixel accuracy (PA) of 90% (computational time ∼90s) and nine spectral bands, achieving a higher average PA of 91.97% (computational time ∼36.5s), and average PA of 91.56% (computational time ∼9.5s) for seven and five new tensor bands, respectively.

Depth Map Enhancement by Revisiting Multi-Scale Intensity Guidance Within Coarse-to-Fine Stages

Being different from the most methods of guided depth map enhancement based on deep convolutional neural network which focus on increasing the depth of networks, this paper is to improve the effectiveness of intensity guidance when the network goes deep. Overall, the proposed network upsamples the low-resolution depth maps from coarse to fine. Within each refinement stage of certain-scale depth features, the current-scale and all coarse-scales of the guidance features are revisited by dense connection. Therefore, the multi-scale guidance is efficiently maintained as the propagation of features. Furthermore, the proposed network maintains the intensity features in the high-resolution domain from which the multi-scale guidance is directly extracted. This design further improves the quality of intensity guidance. In addition, the shallow depth features upsampled via transposed convolution layer are directly transferred to the final depth features for reconstruction, which is called global residual learning in feature domain. Similarly, the global residual learning in pixel domain learns the difference between the depth ground truth and the coarsely upsampled depth map. Also, the local residual learning is to maintain the low frequency within each refinement stage and progressively recover the high frequency. The proposed method is tested for noise-free and noisy cases which compares against 16 state-of-the-art methods. Our experimental results show the improved performances based on the qualitative and quantitative evaluations.

Improving Spectral-Based Endmember Finding by Exploring Spatial Context for Hyperspectral Unmixing

Hyperspectral unmixing, which intends to decompose mixed pixels into a collection of endmembers weighted by their corresponding fraction abundances, has been widely utilized for remote sensing image exploitation. Recent studies have revealed that spatial context of pixels is important complemental information for hyperspectral image processing. However, many well-known endmember finding (EF) algorithms identify spectrally pure spectra from hyperspectral images according to spectral information only, resulting in limited accuracy of hyperspectral unmixing application since they ignore spatial distribution or structure information in the image. Therefore, in this article, several novel spatial exploiting (SE) strategies are proposed to improve the performance of the well-known spectral-based EF (sEF) algorithms by integrating spatial information. Three different spatial exploiting strategies are designed to use pixel spatial context, by which the spectral variation of pixels can be alleviated to improve the performance of hyperspectral unmixing. Specifically, in pixel domain, the pixels are linearly reconstructed using their neighbors in which the spatially derived factor to weight the importance of the spectral information is generated using local linear representation and local sparse representation, while in the feature domain, pixels are revised using dominated features of neighboring pixels in singular value decomposition. The proposed spatial exploiting strategies can not only be used as a preprocessing stage to revise pixels for sEF algorithms, but also be used as a postprocessing step to revise endmembers found via sEF algorithms. Finally, experimental results on both synthetic and real hyperspectral datasets demonstrate that the proposed SE strategies can certainly improve the performance of several well-known sEF algorithms, and obtain more accurate unmixing results than several state-of-the-art spatial preprocessing methods.

Super resolution and recognition of unconstrained ear image

In this paper, a framework is proposed to super-resolve low resolution ear images and to recognise these images, without external dataset. This frame uses linear kernel co-variance function-based Gaussian process regression to super-resolve the ear images. The performance of the proposed framework is evaluated on UERC database by comparing and analysing the peak signal to noise ratio, structural similarity index matrix and visual information fidelity in pixel domain. The results are compared with the state-of-the-art-algorithms. The results demonstrate that the proposed approach outperforms the state-of-the-art super resolution approaches.

Super resolution and recognition of unconstrained ear image

In this paper, a framework is proposed to super-resolve low resolution ear images and to recognise these images, without external dataset. This frame uses linear kernel co-variance function-based Gaussian process regression to super-resolve the ear images. The performance of the proposed framework is evaluated on UERC database by comparing and analysing the peak signal to noise ratio, structural similarity index matrix and visual information fidelity in pixel domain. The results are compared with the state-of-the-art-algorithms. The results demonstrate that the proposed approach outperforms the state-of-the-art super resolution approaches.

Influence of Model Grid Size on the Estimation of Surface Fluxes Using the Two Source Energy Balance Model and sUAS Imagery in Vineyards.

Evapotranspiration (ET) is a key variable for hydrology and irrigation water management, with significant importance in drought-stricken regions of the western US. This is particularly true for California, which grows much of the high-value perennial crops in the US. The advent of small Unmanned Aerial System (sUAS) with sensor technology similar to satellite platforms allows for the estimation of high-resolution ET at plant spacing scale for individual fields. However, while multiple efforts have been made to estimate ET from sUAS products, the sensitivity of ET models to different model grid size/resolution in complex canopies, such as vineyards, is still unknown. The variability of row spacing, canopy structure, and distance between fields makes this information necessary because additional complexity processing individual fields. Therefore, processing the entire image at a fixed resolution that is potentially larger than the plant-row separation is more efficient. From a computational perspective, there would be an advantage to running models at much coarser resolutions than the very fine native pixel size from sUAS imagery for operational applications. In this study, the Two-Source Energy Balance with a dual temperature (TSEB2T) model, which uses remotely sensed soil/substrate and canopy temperature from sUAS imagery, was used to estimate ET and identify the impact of spatial domain scale under different vine phenological conditions. The analysis relies upon high-resolution imagery collected during multiple years and times by the Utah State University AggieAir™ sUAS program over a commercial vineyard located near Lodi, California. This project is part of the USDA-Agricultural Research Service Grape Remote Sensing Atmospheric Profile and Evapotranspiration eXperiment (GRAPEX). Original spectral and thermal imagery data from sUAS were at 10 cm and 60 cm per pixel, respectively, and multiple spatial domain scales (3.6, 7.2, 14.4, and 30 m) were evaluated and compared against eddy covariance (EC) measurements. Results indicated that the TSEB2T model is only slightly affected in the estimation of the net radiation (Rn) and the soil heat flux (G) at different spatial resolutions, while the sensible and latent heat fluxes (H and LE, respectively) are significantly affected by coarse grid sizes. The results indicated overestimation of H and underestimation of LE values, particularly at Landsat scale (30 m). This refers to the non-linear relationship between the land surface temperature (LST) and the normalized difference vegetation index (NDVI) at coarse model resolution. Another predominant reason for LE reduction in TSEB2T was the decrease in the aerodynamic resistance (Ra), which is a function of the friction velocity F*) that varies with mean canopy height and roughness length. While a small increase in grid size can be implemented, this increase should be limited to less than twice the smallest row spacing present in the sUAS imagery. The results also indicated that the mean LE at field scale is reduced by 10% to 20% at coarser resolutions, while the with-in field variability in LE values decreased significantly at the larger grid sizes and ranged between approximately 15% and 45%. This implies that, while the field-scale values of LE are fairly reliable at larger grid sizes, the with-in field variability limits its use for precision agriculture applications.

Bit Rate Transcoding for High Efficiency Video Coding

High efficiency video coding (HEVC) has demonstrated a notable increase in compression performance and is taken as a successor to H.264/AVC. Efficient bit rate adaptation algorithms are required to contain the HEVC standard between real life community facilities. A present issue of bit rate transcoding is its high computational complexity which is related with the encoder of a cascaded pixel domain transcoder. This paper gives Top to Bottom (T2B) approach to reduce complexity by using different complexity schemes. Proposed approach is effective in reducing complexity in Coding Unit (CU) optimization level. Coding Unit has been analyzed in T2B Approach. While examining the coding unit information of the input video is turned to account for decreasing the number of evaluation and early terminate the process. For the Prediction Unit (PU) level the units are powerfully chosen contingent upon likelihood of Prediction Unit sizes and co-found input prediction partitioning. By utilizing this approach, complexity scalable bit rate transcoding has achieved. Machine learning approach can be used to control computational complexity. Additionally, the T2B strategy is able to gain a spread on trade-offs in transrating complexity and coding performance. Using T2B approach 15% encoding time saving is accomplished. From this scheme, for the less resolution video 27% time saving has achieved.

Super-resolution for remote sensing images via dual-domain network learning

Super-resolution (SR), which aims at recovering a high-resolution (HR) image from single or sequential low-resolution (LR) images, is a widely used technology in image processing. In the field of image SR, convolutional neural networks (CNNs) have attracted increasing attention because of their high-quality performance. However, most CNN-based methods treat each channel-wise feature equally, which lacks discriminative learning ability across feature channels. Furthermore, many methods neglect to fully use information of each convolutional layer. To resolve these problems, we propose a remote sensing image SR method named dense channel attention network (DCAN). In our DCAN, a sequence of residual dense channel attention blocks (RDCABs) is cascaded with a densely connected structure. In each RDCAB, we make full use of the information from all convolution layers via densely connected convolutional layers. In addition, RDCABs utilize the channel attention mechanism to adaptively recalibrate channel-wise feature responses by explicitly modeling the interdependencies between the channels. In addition, our DCAN can make full use of the hierarchical features by densely connecting each RDCAB. Finally, to further improve the SR performance, the proposed DCAN is learned in both the pixel and wavelet domains, and a fusion layer is used to fuse the outputs of these two domains. Extensive quantitative and qualitative evaluations verify the superiority of our proposed method over several state-of-the-art methods.

Video Abstraction in H.264/AVC Compressed Domain

Video abstraction allows searching, browsing and evaluating videos only by accessing the useful contents. Most of the studies are using pixel domain, which requires the decoding process and needs more time and process consuming than compressed domain video abstraction. In this paper, we present a new video abstraction method in H.264/AVC compressed domain, AVAIF. The method is based on the normalized histogram of extracted I-frame prediction modes in H.264 standard. The frames’ similarity is calculated by intersecting their I-frame prediction modes’ histogram. Moreover, fuzzy c-means clustering is employed to categorize similar frames and extract key frames. The results show that the proposed method achieves on average 85% accuracy and 22% error rate in compressed domain video abstraction, which is higher than the other tested methods in the pixel domain. Moreover, on average, it generates video key frames that are closer to human summaries and it shows robustness to coding parameters.

An effective image retrieval system using machine learning and fuzzy c- means clustering approach

Recently, Content-based medical image retrieval (CBMIR) systems enable fast diagnosis via the assessment of the visual information in medical application. Most of the state-of-the-art CBMIR systems facing few issues: computationally expensive due to the usage of high dimensional feature vectors and complex classifier/clustering approaches. The reasons behind this are, inability to properly handle the “semantic gap” and the high intra-class versus inter-class variability problem of the medical image database (like radiographic image database). This yields a crucial demand for developing computationally efficient and highly effective retrieval system. For this purpose, the present study proposed an efficient retrieval system which has a four-fold: First, pre-processing and Feature extraction of input image using canonical correlation analysis (CCA). By this approach extracted the feature in both pixel and feature domains and examined more rigorously. Second, applied Fuzzy C means clustering of pixel intensity values as features based on the singular value decomposition. Through this can grouping, the image based on the pixel intensity value. Third, deep convolutional neural network with SVM classifier which makes implementable and requires only a compact feature vector representation of the stored database image with their class levels during retrieval. Finally evaluated the performance based on the measure of Mean Average Precision, Correct rate (CR), Error rate (ER), Accuracy. The classification results and learned features are used for the purpose of retrieving the medical images in a database. The proposed retrieval system performs better than the traditional approach in terms of measuring average value of precision, recall, f-measure and accuracy 95.9%, 94.96%, 95.37% and 95.798% respectively. The suggested approach is best suited towards retrieving the medical images for various part of the body.

Practical Evaluation of VMAF Perceptual Video Quality for WebRTC Applications

WebRTC is the umbrella term for several emergent technologies aimed to exchange real-time media in the Web. Like other media-related services, the perceived quality of WebRTC communication can be measured using Quality of Experience (QoE) indicators. QoE assessment methods can be classified as subjective (users’ evaluation scores) or objective (models computed as a function of different parameters). In this paper, we focus on VMAF (Video Multi-method Assessment Fusion), which is an emergent full-reference objective video quality assessment model developed by Netflix. VMAF is typically used to assess video streaming services. This paper evaluates the use of VMAF in a different type of application: WebRTC. To that aim, we present a practical use case built on the top of well-known open source technologies, such as JUnit, Selenium, Docker, and FFmpeg. In addition to VMAF, we also calculate other objective QoE video metrics such as Visual Information Fidelity in the pixel domain (VIFp), Structural Similarity (SSIM), or Peak Signal-to-Noise Ratio (PSNR) applied to a WebRTC communication in different network conditions in terms of packet loss. Finally, we compare these objective results with a subjective evaluation using a Mean Opinion Score (MOS) scale to the same WebRTC streams. As a result, we found a strong correlation of the subjective video quality perceived in WebRTC video calls with the objective results computed with VMAF and VIFp in comparison with SSIM and PSNR and their variants.

Methods for pixel domain correction of EB leakage

In observation of the cosmic microwave background (CMB) polarization, "$EB$~leakage" refers to the artificial $B$-mode signal coming from the leakage of $E$-mode signal when part of the sky is unavailable or excluded. Correction of such leakage is one of the preconditions for detecting primordial gravitational waves via the CMB $B$-mode signal. In this work, we design two independent methods for correcting the $EB$~leakage directly in the pixel domain using standard definitions of the $E$- and $B$-modes. The two methods give consistent results, and both are fast and easy to implement. Tests on a CMB simulation containing zero initial $B$-mode show an efficient suppression of the $EB$ leakage. When combined with the MASTER method to reconstruct the full-sky $B$-mode spectrum in simulations with a relatively simple mask, the error from EB-leakage is suppressed further by more than one order of magnitude at the recombination bump, and up to three orders of magnitude at higher multipoles, compared to a "pure" MASTER scheme under the same conditions. Meanwhile, although the final power spectrum estimation benefits from apodization, the pixel domain correction itself is done without apodization, and thus the methods offer more freedom in choosing an apodization based on specific requirements.

CISI-net: Explicit Latent Content Inference and Imitated Style Rendering for Image Inpainting

Convolutional neural networks (CNNs) have presented their potential in filling large missing areas with plausible contents. To address the blurriness issue commonly existing in the CNN-based inpainting, a typical approach is to conduct texture refinement on the initially completed images by replacing the neural patch in the predicted region using the closest one in the known region. However, such a processing might introduce undesired content change in the predicted region, especially when the desired content does not exist in the known region. To avoid generating such incorrect content, in this paper, we propose a content inference and style imitation network (CISI-net), which explicitly separate the image data into content code and style code. The content inference is realized by performing inference in the latent space to infer the content code of the corrupted images similar to the one from the original images. It can produce more detailed content than a similar inference procedure in the pixel domain, due to the dimensional distribution of content being lower than that of the entire image. On the other hand, the style code is used to represent the rendering of content, which will be consistent over the entire image. The style code is then integrated with the inferred content code to generate the complete image. Experiments on multiple datasets including structural and natural images demonstrate that our proposed approach out-performs the existing ones in terms of content accuracy as well as texture details.

Deep Policies for Width-Based Planning in Pixel Domains

Width-based planning has demonstrated great success in recent years due to its ability to scale independently of the size of the state space. For example, Bandres et al. (2018) introduced a rollout version of the Iterated Width algorithm whose performance compares well with humans and learning methods in the pixel setting of the Atari games suite. In this setting, planning is done on-line using the “screen” states and selecting actions by looking ahead into the future. However, this algorithm is purely exploratory and does not leverage past reward information. Furthermore, it requires the state to be factored into features that need to be pre-defined for the particular task, e.g., the B-PROST pixel features. In this work, we extend width-based planning by incorporating an explicit policy in the action selection mechanism. Our method, called π-IW, interleaves width-based planning and policy learning using the state-actions visited by the planner. The policy estimate takes the form of a neural network and is in turn used to guide the planning step, thus reinforcing promising paths. Surprisingly, we observe that the representation learned by the neural network can be used as a feature space for the width-based planner without degrading its performance, thus removing the requirement of pre-defined features for the planner. We compare π-IW with previous widthbased methods and with AlphaZero, a method that also interleaves planning and learning, in simple environments, and show that π-IW has superior performance. We also show that π-IW algorithm outperforms previous width-based methods in the pixel setting of Atari games suite.

Multi-Scale Gradient Image Super-Resolution for Preserving SIFT Key Points in Low-Resolution Images

Low-resolution images present challenges to a variety of object recognition problems in a variety of surveillance and navigation applications. In recent years, deep learning has advanced the state of the art in image super-resolution (SR) in terms of pixel domain peak signal to noise ratio (PSNR)/ mean square error (MSE). Inspired by the recent advances of deep convolutional neural networks in general image SR tasks, we develop a computer vision task-driven image SR solution by learning super-resolved gradient images using multiple convolutional neural networks for different scales. Recovering super-resolved gradient images at multiple scales, enables the system to recover more information useful for high level vision tasks than simply SR in the pixel domain. In particular, we propose a residual learning framework to perform image SR in the Difference of Gaussian (DOG) domain. The trained residual network models are then adapted to drive a widely adopted key point algorithm for image recognition, i.e. the SIFT detection and matching. Experimental results show that the proposed approach can significantly improve the SIFT keypoints repeatability compared to the state of the art in pixel domain image SR solutions.

Accurate and Scalable Image Clustering Based on Sparse Representation of Camera Fingerprint

Clustering images according to their acquisition devices is a well-known problem in multimedia forensics, which is typically faced by means of camera sensor pattern noise (SPN). Such an issue is challenging since SPN is a noise-like signal, hard to be estimated, and easy to be attenuated or destroyed by many factors. Moreover, the high dimensionality of SPN hinders large-scale applications. Existing approaches are typically based on the correlation among SPNs in the pixel domain, which might not be able to capture intrinsic data structure in the union of vector subspaces. In this paper, we propose an accurate clustering framework, which exploits linear dependences among SPNs in their intrinsic vector subspaces. Such dependences are encoded under sparse representations, which are obtained by solving an LASSO problem with non-negativity constraint. The proposed framework is highly accurate in a number of clusters’ estimation and image association. Moreover, our framework is scalable to the number of images and robust against double JPEG compression as well as the presence of outliers, owning big potential for real-world applications. Experimental results on Dresden and Vision database show that our proposed framework can adapt well to both medium-scale and large-scale contexts and outperforms the state-of-the-art methods.

Run-Time Deep Learning Enhanced Fast Coding Unit Decision for High Efficiency Video Coding

The state-of-the-art high efficiency video coding (HEVC/H.265) adopts the hierarchical quadtree-structured coding unit (CU) to enhance the coding efficiency. However, the computational complexity significantly increases because of the exhaustive rate-distortion (RD) optimization process to obtain the optimal coding tree unit (CTU) partition. In this paper, we propose a fast CU size decision algorithm to reduce the heavy computational burden in the encoding process. In order to achieve this, the CU splitting process is modeled as a three-stage binary classification problem according to the CU size from [Formula: see text], [Formula: see text] to [Formula: see text]. In each CU partition stage, a deep learning approach is applied. Appropriate and efficient features for training the deep learning models are extracted from spatial and pixel domains to eliminate the dependency on video content as well as on encoding configurations. Furthermore, the deep learning framework is built as a third-party library and embedded into the HEVC simulator to speed up the process. The experiment results show the proposed algorithm can achieve significant complexity reduction and it can reduce the encoding time by 49.65%(Low Delay) and 48.81% (Random Access) on average compared with the traditional HEVC encoders with a negligible degradation (2.78% loss in BDBR, 0.145[Formula: see text]dB loss in BDPSNR for Low Delay, and 2.68% loss in BDBR, 0.128[Formula: see text]dB loss in BDPSNR for Random Access) in the coding efficiency.

Very Low Bitrate Semantic Compression of Airplane Cockpit Screen Content

This paper addresses the problem of encoding the video generated by the screen of an airplane cockpit. As other computer screens, cockpit screens consist of computer-generated graphics often atop a natural background. Existing screen content coding schemes fail notably in preserving the readability of textual information at the low bitrates required in avionic applications. We propose a screen coding scheme where textual information is encoded according to the relative semantics rather than in the pixel domain. The encoder localizes textual information, and the semantics of each character are extracted with a convolutional neural network and predictively encoded. Text is then removed via inpainting, and the residual background video is compressed with a standard codec and transmitted to the receiver together with the text semantics. At the decoder side, text is synthesized using the decoded semantics and superimposed over the decoded residual video recovering the original frame. Our proposed scheme offers two key advantages over a semantics-unaware scheme that encodes text in the pixel domain. First, the text readability at the decoder is not compromised by compression artifacts, whereas the relative bitrate is negligible. Second, removal of high-frequency transform coefficients associated with the inpainted text drastically reduces the bitrate of the residual video. Experiments with real cockpit video sequences show BD-rate gains up to 82% and 69% over a reference H.265/HEVC encoder and its screen content coding extension. Moreover, our scheme achieves quasi-errorless character recognition already at very low bitrates, whereas even HEVC-SCC needs at least three or four times more bitrate to achieve a comparable error rate.

Random grid based color visual cryptography scheme for black and white secret images with general access structures

To reduce the pixel expansion problem in color–black-and-white visual cryptography schemes (CBW-VCSs) and to provide flexible sharing strategy simultaneously, a random grid based CBW-VCS (RG-CBW-VCS) for general access structure (GAS) is proposed in this paper. First of all, the concept of random grid is extended to color pixel domain, and the XOR operation for color pixels is defined. By adopting this operation, a scheme for RG-CBW-VCS which implements the GAS is introduced. The proposed method is further extended to realize the generalized GAS (GGAS), where implementing complicated sharing strategy becomes available. Theoretical analysis on the proposed method are given to prove the security and contrast conditions, as well as the calculation of contrast. Experimental results are exhibited to show that optimal pixel expansion is achieved by the proposed method.