Video Pixels Research Articles

Pixel tampering detection in encrypted surveillance videos on resource-constrained devices

Encryption (naïve/selective) is recommended to secure the recorded visual content; however, intruders can still manipulate encrypted data. Visually, tampering attacks on encrypted video pixels in selectively encrypted videos are difficult to identify. Thus, this paper presents a tampering detection system that performs vulnerability analysis for Regions-of-Interest (ROI) in encrypted videos. To detect the tampering attacks, we explored the pixels’ intensities and proposed a new TampDetect algorithm. The TampDetect applies the Hue, Saturation, and Value (HSV) colour model to detect the encrypted areas in the video. These encrypted pixels are then segmented and selected from the non-encrypted pixels using a minimum and maximum HSV value and a global threshold. The mean intensity of the encrypted pixels is thus calculated and stored. The integrity of the video frame is then validated by comparing the stored mean intensity with the newly calculated mean intensity to validate tampering/attack. The experiments were conducted on an Intel NUC, and its low computational cost demonstrates the lightweight nature of the proposed TampDetect algorithm for detecting tampering in ROI encrypted videos. The developed dataset for experiments, i.e., original, ROI encrypted, and tampered videos (encrypted/decrypted) is made available on kaggle-repository for future researchers.

Adaptive Locally-Aligned Transformer for low-light video enhancement

Low-light enhancement is a crucial task that aims to enhance the under-exposed input in computer vision. While state-of-the-art static single-image enhancement methods have made remarkable progress, yet, few attempts are explored the spatial-temporal sequence problem in low-light video enhancement. In this paper, we propose a simple yet highly effective method, termed as Adaptive Locally-Aligned Transformer (ALAT) for low-light video enhancement based on visual transformers. ALAT consists of three parts: feature encoder, locally-aligned transformer block (LATB) and pyramid feature decoder. Specifically, the transformer block enables the network to model the long-range spatial and appearance dependencies in videos due to its self-attention parallel computing mechanism. However, different from some previous approaches directly using the vanilla transformer, we consider that locality is significant in low-level vision tasks since the misaligned contextual local features (i.e., edges, shapes) may affect the prediction quality. Therefore, the proposed LATB is designed to align the video pixel with its most relevant ones adaptively in the local region to preserve the regional content information. Furthermore, we publish a new real-world low-light video dataset, named ExpressWay, to fill the gaps in the lack of dynamic low-light video scenarios, which contains high-quality videos with moving objects in both dark- and bright-light conditions. We conduct experiments on five benchmarks under three comprehensive settings including synthesized, static and our proposed dynamic low-light video datasets. Extensive experimental results show that our ALAT can outperform the previous state-of-the-arts by a large margin of 0.20∼1.10dB. Our method can be also extended to other video enhancement applications. The project is available at https://github.com/y1wencao/LLVE-ALAT.

A gate‐tunable memristor emulator for motion detection

SummaryWith its low power consumption and small size, the memristor has shown great potential for improving data storage density and computing efficiency. Compared to the dual‐port memristor, greater attention should be paid to researching gate‐tunable memristor for image processing to improve the processing speed and reduce hardware resource consumption. Developing gate‐tunable memristor emulators is highly attractive given the immaturity of current fabrication of the gate‐tunable memristor. This work proposes a digital gate‐tunable memristor emulator based on Raspberry Pi, which addresses the non‐reconfigurability and inflexibility issues of the analog emulators. The proposed emulator can match the behavior of different memristor devices by regulating the gate voltage parameter. Additionally, it can operate at a maximum frequency of 500 MHz. To test the functionality of the proposed emulator, a digital implementation of the memristive circuit for motion detection is designed and verified experimentally. Experiments demonstrate that when moving object detection is performed on a 640 × 350 pixel video stream, low power consumption of 53 mW and a delay of 3.52 μs can be achieved.

Non-contact human respiratory rate measurement based on two-level fusions of video and FMCW radar information

In this paper, we propose a robust respiratory rate (RR) measurement method using a two-level fusion of video and FMCW (frequency modulated continuous wave) radar information. Specifically, the video pixel displacement signals and the radar phase variation signals are extracted independently in shoulders and chest regions, which are then denoised by the serial-CEEMDAN (SCEEMDAN) method to obtain the corresponding single modal RR values. Next, at the feature level, the multivariate singular spectrum analysis (MSSA) is employed to extract the shared respiratory components in the video and radar modalities. Finally, at the decision level, we calculate the target RR value based on a signal-to-noise ratio (SNR) weighting of all single-modality and MSSA results. The proposed two-level fusion method is evaluated on a self-collected dataset, which includes 15 healthy subjects acting five challenging activities. The experimental results show that the proposed fusion method significantly outperform that of the single modality.

Integrated data detection and video restoration for optical camera communications

Optical Camera Communication (OCC) technology enables the use of Image Sensors (IS) with Visible Light Communications (VLC), one of the key technologies for the Sixth Generation (6G) of mobile communications. For Screen-to-Camera (S2C) applications in which Light-Emitting Diode (LED) screens modulate information symbols onto video pixels, increasing either the (transmitted) image resolution or frame rate may have a negative effect on the bit error performance and visual quality of OCC-encoded videos. This work proposes a novel OCC/S2C coding approach that lets both symbol transmission and video quality can be improved simultaneously. For this dual task, a Parallel Factor Analysis (PARAFAC) decomposition is applied for the first time in the literature to model an OCC system. In addition to validating the correct formulation of the proposed models, hypotheses, and algorithms, computational simulations demonstrate that increasing the number of video frames can result in time diversity gains. In accordance with the parameters of the system, this work also addresses the practical aspects of data rate, data volume, and computational complexity of the receiver.

Títul Mobile visual media design based on digital media technology and augmented reality

Abstract The reliance on shape-based parameters for traditional 3D multimedia visual behavior can significantly reduce the stability of image simulation. In this paper, the video feature enhancement method of wavelet and human eye vision is used to set the initialization parameters of the ant colony method calculation and calculate the video pixel to different food source distances as a way to obtain the amount of information of each path. The interface pixel affiliation is calculated to make a judgment on whether the amount of information is greater than the initial threshold. The contrast between the detail components of the interface at different scales is enhanced using the wavelet multistage method, and the video visual feedback enhancement is achieved by extracting the interface edge regions in each scale through the local shareholding method. Then the multimedia video representation of human behavior as silhouette sequence, extract the human behavior motion trajectory as large-scale multimedia visual behavior features, and extract the edge features of multimedia vision, and express the results as visual detail features in multimedia video. Finally, a multimedia visual human behavior state model is formed, and the patio-temporal sequence of the model is estimated by fusing it with particle filtering approximation inference to complete the behavioral simulation analysis of multimedia vision. The experimental results show that the average recognition rate of the method in this paper is as high as 98%, the accuracy of behavioral feature extraction is as high as 96%, and the error rate of modeling is 0.02%.

A target-free video structural motion estimation method based on multi-path optimization

The vibration data are quite important for structural health monitoring (SHM). This paper proposed a novel method, to adaptively estimate video motions of the structure in subpixel accuracy, without attaching any targets. The proposed method includes three steps. In the first step, to remove outliers and simultaneously preserve feature points, the Gaussian range kernel is used along with the Gaussian spatial kernel, and calculated by the polynomial fitting and recursive integrals computing. In the second step, to calculate video pixel motions varied with spatial coordinates in the region of interest (ROI) for testing, the ROI is divided into multiple grid cells. Motions in each grid cell are modeled as local spatially-variant homography matrices, and their spatial consistency are enhanced by a shape-preserving constraint. The third step is to enhance both spatial and temporal correlations of the calculated homography matrices, achieved by the data term and the smoothness term in both space and time domains. The superiority of the proposed method over traditional methods was validated in several case studies for analyzing structural motions. Among the comparisons, the proposed method can produce image denoising, camera motions, structural motions, and structural modal information in subpixel accuracy, and with the best accuracy.

Improving signal-to-noise ratio in ultrasound video pixel difference

Pixel difference is a holistic change measure for ultrasound videos and other image sequences. It is based on Euclidean distance (l2 norm) which is calculated by interpreting each frame as a vector of pixels. Recently, we found that 3D/4D ultrasound data producesignificantly worse signal-to-noise ratio in Pixel Difference curves, which appears to be primarily due to the lower frame rate of 3D/4D ultrasound when compared with 2D ultrasound. We also found that the signal-to-noise ratio is improved when we limit the analysis to tongue-internal pixels. In this study, we seek to quantify the timing behavior of tongue gestures obtained from whole-image and tongue-internalPD. If the timing behavior is similar in both cases, we can increase the signal to noise ratio of 3D/4D ultrasound at the same time as we increase its frame rate, because imaging to a shallower depth requires less time per frame. We also examine the signal-to-noise behavior of PD curves calculated with other norms (e.g., l1, and l3 norms).

Manipulating pixels, graphic images and video using Javascript

Working with pixels, changing, or using their RGBA values, using JavaScript and more precisely the p5.js library is discussed. Filters have been created, which manipulate pixels in images or video. The topic of steganography and storage of information in an image through pixels has also been analysed.

Rate-Distortion Optimal Transform Coefficient Selection for Unoccupied Regions in Video-Based Point Cloud Compression

This paper presents a novel method to determine rate-distortion optimized transform coefficients for efficient compression of videos generated from point clouds. The method exploits a generalized frequency selective extrapolation approach that iteratively determines rate-distortion-optimized coefficients for all basis functions of two-dimensional discrete cosine and sine transforms. The method is applied to blocks containing both occupied and unoccupied pixels in video based point cloud compression for HEVC encoding. In the proposed algorithm, only the values of the transform coefficients are changed such that resulting bit streams are compliant to the V-PCC standard. For all-intra coded point clouds, bitrate savings of more than 4% for geometry and more than 6% for texture error metrics with respect to standard encoding can be observed. These savings are more than twice as high as savings obtained using competing methods from literature. In the randomaccess case, our proposed method outperforms competing V-PCC methods by more than 0.5%.

Iterative Knowledge Exchange Between Deep Learning and Space-Time Spectral Clustering for Unsupervised Segmentation in Videos.

We propose a dual system for unsupervised object segmentation in video, which brings together two modules with complementary properties: a space-time graph that discovers objects in videos and a deep network that learns powerful object features. The system uses an iterative knowledge exchange policy. A novel spectral space-time clustering process on the graph produces unsupervised segmentation masks passed to the network as pseudo-labels. The net learns to segment in single frames what the graph discovers in video and passes back to the graph strong image-level features that improve its node-level features in the next iteration. Knowledge is exchanged for several cycles until convergence. The graph has one node per each video pixel, but the object discovery is fast. It uses a novel power iteration algorithm computing the main space-time cluster as the principal eigenvector of a special Feature-Motion matrix without actually computing the matrix. The thorough experimental analysis validates our theoretical claims and proves the effectiveness of the cyclical knowledge exchange. We also perform experiments on the supervised scenario, incorporating features pretrained with human supervision. We achieve state-of-the-art level on unsupervised and supervised scenarios on four challenging datasets: DAVIS, SegTrack, YouTube-Objects, and DAVSOD. We will make our code publicly available.

RETRACTED: Geometric Construction of Video Stereo Grid Space

The construction of digital twin cities is a current research hotspot. Video data are one of the important aspects of digital twin cities, and their digital modeling is one of the important foundations of its construction. For this reason, the construction and digital analysis of video data space has become an urgent problem to be solved. After in-depth research, this study found that the existing video space construction methods have three shortcomings: first, the problem of high requirements for objective conditions or low accuracy; second, the lack of easy and efficient mapping algorithms from 2D video pixel coordinates to 3D; and third, the lack of efficient correlation mechanisms between video space and external geographic information, making it difficult to integrate video space with external information, and thus prevent a more effective analysis. In view of the above problems, this paper proposes a video stereo grid geometric space construction method based on GeoSOT-3D stereo grid coding and a camera imaging model to form a video stereo grid space model. Finally, targeted experiments of video stereo grid space geometry construction were conducted to analyze the experimental results before and after optimization and compare the variance size to verify the feasibility and effectiveness of the model.

Detection of blood stains using computer vision-based algorithms and their association with postoperative outcomes in thoracoscopic lobectomies.

Our goal was to develop high throughput computer vision (CV) algorithms to detect blood stains in thoracoscopic surgery and to determine how the detected blood stains are associated with postoperative outcomes. Blood pixels in surgical videos were identified by CV algorithms trained with thousands of blood and non-blood pixels randomly selected and manually labelled. The proportion of blood pixels (PBP) was computed for key video frames to summarize the blood stain information during surgery. Statistical regression analyses were utilized to investigate the potential association between PBP and postoperative outcomes, including drainage volume, prolonged tube indwelling duration (≥5 days) and bleeding volume. A total of 275 patients undergoing thoracoscopic lobectomy were enrolled. The sum of PBP after flushing (P < 0.022), age (P = 0.005), immediate postoperative air leakage (P < 0.001), surgical duration (P = 0.001) and intraoperative bleeding volume (P = 0.033) were significantly associated with drainage volume in multivariable linear regression analysis. After adjustment using binary logistic regression analysis, the sum of the PBP after flushing [P = 0.017, odds ratio 1.003, 95% confidence interval (CI) 1.000-1.005] and immediate postoperative air leakage (P < 0.001, odds ratio 4.616, 95% CI 1.964-10.847) were independent predictors of prolonged tube indwelling duration. In the multivariable linear regression analysis, surgical duration (P < 0.001) and the sum of the PBP of the surgery (P = 0.005) were significantly correlated with intraoperative bleeding volume. This is the first study on the correlation between CV and postoperative outcomes in thoracoscopic surgery. CV algorithms can effectively detect from surgical videos information that has good prediction power for postoperative outcomes.

A Crypto-Steganography Approach for Hiding Ransomware within HEVC Streams in Android IoT Devices.

Steganography is a vital security approach that hides any secret content within ordinary data, such as multimedia. This hiding aims to achieve the confidentiality of the IoT secret data; whether it is benign or malicious (e.g., ransomware) and for defensive or offensive purposes. This paper introduces a hybrid crypto-steganography approach for ransomware hiding within high-resolution video frames. This proposed approach is based on hybridizing an AES (advanced encryption standard) algorithm and LSB (least significant bit) steganography process. Initially, AES encrypts the secret Android ransomware data, and then LSB embeds it based on random selection criteria for the cover video pixels. This research examined broad objective and subjective quality assessment metrics to evaluate the performance of the proposed hybrid approach. We used different sizes of ransomware samples and different resolutions of HEVC (high-efficiency video coding) frames to conduct simulation experiments and comparison studies. The assessment results prove the superior efficiency of the introduced hybrid crypto-steganography approach compared to other existing steganography approaches in terms of (a) achieving the integrity of the secret ransomware data, (b) ensuring higher imperceptibility of stego video frames, (3) introducing a multi-level security approach using the AES encryption in addition to the LSB steganography, (4) performing randomness embedding based on RPS (random pixel selection) for concealing secret ransomware bits, (5) succeeding in fully extracting the ransomware data at the receiver side, (6) obtaining strong subjective and objective qualities for all tested evaluation metrics, (7) embedding different sizes of secret data at the same time within the video frame, and finally (8) passing the security scanning tests of 70 antivirus engines without detecting the existence of the embedded ransomware.

Deep-learning-based short-term electricity load forecasting: A real case application

The rising popularity of deep learning can largely be attributed to the big data phenomenon, the surge in the development of new and novel deep neural network architectures, and the advent of powerful computational innovations. However, the application of deep neural networks is rare for time series problems when compared to other application areas. Short-term load forecasting, a typical and difficult time series problem, is considered as the application domain in this study. One-dimensional Convolutional Neural Networks (CNNs) use is rare in time series forecasting problems when compared to Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU), and the efficiency of CNN has been rather remarkable for pattern extraction. Hence, a new method that uses one-dimensional CNNs based on Video Pixel Networks (VPNs) in this study, in which the gating mechanism of Multiplicative Units of the VPNs is modified in some sense, for short term load forecasting. Specifically, the proposed one-dimensional CNNs, LSTM and GRU variants are applied to real-world electricity load data for 1-hour-ahead and 24-hour-ahead prediction tasks which they are the main concerns for the electricity provider firms for short term load forecasting. Statistical tests were conducted to spot the significance of the performance differences in analyses for which ten ensemble predictions of each method were experimented. According to the results of the comparative analyses, the proposed one-dimensional CNN model yielded the best result in total with 2.21% mean absolute percentage error for 24-h ahead predicitions. On the other hand, not a noteworthy difference between the methods was spotted even the proposed one-dimensional CNN method yielded the best results with approximately 1% mean absolute percentage error for 1-h ahead predictions.

Intrinsic Satellite Video Decomposition With Motion Target Energy Constraint

Satellite videos dynamically monitor the Earth’s surface by using staring imaging, which has gained increased attention and enabled target tracking applications. While various tracking methods are processed on the original video, the rapid light changes due to staring imaging are not considered and have a negative effect on tracking. To reduce the effects of illumination and improve the performance of satellite video target tracking, an intrinsic satellite video decomposition model with motion target energy constraint, called MTE-ISVD, is proposed in this paper. The proposed algorithm introduces two main constraints: The first is a temporal constraint of reflectance, which can solve the flicker problem by preserving reflectance coherence in the time domain with the property that the background pixels in satellite videos are nearly consistent between adjacent frames. The second is a motion target energy constraint, which can concentrate the signal energy of the motion targets in the reflectance by representing them with the surrounding background in the shading. The decomposition problem is reformulated as a quadratic function minimization, which can be addressed using the standard conjugate gradient in closed form. For visual and quantitative comparisons, we perform experiments on five Jilin-1 satellite videos and analyze the results in terms of visual comparison, target tracking improvement, stability evaluation and processing time comparison. The experimental results demonstrate that our proposed method outperforms the other representative intrinsic decomposition methods in terms of processing speed, stability, and motion target representation.

AVQBits—Adaptive Video Quality Model Based on Bitstream Information for Various Video Applications

The paper presents <i>AV QBits</i>, a versatile, bitstream-based video quality model. It can be applied in several contexts such as video service monitoring, evaluation of video encoding quality, of gaming video QoE, and even of omnidirectional video quality. In the paper, it is shown that <i>AV QBits</i> predictions closely match video quality ratings obained in various subjective tests with human viewers, for videos up to 4K-UHD resolution (Ultra-High Definition, 3840 x 2180 pixels) and framerates up 120 fps. With the different variants of <i>AV QBits</i> presented in the paper, video quality can be monitored either at the client side, in the network or directly after encoding. The no-reference <i>AV QBits</i> model was developed for different video services and types of input data, reflecting the increasing popularity of Video-on-Demand services and widespread use of HTTP-based adaptive streaming. At its core, <i>AV QBits</i> encompasses the standardized ITU-T P.1204.3 model, with further model instances that can either have restricted or extended input information, depending on the application context. Four different instances of <i>AV QBits</i> are presented, that is, a Mode 3 model with full access to the bitstream, a Mode 0 variant using only metadata such as codec type, framerate, resoution and bitrate as input, a Mode 1 model using Mode 0 information and frame-type and -size information, and a Hybrid Mode 0 model that is based on Mode 0 metadata and the decoded video pixel information. The models are trained on the authors&#x2019; own AVT-PNATS-UHD-1 dataset described in the paper. All models show a highly competitive performance by using AVT-VQDB-UHD-1 as validation dataset, e.g., with the Mode 0 variant yielding a value of 0.890 Pearson Correlation, the Mode 1 model of 0.901, the hybrid no-reference mode 0 model of 0.928 and the model with full bitstream access of 0.942. In addition, all four <i>AV QBits</i> variants are evaluated when applying them out-of-the-box to different media formats such as 360&#x00B0; video, high framerate (HFR) content, or gaming videos. The analysis shows that the ITU-T P.1204.3 and Hybrid Mode 0 instances of <i>AV QBits</i> for the considered use-cases either perform on par with or better than even state-of-the-art full reference, pixel-based models. Furthermore, it is shown that the proposed Mode 0 and Mode 1 variants outperform commonly used no-reference models for the different application scopes. Also, a long-term integration model based on the standardized ITU-T P.1203.3 is presented to estimate ratings of overall audiovisual streaming Quality of Experience (QoE) for sessions of 30 s up to 5 min duration. In the paper, the <i>AV QBits</i> instances with their per-1-sec score output are evaluated as the video quality component of the proposed long-term integration model. All <i>AV QBits</i> variants as well as the long-term integration module are made publicly available for the community for further research.

Automatic Separation of Respiratory Flow from Motion in Thermal Videos for Infant Apnea Detection.

Both Respiratory Flow (RF) and Respiratory Motion (RM) are visible in thermal recordings of infants. Monitoring these two signals usually requires landmark detection for the selection of a region of interest. Other approaches combine respiratory signals coming from both RF and RM, obtaining a Mixed Respiratory (MR) signal. The detection and classification of apneas, particularly common in preterm infants with low birth weight, would benefit from monitoring both RF and RM, or MR, signals. Therefore, we propose in this work an automatic RF pixel detector not based on facial/body landmarks. The method is based on the property of RF pixels in thermal videos, which are in areas with a smooth circular gradient. We defined 5 features combined with the use of a bank of Gabor filters that together allow selection of the RF pixels. The algorithm was tested on thermal recordings of 9 infants amounting to a total of 132 min acquired in a neonatal ward. On average the percentage of correctly identified RF pixels was . Obstructive Apneas (OAs) were simulated as a proof of concept to prove the advantage in monitoring the RF signal compared to the MR signal. The sensitivity in the simulated OA detection improved for the RF signal reaching against the of the MR signal. Overall, the method yielded promising results, although the positioning and number of cameras used could be further optimized for optimal RF visibility.

DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels.

Videos of animal behavior are used to quantify researcher-defined behaviors of interest to study neural function, gene mutations, and pharmacological therapies. Behaviors of interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We created DeepEthogram: software that uses supervised machine learning to convert raw video pixels into an ethogram, the behaviors of interest present in each video frame. DeepEthogram is designed to be general-purpose and applicable across species, behaviors, and video-recording hardware. It uses convolutional neural networks to compute motion, extract features from motion and images, and classify features into behaviors. Behaviors are classified with above 90% accuracy on single frames in videos of mice and flies, matching expert-level human performance. DeepEthogram accurately predicts rare behaviors, requires little training data, and generalizes across subjects. A graphical interface allows beginning-to-end analysis without end-user programming. DeepEthogram's rapid, automatic, and reproducible labeling of researcher-defined behaviors of interest may accelerate and enhance supervised behavior analysis. Code is available at: https://github.com/jbohnslav/deepethogram.

Effective and efficient pixel-level detection for diverse video copy-move forgery types

Video copy-move forgery detection (VCMFD) is a significant and greatly challenging task due to a variety of difficulties, including a huge amount of video information, diverse forgery types, rich forgery objects, and homogenous forgery sources. These difficulties raise four unresolved key challenges in VCMFD: i) ineffective detection in some popular forgery cases; ii) inefficient matching in processing numerous video pixels with hundred-dimensional features under dozens of matching iterations; iii) high false positive (FP) in detecting forgery videos; iv) low trade-off of efficiency and effectiveness in filling forgery region, and even failing in indicating forgeries at the pixel level. In this paper, a novel VCMFD method is proposed to address these issues: i) an innovatively improved SIFT structure that can address the thorough feature extraction in all video copy-move forgery cases; ii) a novel fast keypoint-label matching (FKLM) algorithm is proposed that creates some keypoint-label groups so that every high-dimensional feature is assigned into one of these groups. As a result, matching of video pixels can be directly done on a small number of keypoint-label groups only, leading to a nearly 500% raise in matching efficiency; iii) a new coarse-to-fine filtering relying on intrinsic attributes of exact keypoint-matches is designed to more effectively reduce the false keypoint-matches; iv) the adaptive block filling relying on true keypoint-matches contributes to the accurate and efficient suspicious region filling, even at the pixel level. Finally, the suspicious region locations with the forgery vision persistence concept indicate forgery videos. Compared to the state-of-art methods, the experiments show that our proposed method achieves the best detection accuracy, lowest FP, and improved at least 16% and 8% of F1 scores on the GRIP 2.0 dataset and a combination of SULFA 2.0 & REWIND datasets. Furthermore, the proposed method is with low computational time (4.45 s/Mpixels), which is about 1/2-1/3 times of the latest DFMI-BM (8.02 s/Mpixels) and PM-2D (13.1 s/Mpixels) methods.