Video Rate Research Articles

Content-aware QoE optimization in MEC-assisted Mobile video streaming

The traditional client-based HTTP adaptation strategies do not explicitly coordinate between the clients, servers, and cellular networks. A lack of coordination leads to suboptimal user experience. In addition to optimizing Quality of Experience (QoE), other challenges in adapting HTTP adaptive streaming (HAS) to the cellular environment are overcoming unfair allocation of the video rate and inefficient utilization of the bandwidth under the high-dynamics cellular links. Furthermore, the majority of the adaptive strategies ignore important video content characteristics and HAS client information, such as segment duration, buffer size, and video duration, in the video quality selection process. In this paper, we present a content-aware hybrid multi-access edge computing (MEC)-assisted quality adaptation algorithm by taking advantage of the capabilities of edge cloud computing. The proposed algorithm exploits video content characteristics, HAS client settings, and application-layer information to jointly adapt the bitrates of multiple clients. We design separate strategies to optimize the performance of short and long duration videos. We then demonstrate the efficiency of our algorithm against client-based solutions as well as MEC-assisted algorithms. The proposed algorithm guarantees high QoE, equitably selects video rates for clients, and efficiently utilizes the bandwidth for both short and long duration videos. The results from our extensive experiments reveal that the proposed long video adaptation algorithm outperforms state-of-the-art algorithms, with improvements in average video rate, QoE, fairness, and bandwidth utilization of 0.4%–12.3%, 8%–65%, 3.3%–5.7%, and 60%–130%, respectively. Furthermore, when high bandwidth is available to competing clients, the proposed short video adaptation algorithm improves QoE by 11.1% compared to the long video adaptation algorithm.

Rapid and precise tracking of water influx and efflux across cell membranes induced by a pulsed electric field.

Quantitative measurements of water content within a single cell are notoriously difficult. In this work, we introduce a single-shot optical method for tracking the intracellular water content, by mass and volume, of a single cell at video rate. We utilize quantitative phase imaging and a priori knowledge of a spherical cellular geometry, leveraging a two-component mixture model to compute the intracellular water content. We apply this technique to study CHO-K1 cells responding to a pulsed electric field, which induces membrane permeabilization and rapid water influx or efflux depending upon the osmotic environment. The effects of mercury and gadolinium on water uptake in Jurkat cells following electropermeabilization are also examined.

Rapid detection of neurons in widefield calcium imaging datasets after training with synthetic data

Widefield microscopy can provide optical access to multi-millimeter fields of view and thousands of neurons in mammalian brains at video rate. However, tissue scattering and background contamination results in signal deterioration, making the extraction of neuronal activity challenging, laborious and time consuming. Here we present our deep-learning-based widefield neuron finder (DeepWonder), which is trained by simulated functional recordings and effectively works on experimental data to achieve high-fidelity neuronal extraction. Equipped with systematic background contribution priors, DeepWonder conducts neuronal inference with an order-of-magnitude-faster speed and improved accuracy compared with alternative approaches. DeepWonder removes background contaminations and is computationally efficient. Specifically, DeepWonder accomplishes 50-fold signal-to-background ratio enhancement when processing terabytes-scale cortex-wide functional recordings, with over 14,000 neurons extracted in 17 h.

Learning-Based Rate Control for High Efficiency Video Coding

High efficiency video coding (HEVC) has dramatically enhanced coding efficiency compared to the previous video coding standard, H.264/AVC. However, the existing rate control updates its parameters according to a fixed initialization, which can cause errors in the prediction of bit allocation to each coding tree unit (CTU) in frames. This paper proposes a learning-based mapping method between rate control parameters and video contents to achieve an accurate target bit rate and good video quality. The proposed framework contains two main structural codings, including spatial and temporal coding. We initiate an effective learning-based particle swarm optimization for spatial and temporal coding to determine the optimal parameters at the CTU level. For temporal coding at the picture level, we introduce semantic residual information into the parameter updating process to regulate the bit correctly on the actual picture. Experimental results indicate that the proposed algorithm is effective for HEVC and outperforms the state-of-the-art rate control in the HEVC reference software (HM-16.10) by 0.19 dB on average and up to 0.41 dB for low-delay P coding structure.

Children’s Safety on YouTube: A Systematic Review

Background: With digital transformation and growing social media usage, kids spend considerable time on the web, especially watching videos on YouTube. YouTube is a source of education and entertainment media that has a significant impact on the skill improvement, knowledge, and attitudes of children. Simultaneously, harmful and inappropriate video content has a negative impact. Recently, researchers have given much attention to these issues, which are considered important for individuals and society. The proposed methods and approaches are to limit or prevent such threats that may negatively influence kids. These can be categorized into five main directions. They are video rating, parental control applications, analysis meta-data of videos, video or audio content, and analysis of user accounts. Objective: The purpose of this study is to conduct a systematic review of the existing methods, techniques, tools, and approaches that are used to protect kids and prevent them from accessing inappropriate content on YouTube videos. Methods: This study conducts a systematic review of research papers that were published between January 2016 and December 2022 in international journals and international conferences, especially in IEEE Xplore Digital Library, ACM Digital Library, Web of Science, Google Scholar, Springer database, and ScienceDirect database. Results: The total number of collected articles was 435. The selection and filtration process reduced this to 72 research articles that were appropriate and related to the objective. In addition, the outcome answers three main identified research questions. Significance: This can be beneficial to data mining, cybersecurity researchers, and peoples’ concerns about children’s cybersecurity and safety.

Turbulence video simulation using the spatiotemporal cross correlation of distortion fields.

Previous simulations of atmospheric turbulence in videos are computationally complex. The purpose of this study is to develop an efficient algorithm for simulating spatiotemporal videos affected by atmospheric turbulence, given a static image. We extend an existing method for the simulation of atmospheric turbulence in a single image by incorporating turbulence properties in the time domain and the blurring effect. We accomplish this through analysis of the correlation between turbulence image distortions in time and in space. The significance of this method is the ease with which it will be possible to produce a simulation, given properties of the turbulence (including turbulence strength, object distance, and height). We apply the simulation to low and high frame rate videos, and we show that the spatiotemporal cross correlation of the distortion fields in the simulated video matches the physical spatiotemporal cross correlation function. Such a simulation can be useful when developing algorithms that apply to videos degraded by atmospheric turbulence and require a large amount of imaging data for training.

Dynamic Fusion Technology of Mobile Video and 3D GIS: The Example of Smartphone Video

Mobile videos contain a large amount of data, where the information interesting to the user can either be discrete or distributed. This paper introduces a method for fusing 3D geographic information systems (GIS) and video image textures. For the dynamic fusion of video in 3DGIS where the position and pose angle of the filming device change moment by moment, it integrates GIS 3D visualization, pose resolution and motion interpolation, and proposes a projection texture mapping method for constructing a dynamic depth camera to achieve dynamic fusion. In this paper, the accuracy and time efficiency of different systems of gradient descent and complementary filtering algorithms are analyzed mainly by quantitative analysis method, and the effect of dynamic fusion is analyzed by the playback delay and rendering frame rate of video on 3DGIS as indicators. The experimental results show that the gradient descent method under the Aerial Attitude Reference System (AHRS) is more suitable for the solution of smartphone attitude, and can control the root mean square error of attitude solution within 2°; the delay of video playback on 3DGIS is within 29 ms, and the rendering frame rate is 34.9 fps, which meets the requirements of the minimum resolution of human eyes.

Spectral imaging enables contrast agent–free real-time ischemia monitoring in laparoscopic surgery

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent-free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning-based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

Applying a video recording, video-based rating method in OSCEs

ABSTRACT Introduction Objective structured clinical examination (OSCE) results could be affected by low homogeneity of examiners, non-retrospectiveness of test results, and examiner-cohort effect. In China, many students participate in medical qualification examinations, and this issue is particularly significant. This study aimed to develop a video recording, video-based rating method and compare the reliability of video and on-site ratings to enhance the quality assurance of OSCEs. Methods The subjects of this study were clinical students one year after graduation participating in the clinical skills portion of the National Medical Licensing Examination. The participants were from four cities in Jiangsu province. Participants were randomly allocated to on-site and video rating groups to evaluate the rating methods consistency. We verified the reliability of recording equipment and evaluability of video recording. Moreover, we compared the consistency and equivalence of the two rating methods and analyzed the impact of video recording on scores. Results The reliability of recording equipment and evaluability of video recording were high. Evaluation consistency between experts and examiners was acceptable, and there was no difference in evaluation results (P = 0.61). There was good consistency between video and on-site rating; however, a difference between the two rating methods was detected. The scores of video-based rating group students were lower than those of all students (P < 0.00). Conclusions Video-based rating could be reliable and offer advantages over on-site rating. The video recording, video-based rating method could provide greater content validity based on its traceability and the ability to view details. Video recording, video-based rating offers a promising mthod for improving the effectiveness and fairness of OSCEs.

Three-dimensional single particle tracking using 4π self-interference of temporally phase-shifted fluorescence

Single particle tracking in three dimensions is an indispensable tool for studying dynamic processes in various disciplines, including material sciences, physics, and biology, but often shows anisotropic three-dimensional spatial localization precision, which restricts the tracking precision, and/or a limited number of particles that can be tracked simultaneously over extended volumes. Here we developed an interferometric, three-dimensional fluorescence single particle tracking method based on conventional widefield excitation and temporal phase-shift interference of the emitted, high-aperture-angle, fluorescence wavefronts in a greatly simplified, free-running, triangle interferometer that enables tracking of multiple particles at the same time with <10-nm spatial localization precision in all three dimensions over extended volumes (~35 × 35 × 2 μm3) at video rate (25 Hz). We applied our method to characterize the microenvironment of living cells and up to ~40 μm deep in soft materials.

A GNN-based proactive caching strategy in NDN networks

As people spend more time watching and sharing videos online, it is critical to provide users with a satisfactory quality of experience (QoE). Leveraging the in-network caching and named-based routing features in Named Data Networks (NDNs), our paper aims to improve user experience through caching. We propose a graph neural network-gain maximization (GNN-GM) cache placement algorithm. First, we use a GNN model to predict users’ ratings of unviewed videos. Second, we consider the total predicted rating of a video as the gain of the cached video. Third, we propose a cache placement algorithm to maximize the caching gain and actively cache videos. Cache replacement is implemented based on the cache gain ranking of videos, with higher cache gain videos replacing lower cache gain videos. We compare GNN-GM with two state-of-the-art caching strategies, namely the NMF-based caching strategy and GNN-CPP. GNN-GM is also compared with two traditional caching strategies, LCE and LRU, LCE and FIFO. We evaluate the five caching strategies using real-world datasets in a tree network topology, a real-world network topology GEANT, and various random topologies. The experimental results show that our caching policy significantly improves cache hit ratio, latency and server load. Notably, GNN-GM achieves a 25% higher cache hit rate, 5% lower latency and 7% lower server load than GNN-CPP in GEANT.

Research on the Impact of UGC Quality on Consumers’ Purchase Intention

As the key engine of Internet shopping, online consumer groups in the 5G era has been increasing in recent years, making consumption behavior on Internet a new form of market development. According to the data of China’s Internet Development Status, by 2021, the number of Internet users in China has risen to 1.033 billion, and there are more than 2,000 5G+ industrial network projects under construction, of which the application rate of short video is as high as 90.6%. It can be seen that Internet retail is the current form of market economic development. TikTok, Micro videos, Huoshan short videos, etc. account for half of the short video marketing. The regular operation of the social economy in the face of the COVID-19 outbreak is encouraged by this new form of internet economy, which attracts consumers and encourages purchasing online. This study explains the quality of User-Generated Content (UGC) and examines its impact on consumers’ purchase intentions through a questionnaire poll.

Video-rate Raman-based metabolic imaging by Airy light-sheet illumination and photon-sparse detection

Despite its massive potential, Raman imaging represents just a modest fraction of all research and clinical microscopy to date. This is due to the ultralow Raman scattering cross-sections of most biomolecules that impose low-light or photon-sparse conditions. Bioimaging under such conditions is suboptimal, as it either results in ultralow frame rates or requires increased levels of irradiance. Here, we overcome this tradeoff by introducing Raman imaging that operates at both video rates and 1,000-fold lower irradiance than state-of-the-art methods. To accomplish this, we deployed a judicially designed Airy light-sheet microscope to efficiently image large specimen regions. Further, we implemented subphoton per pixel image acquisition and reconstruction to confront issues arising from photon sparsity at just millisecondintegrations. We demonstrate the versatility of our approach by imaging a variety of samples, including the three-dimensional (3D) metabolic activity of single microbial cells and the underlying cell-to-cell variability. To image such small-scale targets, we again harnessed photon sparsity to increase magnification without a field-of-view penalty, thus, overcoming another key limitation in modern light-sheet microscopy.

Deep Multimodal Habit Tracking System: A User-adaptive Approach for Low-power Embedded Systems

The pace of population ageing is increasing and is currently becoming one of the challenges our society faces. The introduction of Cyber-Physical Systems (CPS) has fostered the development of e-Health solutions that ease the associated economic and social burden. In this work, a CPS-based solution is presented to partially tackle the problem: a Deep Multimodal Habit Tracking system. The aim is to monitor daily life activities to alert in case of life-threatening situations improving their autonomy and supporting healthy lifestyles while living alone at home. Our approach combines video and heart rate cues to accurately identify indoor actions, running the processing locally in embedded edge nodes. Local processing provides inherent protection of data privacy since no image or vital signs are transmitted to the network, and reduces data bandwidth usage. Our solution achieves an accuracy of more than 80% in average, reaching up to a 95% for specific subjects after adapting the system. Adding heart-rate information improves F1-score by 2.4%. Additionally, the precision and recall for critical actions such as falls reaches up to 93.75%. Critical action detection is crucial due to their dramatic consequences, it helps to reduce false alarms, leading to building trust in the system and reducing economic cost. Also, the model is optimized and integrated in a Nvidia Jetson Nano embedded device, reaching real-time performance below 3.75 Watts. Finally, a dataset specifically designed for indoor action recognition using synchronized video and heart rate pulses has been collected.

Assessing calf exercise fidelity among people with Achilles tendinopathy using videos recorded via Zoom®: an observational study

Purpose Exercise fidelity is a fundamental aspect of exercise prescription by physiotherapists. Assessing exercise fidelity via telehealth (videoconference) is potentially useful but its reproducibility has not been assessed. The objective was to evaluate agreement (reproducibility) of video rating of calf exercise fidelity within (over four weeks) and between telehealth raters and compared with live rating. Methods Nineteen videos of participants with AT undertaking calf exercises were rated by physiotherapists in person (live) and by watching pre-recorded videos (replicating telehealth). Three forms of agreement were assessed: (i) between live assessment and recorded video (telehealth) assessment; (ii) between telehealth raters (inter-rater); (iii) agreement within telehealth raters over four weeks (intra-rater). Results There was weak to almost perfect agreement (Kappa = 0.65 to 1.00, rater 1, Kappa = 0.57 to 1.00, rater 2) between the telehealth and live raters. There was moderate to almost perfect agreement (Kappa = 0.65 to 1.00) when assessing intra-rater reliability, aside from one of the six criteria (quality of movement). Conclusion Assessing calf exercise fidelity via videos recorded on Zoom® appears to demonstrate weak or greater agreement versus live rating and over time. Inter-rater agreement was lower, suggesting that the criteria developed may be better suited to use by individual raters over time. Implications for rehabilitation Telehealth assessment (when compared with live, in person assessment) shows a moderate or greater agreement for the majority of assessment criteria when assessing calf exercise fidelity. The calf exercise fidelity criteria developed can be used as an adjunct to assessment and management of people with Achilles Tendinopathy. Intra-rater assessment was shown to be moderately or more consistent over a four-week period when assessing calf exercise fidelity.

Large-scale metagrating complex-based light field 3D display with space-variant resolution for non-uniform distribution of information and energy.

Glasses-free three-dimensional (3D) display has attracted wide interest for providing stereoscopic virtual contents with depth cues. However, how to achieve high spatial and angular resolution while keeping ultrawide field of view (FOV) remains a significant challenge in 3D display. Here, we propose a light field 3D display with space-variant resolution for non-uniform distribution of information and energy. The spatial resolution of each view is modulated according to watching habit. A large-scale combination of pixelated 1D and 2D metagratings is used to manipulate dot and horizontal line views. With the joint modulation of pixel density and view arrangement, the information density and illuminance of high-demand views are at most 5.6 times and 16 times that of low-demand views, respectively. Furthermore, a full-color and video rate light field 3D display with non-uniform information distribution is demonstrated. The prototype provides 3D images with a high spatial resolution of 119.6 pixels per inch and a high angular resolution of 0.25 views per degree in the high-demand views. An ultrawide viewing angle of 140° is also provided. The proposed light field 3D display does not require ultrahigh-resolution display panels and has form factors of thin and light. Thus, it has the potential to be used in portable electronics, window display, exhibition display, as well as tabletop display.

FIAT: A Device for Objective, Optical Measures of Accommodation in Phakic and Pseudophakic Eyes.

To present FIAT, a novel optical instrument and analysis package that is designed to elicit and optically record accommodation in human eyes. FIAT employs a Shack-Hartmann wavefront sensor and a retro-illumination pupil camera that records from a single eye at video rates. It is effective at eliciting accommodation by offering the subject a full-field binocular view of an alternating distant target and a near-eye display. FIAT analysis software computes wave aberrations for each video frame over full- or subpupil sizes and computes accommodative dynamics and accommodative range. The system is validated by showing accurate refraction measurements in model eyes and human eyes with trial lenses. Robust accommodative responses are shown for young eyes, and a lack of accommodative response is shown for a known presbyopes. Accommodative stimulus-response curves from five phakic subjects over a range of ages show expected results. Results from two individuals with monofocal intraocular lenses are shown. FIAT is an effective instrument for making accurate, objective measures of accommodation in phakic and pseudophakic eyes. We present a device that can play an important role in the development and testing of accommodating intraocular lenses.

Diffuser-based fiber endoscopy for single-shot 3D fluorescence imaging

Minimally invasive endoscopy using coherent fiber bundles shows great potential for numerous applications in biomedical imaging. With a diffuser on the distal side of the fiber bundle and computational image recovery, single-shot 3D imaging is possible by encoding the image volume into 2D speckle patterns. In comparison to equivalent lens systems, a higher space-bandwidth product can be achieved. However, decoding the image with iterative algorithms is time-consuming. Thus, we propose utilizing a neural network for fast 2D and 3D image reconstruction at video rate. In this work, single-shot 3D fluorescence imaging with an ultra-thin endoscope is demonstrated, enabling applications like calcium imaging for in vivo brain diagnostics at cellular resolution.

Hyperspectral Image Superresolution via Subspace-Based Deep Prior Regularization

Hyperspectral imaging is able to provide a finer delivery of various material properties than conventional imaging systems. Yet in reality, an optical system can only generate data with high spatial resolution but low spectral one, or vice versa, at video rates. As a result, an issue that fuses low-resolution hyperspectral (LRHS) and high-resolution multispectral (HRMS) images has gained great attention. However, most fusion approaches depend purely on hand-crafted regularizers or data-driven priors, leading to the issues of tricky parameter selection or poor interpretability. In this work, a subspace-based deep prior regularization (SDPR) is proposed to tackle these problems, which takes both hand-crafted regularizer and data-driven prior into account. Specifically, we leverage the spectral correlation of the images and transfer them from the original space to the subspace domain, within which a modified U-net-based deep prior learning network (SDPL-net) is designed for the fusion issue. Moreover, instead of taking the output of SDPL-net directly as the result, we further feed the output back to the model-based optimization. Under such prior regularization, the recovered high-resolution hyperspectral (HRHS) image holds a high consistency to its inherent structure and hence tends to present enhanced reliability and accuracy. Experimental results on simulated and real data reveal that the proposed method excels other state-of-the-art (SOTA) methods in both quantitative and qualitative metrics.

A Residual Learning Approach to Deblur and Generate High Frame Rate Video With an Event Camera

Event cameras are bio-inspired cameras that can measure the intensity change asynchronously with high temporal resolution. One of the advantages of event cameras is that they suffer less from motion blur than traditional frame cameras when recording daily scenes with fast-moving objects. In this paper, we formulate the deblurring task on traditional cameras directed by events to be a residual learning one, and propose corresponding network architectures for effective learning of deblurring and high frame rate video generation tasks. We first train a modified U-Net network to restore a sharp image from a blurry image using the corresponding events. Then we train another similar network by replacing the downsampling blocks with blocks of the convolutional long short-term memory (Conv-LSTM) to recurrently generate high frame rate video using the restored sharp image and part of the events. Benefitting from the blur-free events and the proposed learning strategy, the experimental results show that the proposed method outperforms state-of-the-art methods for generating sharp images and high frame rate videos.