Raw Frames Research Articles

Masked Contrastive Representation Learning for Reinforcement Learning.

In pixel-based reinforcement learning (RL), the states are raw video frames, which are mapped into hidden representation before feeding to a policy network. To improve sample efficiency of state representation learning, recently, the most prominent work is based on contrastive unsupervised representation. Witnessing that consecutive video frames in a game are highly correlated, to further improve data efficiency, we propose a new algorithm, i.e., masked contrastive representation learning for RL (M-CURL), which takes the correlation among consecutive inputs into consideration. In our architecture, besides a CNN encoder for hidden presentation of input state and a policy network for action selection, we introduce an auxiliary Transformer encoder module to leverage the correlations among video frames. During training, we randomly mask the features of several frames, and use the CNN encoder and Transformer to reconstruct them based on context frames. The CNN encoder and Transformer are jointly trained via contrastive learning where the reconstructed features should be similar to the ground-truth ones while dissimilar to others. During policy evaluation, the CNN encoder and the policy network are used to take actions, and the Transformer module is discarded. Our method achieves consistent improvements over CURL on 14 out of 16 environments from DMControl suite and 23 out of 26 environments from Atari 2600 Games. The code is available at https://github.com/teslacool/m-curl.

Through-Foliage Tracking with Airborne Optical Sectioning

Detecting and tracking moving targets through foliage is difficult, and for many cases even impossible in regular aerial images and videos. We present an initial light-weight and drone-operated 1D camera array that supports parallel synthetic aperture aerial imaging. Our main finding is that color anomaly detection benefits significantly from image integration when compared to conventional raw images or video frames (on average 97% vs. 42% in precision in our field experiments). We demonstrate that these two contributions can lead to the detection and tracking of moving people through densely occluding forest.

CalibBD: Extrinsic Calibration of the LiDAR and Camera Using a Bidirectional Neural Network

With the rapid growth of self-driving vehicles, automobiles demand diverse data from multiple sensors to perceive the surrounding environment. Calibrating preprocessing between multiple sensors is necessary to utilize the data effectively. In particular, the LiDAR-camera pair, a suitable complement with 2D-3D information for each other, has been widely used in autonomous vehicles. Most traditional calibration methods require specific calibration targets set up under complicated environmental conditions, which require expensive human manual work. In this study, we propose a deep neural network that does not require any specific targets and offline setup to find the six degrees of freedom (6 DoF) transformation between LiDAR and the camera. Unlike previous deep learning CNN-based methods, which use raw 3D point clouds and 2D images frame by frame, CalibBD utilizes Bi-LSTM for sequence data to extract temporal features between consecutive frames. It not only predicts the calibration parameters by minimizing both transformation and depth losses but also calibrates the camera parameters by using temporal loss to refine the calibration parameters. The proposed model achieves a steady performance under various deviations of mis-calibration parameters and achieved higher results in terms of accuracy than the state-of-the-art CNN-based method on the KITTI datasets.

An attention based dual learning approach for video captioning

Video captioning aims to generate sentences/captions to describe video contents. It is one of the key tasks in the field of multimedia processing. However, most of the current video captioning approaches utilize only the visual information of a video to generate captions. Recently, a new encoder–decoder–reconstructorarchitecture was developed for video captioning, which can capture the information in both raw videos and the generated captions through dual learning. Based on this architecture, this paper proposes a novel attention based dual learning approach (ADL) for video captioning. Specifically, ADL is composed of a caption generation module and a video reconstruction module. The caption generation module builds a translatable mapping between raw video frames and the generated video captions, i.e., using the visual features extracted from videos by an Inception-V4 network to produce video captions. Then the video reconstruction module reproduces raw video frames using the generated video captions, i.e., using the hidden states of the decoder in the caption generation module to reproduce/synthesize raw visual features. A multi-head attention mechanism is adopted to help the two modules focus on the most effective information in videos and captions, and a dual learning mechanism is adopted to fine-tune the performance of the two modules to generate final video captions. Therefore, ADL can minimize the semantic gap between raw videos and the generated captions by minimizing the differences between the reproduced and the raw videos, thereby improving the quality of the generated video captions. Experimental results demonstrate that ADL is superior to the state-of-the-art video captioning approaches on benchmark datasets.

Slant-Stack Migration Applied to Plane-Wave Ultrasound Imaging.

Ultrafast plane-wave ultrasound imaging replaces numerous focused-beam transmissions with a single emitted plane-wave pulse, insonifying the entire subsurface region of interest all at once. To improve image quality, one can employ coherent plane wave compounding (CPWC), whereby several pulses are emitted sequentially at different steering angles, and their corresponding acquired raw data frames are individually beamformed and then combined to form a final reconstructed image frame. We describe a classic geophysical reconstruction technique called slant-stack migration, adapted here to CPWC imaging. Our evaluation results, based on two public-domain datasets featuring both anechoic and hyperechoic targets, demonstrate that the presented approach compares favorably with conventional delay-and-sum beamforming.

Integrating Domain Knowledge Into Deep Networks for Lung Ultrasound With Applications to COVID-19.

Lung ultrasound (LUS) is a cheap, safe and non-invasive imaging modality that can be performed at patient bed-side. However, to date LUS is not widely adopted due to lack of trained personnel required for interpreting the acquired LUS frames. In this work we propose a framework for training deep artificial neural networks for interpreting LUS, which may promote broader use of LUS. When using LUS to evaluate a patient’s condition, both anatomical phenomena (e.g., the pleural line, presence of consolidations), as well as sonographic artifacts (such as A- and B-lines) are of importance. In our framework, we integrate domain knowledge into deep neural networks by inputting anatomical features and LUS artifacts in the form of additional channels containing pleural and vertical artifacts masks along with the raw LUS frames. By explicitly supplying this domain knowledge, standard off-the-shelf neural networks can be rapidly and efficiently finetuned to accomplish various tasks on LUS data, such as frame classification or semantic segmentation. Our framework allows for a unified treatment of LUS frames captured by either convex or linear probes. We evaluated our proposed framework on the task of COVID-19 severity assessment using the ICLUS dataset. In particular, we finetuned simple image classification models to predict per-frame COVID-19 severity score. We also trained a semantic segmentation model to predict per-pixel COVID-19 severity annotations. Using the combined raw LUS frames and the detected lines for both tasks, our off-the-shelf models performed better than complicated models specifically designed for these tasks, exemplifying the efficacy of our framework.

GPU-accelerated real-time reconstruction in Python of three-dimensional datasets from structured illumination microscopy with hexagonal patterns

We present a structured illumination microscopy system that projects a hexagonal pattern by the interference among three coherent beams, suitable for implementation in a light-sheet geometry. Seven images acquired as the illumination pattern is shifted laterally can be processed to produce a super-resolved image that surpasses the diffraction-limited resolution by a factor of over 2 in an exemplar light-sheet arrangement. Three methods of processing data are discussed depending on whether the raw images are available in groups of seven, individually in a stream or as a larger batch representing a three-dimensional stack. We show that imaging axially moving samples can introduce artefacts, visible as fine structures in the processed images. However, these artefacts are easily removed by a filtering operation carried out as part of the batch processing algorithm for three-dimensional stacks. The reconstruction algorithms implemented in Python include specific optimizations for calculation on a graphics processing unit and we demonstrate its operation on experimental data of static objects and on simulated data of moving objects. We show that the software can process over 239 input raw frames per second at 512 × 512 pixels, generating over 34 super-resolved frames per second at 1024 × 1024 pixels.This article is part of the Theo Murphy meeting issue ‘Super-resolution structured illumination microscopy (part 1)’.

ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning

Structured illumination microscopy (SIM) has become an important technique for optical super-resolution imaging because it allows a doubling of image resolution at speeds compatible with live-cell imaging. However, the reconstruction of SIM images is often slow, prone to artefacts, and requires multiple parameter adjustments to reflect different hardware or experimental conditions. Here, we introduce a versatile reconstruction method, ML-SIM, which makes use of transfer learning to obtain a parameter-free model that generalises beyond the task of reconstructing data recorded by a specific imaging system for a specific sample type. We demonstrate the generality of the model and the high quality of the obtained reconstructions by application of ML-SIM on raw data obtained for multiple sample types acquired on distinct SIM microscopes. ML-SIM is an end-to-end deep residual neural network that is trained on an auxiliary domain consisting of simulated images, but is transferable to the target task of reconstructing experimental SIM images. By generating the training data to reflect challenging imaging conditions encountered in real systems, ML-SIM becomes robust to noise and irregularities in the illumination patterns of the raw SIM input frames. Since ML-SIM does not require the acquisition of experimental training data, the method can be efficiently adapted to any specific experimental SIM implementation. We compare the reconstruction quality enabled by ML-SIM with current state-of-the-art SIM reconstruction methods and demonstrate advantages in terms of generality and robustness to noise for both simulated and experimental inputs, thus making ML-SIM a useful alternative to traditional methods for challenging imaging conditions. Additionally, reconstruction of a SIM stack is accomplished in less than 200 ms on a modern graphics processing unit, enabling future applications for real-time imaging. Source code and ready-to-use software for the method are available at http://ML-SIM.github.io.

Deep learning based one-shot optically-sectioned structured illumination microscopy for surface measurement.

Optically-sectioned structured illumination microscopy (OS-SIM) is broadly used for biological imaging and engineering surface measurement owing to its simple, low-cost, scanning-free experimental setup and excellent optical sectioning capability. However, the efficiency of current optically-sectioned methods in OS-SIM is yet limited for surface measurement because a set of wide-field images under uniform or structured illumination are needed to derive an optical section at each scanning height. In this paper, a deep-learning-based one-shot optically-sectioned method, called Deep-OS-SIM, is proposed to improve the efficiency of OS-SIM for surface measurement. Specifically, we develop a convolutional neural network (CNN) to learn the statistical invariance of optical sectioning across structured illumination images. By taking full advantage of the high entropy properties of structured illumination images to train the CNN, fast convergence and low training error are achieved in our method even for low-textured surfaces. The well-trained CNN is then applied to a plane mirror for testing, demonstrating the ability of the method to reconstruct high-quality optical sectioning from only one instead of two or three raw structured illumination frames. Further measurement experiments on a standard step and milled surface show that the proposed method has similar accuracy to OS-SIM techniques but with higher imaging speed.

A Tunable Parameter, High Linearity Time-to-Digital Converter Implemented in 28-nm FPGA

This article presents a high linearity time-to-digital converter (TDC) with tunable parameters, based on a field-programmable gate array (FPGA) device. Specifically, a nonuniform monotonic multiphase (NUMMP) method is proposed to reestablish equivalent delay units with nonuniform delay time to overcome the limitation of the intrinsic delay and the uniformity of delay element in FPGA firstly. Then, the raw data frame composed of the equivalent delay units is marked by a time scale marking (TSM) method. Thirdly, a feature extraction (FE) method is proposed to extract the marked raw data frame to form a feature frame, which can reduce the data volume by 79.1%. After that, the feature frames are decoded by a proposed coarse-fine decoding (CFD) algorithm, which can increase the decoding speed, shorten the dead time to 8 clock cycles and reduce the amount of decoding calculation by 80%. Finally, the proposed TDC has been verified with a Xilinx Kintex-7 FPGA. The measurement results demonstrate that the proposed NUMMP TDC with highly adjustable linearity and resolution has been realized. For a high linearity configuration with a LSB of 20 ps, the differential nonlinearity (DNL) and the integral nonlinearity (INL) are only +0.06/-0.05 and +0.08/-0.15 LSB, respectively. And for a high-resolution configuration with a LSB of 1.87 ps, the root mean square (RMS) for the resolution can achieve 2.79 ps, and the values of the DNL and INL are +1.30/-0.54 and +3.51/-2.21 LSB, respectively.

Processing GOTO data with the Rubin Observatory LSST Science Pipelines I: Production of coadded frames

Abstract The past few decades have seen the burgeoning of wide-field, high-cadence surveys, the most formidable of which will be the Legacy Survey of Space and Time (LSST) to be conducted by the Vera C. Rubin Observatory. So new is the field of systematic time-domain survey astronomy; however, that major scientific insights will continue to be obtained using smaller, more flexible systems than the LSST. One such example is the Gravitational-wave Optical Transient Observer (GOTO) whose primary science objective is the optical follow-up of gravitational wave events. The amount and rate of data production by GOTO and other wide-area, high-cadence surveys presents a significant challenge to data processing pipelines which need to operate in near-real time to fully exploit the time domain. In this study, we adapt the Rubin Observatory LSST Science Pipelines to process GOTO data, thereby exploring the feasibility of using this ‘off-the-shelf’ pipeline to process data from other wide-area, high-cadence surveys. In this paper, we describe how we use the LSST Science Pipelines to process raw GOTO frames to ultimately produce calibrated coadded images and photometric source catalogues. After comparing the measured astrometry and photometry to those of matched sources from PanSTARRS DR1, we find that measured source positions are typically accurate to subpixel levels, and that measured L-band photometries are accurate to $\sim50$ mmag at $m_L\sim16$ and $\sim200$ mmag at $m_L\sim18$ . These values compare favourably to those obtained using GOTO’s primary, in-house pipeline, gotophoto, in spite of both pipelines having undergone further development and improvement beyond the implementations used in this study. Finally, we release a generic ‘obs package’ that others can build upon, should they wish to use the LSST Science Pipelines to process data from other facilities.

Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification.

Pain is an unpleasant feeling that has been shown to be an important factor for the recovery of patients. Since this is costly in human resources and difficult to do objectively, there is the need for automatic systems to measure it. In this paper, contrary to current state-of-the-art techniques in pain assessment, which are based on facial features only, we suggest that the performance can be enhanced by feeding the raw frames to deep learning models, outperforming the latest state-of-the-art results while also directly facing the problem of imbalanced data. As a baseline, our approach first uses convolutional neural networks (CNNs) to learn facial features from VGG_Faces, which are then linked to a long short-term memory to exploit the temporal relation between video frames. We further compare the performances of using the so popular schema based on the canonically normalized appearance versus taking into account the whole image. As a result, we outperform current state-of-the-art area under the curve performance in the UNBC-McMaster Shoulder Pain Expression Archive Database. In addition, to evaluate the generalization properties of our proposed methodology on facial motion recognition, we also report competitive results in the Cohn Kanade+ facial expression database.

Multi-Encoder Towards Effective Anomaly Detection in Videos

Given normal training samples, anomaly detection in videos can be regarded as a challenging problem of identifying unexpected events. The state-of-the-art approaches generally resort to the autoencoder model by using a single encoder to capture the motion and content patterns jointly. Nevertheless, due to the lack of accurate labels of normal and abnormal samples, how to detect anomalies is decided by the subjective understanding of models. It infers that different models will prefer to mine different patterns according to the characteristics of models. We call this problem as a pattern bias problem. To alleviate this problem, a novel Multi-Encoder Single-Decoder network, termed as MESDnet, is proposed in the spirit of encoding motion and content cues individually with multiple encoders. MESDnet is of end-to-end learning ability and real-time running speed. Particularly, the differences between adjacent frames and the raw frames are used as the motion and content sources, respectively. Then, a decoder takes charge of detecting anomalies in the way of observing reconstructing error towards the video frames by using the multi-stream encoded motion and content features simultaneously. The experiments on the CUHK Avenue dataset, the UCSD Pedestrian dataset, and the ShanghaiTech Campus dataset verify the effectiveness of MESDnet.

Electron-event representation data enable efficient cryoEM file storage with full preservation of spatial and temporal resolution.

Direct detector device (DDD) cameras have revolutionized electron cryomicroscopy (cryoEM) with their high detective quantum efficiency (DQE) and output of movie data. A high ratio of camera frame rate (frames per second) to camera exposure rate (electrons per pixel per second) allows electron counting, which further improves the DQE and enables the recording of super-resolution information. Movie output also allows the correction of specimen movement and compensation for radiation damage. However, these movies come at the cost of producing large volumes of data. It is common practice to sum groups of successive camera frames to reduce the final frame rate, and therefore the file size, to one suitable for storage and image processing. This reduction in the temporal resolution of the camera requires decisions to be made during data acquisition that may result in the loss of information that could have been advantageous during image analysis. Here, experimental analysis of a new electron-event representation (EER) data format for electron-counting DDD movies is presented, which is enabled by new hardware developed by Thermo Fisher Scientific for their Falcon DDD cameras. This format enables the recording of DDD movies at the raw camera frame rate without sacrificing either spatial or temporal resolution. Experimental data demonstrate that themethod retains super-resolution information and allows the correction of specimen movement at the physical frame rate of the camera while maintaining manageable file sizes. The EER format will enable the development of new methods that can utilize the full spatial and temporal resolution of DDD cameras.

Fast structured illumination microscopy via deep learning

This study shows that convolutional neural networks (CNNs) can be used to improve the performance of structured illumination microscopy to enable it to reconstruct a super-resolution image using three instead of nine raw frames, which is the standard number of frames required to this end. Owing to the isotropy of the fluorescence group, the correlation between the high-frequency information in each direction of the spectrum is obtained by training the CNNs. A high-precision super-resolution image can thus be reconstructed using accurate data from three image frames in one direction. This allows for gentler super-resolution imaging at higher speeds and weakens phototoxicity in the imaging process.

Physics‐based dynamic texture analysis and synthesis model using GPU

The texture is a repetition of a particular structure. Textures classified into static texture and dynamic texture. There are two approaches for synthesising dynamic textures: image-based approach and physics-based approach. In the proposed work, the synthesis of different dynamic textures is shown. The physics-based approach is used to synthesise dynamic texture videos. Different mathematical models are proposed which are suitable to give appropriate motion to the dynamic textures. The raw frames are created, and these frames are further used to synthesise the dynamic textures using physics laws and mathematical formulae. The flexibility of each model is demonstrated. The proposed model has less specificity and less computational complexity. The proposed algorithms are implemented on the graphics processing unit to reduce the overall execution time and time complexity. High-quality videos are produced, and the evaluation of every frame assures the quality.

Molecular Dynamics of Rhodamine 6G Solutions as Revealed by the Computer Processing of Fluorescence Microscopy Images

In this article the image processing of workflow from raw camera frames is detailed. The results of the visualization and quantitative analysis of the images from the rhodamine 6 G solution obtained using total internal reflection fluorescence microscopy are presented. The main stages for image processing of fluorescent molecules are considered. The following actions are described: acquisition of raw data, noise reduction, segmentation of areas of interest by characteristic features, localization of potential fluorophores, postprocessing (assessment of area parameters). Spatial information (time dependence of relative movement of the molecule positions, the concentration distribution depending on the wavelength) is obtained. These results confirm the possibility of applying computer analysis to study the nature of the movement of molecules in a system, their interactions, and dynamic properties of various molecular systems.

Noise-Resilient Reconstruction of Panoramas and 3D Scenes Using Robot-Mounted Unsynchronized Commodity RGB-D Cameras

We present a two-stage approach to first constructing 3D panoramas and then stitching them for noise-resilient reconstruction of large-scale indoor scenes. Our approach requires multiple unsynchronized RGB-D cameras, mounted on a robot platform, which can perform in-place rotations at different locations in a scene. Such cameras rotate on a common (but unknown) axis, which provides a novel perspective for coping with unsynchronized cameras, without requiring sufficient overlap of their Field-of-View (FoV). Based on this key observation, we propose novel algorithms to track these cameras simultaneously. Furthermore, during the integration of raw frames onto an equirectangular panorama, we derive uncertainty estimates from multiple measurements assigned to the same pixels. This enables us to appropriately model the sensing noise and consider its influence, so as to achieve better noise resilience, and improve the geometric quality of each panorama and the accuracy of global inter-panorama registration. We evaluate and demonstrate the performance of our proposed method for enhancing the geometric quality of scene reconstruction from both real-world and synthetic scans.

Relationship Of Mets And Muscle Engagement To Learning Of Yoga Postures

PURPOSE: We are developing an exergame that provides real-time assessment of performance for yoga postures by measuring basic physiologic parameters analyzed to assess yoga skill acquisition as a means to promote healthy physical activity and wellness. METHODS: A convenience sample of 20 adult students in a college yoga course were recorded by a Microsoft Kinect 3D digital camera attached to a PC while following instructions from a yoga instructor. Three yoga sessions scheduled as pre-test, mid-way and a post-test were captured during the regularly scheduled yoga class which met twice weekly for 75 minutes, over a 10-week period. In addition for a positive control, we recorded six yoga instructors performing the same series of five yoga postures as the “gold standard” for training using a machine learning classifier. Scoring of frames were performed by at least two yoga instructors. We examined various statistical functions derived from raw frame scores of false and true positives and negatives. The statistical measure of sensitivity showed consistent increasing trends for Mountain, Forward Bend, and Upward Salute postures. For Mountain, sensitivity went from 0.78 to 0.87, while the expert’s test clips scored 0.94. Which suggests greater training has occurred as the student postures were closer to the yoga instructor’s poses as the “gold standard”. VO2 was measured as METS. RESULTS: We found more difficult or strenuous yoga postures measured higher METS. Repeated Measures analysis of posture learning found significance for majority of yoga postures. We sought to determine if different yoga poses that scored easier to learn based on the linear fit slope of sensitivity correlate with muscle-skeleton complexity of that pose as estimated with total engaged muscle mass. Using literature values of estimated standard muscle masses and identity of muscle engagements in a yoga pose, we ranked yoga poses by total muscle mass engaged. From linear fit slopes, the hardest to learn are Upward Salute and Side Bend poses, while Forward Bend was easiest with greatest slope of learning. CONCLUSIONS: We developed a score based on the muscle-skeleton complexity of that pose from engaged muscle masses. We find that engaged muscle mass relates to the magnitude of VO2 and that the greater the engaged muscle mass the easier the posture is to learn.

A Hybrid Deep Learning Architecture Using 3D CNNs and GRUs for Human Action Recognition

Video contents have variations in temporal and spatial dimensions, and recognizing human actions requires considering the changes in both directions. To this end, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) and their combinations have been used to tackle the video dynamics. However, a hybrid architecture usually results in a more complex model and hence a greater number of parameters to be optimized. In this study, we propose to use a stack of gated recurrent unit (GRU) layers on top of a two-stream inflated convolutional neural network. Raw frames and optical flow of the video are processed in the first and second streams, respectively. We first segment the video frames in order to be able to track the video contents in more details and by using 3D CNNs extract spatial-temporal features, called local features. We then import the sequence of local features to the GRU‌ network, and use a weighted averaging operator to aggregate the outcome of the two processing flows, called global features. The evaluations confirm acceptable results for the two HMDB51 and UCF101 datasets. The proposed method resulted in a 1.6% improvement in the classification accuracy of the HMDB51 challenging dataset compared to the best reported results.