Multiple RGB Research Articles

Multi-person 3D pose estimation from unlabelled data

Its numerous applications make multi-human 3D pose estimation a remarkably impactful area of research. Nevertheless, it presents several challenges, especially when approached using multiple views and regular RGB cameras as the only input. First, each person must be uniquely identified in the different views. Secondly, it must be robust to noise, partial occlusions, and views where a person may not be detected. Thirdly, many pose estimation approaches rely on environment-specific annotated datasets that are frequently prohibitively expensive and/or require specialised hardware. Specifically, this is the first multi-camera, multi-person data-driven approach that does not require an annotated dataset. In this work, we address these three challenges with the help of self-supervised learning. In particular, we present a three-staged pipeline and a rigorous evaluation providing evidence that our approach performs faster than other state-of-the-art algorithms, with comparable accuracy, and most importantly, does not require annotated datasets. The pipeline is composed of a 2D skeleton detection step, followed by a Graph Neural Network to estimate cross-view correspondences of the people in the scenario, and a Multi-Layer Perceptron that transforms the 2D information into 3D pose estimations. Our proposal comprises the last two steps, and it is compatible with any 2D skeleton detector as input. These two models are trained in a self-supervised manner, thus avoiding the need for datasets annotated with 3D ground-truth poses.

Anatomical-Marker-Driven 3D Markerless Human Motion Capture.

Marker-based motion capture (mocap) is a conventional method used in biomechanics research to precisely analyze human movement. However, the time-consuming marker placement process and extensive post-processing limit its wider adoption. Therefore, markerless mocap systems that use deep learning to estimate 2D keypoint from images have emerged as a promising alternative, but annotation errors in training datasets used by deep learning models can affect estimation accuracy. To improve the precision of 2D keypoint annotation, we present a method that uses anatomical landmarks based on marker-based mocap. Specifically, we use multiple RGB cameras synchronized and calibrated with a marker-based mocap system to create a high-quality dataset (RRIS40) of images annotated with surface anatomical landmarks. A deep neural network is then trained to estimate these 2D anatomical landmarks and a ray-distance-based triangulation is used to calculate the 3D marker positions. We conducted extensive evaluations on our RRIS40 test set, which consists of 10 subjects performing various movements. Compared against a marker-based system, our method achieves a mean Euclidean error of 13.23 mm in 3D marker position, which is comparable to the precision of marker placement itself. By learning directly to predict anatomical keypoints from images, our method outperforms OpenCap's augmentation of 3D anatomical landmarks from triangulated wild keypoints. This highlights the potential of facilitating wider integration of markerless mocap into biomechanics research. The RRIS40 test set is made publicly available for research purposes at koonyook.github.io/rris40.

EV-LFV: Synthesizing Light Field Event Streams From an Event Camera and Multiple RGB Cameras.

Light field videos captured in RGB frames (RGB-LFV) can provide users with a 6 degree-of-freedom immersive video experience by capturing dense multi-subview video. Despite its potential benefits, the processing of dense multi-subview video is extremely resource-intensive, which currently limits the frame rate of RGB-LFV (i.e., lower than 30 fps) and results in blurred frames when capturing fast motion. To address this issue, we propose leveraging event cameras, which provide high temporal resolution for capturing fast motion. However, the cost of current event camera models makes it prohibitive to use multiple event cameras for RGB-LFV platforms. Therefore, we propose EV-LFV, an event synthesis framework that generates full multi-subview event-based RGB-LFV with only one event camera and multiple traditional RGB cameras. EV-LFV utilizes spatial-angular convolution, ConvLSTM, and Transformer to model RGB-LFV's angular features, temporal features, and long-range dependency, respectively, to effectively synthesize event streams for RGB-LFV. To train EV-LFV, we construct the first event-to-LFV dataset consisting of 200 RGB-LFV sequences with ground-truth event streams. Experimental results demonstrate that EV-LFV outperforms state-of-the-art event synthesis methods for generating event-based RGB-LFV, effectively alleviating motion blur in the reconstructed RGB-LFV.

Domain embedding transfer for unequal RGB-D image recognition

Most recent unsupervised domain adaptation (UDA) approaches concentrate on single RGB source to single RGB target task. They have to face the real-world scenario, where the source domain can be collected from multiple modalities, e.g., RGB data and depth data. Our work focuses on a more practical and challenging scenario which recognizes RGB images by learning from RGB-D data under the label inequality scenario. We are confronted with three challenges: multiple modalities in the source domain, domain shifting problem and unequal label numbers. To address the aforementioned settings, a novel method, referred to as Domain depth Embedding Transfer (DdET) is proposed, which takes advantage of the depth data in the source domain and handles the domain distribution mismatch under label inequality scenario simultaneously. We conduct comprehensive experiments on five cross domain image classification tasks and observe that DdET can perform favorably against state-of-the-art methods, especially under label inequality scenario.

Multibiometric Images Encryption Method Based on Fast Fourier Transform and Hyperchaos

Biometric images are an important means of personal identity verification and identification and are related to personal privacy and property security. To address the problems of poor security and low image reconstruction quality in the encryption and transmission of multibiometric images, a multibiometric images encryption method based on Fast Fourier Transform (FFT) and hyperchaotic system is proposed. First, the FFT is used to transform the multibiometric images from spatial to frequency domain. Then, the initial values of the hyperchaotic Lorenz system are generated using a one-dimensional chaotic logistic system to generate the key stream. Combined with the recoding rules of matrix reconstruction and scrambling without repetition of using multimatrix, the multiple matrices of amplitude and phase in the transform domain are reconstructed to be multiple RGB three-channel color images by using the inverse fast Fourier transform. Then, we combine the two diffusion methods of additive mode and Galois domain diffusion on each color channel to perform confusion and diffusion. Finally, the multiple grayscale images are encrypted to become multiple color images. The experimental results demonstrate that the method can effectively defend against various attacks. In addition, it solves the problem of low reconstruction accuracy that exists in the field of multiple images security.

SmartMocap: Joint Estimation of Human and Camera Motion Using Uncalibrated RGB Cameras

Markerless human motion capture (mocap) from multiple RGB cameras is a widely studied problem. Existing methods either need calibrated cameras or calibrate them relative to a static camera, which acts as the reference frame for the mocap system. The calibration step has to be done <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> for every capture session, which is a tedious process, and re-calibration is required whenever cameras are intentionally or accidentally moved. In this paper, we propose a mocap method which uses multiple static and moving extrinsically uncalibrated RGB cameras. The key components of our method are as follows. First, since the cameras and the subject can move freely, we select the ground plane as a common reference to represent both the body and the camera motions unlike existing methods which represent bodies in the camera coordinate system. Second, we learn a probability distribution of short human motion sequences ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 1 sec) relative to the ground plane and leverage it to disambiguate between the camera and human motion. Third, we use this distribution as a motion prior in a novel multi-stage optimization approach to fit the SMPL human body model and the camera poses to the human body keypoints on the images. Finally, we show that our method can work on a variety of datasets ranging from aerial cameras to smartphones. It also gives more accurate results compared to the state-of-the-art on the task of monocular human mocap with a static camera. A video demo and our code are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://tinyurl.com/yeykrb67</uri> and <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://tinyurl.com/2p9rme9y</uri> .

VioNets: efficient multi-modal fusion method based on bidirectional gate recurrent unit and cross-attention graph convolutional network for video violence detection

Recent research on video violence detection has made great progress with the development of multi-modal fusion techniques. However, existing approaches still pose huge challenges in real-time violence detection due to fusion features being fixed and not able to be further fine tuned. We aim to address this challenge by exploring the multi-modal fusion method for analyzing multiple modal violent information, i.e., audio, optical flows, and RGB images. We propose a unified network called VioNets, which contains both a cross-attention graph convolutional network (GCN) module and a bidirectional gate recurrent unit (Bi-GRU) module for fusing different modalities of information. First, the cross-attention GCN module is utilized to extract the cross-modal spatial–temporal features. The Bi-GRU module is then applied to accurately capture both past and future context features for each time step of the single-modal features. As a result, the model retains important single-modal information in the extracted features while using the cross-modal features to improve the detection accuracy. Experiments conducted on the XD-Violence dataset show that the proposed method achieves an average precision of 80.59% and an inference time of 0.16 s with 1.82 M parameters.

Design of an automotive platform for computer vision research

The goal of our work is to design an automotive platform for AD/ADAS data acquisition and to demonstrate its application to behavior analysis of vulnerable road users. We present a novel data capture platform mounted on a Mercedes GLC vehicle. The car is equipped with an array of sensors and recording hardware including multiple RGB cameras, Lidar, GPS and IMU. For subsequent research on human behavior analysis in traffic scenes, we have conducted two kinds of data recordings. Firstly, we have designed a range of artificial test cases which we recorded on a safety regulated proving ground with stunt persons to capture rare events in traffic scenes in a predictable and structured way. Secondly, we have recorded data on public streets of Vienna, Austria, showing unconstrained pedestrian behavior in an urban setting, while also considering European General Data Protection Regulation (GDPR) requirements. We describe the overall framework including data acquisition and ground truth annotation, and demonstrate its applicability for the implementation and evaluation of selected deep learning models for pedestrian behavior prediction.

Vision-Based Autonomous Landing on Unprepared Field With Rugged Surface

Landing on unprepared terrains is a challenging task for fully autonomous micro unmanned aerial vehicles (UAVs). Most of the existing methods mainly rely on manual control when the terrain of the target area is unknown in advance. In this article, we propose an autonomous landing method based on a learning-based multi-view stereo (MVS) system. UAV acquires multiple RGB pictures of the terrain after cruise, and then extracts speed-up robust features (SURF) and perform structure from motion (SFM) to obtain sparse feature point clouds. By utilizing the generated initial depth map, we further propose a novel 3-D reconstruction algorithm named PatchmatchNet-A, which can help to obtain precise and stable estimates of the depth information. In PatchmatchNet-A, we also use a new activation function, adjustable-arctangent linear units (ALU), to improve the accuracy and robustness. We have tested our algorithms in a DTU dataset and found that it can speed up the processing by around 25% while guaranteeing competitive performance. Flight experiments have also been conducted to verify the effectiveness of the whole landing system by using a commercial M210V2 quadcopter.

다중 시점 RGB 영상의 심층 합성곱 신경망 기반 3 차원 형상 추정 방법

In this paper, we describe point cloud generation method from multiple RGB images based on convolutional neural network. The proposed method is motivated from human’s 3d understanding from orthographic sketches, which infer spatial relationship from front, top, side view of the 3d object. The network of the proposed method employed generative model, to make it predict point clouds of atypical objects. Auto encoder network is utilized to encode three RGB images into latent vectors, and generate point clouds. Loss functions are defined which measure reconstruction performance and uniformity of the point clouds to make the network generate point clouds similar to the original and distribute uniformly along entire region. From this structure, we expected the network to learn spatial relation of the original 3d model from multiple RGB images. The result shows that the proposed method can predict overall shape of the objects, and it is hard to express detailed geometry. As a further work, network structure improvement to generate detailed shape of the object, predict occluded region will be performed.

Enhanced Perception for Autonomous Driving Using Semantic and Geometric Data Fusion.

Environment perception remains one of the key tasks in autonomous driving for which solutions have yet to reach maturity. Multi-modal approaches benefit from the complementary physical properties specific to each sensor technology used, boosting overall performance. The added complexity brought on by data fusion processes is not trivial to solve, with design decisions heavily influencing the balance between quality and latency of the results. In this paper we present our novel real-time, 360 enhanced perception component based on low-level fusion between geometry provided by the LiDAR-based 3D point clouds and semantic scene information obtained from multiple RGB cameras, of multiple types. This multi-modal, multi-sensor scheme enables better range coverage, improved detection and classification quality with increased robustness. Semantic, instance and panoptic segmentations of 2D data are computed using efficient deep-learning-based algorithms, while 3D point clouds are segmented using a fast, traditional voxel-based solution. Finally, the fusion obtained through point-to-image projection yields a semantically enhanced 3D point cloud that allows enhanced perception through 3D detection refinement and 3D object classification. The planning and control systems of the vehicle receives the individual sensors’ perception together with the enhanced one, as well as the semantically enhanced 3D points. The developed perception solutions are successfully integrated onto an autonomous vehicle software stack, as part of the UP-Drive project.

Accurate Power Modelling Framework for Medical Images in Embedded System

The display in embedded devices is a significant energy-consuming device, and the display content determines the degree of energy consumption. In the practice of energy saving and emission reduction, it is necessary to build an accurate medical image power prediction model for the display. Current power consumption prediction models for medical images often focus on the performance of a single prediction model, ignoring the ability of multiple single prediction models to improve performance. This paper proposes an accurate medical power framework (AMPF), which consists of three steps. In the first step, the strong predictor is used to compose multiple RGB single channels to obtain the power model. In the second step, the power between RGB channels is represented by errors, and the power generated by RGB channel dependence is obtained by the strong predictor method. The third step is to compose the power of the first step and the second step to get accurate medical image power prediction results. The experimental results show that the accuracy of the AMFP proposed in this paper reaches 0.9798, which is 6.836% higher than that of the single power model. The AMPF implemented by AdaBoost is superior to that implemented by EWM in performance and has about 1.3 times the time advantage in training.

Transfer learning from citizen science photographs enables plant species identification in UAV imagery

Accurate information on the spatial distribution of plant species and communities is in high demand for various fields of application, such as nature conservation, forestry, and agriculture. A series of studies has shown that Convolutional Neural Networks (CNNs) accurately predict plant species and communities in high-resolution remote sensing data, in particular with data at the centimeter scale acquired with Unoccupied Aerial Vehicles (UAV). However, such tasks often require ample training data, which is commonly generated in the field via geocoded in-situ observations or labeling remote sensing data through visual interpretation. Both approaches are laborious and can present a critical bottleneck for CNN applications. An alternative source of training data is given by using knowledge on the appearance of plants in the form of plant photographs from citizen science projects such as the iNaturalist database. Such crowd-sourced plant photographs typically exhibit very different perspectives and great heterogeneity in various aspects, yet the sheer volume of data could reveal great potential for application to bird’s eye views from remote sensing platforms. Here, we explore the potential of transfer learning from such a crowd-sourced data treasure to the remote sensing context. Therefore, we investigate firstly, if we can use crowd-sourced plant photographs for CNN training and subsequent mapping of plant species in high-resolution remote sensing imagery. Secondly, we test if the predictive performance can be increased by a priori selecting photographs that share a more similar perspective to the remote sensing data. We used two case studies to test our proposed approach with multiple RGB orthoimages acquired from UAV with the target plant species Fallopia japonica and Portulacaria afra respectively. Our results demonstrate that CNN models trained with heterogeneous, crowd-sourced plant photographs can indeed predict the target species in UAV orthoimages with surprising accuracy. Filtering the crowd-sourced photographs used for training by acquisition properties increased the predictive performance. This study demonstrates that citizen science data can effectively anticipate a common bottleneck for vegetation assessments and provides an example on how we can effectively harness the ever-increasing availability of crowd-sourced and big data for remote sensing applications.

CamSense: A camera-based contact-less heart activity monitoring

Remote Photoplythysmograpy (rPPG) systems enable contactless heart activities (heart rate, heart rate variability) monitoring by estimating Photoplethysmogram (PPG) signal, blood’s volumetric variation in the skin tissues, leveraging the occurred diffused reflection from the exposed skin in skin video. They can primarily monitor heart activities using off-the-shelf video sensors while ensuring the safety of concerned individuals during contagious diseases. However, developing the rPPG systems is challenging due to the marginal presence of PPG signal in the video stream, data variations, limited and noisy rPPG data. In this regard, we propose an end-to-end deep learning-based approach for camera-based contactless sensing CamSense for recovering PPG signals from consecutive raw video frames. Firstly, we design and validate a personalized model to bypass data variation and noise across data collection with a modified objective function under realistic settings. Secondly, we explore and design multi-task learning (MTL) network to address the rPPG data variabilities for learning generalized rPPG representations. We also propose a transfer learning approach that integrates an efficient weight initialization to scale the rPPG systems under different domains and settings for fast and generalized training and inferences. Finally, we offer a new dataset, MPSC-rPPG dataset, containing multiple RGB videos and corresponding PPG ground truth for end–end rPPG network training. We evaluate CamSense on two public datasets and our MPSC-rPPG dataset across multiple subjects and heterogeneous camera sensors such as DSLR and near-infrared sensors with different ground truth provider PPG sensors (wrist, finger) to showcase its’ generalizability. We further validate our components’ design choices by performing ablation studies using different settings. Our developed model approximates accurate PPG signals with an average root mean square error (RMSE) of 0.08, 0.10, and 0.06 for personalized models, MTL model, and transfer learnings on the held-out test videos.

Multi-Modality Self-Distillation for Weakly Supervised Temporal Action Localization.

As a challenging task of high-level video understanding, Weakly-supervised Temporal Action Localization (WTAL) has attracted increasing attention in recent years. However, due to the weak supervisions of whole-video classification labels, it is challenging to accurately determine action instance boundaries. To address this issue, pseudo-label-based methods [Alwassel et al. (2019), Luo et al. (2020), and Zhai et al. (2020)] were proposed to generate snippet-level pseudo labels from classification results. In spite of the promising performance, these methods hardly take full advantages of multiple modalities, i.e., RGB and optical flow sequences, to generate high quality pseudo labels. Most of them ignored how to mitigate the label noise, which hinders the capability of the network on learning discriminative feature representations. To address these challenges, we propose a Multi-Modality Self-Distillation (MMSD) framework, which contains two single-modal streams and a fused-modal stream to perform multi-modality knowledge distillation and multi-modality self-voting. On the one hand, multi-modality knowledge distillation improves snippet-level classification performance by transferring knowledge between single-modal streams and a fused-modal stream. On the other hand, multi-modality self-voting mitigates the label noise in a modality voting manner according to the reliability and complementarity of the streams. Experimental results on THUMOS14 and ActivityNet1.3 datasets demonstrate the effectiveness of our method and superior performance over state-of-the-art approaches. Our code is available at https://github.com/LeonHLJ/MMSD.

High-quality Voxel Reconstruction from Stereoscopic Images

Volumetric reconstruction from one or multiple RGB images has shown significant advances in recent years, but the approaches used so far do not take advantage of stereoscopic features such as distance blur, perspective disparity, textures, etc. that are useful to shape the object volumes. Our study is to evaluate a convolutional neural network architecture for reconstruction of 128³ voxel models from 960 pairs of stereoscopic images. The preliminary results show an 80% of coincidence with the original models in 2 categories using the Intersection over Union metric. These results indicate that good reconstructions can be made from a small dataset. This will reduce the time and memory usage for this task.

Multiple RGB images encryption algorithm based on elliptic curve, improved Diffie Hellman protocol

Multiple RGB images encryption algorithm based on elliptic curve, improved Diffie Hellman protocol

Fast calculation method based on hidden region continuity for computer-generated holograms with multiple cameras in a sports scene.

We propose, to the best of our knowledge, the world's first system capable of fast calculating computer-generated holograms (CGHs) from a large-scale outdoor sports scene captured with multiple RGB cameras. In the system, we introduce a fast calculation method focusing on hidden region continuity (HRC) that frequently appears in a point cloud of a 3D sports scene generated from free-viewpoint video technology. The experimental results show that the calculation time of the proposed HRC method is five to 10 times faster than that of the point-based method, which is one of the common CGH calculation methods.

Security of multiple RGB images in the time domain and frequency domain

Secure and efficient transmission of images from a sender to a receiver through the public channel has become a prime agenda. An encryption algorithm to secure three color images simultaneously using matrix affine cipher, factional discrete cosine transform and Arnold transform is proposed in this paper. In earlier proposed algorithms, pixel intensities were not changed but rather shifted. In our proposed algorithm, pixel intensities are both shifted and changed. The color maps of the three color images are extracted to convert three color images into three indexed images respectively. The three indexed images are taken as R(red), G(green) and B(blue) channels of a color image. The color image is encrypted using the proposed encryption algorithm. The encryption algorithms for the security of color images proposed so far are secure either in time domain or in frequency domain, but the proposed encryption algorithm is secure in both the time and the frequency domain without any loss of data. The security of the proposed encryption algorithm depends not only upon secret keys but also upon the correct arrangement of the secret keys as compared to existing similar encryption algorithms whose security depends only upon secret keys. The encrypted(cipher) image is a single color image and is favorable for secure and efficient transmission over the public channel. Simulation is done with standard examples to analyze the validity of the proposed encryption scheme. We have provided the complete security analysis, statistical analysis and comparison with the existing similar encryption algorithms.

Towards Better Visualisation of Alpine Quaternary Landform Features on High-Resolution Digital Elevation Models

Alpine topography is formed by a complex series of geomorphological processes that result in a vast number of different landforms. The youngest and most diverse landforms are various Quaternary sedimentary bodies, each characterised by its unique landform features. The formation of Quaternary sedimentary bodies and their features derive from the dominant building sedimentary processes. In recent years, studies of Quaternary sedimentary bodies and processes have been greatly aided by the use of digital elevation models (DEMs) derived by airborne laser scanning (ALS). High-resolution DEMs allow detailed mapping of sedimentary bodies, detection of surface changes, and recognition of the building sedimentary processes. DEMs are often displayed as hillshaded reliefs, the most common visualisation technique, which suffers from the limitation of a single illumination source. As a result, features can be barely visible or even invisible to the viewer if they are parallel to the light source or hidden in the shadow. These limitations become challenging when representing landforms and subtle landscape features in a diverse alpine topography. In this study, we focus on eleven visualisations of Quaternary sedimentary bodies and their sedimentary and morphological features on a 0.5 m resolution DEM. We qualitatively compare analytical hillshading with a set of visualisation techniques contained in the Raster Visualisation Toolbox software, primarily hillshading from multiple directions RGB, 8-bit sky view factor and 8-bit slope. The aim is to determine which visualisation technique is best suited for visual recognition of sedimentary bodies and sedimentation processes in complex alpine landscapes. Detailed visual examination of previously documented Pleistocene moraine and lacustrine deposits, Holocene alluvial fans, scree deposits, debris flow and fluvial deposits on the created visualisations revealed several small-scale morphological and sedimentary features that were previously difficult or impossible to detect on analytical hillshading and aerial photographs. Hillshading from multiple directions resulted in a visualisation that could be universally applied across the mountainous and hilly terrains. In contrast, 8-bit sky view factor and 8-bit slope visualisations created better visibility and facilitated interpretation of subtle and small-scale (less than ten metres) sedimentary and morphological features.