Camera Viewpoint Research Articles

Manually Acquiring Targets From Multiple Viewpoints Using Video Feedback.

The effect of camera viewpoint was studied when performing visually obstructed psychomotor targeting tasks. Previous research in laparoscopy and robotic teleoperation found that complex perceptual-motor adaptations associated with misaligned viewpoints corresponded to degraded performance in manipulation. Because optimal camera positioning is often unavailable in restricted environments, alternative viewpoints that might mitigate performance effects are not obvious. A virtual keyboard-controlled targeting task was remotely distributed to workers of Amazon Mechanical Turk. The experiment was performed by 192 subjects for a static viewpoint with independent parameters of target direction, Fitts' law index of difficulty, viewpoint azimuthal angle (AA), and viewpoint polar angle (PA). A dynamic viewpoint experiment was also performed by 112 subjects in which the viewpoint AA changed after every trial. AA and target direction had significant effects on performance for the static viewpoint experiment. Movement time and travel distance increased while AA increased until there was a discrete improvement in performance for 180°. Increasing AA from 225° to 315° linearly decreased movement time and distance. There were significant main effects of current AA and magnitude of transition for the dynamic viewpoint experiment. Orthogonal direction and no-change viewpoint transitions least affected performance. Viewpoint selection should aim to minimize associated rotations within the manipulation plane when performing targeting tasks whether implementing a static or dynamic viewing solution. Because PA rotations had negligible performance effects, PA adjustments may extend the space of viable viewpoints. These results can inform viewpoint selection for visual feedback during psychomotor tasks.

An Episodic Learning Network for Text Detection on Human Bodies in Sports Images

Due to the proliferation of sports-related multimedia content on the WWW, effective visual search and retrieval present interesting research challenges. These are caused by poor image quality, a wide range of possible camera points of view, pose variations on the part of athletes engaged in playing a sport, deformations of text appearing on sports person’s clothing and uniforms in motion, occlusions caused by other objects, etc. To address these challenges, this paper presents a new method for detecting text on human bodies in sports images. Unlike most existing methods, which attempt to exploit locations of a player’s torso, face, and skin, we propose an end-to-end episodic learning approach that employs inductive learning criteria for detecting clothing regions in an image, which are, in turn, then used for text detection. Our method integrates a Residual Network (ResNet) and Pyramidal Pooling Module (PPM) for generating a spatial attention map. The Progressive Scalable Expansion Algorithm (PSE) is adapted for text detection from these regions. Experimental results on our own dataset as well as several benchmarks (like RBNR and MMM which contain images of runners in marathons, and Re-ID which is a person re-identification dataset) demonstrate that the proposed method outperforms existing methods in terms of precision and F1-score. We also present results for sports images chosen from natural scene text detection datasets such as CTW1500 and MS-COCO to show the proposed method is effective and reliable across a range of inputs.

Vision-Only Robot Navigation in a Neural Radiance World

Neural Radiance Fields (NeRFs) have recently emerged as a powerful paradigm for the representation of natural, complex 3D scenes. Neural Radiance Fields (NeRFs) represent continuous volumetric density and RGB values in a neural network, and generate photo-realistic images from unseen camera viewpoints through ray tracing. We propose an algorithm for navigating a robot through a 3D environment represented as a NeRF using only an onboard RGB camera for localization. We assume the NeRF for the scene has been pre-trained offline, and the robot’s objective is to navigate through unoccupied space in the NeRF to reach a goal pose. We introduce a trajectory optimization algorithm that avoids collisions with high-density regions in the NeRF based on a discrete time version of differential flatness that is amenable to constraining the robot’s full pose and control inputs. We also introduce an optimization based filtering method to estimate 6DoF pose and velocities for the robot in the NeRF given only an onboard RGB camera. We combine the trajectory planner with the pose filter in an online replanning loop to give a vision-based robot navigation pipeline. We present simulation results with a quadrotor robot navigating through a jungle gym environment, the inside of a church, and Stonehenge using only an RGB camera. We also demonstrate an omnidirectional ground robot navigating through the church, requiring it to reorient to fit through a narrow gap.

GKNet: Grasp keypoint network for grasp candidates detection

Contemporary grasp detection approaches employ deep learning to achieve robustness to sensor and object model uncertainty. The two dominant approaches design either grasp-quality scoring or anchor-based grasp recognition networks. This paper presents a different approach to grasp detection by treating it as keypoint detection in image-space. The deep network detects each grasp candidate as a pair of keypoints, convertible to the grasp representation g = {x,y,w,θ} T , rather than a triplet or quartet of corner points. Decreasing the detection difficulty by grouping keypoints into pairs boosts performance. To promote capturing dependencies between keypoints, a non-local module is incorporated into the network design. A final filtering strategy based on discrete and continuous orientation prediction removes false correspondences and further improves grasp detection performance. GKNet, the approach presented here, achieves a good balance between accuracy and speed on the Cornell and the abridged Jacquard datasets (96.9% and 98.39% at 41.67 and 23.26 fps). Follow-up experiments on a manipulator evaluate GKNet using four types of grasping experiments reflecting different nuisance sources: static grasping, dynamic grasping, grasping at varied camera angles, and bin picking. GKNet outperforms reference baselines in static and dynamic grasping experiments while showing robustness to varied camera viewpoints and moderate clutter. The results confirm the hypothesis that grasp keypoints are an effective output representation for deep grasp networks that provide robustness to expected nuisance factors.

Pre-Training Without Natural Images

Is it possible to use convolutional neural networks pre-trained without any natural images to assist natural image understanding? The paper proposes a novel concept, Formula-driven Supervised Learning (FDSL). We automatically generate image patterns and their category labels by assigning fractals, which are based on a natural law. Theoretically, the use of automatically generated images instead of natural images in the pre-training phase allows us to generate an infinitely large dataset of labeled images. The proposed framework is similar yet different from Self-Supervised Learning because the FDSL framework enables the creation of image patterns based on any mathematical formulas in addition to self-generated labels. Further, unlike pre-training with a synthetic image dataset, a dataset under the framework of FDSL is not required to define object categories, surface texture, lighting conditions, and camera viewpoint. In the experimental section, we find a better dataset configuration through an exploratory study, e.g., increase of #category/#instance, patch rendering, image coloring, and training epoch. Although models pre-trained with the proposed Fractal DataBase (FractalDB), a database without natural images, do not necessarily outperform models pre-trained with human annotated datasets in all settings, we are able to partially surpass the accuracy of ImageNet/Places pre-trained models. The FractalDB pre-trained CNN also outperforms other pre-trained models on auto-generated datasets based on FDSL such as Bezier curves and Perlin noise. This is reasonable since natural objects and scenes existing around us are constructed according to fractal geometry. Image representation with the proposed FractalDB captures a unique feature in the visualization of convolutional layers and attentions.

Resident Education on the Surgical Treatment of Distal Radius Fractures Using a Point-of-View Camera.

In the coronavirus disease 2019 (COVID-19) era, surgical resident education depends largely on virtual materials. With the help of point-of-view (POV) cameras, educational videos have become widely used for surgical training. A video recorded from the surgeon's POV helps demonstrate the procedure. We made training movies of the surgical approach to distal radius fractures for residents using a head-mounted video recording system with a laser point targeting device (LPTD). A 15-minnute movie of the trans-flexor carpi radialis approach for distal radius fractures was made. A POV camera was assembled with an LPTD and strapped on the surgeon's head. This enabled maintenance of the surgical field while recording the procedure. A shorter version of the clip was also made to investigate trainee preference. We asked 24 trainees to watch the two versions of the video and complete a short questionnaire. All trainees felt that the movie made with a POV camera was more efficient than existing materials. Only 1 (4.2%) felt that the laser pointer hindered the view. Four of the 23 trainees (16.7%) felt dizzy while watching the video. Of the two versions, 16 trainees (66.7%) preferred the shorter, edited version. The average score for the video was 8.42 out of 10. A video recording system in the operating room that uses an LPTD-POV camera is an efficient way to produce educational material, particularly for surgical residents during the COVID-19 era.

Automated Coverage Path Planning for Inspection of Panels Using Camera Viewpoints

Automated Coverage Path Planning for Inspection of Panels Using Camera Viewpoints

Progressive Transfer Learning.

Model fine-tuning is a widely used transfer learning approach in person Re-identification (ReID) applications, which fine-tuning a pre-trained feature extraction model into the target scenario instead of training a model from scratch. It is challenging due to the significant variations inside the target scenario, e.g., different camera viewpoint, illumination changes, and occlusion. These variations result in a gap between each mini-batch's distribution and the whole dataset's distribution when using mini-batch training. In this paper, we study model fine-tuning from the perspective of the aggregation and utilization of the dataset's global information when using mini-batch training. Specifically, we introduce a novel network structure called Batch-related Convolutional Cell (BConv-Cell), which progressively collects the dataset's global information into a latent state and uses it to rectify the extracted feature. Based on BConv-Cells, we further proposed the Progressive Transfer Learning (PTL) method to facilitate the model fine-tuning process by jointly optimizing BConv-Cells and the pre-trained ReID model. Empirical experiments show that our proposal can greatly improve the ReID model's performance on MSMT17, Market-1501, CUHK03, and DukeMTMC-reID datasets. Moreover, we extend our proposal to the general image classification task. The experiments in several image classification benchmark datasets demonstrate that our proposal can significantly improve baseline models' performance. The code has been released at https://github.com/ZJULearning/PTL.

Person Re-Identification With Hierarchical Discriminative Spatial Aggregation

Practically, person re-identification (re-ID) may suffer from the critical spatial misalignment problem due to inaccurate human detection, variation on human pose and camera viewpoint, etc. To address this, a hierarchical discriminative spatial aggregation method is proposed. The key idea is to conduct spatial aggregation on local human parts via global average-pooling to acquire the strong spatial misalignment tolerance, with VALD encoding on the local parts for facilitating discriminative power jointly. This proposition is built on NetVLAD to ensure end-to-end deep learning capacity. Due to the fine-grained property of person re-ID task that has not been well concerned by the original NetVLAD model for scene recognition, a feature refinement layer that consists of 1 fully-connected (FC) layer and 2 batch normalization (BN) layers is added on top of the raw NetVLAD layer to enhance the discriminative power and training convergence. And, a human body occlusion and background component dropout manner is also proposed to resist the effect of serious occlusion. Technically, a refined codeword initialization manner is proposed to alleviate the potential codeword imbalance problem caused by naive random initialization. The proposed discriminative spatial aggregation approach is then conducted on multi-resolution convolutional feature map layers hierarchically via early feature fusion, to involve richer semantic and fine-grained visual clues jointly. Wide-range experiments on 6 datasets (i.e., CUHK03, DukeMTMC-reID, Occluded-DukeMTMC, Market-1501, MSMT17 and Occluded-REID) verifies the effectiveness of our proposition. The source code and supporting material is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zmyme/HDSA-reID</uri> .

LasHeR: A Large-Scale High-Diversity Benchmark for RGBT Tracking.

RGBT tracking receives a surge of interest in the computer vision community, but this research field lacks a large-scale and high-diversity benchmark dataset, which is essential for both the training of deep RGBT trackers and the comprehensive evaluation of RGBT tracking methods. To this end, we present a La rge- s cale H igh-diversity [Formula: see text]nchmark for short-term R GBT tracking (LasHeR) in this work. LasHeR consists of 1224 visible and thermal infrared video pairs with more than 730K frame pairs in total. Each frame pair is spatially aligned and manually annotated with a bounding box, making the dataset well and densely annotated. LasHeR is highly diverse capturing from a broad range of object categories, camera viewpoints, scene complexities and environmental factors across seasons, weathers, day and night. We conduct a comprehensive performance evaluation of 12 RGBT tracking algorithms on the LasHeR dataset and present detailed analysis. In addition, we release the unaligned version of LasHeR to attract the research interest for alignment-free RGBT tracking, which is a more practical task in real-world applications. The datasets and evaluation protocols are available at: https://github.com/mmic-lcl/Datasets-and-benchmark-code.

A Study on the Accuracy of Estimated Viewpoint by the Camera Calibration with 3D Point Clouds

In this study, I used a simulation to investigate the effect of the quantity and the position of correspondence points between 2D and 3D on the accuracy of camera calibration. The 3D points for simulation of the camera calibration were measured by a 3D laser scanner and were added variability (mean = 0 mm, SD = 10 mm). The 2D points for simulation of the camera calibration were calculated projecting the 3D points onto a 2D image with the camera parameters, and were added variability (mean = 0 pixel, SD = 0 ~ 4 pixel). In the case of the relatively accurate camera calibration (54 pair points, RMSE = 1.8), the simulation results showed that the camera calibration were stable with respect to the 2D points variability. And, the camera viewpoint error was estimated to be within 100 mm. In the case of the inaccurate camera calibration (36 pair points, RMSE = 2.7), the simulation results showed that the camera calibration were unstable with respect to 2D points variability. And, the camera viewpoint error was estimated to be 100mm or more. Therefore, by running the simulation after calibrating the camera, we were able to estimate the stability of calculation results and statistical evaluation of the camera parameters.

Unreliability and Point of View in Filmic Narration

Novels like Fight Club or American Psycho are said to be instances of unreliable narration: the first person narrator presents an evidently distorted picture of the fictional world. The film adaptations of these novels are likewise said to involve unreliable narration. I resist this extension of the term ‘unreliable narration’ to film. My argument for this rests on the observation that unreliable narration requires a personal narrator while film typically involves an impersonal narrator (corresponding to the camera viewpoint). The kind of ambiguous story-telling that we find in literary fiction with unreliable narrators, where for certain descriptions it is unclear whether what we’re told is an accurate account of what’s happening in the story world or not, can instead be achieved by conventionalized filmmaking techniques for reporting the contents of mental states, like the point of view shot, but especially the more ambiguous blended perspective shot.

Multi‐label based view learning for vehicle re‐identification

Due to the high similarity of different vehicles with similar appearances and the great diversity of camera viewpoints, vehicle re-identification (ReID) is still a challenging task. It commonly maps query set image into a high dimensional embedding space and then retrieve gallery vehicle images according to the distance. In this letter, a multi-label based view learning (MLVL) model to enhance the distinguishability of intra-class and inter-class through multiple labels learning, and decrease the distance of intra-class features caused by multi-view appearances of vehicles. Specifically, the model includes two main parts. First, the vehicle orientation estimation module is responsible for detecting key-points of the vehicle and predicting its orientation, which provides the orientation label for the multi-label based learning network to enhance view-invariant representation. Second, a novel dual-triplet and quadruplet loss function is designed to optimise intra-class and inter-class distance with the help of multi-label information, i.e. vehicle colour, type and orientation. In particular, a multi-label based sampler is proposed to generate training mini-batches instead of random sampler. Extensive experiments results show that the proposed MLVL model achieves 5.6 % $\%$ mAP improvement on VeRi-776 and 1.9 % $\%$ on VERI-Wild datasets compared with baseline model.

Performance analysis of real-time plant species recognition using bilateral network combined with machine learning classifier

A real-time plant species recognition under an unconstrained environment is a challenging and time-consuming process. The recognition model should cope up with the computer vision challenges such as scale variations, illumination changes, camera viewpoint or object orientation changes, cluttered backgrounds and structure of leaf (simple or compound). In this paper, a bilateral convolutional neural network (CNN) with machine learning classifiers are investigated in relation to the real-time implementation of plant species recognition. The CNN models considered are MobileNet, Xception and DenseNet-121. In the bilateral CNNs (Homogeneous/Heterogeneous type), the models are connected using the cascade early fusion strategy. The Bilateral CNN is used in the process of feature extraction. Then, the extracted features are classified using different machine learning classifiers such as Linear Discriminant Analysis (LDA), multinomial Logistic Regression (MLR), Naïve Bayes (NB), k-Nearest Neighbor (k−NN), Classification and Regression Tree (CART), Random Forest Classifier (RF), Bagging Classifier (BC), Multi-Layer Perceptron (MLP) and Support Vector Machine (SVM). From the experimental investigation, it is observed that the multinomial Logistic Regression classifier performed better compared to other classifiers, irrespective of the bilateral CNN models (Homogeneous - MoMoNet, XXNet, DeDeNet; Heterogeneous - MoXNet, XDeNet, MoDeNet). It is also observed that the MoDeNet + MLR model attained the state-of-the-art results (Flavia: 98.71%, Folio: 96.38%, Swedish Leaf: 99.41%, custom created Leaf-12: 99.39%), irrespective of the dataset. The number of misprediction/class is highly reduced by utilizing the MoDeNet + MLR model for real-time plant species recognition.

Real-Time Closed-Loop Detection Method of vSLAM Based on a Dynamic Siamese Network.

As visual simultaneous localization and mapping (vSLAM) is easy disturbed by the changes of camera viewpoint and scene appearance when building a globally consistent map, the robustness and real-time performance of key frame image selections cannot meet the requirements. To solve this problem, a real-time closed-loop detection method based on a dynamic Siamese networks is proposed in this paper. First, a dynamic Siamese network-based fast conversion learning model is constructed to handle the impact of external changes on key frame judgments, and an elementwise convergence strategy is adopted to ensure the accurate positioning of key frames in the closed-loop judgment process. Second, a joint training strategy is designed to ensure the model parameters can be learned offline in parallel from tagged video sequences, which can effectively improve the speed of closed-loop detection. Finally, the proposed method is applied experimentally to three typical closed-loop detection scenario datasets and the experimental results demonstrate the effectiveness and robustness of the proposed method under the interference of complex scenes.

Perspective Distortion Correction for Multi-Modal Registration between Ultra-Widefield and Narrow-Angle Retinal Images.

Multi-modal retinal image registration between 2D Ultra-Widefield (UWF) and narrow-angle (NA) images has not been well-studied, since most existing methods mainly focus on NA image alignment. The stereographic projection model used in UWF imaging causes strong distortions in peripheral areas, which leads to inferior alignment quality. We propose a distortion correction method that remaps the UWF images based on estimated camera view points of NA images. In addition, we set up a CNN-based registration pipeline for UWF and NA images, which consists of the distortion correction method and three networks for vessel segmentation, feature detection and matching, and outlier rejection. Experimental results on our collected dataset shows the effectiveness of the proposed pipeline and the distortion correction method.

Shape transformer nets: Generating viewpoint-invariant 3D shapes from a single image

Single-view 3D shapes generation has achieved great success in recent years. However, current methods always blind the learning of shapes and viewpoints. The generated shape only fit the observed viewpoints and would not be optimal from unknown viewpoints. In this paper, we propose a novel encoder–decoder based network which contains a disentangled transformer to generate the viewpoint-invariant 3D shapes. The differentiable and parametric Non-uniform B-spline (NURBS) surface generation and 3D-to-3D viewpoint transformation are incorporated to learn the viewpoint-invariant shape and the camera viewpoint, respectively. Our new framework allows us to learn the latent geometric parameters of shapes and viewpoints without knowing the ground truth viewpoint. That can simultaneously generate camera-viewpoint and viewpoint-invariant 3D shapes of the object. We analyze the effects of disentanglement and show both quantitative and qualitative results of shapes generated at various unknown viewpoints.

SchiNet: Automatic Estimation of Symptoms of Schizophrenia from Facial Behaviour Analysis

Patients with schizophrenia often display impairments in the expression of emotion and speech and those are observed in their facial behaviour. Automatic analysis of patients’ facial expressions that is aimed at estimating symptoms of schizophrenia has received attention recently. However, the datasets that are typically used for training and evaluating the developed methods, contain only a small number of patients (4-34) and are recorded while the subjects were performing controlled tasks such as listening to life vignettes, or answering emotional questions. In this paper, we use videos of professional-patient interviews, in which symptoms were assessed in a standardised way as they should/may be assessed in practice, and which were recorded in realistic conditions (i.e., varying illumination levels and camera viewpoints) at the patients’ homes or at mental health services. We automatically analyse the facial behaviour of 91 out-patients – this is almost 3 times the number of patients in other studies – and propose SchiNet, a novel neural network architecture that estimates expression-related symptoms in two different assessment interviews. We evaluate the proposed SchiNet for patient-independent prediction of symptoms of schizophrenia. Experimental results show that some automatically detected facial expressions are significantly correlated to symptoms of schizophrenia, and that the proposed network for estimating symptom severity delivers promising results.

Internal and external spatial–temporal constraints for person reidentification

Spatial–temporal information is easy to achieve in a practical surveillance scene, but it is often neglected in most current person reidentification (ReID) methods. Employing spatial–temporal information as a constrain has been verified as beneficial for ReID. However, there is no effective modeling according to the pedestrian movement law. In this paper, we present a ReID framework with internal and external spatial–temporal constraints, termed as IESC-ReID. A novel residual spatial attention module is proposed to build a spatial–temporal constraint and increase the robustness to partial occlusions or camera viewpoint changes. A Laplace-based spatial–temporal constraint is also introduced to eliminate irrelevant gallery images, which are gathered by the internal learning network. IESC-ReID constrains the attention within the functioning range of the channel space, and utilizes additional spatial–temporal constrains to further constrain results. Intensive experiments show that these constraints consistently improve the performance. Extensive experimental results on numerous publicly available datasets show that the proposed method outperforms several state-of-the-art ReID algorithms. Our code is publicly available at https://github.com/jiaming-wang/IESC.

AttributeNet: Attribute enhanced vehicle re-identification

Vehicle Re-Identification (V-ReID) is a critical task that associates the same vehicle across images from different camera viewpoints. Many works explore attribute clues to enhance V-ReID; however, there is usually a lack of effective interaction between the attribute-related modules and final V-ReID objective. In this work, we propose a new method to efficiently explore discriminative information from vehicle attributes (for instance, color and type). We introduce AttributeNet (ANet) that jointly extracts identity-relevant features and attribute features. We enable the interaction by distilling the ReID-helpful attribute feature and adding it into the general ReID feature to increase the discrimination power. Moreover, we propose a constraint, named Amelioration Constraint (AC), which encourages the feature after adding attribute features onto the general ReID feature to be more discriminative than the original general ReID feature. We validate the effectiveness of our framework on three challenging datasets. Experimental results show that our method achieves the state-of-the-art performance.