Uncontrolled Scenes Research Articles

PSDF: Prior-Driven Neural Implicit Surface Learning for Multi-view Reconstruction.

Surface reconstruction has traditionally relied on the Multi-View Stereo (MVS)-based pipeline, which often suffers from noisy and incomplete geometry. This is due to that although MVS has been proven to be an effective way to recover the geometry of the scenes, especially for locally detailed areas with rich textures, it struggles to deal with areas with low texture and large variations of illumination where the photometric consistency is unreliable. Recently, Neural Implicit Surface Reconstruction (NISR) combines surface rendering and volume rendering techniques and bypasses the MVS as an intermediate step, which has emerged as a promising alternative to overcome the limitations of traditional pipelines. While NISR has shown impressive results on simple scenes, it remains challenging to recover delicate geometry from uncontrolled real-world scenes which is caused by its underconstrained optimization. To this end, the framework PSDF is proposed which resorts to external geometric priors from a pretrained MVS network and internal geometric priors inherent in the NISR model to facilitate high-quality neural implicit surface learning. Specifically, the visibility-aware feature consistency loss and depth prior-assisted sampling based on external geometric priors are introduced. These proposals provide powerfully geometric consistency constraints and aid in locating surface intersection points, thereby significantly improving the accuracy and delicate reconstruction of NISR. Meanwhile, the internal prior-guided importance rendering is presented to enhance the fidelity of the reconstructed surface mesh by mitigating the biased rendering issue in NISR. Extensive experiments on Tanks and Temples datasets show that PSDF achieves state-of-the-art performance on complex uncontrolled scenes.

Boosting cattle face recognition under uncontrolled scenes by embedding enhancement and optimization

Accurate individual cattle identification is crucial for modern precision cattle farming. However, practical applications encounter challenges due to factors such as shooting distance and angles, cattle movements, weather conditions, and cattle face posture. Existing models struggle with low recognition accuracy for low-recognizable images and poor robustness to facial pose variations, presenting urgent problems that need to be addressed. In this study, a novel cattle face recognition method was proposed, aiming to solve the aforementioned issues by analyzing and optimizing the embedding distribution of cattle faces. Firstly, MobileFaceNet was employed as the feature extraction network to sufficiently extract discriminative representations. Secondly, due to the tendency of low-recognizable samples to cluster in specific areas of the embedding space, an Embedding Enhancement Module (EEM) was proposed. This module drives embedding features away from ineffective embedding spaces, thereby enhancing the model's ability to extract identity features from less recognizable cattle faces. Finally, an Embedding Optimization Module (EOM) was proposed, which utilized a Sub-Center method to alleviate the model's learning difficulty during early training stages and achieved "pose-irrelevant" clusters by merging Sub-Centers, enhancing robustness to facial pose variations. Experimental results confirmed the effectiveness of the proposed method, with it outperforming the baseline by up to 2.12 % in accuracy, achieving a recognition accuracy of 98.69 %. Furthermore, the application of the proposed method in cattle farms further demonstrated the significant potential value of this work.

Learning a High Fidelity Identity Representation for Face Frontalization

This paper considers the problem of face frontalization in the wild, which transforms a face image with profile views into a frontal face. Face frontalization provides an effective solution to the face recognition problem in uncontrolled scenes. However, the existing methods either focus on deep learning techniques as an end-to-end framework or combine other explicit facial prior estimation tasks, such as 3D representation, optical flow estimation and so on, where computation is highly redundant and facial identity cannot be well represented. In this paper, we focus on how to maximise the potential of the model for identity learning and representation, and propose an accurate and lightweight face frontalization approach, named identity-preserving model (IPM). IPM has a well-designed encoder-decoder architecture which restores input face to a frontal counterpart. The encoder is constructed to extract representation from the input face, where a contrastive loss function is applied that encourages representations to form compact clusters, while preserving their relationships across the corpora. Then a cross-domain rectification module is proposed to eliminate the representation differences between the recognition and reconstruction domains, thus improving the accuracy of the reconstructed face. Extensive experiments on benchmark datasets show that the proposed IPM approach not only outperforms the state-of-the-art on public datasets but also can cope with images in the uncontrolled scenes.

Simple-action-guided dictionary learning for complex action recognition

Complex action recognition is an important yet challenging problem because of uncontrolled scenes, such as partial occlusions, viewpoint changes and dynamically changing backgrounds. Meanwhile obtaining a robust and performance model requires sufficient labeled training data, and it is hard to obtain. Due to the fact that each complex action is composed of a sequence of simple actions, they can share the common dictionary. To utilize such decomposition form, we build a simple-action-guided dictionary learning model (SAG-DLM) for complex action recognition. Especially, a common dictionary is learned by simple actions to model action-shared features, and we will transfer the common dictionary to help complex action learning. Further the difference complex action dictionary is studied to better obtain sparse representation. Finally, the complex actions are reconstructed by the common dictionary and the difference complex action dictionary. We validate the proposed SAG-DLM on two complex datasets: Olympic Sports dataset and UCF50 dataset. Extensive experiments prove that the effectiveness of the proposed SAG-DLM, and the learned common dictionary can provide promising improvement.

Face frontalization for uncontrolled scenes

Face frontalization for uncontrolled scenes

Free-view gait recognition.

Human gait has been shown to be an effective biometric measure for person identification at a distance. On the other hand, changes in the view angle pose a major challenge for gait recognition as human gait silhouettes are usually different from different view angles. Traditionally, such a multi-view gait recognition problem can be tackled by View Transformation Model (VTM) which transforms gait features from multiple gallery views to the probe view so as to evaluate the gait similarity. In the real-world environment, however, gait sequences may be captured from an uncontrolled scene and the view angle is often unknown, dynamically changing, or does not belong to any predefined views (thus VTM becomes inapplicable). To address this free-view gait recognition problem, we propose an innovative view-adaptive mapping (VAM) approach. The VAM employs a novel walking trajectory fitting (WTF) to estimate the view angles of a gait sequence, and a joint gait manifold (JGM) to find the optimal manifold between the probe data and relevant gallery data for gait similarity evaluation. Additionally, a RankSVM-based algorithm is developed to supplement the gallery data for subjects whose gallery features are only available in predefined views. Extensive experiments on both indoor and outdoor datasets demonstrate that the VAM outperforms several reference methods remarkably in free-view gait recognition.

非控场景下主成分稀疏表示与低秩分解的人脸识别

非控场景下主成分稀疏表示与低秩分解的人脸识别

A Richly Annotated Pedestrian Dataset for Person Retrieval in Real Surveillance Scenarios.

Retrieving specific persons with various types of queries, e.g., a set of attributes or a portrait photo has great application potential in large-scale intelligent surveillance systems. In this paper, we propose a richly annotated pedestrian (RAP) dataset which serves as a unified benchmark for both attribute-based and image-based person retrieval in real surveillance scenarios. Typically, previous datasets have three improvable aspects, including limited data scale and annotation types, heterogeneous data source, and controlled scenarios. Differently, RAP is a large-scale dataset which contains 84928 images with 72 types of attributes and additional tags of viewpoint, occlusion, body parts, and 2589 person identities. It is collected in the real uncontrolled scene and has complex visual variations in pedestrian samples due to the change of viewpoints, pedestrian postures, and cloth appearance. Towards a high-quality person retrieval benchmark, an amount of state-of-the-art algorithms on pedestrian attribute recognition and person re-identification (ReID), are performed for quantitative analysis with three evaluation tasks, i.e., attribute recognition, attribute-based and image-based person retrieval, where a new instance-based metric is proposed to measure the dependency of the prediction of multiple attributes. Finally, some interesting problems, e.g., the joint feature learning of attribute recognition and ReID, and the problem of cross-day person ReID, are explored to show the challenges and future directions in person retrieval.

A comprehensive solution for detecting events in complex surveillance videos

Event detection have long been a fundamental problem in computer vision society. Various datasets for recognizing human events and activities have been proposed to help developing better models and methods, such as UCF101, HMDB51, etc. These datasets all share the same properties that either predefined scripts are provided or the images are almost actor-oriented with little background noise. These properties, however, are completely different from that of surveillance event detection, making the effective solutions on these datasets totally not suitable. Event detection in complex surveillance video is a much more difficult task with several challenges: heavy occlusions between pedestrians, low image resolution and uncontrolled scene condition. TRECVID-SED evaluation, aiming at detecting events in highly crowded airport, is well-known for its great difficulties. To deal with event detection in realistic scene, such as TRECVID-SED, we introduce a comprehensive solution framework based on pedestrian detection, deep key-pose detection and trajectory analysis. Explicitly, instead of detecting whole body of one person, we detect the head-shoulder of pedestrian, addressing the issue of heavy occlusion of pedestrians in complex scene. We also propose a trajectory-based event detection method so as to better focus on the key actors of events. For those events with discriminative poses, we model the event detection as key pose detection by taking advantages of Faster R-CNN. The presented framework achieves the best result in TRECVID-SED 2016 evaluation.

Event-Based Color Segmentation With a High Dynamic Range Sensor.

This paper introduces a color asynchronous neuromorphic event-based camera and a methodology to process color output from the device to perform color segmentation and tracking at the native temporal resolution of the sensor (down to one microsecond). Our color vision sensor prototype is a combination of three Asynchronous Time-based Image Sensors, sensitive to absolute color information. We devise a color processing algorithm leveraging this information. It is designed to be computationally cheap, thus showing how low level processing benefits from asynchronous acquisition and high temporal resolution data. The resulting color segmentation and tracking performance is assessed both with an indoor controlled scene and two outdoor uncontrolled scenes. The tracking's mean error to the ground truth for the objects of the outdoor scenes ranges from two to twenty pixels.

Robust, Efficient Depth Reconstruction With Hierarchical Confidence-Based Matching.

In recent years, taking photos and capturing videos with mobile devices have become increasingly popular. Emerging applications based on the depth reconstruction technique have been developed, such as Google lens blur. However, depth reconstruction is difficult due to occlusions, non-diffuse surfaces, repetitive patterns, and textureless surfaces, and it has become more difficult due to the unstable image quality and uncontrolled scene condition in the mobile setting. In this paper, we present a novel hierarchical framework with multi-view confidence-based matching for robust, efficient depth reconstruction in uncontrolled scenes. Particularly, the proposed framework combines local cost aggregation with global cost optimization in a complementary manner that increases efficiency and accuracy. A depth map is efficiently obtained in a coarse-to-fine manner by using an image pyramid. Moreover, confidence maps are computed to robustly fuse multi-view matching cues, and to constrain the stereo matching on a finer scale. The proposed framework has been evaluated with challenging indoor and outdoor scenes, and has achieved robust and efficient depth reconstruction.

Grapevine flower estimation by applying artificial vision techniques on images with uncontrolled scene and multi-model analysis

New technologies in precision viticulture are increasingly being used to improve grape quality. One of the main challenges being faced by the scientific community in viticulture is early yield prediction. Within this framework, flowering as well as fruit set assessment is of special interest since these two physiological processes highly influence grapevine yield. In addition, an accurate fruit set evaluation can only be performed by means of flower counting. Herein a new methodology for segmenting inflorescence grapevine flowers in digital images is presented. This approach, based on mathematical morphology and pyramidal decomposition, constitutes an outstanding advance with respect to other previous approaches since it can be applied on images with uncontrolled background. The algorithm was tested on 40 images of 4 different Vitis vinifera L. varieties, and resulted in high performance. Specifically, values for Precision and Recall were 83.38% and 85.01%, respectively. Additionally, this paper also proposes a comprehensive study on models for estimating actual flower number per inflorescence. Results and conclusions that are developed in the literature and treated herewith are also clarified. Furthermore, the use of non-linear models as a promising alternative to previously-proposed linear models is likewise suggested in this study.

Adaptive moving shadow detection and removal by new semi-supervised learning technique

The efficient application of current methods of shadow detection in video is hindered by the difficulty in defining their parameters or models and/or their application domain dependence. This paper presents a new shadow detection and removal method that aims to overcome these inefficiencies. It proposes a semi-supervised learning rule using a new variant of co-training technique for shadow detection and removal in uncontrolled scenes. The new variant both reduces the run-time through a periodical execution of a co-training process according to a novel temporal framework, and generates a more generic prediction model for an accurate classification. The efficiency of the proposed method is shown experimentally on a testbed of videos that were recorded by a static camera and that included several constraints, e.g., dynamic changes in the natural scene and various visual shadow features. The conducted experimental study produced quantitative and qualitative results that highlighted the robustness of our shadow detection method and its accuracy in removing cast shadows. In addition, the practical usefulness of the proposed method was evaluated by integrating it in a Highway Control and Management System software called RoadGuard.

Real-time shading-based refinement for consumer depth cameras

We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes. Per frame, our real-time algorithm takes raw noisy depth data and an aligned RGB image as input, and approximates the time-varying incident lighting, which is then used for geometry refinement. This leads to dramatically enhanced depth maps at 30Hz. Our algorithm makes few scene assumptions, handling arbitrary scene objects even under motion. To enable this type of real-time depth map enhancement, we contribute a new highly parallel algorithm that reformulates the inverse rendering optimization problem in prior work, allowing us to estimate lighting and shape in a temporally coherent way at video frame-rates. Our optimization problem is minimized using a new regular grid Gauss-Newton solver implemented fully on the GPU. We demonstrate results showing enhanced depth maps, which are comparable to offline methods but are computed orders of magnitude faster, as well as baseline comparisons with online filtering-based methods. We conclude with applications of our higher quality depth maps for improved real-time surface reconstruction and performance capture.

Pi-Tag: a fast image-space marker design based on projective invariants

Visual marker systems have become an ubiquitous tool to supply a reference frame onto otherwise uncontrolled scenes. Throughout the last decades, a wide range of different approaches have emerged, each with dif- ferent strengths and limitations. Some tags are optimized to reach a high accuracy in the recovered camera pose, others are based on designs that aim to maximizing the detection speed or minimizing the effect of occlusion on the detec- tion process. Most of them, however, employ a two-step pro- cedure where an initial homography estimation is used to translate the marker from the image plane to an orthonormal world, where it is validated and recognized. In this paper, we present a general purpose fiducial marker system that performs both steps directly in image-space. Specifically, by exploiting projective invariants such as collinearity and cross-ratios, we introduce a detection and recognition algo- rithm that is fast, accurate and moderately robust to occlu- sion. The overall performance of the system is evaluated in an extensive experimental section, where a comparison with a well-known baseline technique is presented. Additionally, several real-world applications are proposed, ranging from camera calibration to projector-based augmented reality.

Dissociating object and space representations in scene-selective visual cortex

Neuroimaging studies have identified a network of scene-selective cortical regions: Parahippocampal Place Area (PPA), Retrosplenial Complex (RSC) and Transverse Occipital Sulcus (TOS). However, the different contributions of these areas to scene recognition are unclear. Further, while PPA shows a stronger response to scenes than objects it also contains significant object information. To reconcile these findings and elucidate the nature of scene processing in each region, we assessed the relative impact of objects and spatial backgrounds on responses. We manipulated object and spatial information by generating minimal scenes comprising one of seven objects (or no object), presented on one of three different backgrounds differing in the spatial information they contain (room interior, horizon, and luminance gradient). The horizon and room backgrounds conveyed depth information, but only the room defined an enclosed space. Unlike other studies using real-world, but visually uncontrolled scenes, these minimal scenes provide a simple controlled test of the relative contribution of objects and backgrounds. Individual scenes were presented in an ungrouped event-related fMRI experiment and the distributed response patterns to each were analyzed using split-half correlations. Response patterns in scene-selective cortex discriminated both objects and backgrounds. However, different scene-selective regions emphasized different aspects of the scenes. RSC primarily reflected the spatial aspects of the scenes, discriminating backgrounds regardless of objects and showing no modulation of response strength by object presence. In contrast, PPA and TOS responded more strongly in the presence of an object and showed strong discrimination of objects as well as backgrounds. However, object discrimination in PPA, but not TOS, was modulated by the type of background, with the strongest discrimination against room backgrounds. We conclude that while RSC mainly represents spatial information, PPA and TOS represent both object and spatial information to provide a rich representation of the visual environment.

Estimating the scene illumination chromaticity by using a neural network.

A neural network can learn color constancy, defined here as the ability to estimate the chromaticity of a scene's overall illumination. We describe a multilayer neural network that is able to recover the illumination chromaticity given only an image of the scene. The network is previously trained by being presented with a set of images of scenes and the chromaticities of the corresponding scene illuminants. Experiments with real images show that the network performs better than previous color constancy methods. In particular, the performance is better for images with a relatively small number of distinct colors. The method has application to machine vision problems such as object recognition, where illumination-independent color descriptors are required, and in digital photography, where uncontrolled scene illumination can create an unwanted color cast in a photograph.