Scene Features Research Articles

목표 기반 시각적 이동 작업을 위한 시각적 맥락정보 임베딩과 교착상태 처리

In this paper, we propose a novel deep neural network-based agent model, (VCENet), for performing target-driven visual navigation tasks. Most recent agent models for visual navigation try to recognize the real-time task context by using only objects and their relationships detected from RGB input images. However, such object-oriented visual context does not contain detailed information about the background scene, which is important for ground navigation. Moreover, it may result in a misrecognized context due to because of errors in of the object and relation detectors. To overcome these problems, the proposed VCENet model represents the real-time task context by using as the appearance and geometric features of the background scene extracted from RGB-D input images well as the object relation features derived through from graph embedding. Many existing models, which learn an action policiesy based upon on reinforcement, learning, do not provide any reward functions to avoid deadlocks ahead during of navigation or any effective deadlock recovery policyy learning mechanisms to escape the deadlocks. To address these problems, the proposed VCENet model provides not only reward functions for deadlock avoidance and recovery, but also a deadlock recovery policy based on imitation learning. Conducting a variety of various experiments in photo-realistic virtual indoor environments provided by the 3D simulator AI2THOR, we made sure of demonstrated the superiority of the proposed VCENet model.

Muti-class Image Classification using Transfer Learning

Abstract: Humans are very proficient at perceiving natural scenes and understanding their contents. Everyday image content across the globe is rapidly increasing and there is a need for classifying these images for further research. Scene classification is a challenging task, because in some natural scenes there will be common features in images and some images may contain half indoor and half outdoor scene features. In this project we are going to classify natural scenery in images using Artificial Intelligence. Based on the analysis of the error backpropagation algorithm, we propose an innovative training criterion of depth neural network for maximum interval minimum classification error. At the same time, the cross entropy and M3CE are analyzed and combined to obtain better results. Finally, we tested our proposed M3 CE-CEc on two deep learning standard databases, MNIST and CIFAR-10. The experimental results show that M3 CE can enhance the cross-entropy, and it is an effective supplement to the cross-entropy criterion. M3 CE-CEc has obtained good results in both databases.

Encoding of environmental illumination by primate melanopsin neurons.

Light regulates physiology, mood, and behavior through signals sent to the brain by intrinsically photosensitive retinal ganglion cells (ipRGCs). How primate ipRGCs sense light is unclear, as they are rare and challenging to target for electrophysiological recording. We developed a method of acute identification within the live, ex vivo retina. Using it, we found that ipRGCs of the macaque monkey are highly specialized to encode irradiance (the overall intensity of illumination) by blurring spatial, temporal, and chromatic features of the visual scene. We describe mechanisms at the molecular, cellular, and population scales that support irradiance encoding across orders-of-magnitude changes in light intensity. These mechanisms are conserved quantitatively across the ~70 million years of evolution that separate macaques from mice.

Multi-exposure electric power monitoring image fusion method without ghosting based on exposure fusion framework and color dissimilarity feature.

To solve the ghosting artifacts problem in dynamic scene multi-scale exposure fusion, an improved multi-exposure fusion method has been proposed without ghosting based on the exposure fusion framework and the color dissimilarity feature of this study. This fusion method can be further applied to power system monitoring and unmanned aerial vehicle monitoring. In this study, first, an improved exposure fusion framework based on the camera response model was applied to preprocess the input image sequence. Second, the initial weight map was estimated by multiplying four weight items. In removing the ghosting weight term, an improved color dissimilarity feature was used to detect the object motion features in dynamic scenes. Finally, the improved pyramid model as adopted to retain detailed information about the poor exposure areas. Experimental results indicated that the proposed method improves the performance of images in terms of sharpness, detail processing, and ghosting artifacts removal and is superior to the five existing multi-exposure image fusion (MEF) methods in quality evaluation.

Integrating audio and visual modalities for multimodal personality trait recognition via hybrid deep learning.

Recently, personality trait recognition, which aims to identify people's first impression behavior data and analyze people's psychological characteristics, has been an interesting and active topic in psychology, affective neuroscience and artificial intelligence. To effectively take advantage of spatio-temporal cues in audio-visual modalities, this paper proposes a new method of multimodal personality trait recognition integrating audio-visual modalities based on a hybrid deep learning framework, which is comprised of convolutional neural networks (CNN), bi-directional long short-term memory network (Bi-LSTM), and the Transformer network. In particular, a pre-trained deep audio CNN model is used to learn high-level segment-level audio features. A pre-trained deep face CNN model is leveraged to separately learn high-level frame-level global scene features and local face features from each frame in dynamic video sequences. Then, these extracted deep audio-visual features are fed into a Bi-LSTM and a Transformer network to individually capture long-term temporal dependency, thereby producing the final global audio and visual features for downstream tasks. Finally, a linear regression method is employed to conduct the single audio-based and visual-based personality trait recognition tasks, followed by a decision-level fusion strategy used for producing the final Big-Five personality scores and interview scores. Experimental results on the public ChaLearn First Impression-V2 personality dataset show the effectiveness of our method, outperforming other used methods.

Study on Recognition Method of Similar Weather Scenes in Terminal Area

Weather is a key factor affecting the control of air traffic. Accurate recognition and classification of similar weather scenes in the terminal area is helpful for rapid decision-making in air traffic flow management. Current researches mostly use traditional machine learning methods to extract features of weather scenes, and clustering algorithms to divide similar scenes. Inspired by the excellent performance of deep learning in image recognition, this paper proposes a terminal area similar weather scene classification method based on improved deep convolution embedded clustering (IDCEC), which uses the combination of the encoding layer and the decoding layer to reduce the dimensionality of the weather image, retaining useful information to the greatest extent, and then uses the combination of the pre-trained encoding layer and the clustering layer to train the clustering model of the similar scenes in the terminal area. Finally, terminal area of Guangzhou Airport is selected as the research object, the method proposed in this article is used to classify historical weather data in similar scenes, and the performance is compared with other state-of-the-art methods. The experimental results show that the proposed IDCEC method can identify similar scenes more accurately based on the spatial distribution characteristics and severity of weather; at the same time, compared with the actual flight volume in the Guangzhou terminal area, IDCEC's recognition results of similar weather scenes are consistent with the recognition of experts in the field.

From Front to Rear: 3D Semantic Scene Completion Through Planar Convolution and Attention-Based Network

Semantic Scene Completion (SSC) aims to reconstruct complete 3D scenes with precise voxel-wise semantics from the single-view incomplete input data, a crucial but highly challenging problem for scene understanding. Although SSC has seen significant progress due to the introduction of 2D semantic priors in recent years, the occluded parts, especially the rear-view of the scenes, are still poorly completed and segmented. To ameliorate this issue, we propose a novel deep learning framework for 3D SSC, named Planar Convolution and Attention-based Network (PCANet), to effectively extend high-precision predictions of the front-view surface to the rear-view occluded areas. Specifically, we decompose the traditional convolutional layer into three successive planar convolutions to form a Planar Convolution Residual (PCR) block, which maintains the planar features of the 3D scene. Afterward, the Planar Attention Module (PAM) is proposed to capture three different planar attentions and harvest the global context from the front surface to the rear occluded areas to improve the overall accuracy. Extensive experiments on the real NYU and NYUCAD datasets and the synthetic SUNCG-RGBD dataset demonstrate that our proposed framework can generate high-quality SSC results in both front and rear views and outperforms the state-of-the-art approaches trained in an end-to-end manner without additional data.

Influence of auditory working memory in discriminating children with good musical abilities from children with poor musical abilities

Abstract Purpose: Musical abilities are associated with the perception of complex acoustic features in an auditory scene, which requires a good load of cognitive processing. Musical sleepers (individuals with good musical abilities without formal music training) were proven to be in adults, and their enhanced cognitive abilities were established, but such a phenomenon in children is not explored yet. Hence, the present study was carried out to assess auditory working memory (AWM) abilities in children with widespread musical abilities. Materials and Methods: Twenty-nine children within the age range of 7–13 years participated in the study. The children’s musical abilities were assessed using the Montreal Battery for Evaluation of Musical Abilities and scores were recorded. Sixteen and thirteen children were categorized into individuals with good and poor musical abilities, respectively, based on the 50th percentile score as the cutoff. The tests for AWM, such as forward span, backward span, and N-back, were administered. Results: Point biserial correlation showed that groups had a significant positive association with forward span (r = 0.65; P = 0.00), backward span test (r = 0.41; P = 0.02), and N-back test (r = 0.70; P = 0.00). Multivariate analysis of variance (MANOVA) indicated a significant main effect of groups, and post hoc analysis showed that children with good musical abilities outperformed the ones with poor musical abilities in all three working memory measures. Further, Fisher’s discriminant analysis revealed that the N-back test, with discriminant coefficient of 0.75, is the best auditory-cognitive predictor of musical abilities in children. Conclusions: Children who had no musical training exhibited better musical ability. This may be mediated by improved AWM, but an additional investigation into the relationship between musical aptitude and other psychophysical abilities in children without musical training is necessary.

Exploiting Human Pose and Scene Information for Interaction Detection

Identifying human actions and interactions finds its use in many areas, such as security, surveillance, assisted living, patient monitoring, rehabilitation, sports, and e-learning. This wide range of applications has attracted many researchers to this field. Inspired by the existing recognition systems, this paper proposes a new and efficient human-object interaction recognition (HOIR) model which is based on modeling human pose and scene feature information. There are different aspects involved in an interaction, including the humans, the objects, the various body parts of the human, and the background scene. The main objectives of this research include critically examining the importance of all these elements in determining the interaction, estimating human pose through image foresting transform (IFT), and detecting the performed interactions based on an optimized multi-feature vector. The proposed methodology has six main phases. The first phase involves preprocessing the images. During preprocessing stages, the videos are converted into image frames. Then their contrast is adjusted, and noise is removed. In the second phase, the human-object pair is detected and extracted from each image frame. The third phase involves the identification of key body parts of the detected humans using IFT. The fourth phase relates to three different kinds of feature extraction techniques. Then these features are combined and optimized during the fifth phase. The optimized vector is used to classify the interactions in the last phase. The MSR Daily Activity 3D dataset has been used to test this model and to prove its efficiency. The proposed system obtains an average accuracy of 91.7% on this dataset.

Visual Navigation Based on Language Assistance and Memory

In order to solve outdoor mobile robots’ dependence on geographic information systems, and to realize automatic navigation in the face of complex and changeable scenes, we propose a method that selects landmark and adds prompt guidance so that the mobile robot can navigate relying on visual-language and memory. Visual-language can guide the direction of the mobile robot’s movement, obeying the annotation of people and according to its memory of the scene, which refers to the strategy of selecting passed-by landmarks for the route and remembering the scene features. When passing it, the agent can ascertain the position and match it to carry out the action. Experiments showed that our proposed method can achieve the purpose of independent navigation without GIS, and is superior to existing methods.

Self-Parameter Distillation Dehazing.

In this paper, we propose a novel dehazing method based on self-distillation. In contrast to conventional knowledge distillation approaches that transfer large models (teacher networks) to small models (student networks), we introduce a single knowledge distillation network that transfers network parameters to itself for dehazing. In the early stages, the proposed network transfers scene content (identity) information to the next stage of itself using haze-free data. However, in the later stages, the network transfers haze information to itself using haze data, enabling the accurate dehazing of input images using scene information from the early stages. In a single network, parameters are seamlessly updated from extracting global scene features to dehazing the scene. During the training, forward propagation acts as a teacher network, whereas backward propagation acts as a student network. The experimental results demonstrate that the proposed method considerably outperforms other state-of-the-art dehazing methods.

SAR Image Reconstruction and Autofocus Using Complex-Valued Feature Prior and Deep Network Implementation

Synthetic aperture radar (SAR) plays an important role in remote sensing by providing electromagnetic images of the observation scene. The prior knowledge-based SAR image reconstruction method can reduce the requirement of data sampling ratio and improve image quality. The existing prior knowledge-based method usually uses the magnitude information of SAR images while ignoring the phase information. However, since the echo and backscattering coefficients are complex values, the phase information of SAR images will help improve the reconstruction accuracy in the image reconstruction process. To improve the reconstruction performance, this paper proposes a SAR image reconstruction and autofocus method using complex-valued feature prior. In the proposed method, the complex-valued feature prior is learned from data by a complex-valued feature projection operator (CFPO), which can characterize and extract scene features in Range-Doppler domain and 2D frequency domain. The proposed CFPO enables more efficient use of echo data and helps to improve image reconstruction and autofocus performance. In addition, the proposed method is implemented by an unfolded deep network, which enables data-driven feature learning and efficient computation. The proposed method is verified by simulated and measured data.

Refined Voting and Scene Feature Fusion for 3D Object Detection in Point Clouds.

An essential task for 3D visual world understanding is 3D object detection in lidar point clouds. To predict directly bounding box parameters from point clouds, existing voting-based methods use Hough voting to obtain the centroid of each object. However, it may be difficult for the inaccurately voted centers to regress boxes accurately, leading to the generation of redundant bounding boxes. For objects in indoor scenes, there are several co-occurrence patterns for objects in indoor scenes. Concurrently, semantic relations between object layouts and scenes can be used as prior context to guide object detection. We propose a simple, yet effective network, RSFF-Net, which adds refined voting and scene feature fusion for indoor 3D object detection. The RSFF-Net consists of three modules: geometric function, refined voting, and scene constraint. First, a geometric function module is used to capture the geometric features of the nearest object of the voted points. Then, the coarse votes are revoted by a refined voting module, which is based on the fused feature between the coarse votes and geometric features. Finally, a scene constraint module is used to add the association information between candidate objects and scenes. RSFF-Net achieves competitive results on indoor 3D object detection benchmarks: ScanNet V2 and SUN RGB-D.

Wiki Loves Monuments: Crowdsourcing the Collective Image of the Worldwide Built Heritage

The wide adoption of digital technologies in the cultural heritage sector has promoted the emergence of new, distributed ways of working, communicating, and investigating cultural products and services. In particular, collaborative online platforms and crowdsourcing mechanisms have been widely adopted in the effort to solicit input from the community and promote engagement. In this work, we provide an extensive analysis of the Wiki Loves Monuments initiative, an annual, international photography contest in which volunteers are invited to take pictures of the built cultural heritage and upload them to Wikimedia Commons. We explore the geographical, temporal, and topical dimensions across the 2010–2021 editions. We first adopt a set of CNN-based artificial systems that allow the learning of deep scene features for various scene recognition tasks, exploring cross-country (dis)similarities. To overcome the rigidity of the framework based on scene descriptors, we train a deep convolutional neural network model to label a photo with its country of origin. The resulting model captures the best representation of a heritage site uploaded in a country, and it allows the domain experts to explore the complexity of cross-national architectural styles. Finally, as a validation step, we explore the link between architectural heritage and intangible cultural values, operationalized using the framework developed within the World Value Survey research program. We observe that cross-country cultural similarities match to a fair extent the interrelations emerging in the architectural domain. We think this study contributes to highlighting the richness and the potential of the Wikimedia data and tools ecosystem to act as a scientific object for art historians, iconologists, and archaeologists.

A Light Field Full-Focus Image Feature Point Matching Method with an Improved ORB Algorithm

Most of the traditional image feature point extraction and matching methods are based on a series of light properties of images. These light properties easily conflict with the distinguishability of the image features. The traditional light imaging methods focus only on a fixed depth of the target scene, and subjects at other depths are often easily blurred. This makes the traditional image feature point extraction and matching methods suffer from a low accuracy and a poor robustness. Therefore, in this paper, a light field camera is used as a sensor to acquire image data and to generate a full-focus image with the help of the rich depth information inherent in the original image of the light field. The traditional ORB feature point extraction and matching algorithm is enhanced with the goal of improving the number and accuracy of the feature point extraction for the light field full-focus images. The results show that the improved ORB algorithm extracts not only most of the features in the target scene but also covers the edge part of the image to a greater extent and produces extracted feature points which are evenly distributed for the light field full-focus image. Moreover, the extracted feature points are not repeated in a large number in a certain part of the image, eliminating the aggregation phenomenon that exists in traditional ORB algorithms.

Complete trajectory extraction for moving targets in traffic scenes that considers multi-level semantic features

Complex traffic scenes are susceptible to the occlusion of moving targets and blurred target images, resulting in incomplete extraction of target trajectories within the field of view (FOV). To address this issue, we proposed a complete trajectory extraction algorithm for moving targets (TraEA) that considers the multi-level semantic features of traffic scenes following the FOV of single-camera surveillance videos. TraEA utilizes semantic information from the multi-level scene structure, such as image pixel features, geometric features, as well as temporal and spatial relationships between targets within each keyframe image. The semantic information helps extract and optimize the complete trajectory of a target, to assemble the target’s moving trajectory with improved integrity. An experiment with real-world traffic videos collected in Beijing, China validated the proposed algorithm and showed that TraEA improved the completeness, accuracy and robustness of target trajectory extraction. Traffic trajectories from the experiment can facilitate traffic supervision by analyzing the moving behaviors of individual targets. While the proposed TraEA algorithm is applicable to extract trajectories from videos in other domains, such as animal movements, potential privacy infringement and abusive applications warrant cautions of ethical considerations.

ФИЛЬМ «ЛЕТЯТ ЖУРАВЛИ»: ОПЫТ СИСТЕМНОГО КИНОАНАЛИЗА

The article provides a film analysis of the movie "The Cranes Are Flying", created in 1957 by a team of Soviet creative figures – director Mikhail Kalatozov, cameraman Sergei Urusevsky, screenwriter Viktor Rozov, composer Moisei Weinberg. The methodology of the study was the philosophical and art history analysis of works of art by V.I. Zhukovsky and N.P. Koptseva and the method of system film analysis by G. Korte. According to the results of the study, artistic differences were revealed between the text of the play by V. Rozov "Forever Alive", which is the basis of the script, and the plot of the film "The Cranes Are Flying"; analyzed the key characters of the film and the features of their representation on the screen with the help of visual images; the aesthetic features of the key scenes of the film are identified and analyzed, and it is also established that the film is a masterpiece of the world cinema art, as it contains the transition to the symbolic status of an artistic image.

GhostCount: A lightweight convolution network based on high‐altitude video for vehicle instantaneous counting in dense traffic scenes

AbstractInstantaneous vehicle counting of traffic scenes based on high‐altitude video is an important way for real‐time traffic information collection in intelligent transportation systems (ITS). However, vehicle counts based on high‐altitude video are susceptible to problems such as denseness, occlusion and small size. The mainstream method is to use a Convolutional Neural Network (CNN) to output density maps and obtain vehicle count results. However, most CNNs are computationally expensive and have poor real‐time performance. Therefore, we propose a lightweight CNN named GhostCount, specially designed for high‐accuracy vehicle counts on edge devices. First, we combine ResNet‐18 and Lightweight RefineNet to build an encoder–decoder network architecture to effectively extract vehicle features in complex traffic scenes. Next, we replace the ordinary convolutional layers in ResNet‐18 with Ghost modules to lighten the network. Finally, a binary cross‐entropy loss function is introduced to suppress background noise. We demonstrate GhostCount on public datasets (TRANCOS, CARPK, PUCPR+) and our self‐built dataset (CSCAR). Results show that GhostCount can perform instantaneous vehicle counting with higher accuracy and faster inference speed than other representative lightweight CNNs. The method we propose would provide new solutions and ideas for ITS applications such as traffic information collection and smart parking management.

Systematic reduction of the dimensionality of natural scenes allows accurate predictions of retinal ganglion cell spike outputs

The mammalian retina engages a broad array of linear and nonlinear circuit mechanisms to convert natural scenes into retinal ganglion cell (RGC) spike outputs. Although many individual integration mechanisms are well understood, we know less about how multiple mechanisms interact to encode the complex spatial features present in natural inputs. Here, we identified key spatial features in natural scenes that shape encoding by primate parasol RGCs. Our approach identified simplifications in the spatial structure of natural scenes that minimally altered RGC spike responses. We observed that reducing natural movies into 16 linearly integrated regions described ∼80% of the structure of parasol RGC spike responses; this performance depended on the number of regions but not their precise spatial locations. We used simplified stimuli to design high-dimensional metamers that recapitulated responses to naturalistic movies. Finally, we modeled the retinal computations that convert flashed natural images into one-dimensional spike counts.

Mobile phone indoor scene features recognition localization method based on semantic constraint of building map location anchor

Abstract Visual features play a key role in indoor positioning and navigation services as the main semantic information to help people understand the environment. However, insufficient semantic constraint information and mismatching localization without building map have hindered the ubiquitous application services. To address the problem, we propose a smartphone indoor scene features recognition localization method with building map semantic constraints. First, based on Geographic Information System and Building Information Modeling techniques, a geocoded entity library of building Map Location Anchor (MLA) is constructed, which is able to provide users with “immersive” meta-building-map and semantic anchor constraints for mobile phone positioning when map matching. Second, using the MYOLOv5s deep learning model improved on indoor location scenario, the nine types of ubiquitous anchor features in building scenes are recognized in real time by acquiring video frames from the smartphone camera. Lastly, the spatial locations of the ubiquitous indoor facilities obtained using smartphone video recognition are matched with the MLA P3P algorithm to achieve real-time positioning and navigation. The experimental results show that the MLA recognition accuracy of the improved MYOLOv5s is 97.2%, and the maximum localization error is within the range of 0.775 m and confined to the interval of 0.5 m after applying the Building Information Modeling based Positioning and Navigation road network step node constraint, which can effectively achieve high positioning accuracy in the building indoor scenarios with adequate MLA and road network constraint.