Viewpoint Changes Research Articles

Human activity recognition based on integration of multilayer information of convolutional neural network architecture

SummaryHuman activity recognition (HAR) has gained researcher's interest due to its increasing demand in automated monitoring applications. Development of efficient HAR algorithm is still an open research area due to the challenges like inter and intra‐class variations, diversity in lighting conditions, view point changes, and complex object motions. Convolutional neural network (CNN) based methods have achieved significant improvement in HAR. However, CNN implementations have drawback that it require a lot of computational resources due to the use of large number of learnable parameters. To overcome this drawback, we propose a simple and computationally efficient deep CNN architecture using multi‐layer information fusion for HAR. In this study, we explore the impact of information fusion at intermediate layers of the network, as each convolutional layer of the network hierarchically extracts information at different level of abstraction of the objects from the video frames. In this work, first we designed a simple and computationally efficient deep CNN architecture and then we introduce a feature fusion strategy to integrate the complementary information of intermediate layers to the layer of the proposed CNN architecture. The proposed architecture is fine‐tuned and trained from scratch with raw RGB data. Softmax classifier is used at the last layer of network for activity classification. Benefits of the proposed architecture over standard deep learning architectures is it's high computational efficiency and reduced requirement of computational resources. To prove the effectiveness of the proposed method, we performed several extensive experiments on publically available datasets. The experimental results of the proposed method have demonstrated its superiority over other existing state‐of‐the‐art methods.

Facial Identity And Facial Speech Processing In Developmental Prosopagnosia

Developmental prosopagnosia is the life-long impaired ability to recognize familiar faces. The neural basis of this condition remains a matter of investigation. Face processing networks are bilateral but have right hemisphere dominance, and the location of lesions in acquired prosopagnosia is in keeping with this. There may be a complementary contribution from left hemisphere regions to certain types of face processing, and prior reports suggest that this might involve processing faces depicted by line contours or facial speech patterns by lip reading. We performed two studies in seven subjects with developmental prosopagnosia. The first task tested participants’ ability to match faces across viewpoint changes, with either unaltered photographs or high contrast line-drawing-like images. The second task involved a series of lip-reading tests to examine ability to perceive facial speech patterns. Performance on these two tasks was compared to twenty-three age- and education-matched control subjects. Prosopagnosic subjects had normal performance with line-contour faces, but facial processing showed no improvement with the added information in unaltered photographs. In facial speech pattern tests developmental prosopagnosic subjects could detect, discriminate, and identify facial speech patterns, but compared to controls, there was reduced use of visual cues or anomalous audiovisual integration in the McGurk effect, with only one subject performing normally. We conclude that developmental prosopagnosia may have a broader range of face processing impairment beyond just identity recognition. There can be association with a subtle impairment in lip-reading, which has been associated more with left than right fusiform damage after acquired lesions.

Fusing surveillance videos and three‐dimensional scene: A mixed reality system

AbstractAugmented Virtual Environments (AVE) or Virtual‐Reality Fusion systems fuse dynamic videos with static three‐dimensional (3D) models of a virtual environment to provide an optimal solution for visualizing and understanding multichannel surveillance systems. However, texture distortion caused by viewpoint changes in such systems is a critical issue that needs to be addressed. To minimize texture fusion distortion, this paper presents a novel virtual environment system in two phases, offline and online phases, to dynamically fuse multiple surveillance videos with a virtual 3D scene. In the offline phase, a static virtual environment is obtained by performing a 3D photogrammetric reconstruction from the input images of the scene. In the online phase, the virtual environment is augmented by fusing multiple videos through two optional strategies. One strategy is to dynamically map images of different videos onto a 3D model of the virtual environment, and the other is to extract moving objects and represent them as billboards. The system can be used to visualize a 3D environment from any viewpoint augmented by real‐time videos. Experiments and user studies in different scenarios demonstrate the superiority of our system.

Disentangled Representations for Short-Term and Long-Term Person Re-Identification.

We address the problem of person re-identification (reID), that is, retrieving person images from a large dataset, given a query image of the person of interest. A key challenge is to learn person representations robust to intra-class variations, as different persons could have the same attribute, and persons' appearances look different, e.g., with viewpoint changes. Recent reID methods focus on learning person features discriminative only for a particular factor of variations (e.g., human pose), which also requires corresponding supervisory signals (e.g., pose annotations). To tackle this problem, we propose to factorize person images into identity-related and -unrelated features. Identity-related features contain information useful for specifying a particular person (e.g., clothing), while identity-unrelated ones hold other factors (e.g., human pose). To this end, we propose a new generative adversarial network, dubbed identity shuffle GAN (IS-GAN). It disentangles identity-related and -unrelated features from person images through an identity-shuffling technique that exploits identification labels alone without any auxiliary supervisory signals. We restrict the distribution of identity-unrelated features, or encourage the identity-related and -unrelated features to be uncorrelated, facilitating the disentanglement process. Experimental results validate the effectiveness of IS-GAN, showing state-of-the-art performance on standard reID benchmarks, including Market-1501, CUHK03 and DukeMTMC-reID. We further demonstrate the advantages of disentangling person representations on a long-term reID task, setting a new state of the art on a Celeb-reID dataset. Our code and models are available online: https://cvlab-yonsei.github.io/projects/ISGAN/.

Robust AUV Visual Loop-Closure Detection Based on Variational Autoencoder Network

The visual loop-closure detection for autonomous underwater vehicles (AUVs) is a key component to reduce the drift error accumulated in simultaneous localization and mapping tasks. However, due to viewpoint changes, textureless images, and fast-moving objects, the loop closure detection in dramatically changing underwater environments remains a challenging problem to traditional geometric methods. Inspired by strong feature learning ability of deep neural networks, we propose an underwater loop-closure detection method based on a variational autoencoder network in this article. Our proposed method can learn effective image representations to deal with the challenges caused by dynamic underwater environments. Specifically, the proposed network is an unsupervised method, which avoids the difficulty and cost of labeling a great quantity of underwater data. Also included is a semantic object segmentation module, which is utilized to segment the underwater environments and assign weights to objects in order to alleviate the impact of fast-moving objects. Furthermore, an underwater image description scheme is used to enable efficient access to geometric and object-level semantic information, which helps to build a robust and real-time system in dramatically changing underwater scenarios. Finally, we test the proposed system under complex underwater environments and get a recall rate of 92.31% in the tested environments.

Unsupervised Learning of Local Equivariant Descriptors for Point Clouds.

Correspondences between 3D keypoints generated by matching local descriptors are a key step in 3D computer vision and graphic applications. Learned descriptors are rapidly evolving and outperforming the classical handcrafted approaches in the field. Yet, to learn effective representations they require supervision through labeled data, which are cumbersome and time-consuming to obtain. Unsupervised alternatives exist, but they lag in performance. Moreover, invariance to viewpoint changes is attained either by relying on data augmentation, which is prone to degrading upon generalization on unseen datasets, or by learning from handcrafted representations of the input which are already rotation invariant but whose effectiveness at training time may significantly affect the learned descriptor. We show how learning an equivariant 3D local descriptor instead of an invariant one can overcome both issues. LEAD (Local EquivAriant Descriptor) combines Spherical CNNs to learn an equivariant representation together with plane-folding decoders to learn without supervision. Through extensive experiments on standard surface registration datasets, we show how our proposal outperforms existing unsupervised methods by a large margin and achieves competitive results against the supervised approaches, especially in the practically very relevant scenario of transfer learning.

Viewpoint-Invariant Loop Closure Detection Using Step-Wise Learning With Controlling Embeddings of Landmarks

The research on loop closure detection has been carried out for the last many years; however, loop closure detection efficiency is still not that good and has been affected by many factors, including illumination conditions, weather conditions, seasons, and viewpoint changes. The research on loop closure detection from a different viewpoint is still an open research problem. The paper proposes an efficient solution to loop closure detection from different viewpoints by using landmarks instead of whole frames and taking the deep and robust features of deep learning instead of handcrafted features. A different kind of training approach is used to train the deep CNN to get highly abstract embeddings of input landmarks. The approach has many advantages over the traditional training approach and can solve many complex problems. This paper has used this approach to force similar or closer embeddings for similar landmarks and is forced to have a large gap in embeddings of different landmarks. The proposed method endeavors viewpoint invariant features, and the astonishing power of deep learning makes the features robust to viewpoint changes, occlusions, and illumination variations. The proposed visual SLAM system is tested on six publicly available datasets, and the results are compared with the most popular Bag of Words methods like DBoW2, DBoW3, and state-of-the-art deep learning methods AlexNet, ResNeXt, FlyNet, AFDPR and Siamese network. The results show that our method is efficient in finding loop closures candidates from different viewpoints. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/IRMVLab/Step-wise-Learning</uri> .

Spatial and Temporal Hand-Raising Recognition from Classroom Videos Using Locality, Relative Position-Aware Non-Local Networks and Hand Tracking

Hand-raising gesture is one of the most popular signs of communication, whose frequency is related to the classroom’s atmosphere, the attractiveness of the subject, and the level of interactions between students and teachers. However, automatic hand-raising gesture detection and recognition remains still a challenging problem mainly due to low hand resolution, hand occlusion, various backgrounds, and viewpoint changes. While majority of the existing methods focus on static hand-raising posture detection, in this paper, we propose a framework for dynamic hand gesture recognition from classroom videos consisting of two main stages: hand posture detection and dynamic hand gesture recognition. In hand posture detection stage, we extend the previous work by adding relative position-aware in non-local network. After detecting the hand-raising posture on static pixels, which in turn accelerates the performance images, at the dynamic hand gesture recognition stage, we incorporate object detection and tracking to associate the hand-raising detection results at consecutive frames to supplement the missing detection due to the occlusion issue and obtain hand-raising gesture recognition at the event level. The experimental results show that the proposed method outperforms three benchmark models for static hand posture detection that are Faster-RCNN, Libra-RCNN, and Libra-RCNN+RDA on our dataset with the gains obtained for mAP being 7.68%, 5.76%, and 0.35% higher, respectively. In terms of event-level recognition results, the proposed method achieves the value of frame-wise accuracy, temporal_IoU, F1-score@0.3, and Levenshtein-score are 90.0%, 84.4%, 83.2%, and 84.3%, respectively. The code and dataset used in the paper will be made publicity available for research community.

Autonomous Surveillance for an Indoor Security Robot

Conventional surveillance for a security robot suffers from severe limitations, perceptual aliasing (e.g., different places/objects can appear identical), occlusion (e.g., place/object appearance changes between visits), illumination changes, significant viewpoint changes, etc. This paper proposes an autonomous robotic system based on CNN (convolutional neural network) to perform visual perception and control tasks. The visual perception aims to identify all objects moving in the scene and to verify whether the target is an authorized person. The visual perception system includes a motion detection module, a tracking module, face detection, and a recognition module. The control system includes motion control and navigation (path planning and obstacle avoidance). The empirical validation includes the evaluation metrics, such as model speed, accuracy, precision, recall, ROC (receiver operating characteristic) curve, P-R (precision–recall) curve, F1-score for AlexNet, VggNet, and GoogLeNet, and RMSE (root-mean-square error) value of mapping errors. The experimental results showed that the average accuracy of VggNet under four different illumination changes is 0.95, and it has the best performance under all unstable factors among three CNN architectures. For the accuracy of building maps in real scenes, the mapping error is 0.222 m.

Automatic Detection of Construction Workers’ Helmet Wear Based on Lightweight Deep Learning

To reduce the risk of head trauma to workers working in high-risk workplaces such as construction sites, we designed a new automated lightweight end-to-end convolutional neural network to identify whether all people on a construction site are wearing helmets. Firstly, we used GhostNet as the backbone feature extraction network to take advantage of its low running cost and make the model lighter overall while ensuring efficient automatic feature extraction. Secondly, we designed a multi-scale segmentation and feature fusion network (MSFFN) in the feature-processing stage to improve the algorithm’s robustness in detecting objects at different scales. In contrast, the design of the feature fusion network can enrich the diversity of helmet features and improve the accuracy of helmet detection when distance changes, viewpoint changes, and occlusion phenomena occur. Thirdly, we proposed an improved version of the attention mechanism, the lightweight residual convolutional attention network version 2 (LRCA-Netv2). The main idea of the improvement is implemented around the spatial dimension by fusing the combined features along with the horizontal and vertical directions and then weighting them separately. Such an operation allows the establishment of dependencies between the more distant features with improved accuracy compared to the original LRCA-Net. Finally, when tested on the dataset, the proposed lightweight helmet-wearing detection network has a mAP and FPS of 93.5% and 42.

Forest: A Lightweight Semantic Image Descriptor for Robust Visual Place Recognition

Visual place recognition (VPR) is the process of identifying previously visited places using visual information. It is crucial for a robot to achieve fast and accurate VPR under appearance and viewpoint changes. Inspired by human perception intuition, this letter proposes a lightweight semantic image descriptor called Forest for robust VPR. Specifically, for each semantic object in the image, three most recognizable reference objects are first found and ranked in descending order according to a customized measure. And then, categories, position and area of the semantic object itself, along with the categories of the three recognized reference objects, are used to encode the local descriptor of the object. As a result, the set of local descriptors is used as the global descriptor of the image. Performance of the proposed descriptor is evaluated and compared with six state-of-the-art VPR methods, Ibow-LCD, CoHOG, NetVLAD, LOST-X, Region VLAD, Patch-NetVLAD, over four public datasets of urban scenes, Extended-CMU Season, RobotCar Seasons v2, SYNTHIA and Kitti, using three Precision-Recall based metrics, AUC(Area Under Curve), Recall@100%Precision and Precision@100%Recall. The results show that our method maintains competitive accuracy in scenes with appearance and viewpoint changes and exhibits high robustness. At the same time, our method is robust to the noise of semantic segmentation and exhibits high real-time performance.

Object-Plane Co-Represented and Graph Propagation-Based Semantic Descriptor for Relocalization

Relocalization is a critical component of robotics applications, it poses challenges due to changes in lighting conditions, weather, and viewing point. Image feature-based approaches are appearance-sensitive, high-level semantic landmark-based methods are ambiguous, and topological map matching-based methods are not robust enough among available solutions. We propose a highly robust and highly expressive semantic descriptor for graph matching. Specifically, we begin by introducing an object-plane co-represented topological graph and a graph propagation algorithm to formulate descriptors for high-level landmarks; we then solve graph matching using the sKM algorithm. Finally, we develop a relocalization system that combines semantic objects and geometric planes for pose optimization and conducts experimental validation on various datasets. Experimental results demonstrate that the suggested method remains effective even when the viewpoint changes by more than 80 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\circ$</tex-math></inline-formula> on the 2D-3D-S dataset, and the mean orientation error is less than 5 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\circ$</tex-math></inline-formula> on the sceneNN dataset.

Challenges and Possibilities of Educational Equity and Equality in the Post-COVID-19 Realm in the Nordic Countries

ABSTRACT The aim of the article is to offer a base for discussions on educational equality from the viewpoint of changes in the world causing widening disparities between pupils and students. The essence and possibilities of educational equity and equality in the Nordic educational realm are discussed from six viewpoints. The outcome is that all children have the moral right to access equal opportunities even though they are different and unique. Second, we cannot find absolute equality between individuals because all individuals are different and unique. Third, we should find political and pedagogical ways in which to handle educational inequalities. Fourth, students who are not motivated and skilled to undertake independent work may be in danger of being left behind. Fifth, the COVID-19 pandemic has exacerbated the differences between students and diminished the possibilities of students who are vulnerable. Finally, new technologies may open new paths to reduce educational inequality.

DIMENSIONS OF YOUNG ADULT LITERATURE: THE READING ON FIVE FEET APART

This study aims to describe the characteristics of Young Adult Literature genre found in the novel. Young Adult Literature’s characteristics relate to the novel’s intrinsic elements which consists of character, theme, plot, and point of view. The approach used in analysing the novel is structuralism approach. This research uses descriptive qualitative method. The result of the analysis states that Five Feet Apart is categorized into young adult literature since it conforms the genre’s characteristics. The characters of the novel have a physical appearance and the way they behave as well as interact with each other. Moreover, the characters in the novel have a change in attitude and way of thinking. Seen from the analysis, the plot is going fast and the storyline is written in the present. Theme in the novel is about romance because the novel tells the love story of the two young adult people who suffer from lungs illness, named Stella and Will. There are two points of view from young adult protagonists as the main characters in the novel. The point of view changes in every chapter because the story is sometimes told by Stella and sometimes by Will. Finally, it can be concluded that characteristics of Young Adult Literature genre are found in the novel Five Feet Apart. &#x0D;

Robust and Lightweight System for Gait-based Age Estimation towards Viewing Angle Variations

Abstract: Gait based age recognition is a very challenging task, as it involves multiple hurdles such as change in viewpoint of the person. The proposed system handles this problem by performing a sequence of tasks such as GEI formation from silhouette, applying DCT on GEI and extracting the features and finally using MLP for age estimation. The proposed system proves its effectiveness comparing the performance with state of art methods -conventional methods and deep learning based methods. The performance of the system is estimated on OU-MVLP and OULP-Age datasets. The experimental results show the robustness of the system against viewing angle variations. Background: In computer vision applications, gait-based age estimation across several cameras is critical, especially when following the same person in various viewpoints. Objective: To implement the system which adopts lightweight approach for gait-based age estimation. Method: The proposed system uses a combination of the discrete cosine transform (DCT) and multi-layer perceptron (MLP) on gait energy image (GEI) to perform age estimation. Result: The performance of the system is extensively evaluated on the OU-MVLP and OULP-Age datasets. Conclusion: The proposed system attains best mean absolute error (MAE) of 5.05 (in years) for the OU-MVLP dataset, and 5.65 for the OULP dataset.

Multi-Granularity Mutual Learning Network for Object Re-Identification

Object re-identification (re-ID), which is key and fundamental technology for intelligent transportation systems, is a challenging task including person re-ID and vehicle re-ID. It aims to retrieve a given target object from the gallery images captured by different cameras. In this task, it is necessary to extract fine-grained and discriminative features to deal with complex inter-class and intra-class variations caused by the changes of camera viewpoints and object poses. Existing methods focus on learning discriminative local features to improve the re-ID performance. Some state-of-the-art methods use key point detection model to locate local features, which also increases the additional computational cost as side effect. Another type of method focuses on how to learn features of different granularity from rigid stripes of different scales. However, there is little attention paid to how to effectively coalesce multi-granularity features without additional calculation cost. To tackle this issue, this paper proposes the Multi-granularity Mutual Learning Network (MMNet) and makes two contributions. 1) We introduce the multi-granularity jigsaw puzzle module into object re-ID to impel the network to learn local discriminative features from multiple visual granularities by breaking spatial correlation in original images. 2) We propose a parameter-free multi-scale feature reconstruction module to facilitate mutual learning of features at multiple grain levels, thereby both global features and local features have strong representation capabilities. Extensive experiments demonstrate the effectiveness of our proposed modules and the superiority of our method over various state-of-the-art methods on both person and vehicle re-ID benchmarks.

Stress State Change and Fault-Slip Tendency Assessment Associated with Gas Injection and Extraction in the Hutubi (China) Underground Gas Storage

Summary The Hutubi (HTB) gas reservoir in northwest China was active in production from 1998 to 2012; beginning in 2013, it was used for gas storage. Several earthquakes occurred during the early operation of the Hutubi underground gas storage (HTB-UGS) project, so it is important to explore their relationship with gas injection and extraction from the viewpoint of stress state change and faulting susceptibility. We study the evolution of the stress state and slip tendency associated with the operation of the HTB-UGS to understand the relationship between pore pressure variation and induced seismicity. We constrain the stress state in the reservoir at the time of cessation of production in 2012 using image-logged wellbore breakouts. We then estimate stress state variation over time as a result of pore pressure changes, based on the assumption of poroelastic coupling between horizontal elastic stresses and pore pressure. The maximum principal stress in the reservoir region before the HTB-UGS operation is oriented N23°±6°E, which is consistent with that of the prevailing tectonic stress derived from earthquake focal mechanisms. Our results show that the vertical stress (Sv) and the minimum horizontal principal stress (Shmin) were initially similar in magnitude before petroleum production; the magnitude of Shmin gradually decreased during petroleum production. Gas injection, initiated in 2013, raised the Shmin magnitude, returning it to approximately that of Sv. We use the estimated variations in the reservoir stress state over time to calculate temporal changes in slip tendency of the main faults in the reservoir. The fault-slip tendency decreased continuously with petroleum production and then increased with gas injection. The first earthquake swarm associated with gas injection occurred approximately 2 months after the start of injection, possibly due to the slow pore pressure diffusion. Thereafter, earthquakes were induced whenever gas was injected. Our assessment of slip tendency suggests that earthquake swarms were induced during increasing pore pressure phases when slip tendency reached a value of 0.43 ± 0.04. The maximum allowable pore pressure for avoiding earthquakes (ML &amp;gt; 1.5) is 29.0 ± 4.5 MPa, which might increase through stress adjustments and a newly attained balance. This study provides a possible method for induced earthquake analysis associated with gas injection and extraction from the perspective of stress state change and faulting susceptibility.

Train rolling stock video segmentation and classification for bogie part inspection automation: a deep learning approach

Train rolling stock examination (TRSE) is a physical procedure for inspecting the bogie parts during transit at a little over 30 kmph. Currently, this process is manually performed across many railway networks across the world. This work proposes to automate the process of TRSE using artificial intelligence techniques. The previous works have proposed active contour-based models for the segmentation of bogie parts. Though accurate, the models require manual intervention and are found to be iterative making them unsuitable for real-time operations. In this work, we propose a segmentation model followed by a deep learning classifier that can accurately increase the deployability of such systems in real time. We apply the UNet model for the segmentation of bogie parts which are further classified using an attention-based convolutional neural network (CNN) classifier. In this work, we propose a shape deformable attention model to identify shape variations occurring in the video sequence due to viewpoint changes during the train movement. The TRSNet is trained and tested on the high-speed train bogie videos captured across four different trains. The results of the experimentation have been shown to improve the recognition accuracy of the proposed system by 6% over the state-of-the-art classifiers previously developed for TRSE.

Optimal production of remanufactured products with collaboration and co-branding of luxury brands

Remanufactured products belong to the category of environmentally friendly products. Here, we develop models and strategies for collaborating and co-branding of luxury brands with other companies for producing remanufactured products of luxury brands. Then, we examine whether the strategies can coordinate them for producing remanufactured products or not, because of small market share of luxury brands, they should co-brand with a company which has large market share for expanding the market share and selling the remanufactured product. We use a specific name for a brand of remanufactured product because of preventing change of viewpoint of customers to the luxury brand and avoiding brand switching. Producing a new type product should not hurt the status of the luxury brand. Thus, all the possible circumstances should be considered in the process of introduction of a new brand. After development of remanufactured products, we describe different ways and models of advertising for expanding sales by employing differential game models. Overall, we describe the complete activities that are necessary for development of a new brand. As a specific case, we choose the production of remanufactured products by luxury brands which can be a new subject for future research. We find expansion of luxury brand’s market share a new issue. Moreover, producing remanufactured and environmentally friendly products with secondhand materials by luxury brands is a new subject. In the last part, we perform sensitive analysis to examine the effect of parameters on decision variables and presenting managerial insights.

Indian Paper Currency Recognition with Audio Output System for Visually Impaired based on Image Processing using Transfer Learning

Abstract: We give an outline of camera-based computer vision technology in this paper. It can aid those who are not visually blessed in instantly identifying paper currency. For those who are blind or visually challenged, an effective paper money detection algorithm should have the following qualities: 1) Complete precision; and 2) adaptability in a range of circumstances in various environments and emergence. Many of the currency identification algorithms in use today are constrained to specific situations. We suggest a deep learning strategy focused on high accuracy in this project. This method works well for gathering more class-specific data and is better at handling partial closure and viewpoint changes. Evaluation of transfer learning also demonstrate its efficiency in coping with visual rotation, measures, and illumination changes.