Semantic Vision Research Articles

Learning Dual Encoding Model for Adaptive Visual Understanding in Visual Dialogue.

Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue task involves multiple rounds of dialogues which cover a broad range of visual content that could be related to any objects, relationships or high-level semantics. Thus one of the key challenges in Visual Dialogue task is to learn a more comprehensive and semantic-rich image representation that can adaptively attend to the visual content referred by variant questions. In this paper, we first propose a novel scheme to depict an image from both visual and semantic views. Specifically, the visual view aims to capture the appearance-level information in an image, including objects and their visual relationships, while the semantic view enables the agent to understand high-level visual semantics from the whole image to the local regions. Furthermore, on top of such dual-view image representations, we propose a Dual Encoding Visual Dialogue (DualVD) module, which is able to adaptively select question-relevant information from the visual and semantic views in a hierarchical mode. To demonstrate the effectiveness of DualVD, we propose two novel visual dialogue models by applying it to the Late Fusion framework and Memory Network framework. The proposed models achieve state-of-the-art results on three benchmark datasets. A critical advantage of the DualVD module lies in its interpretability. We can analyze which modality (visual or semantic) has more contribution in answering the current question by explicitly visualizing the gate values. It gives us insights in understanding of information selection mode in the Visual Dialogue task. The code is available at https://github.com/JXZe/Learning_DualVD.

The Historical Change in the Role of the Two Hungarian Words for Red (Piros and Vörös) and Their Visual Semantics

The present study summarizes the key findings of a multi-year interdisciplinary investigation, performed using specific (ethnographic, anthropological, and linguistic) research methods, into the two color terms mentioned in the title. Originally intended as empirical research involving all Hungarian color terms and individual community-dependent relationships with colors, it was eventually supplemented by a text-based examination of the history of the color terms piros and veres/vörös. A further objective was to answer questions raised in the course of international research concerning the reason for the existence of two color terms with similar meanings in the Hungarian language to denote the red color range. Earlier studies had already suggested that the modern use of vörös, which has more ancient roots in the Hungarian language, may be related to the fact that this color term was previously used more extensively. However, the present research is unique in demonstrating the substantial changes that have taken place in the Hungarian language in relation to the role and meaning of these color terms. It has already been established that the two color terms switched places historically, and that piros today fulfills precisely the same function that for centuries belonged to veres/vörös, until the color term piros began to gain ground in the 19th century.

RETRACTED: Basketball action recognition based on FPGA and particle image

Fine-grained motion recognition is most important for such video retrieval, and most work nowadays focuses on coarse-grained and fine-grained actions in motion recognition, without being involved in many uses. To solve this problem, in this system, it have a dataset that challenged a basketball game by annotating detailed actions in a video. Adaptive Multi-Label Classification methods for basketball action recognition benchmark also provides data about the system. In addition, this method is proposed to integrate the FPGA into a network of two data streams in order to find the finest areas of basketball action recognition and extracts the features of the recognition system. This proposed system gives significantly a better and superior results than the existing methods. Taken individually, the surrounding first-person footage can be associated with similar situations in the past and compared with the visual semantics of the spatial and social layout of personal records. In general, first-person videos can track common interests, and can be linked to group of individuals in this system.

Cortical network responses map onto data-driven features that capture visual semantics of movie fragments

Research on how the human brain extracts meaning from sensory input relies in principle on methodological reductionism. In the present study, we adopt a more holistic approach by modeling the cortical responses to semantic information that was extracted from the visual stream of a feature film, employing artificial neural network models. Advances in both computer vision and natural language processing were utilized to extract the semantic representations from the film by combining perceptual and linguistic information. We tested whether these representations were useful in studying the human brain data. To this end, we collected electrocorticography responses to a short movie from 37 subjects and fitted their cortical patterns across multiple regions using the semantic components extracted from film frames. We found that individual semantic components reflected fundamental semantic distinctions in the visual input, such as presence or absence of people, human movement, landscape scenes, human faces, etc. Moreover, each semantic component mapped onto a distinct functional cortical network involving high-level cognitive regions in occipitotemporal, frontal and parietal cortices. The present work demonstrates the potential of the data-driven methods from information processing fields to explain patterns of cortical responses, and contributes to the overall discussion about the encoding of high-level perceptual information in the human brain.

A novel deep quality-aware CNN for image edge smoothening

Image-edge smoothing is a practical technique in many image-editing implementations such as Visual Aesthetic Enhancement, Action Movie Production, and Privacy Protection. In the literature, there have been many well-adapted smoothing techniques used for image edges such as Gaussian and Bilateral Blurred utilizing visual semantics perceived by the human visual system. However, it is observed that the previously employed image-edge smoothing techniques are not both content-aware and semantically aware. In other words, the image edges are not smoothsimple and efficient, its performance might be less satisfactory. We tend to smooth semantically non-important regions while smoothing semantically important regions slightly in practice. The motivation behind this is that we propose a deep architecture for semantically aware image-edge smoothing. Hence, provided massive image patches extracted from highly aesthetic images crawled from Flickr, we extract multimodal visual features to characterize each. Then, these multimodal features are optimally combined via a multi-view learning framework. Hence, we employ an active learning framework to extract the visually attractive image patches from each image. Thus, we train a deep CNN to find out the edge attributes of actively selected image patches. Utilizing the trained CNN feature of each image patch, we construct a Gaussian mixture model to conduct image-edge smoothing. To verify the proposed method, extensive experimental runs on images with different styles are employed to present the attractiveness of image edges processed by the proposed method. Besides, the efficiency of the proposed method is compared to the well-known edge smoothing algorithms.

Real distorted images quality assessment based on multi-layer visual perception mechanism and high-level semantics

Most of the existing image quality assessment (IQA) methods are directed to artificially synthesized distorted images, in which the types and characteristics of distortion are different from those in the real world. In view of the fact that the existing non-reference IQA methods can not accurately evaluate the quality of the real distortion image, combined with the theoretical analysis of multi-layer visual perception mechanism, we propose a real image distortion IQA method based on image underlying features and high-level semantics. Considering non-linear hierarchical structure of human visual perception, firstly, k-means clustering algorithm is performed according to the underlying feature indexs of the image so that the used image database can be divided into several groups, which aims to improve the accuracy of predicted quality score. Secondly, the deep convolutional neural network (DCNN) is used to extract the first-grade high-level semantic features in each group. Then, second-grade high-level semantic features that can provide better representation of image features are obtained by performing multiple statistical functions on first-grade high-level semantics. Besides, we establish an effective high-capacity regressor with high-level semantics and subjective mean opinion scores (MOS) values of the human eyes. The experimental results show that the proposed model on the KonIQ-10 k image database can predict the quality score effectively and achieve a high consistency with the corresponding MOS value, which is helpful for the subsequent image enhancement.

B5G and Explainable Deep Learning Assisted Healthcare Vertical at the Edge: COVID-I9 Perspective

B5G-based tactile edge learning shows promise as a solution to handle infectious diseases such as COVID-19 at a global level. By leveraging edge computing with the 5G RAN, management of epidemic diseases such as COVID-19 can be conducted efficiently. Deploying a hierarchical edge computing architecture offers several benefits such as scalability, low latency, and privacy for the data and the training model, which enables analysis of COVID-19 by a local trusted edge server. However, existing deep learning (DL) algorithms suffer from two crucial drawbacks: first, the training requires a large COVID-19 dataset on various dimensions, which is difficult for any local authority to manage. Second, the DL results require ethical approval and explanations from healthcare providers and other stakeholders in order to be accepted. In this article, we propose a B5G framework that supports COVID-19 diagnosis, leveraging the low-latency, high-bandwidth features of the 5G network at the edge. Our framework employs a distributed DL paradigm where each COVID-19 edge employs its own local DL framework and uses a three-phase reconciliation with the global DL framework. The local DL model runs on edge nodes while the global DL model runs on a cloud environment. The training of a local DL model is performed with the dataset available from the edge; it is applied to the global model after receiving approval from the subject matter experts at the edge. Our framework adds semantics to existing DL models so that human domain experts on COVID-19 can gain insight and semantic visualization of the key decision-making activities that take place within the deep learning ecosystem. We have implemented a subset of various COVID-19 scenarios using distributed DL at the edge and in the cloud. The test results are promising.

Interactive Natural Language Grounding via Referring Expression Comprehension and Scene Graph Parsing.

Natural language provides an intuitive and effective interaction interface between human beings and robots. Currently, multiple approaches are presented to address natural language visual grounding for human-robot interaction. However, most of the existing approaches handle the ambiguity of natural language queries and achieve target objects grounding via dialogue systems, which make the interactions cumbersome and time-consuming. In contrast, we address interactive natural language grounding without auxiliary information. Specifically, we first propose a referring expression comprehension network to ground natural referring expressions. The referring expression comprehension network excavates the visual semantics via a visual semantic-aware network, and exploits the rich linguistic contexts in expressions by a language attention network. Furthermore, we combine the referring expression comprehension network with scene graph parsing to achieve unrestricted and complicated natural language grounding. Finally, we validate the performance of the referring expression comprehension network on three public datasets, and we also evaluate the effectiveness of the interactive natural language grounding architecture by conducting extensive natural language query groundings in different household scenarios.

MRA-Net: Improving VQA Via Multi-Modal Relation Attention Network.

Visual Question Answering (VQA) is a task to answer natural language questions tied to the content of visual images. Most recent VQA approaches usually apply attention mechanism to focus on the relevant visual objects and/or consider the relations between objects via off-the-shelf methods in visual relation reasoning. However, they still suffer from several drawbacks. First, they mostly model the simple relations between objects, which results in many complicated questions cannot be answered correctly, because of failing to provide sufficient knowledge. Second, they seldom leverage the harmony cooperation of visual appearance feature and relation feature. To solve these problems, we propose a novel end-to-end VQA model, termed Multi-modal Relation Attention Network (MRA-Net). The proposed model explores both textual and visual relations to improve performance and interpretability. In specific, we devise 1) a self-guided word relation attention scheme, which explore the latent semantic relations between words; 2) two question-adaptive visual relation attention modules that can extract not only the fine-grained and precise binary relations between objects but also the more sophisticated trinary relations. Both kinds of question-related visual relations provide more and deeper visual semantics, thereby improving the visual reasoning ability of question answering. Furthermore, the proposed model also combines appearance feature with relation feature to reconcile the two types of features effectively. Extensive experiments on five large benchmark datasets, VQA-1.0, VQA-2.0, COCO-QA, VQA-CP v2, and TDIUC, demonstrate that our proposed model outperforms state-of-the-art approaches.

Deep Multi-Patch Matching Network for Visible Thermal Person Re-Identification

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Visible Thermal Person Re-Identification</i> (VTReID) is a cross-modality retrieval problem in computer vision. Accurate VTReID is very challenging due to large modality discrepancies. In this work, we design a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Multi-Patch Matching Network</i> (MPMN) framework to simultaneously mitigate the heterogeneity of coarse-grained and fine-grained visual semantics. In view of cross-modality matching, we verify that aligning modality distributions of the original features is likely to suffer from the selective alignment behavior, i.e., only focuses on easiest dimensions or subspaces. Inspired by adversarial learning, we propose a new <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Multi-Patch Modality Alignment</i> (MPMA) loss to jointly balance and reduce the modality discrepancies of multi-patch features by mining hard subspaces and abandoning easy subspaces. Since multi-patch features are potentially complementary to each other, the semantic correlations between different patches should be exploited during training. Motivated by knowledge distillation, we put forward a new <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cross-Patch Correlation Distillation</i> (CPCD) loss to transfer the semantic knowledges across different patches. To balance multi-patch tasks, an effective <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Patch-Aware Priority Attention</i> (PAPA) method is further introduced to dynamically prioritize hard patch tasks during training. This paper experimentally demonstrates the effectiveness of the proposed methods, achieving superior performance over the state-of-the-art methods on RegDB and SYSU-MM01 datasets.

DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue

Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue involves multiple questions which cover a broad range of visual content that could be related to any objects, relationships or semantics. The key challenge in Visual Dialogue task is thus to learn a more comprehensive and semantic-rich image representation which may have adaptive attentions on the image for variant questions. In this research, we propose a novel model to depict an image from both visual and semantic perspectives. Specifically, the visual view helps capture the appearance-level information, including objects and their relationships, while the semantic view enables the agent to understand high-level visual semantics from the whole image to the local regions. Futhermore, on top of such multi-view image features, we propose a feature selection framework which is able to adaptively capture question-relevant information hierarchically in fine-grained level. The proposed method achieved state-of-the-art results on benchmark Visual Dialogue datasets. More importantly, we can tell which modality (visual or semantic) has more contribution in answering the current question by visualizing the gate values. It gives us insights in understanding of human cognition in Visual Dialogue.

Deep hard modality alignment for visible thermal person re-identification

Visible Thermal Person Re-Identification (VTReID) is essentially a cross-modality problem and widely encountered in real night-time surveillance scenarios, which is still in need of vigorous performance improvement. In this work, we design a simple but effective Hard Modality Alignment Network (HMAN) framework to learn modality-robust features. Since current VTReID works do not consider the cross-modality discrepancy imbalance, their models are likely to suffer from the selective alignment behavior. To solve this problem, we propose a novel Hard Modality Alignment (HMA) loss to simultaneously balance and reduce the modality discrepancies. Specifically, we mine the hard feature subspace with large modality discrepancies and abandon the easy feature subspace with small modality discrepancies to make the modality distributions more distinguishable. For mitigating the discrepancy imbalance, we pay more attention on reducing the modality discrepancies of the hard feature subspace than that of the easy feature subspace. Furthermore, we propose to jointly relieve the modality heterogeneity of global and local visual semantics to further boost the cross-modality retrieval performance. This paper experimentally demonstrates the effectiveness of the proposed method, achieving superior performance over the state-of-the-art methods on RegDB and SYSU-MM01 datasets.

Reasoning on the Relation: Enhancing Visual Representation for Visual Question Answering and Cross-Modal Retrieval

Cross-modal analysis has become a promising direction for artificial intelligence. Visual representation is crucial for various cross-modal analysis tasks that require visual content understanding. Visual features which contain semantical information can disentangle the underlying correlation between different modalities, thus benefiting the downstream tasks. In this paper, we propose a Visual Reasoning and Attention Network (VRANet) as a plug-and-play module to capture rich visual semantics and help to enhance the visual representation for improving cross-modal analysis. Our proposed VRANet is built based on the bilinear visual attention module which identifies the critical objects. We propose a novel Visual Relational Reasoning (VRR) module to reason about pair-wise and inner-group visual relationships among objects guided by the textual information. The two modules enhance the visual features at both relation level and object level. We demonstrate the effectiveness of the proposed VRANet by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments conducted on VQA 2.0, CLEVR, CMPlaces, and MS-COCO datasets indicate superior performance comparing with state-of-the-art work.

Hand gesture understanding by weakly-supervised fusing shallow/deep image attributes

Accurately recognizing human hand gestures is a useful component in many modern intelligent systems, such as identification authentication, human–computer interaction, and sign language recognition. Conventional approaches are typically based on shallow visual features and relatively simple backgrounds, which cannot readily recognize partially occluded hand gestures with sophisticated backgrounds. In this work, we propose a unified hand gesture recognition framework by optimally fusing a set of shallow/deep finger-level image attributes, based on which a weakly-supervised ranking algorithm is designed to select semantically salient regions for gesture understanding. More specifically, given a rich number of hand gesture images, we employ the well-known BING object proposal generator to extract hundreds of object patches that potentially draw human visual attention. Since the hundreds of object patches are still too many for building an effective recognition system, a weakly-supervised metric is proposed to rank them by extracting multiple shallow/deep features. And visual semantics are encoded at region-level by transferring the image-level semantic tags into various human gesture image regions by a weakly-supervised learning paradigm Apparently, the top-ranking highly salient object patches are highly indicative to human visual perception of human hand gesture, Thus we extract their ImageNet-CNN features and further concatenate them. Finally, the concatenated deep feature is fed into a multi-class SVM for classifying each hand gesture image into a particular type. Comprehensive experimental validations have demonstrated the effectiveness and robustness of our proposed hybrid-feature-based hand gesture categorization.

Classification and Risk-Mapping of River Water Quality in Surabaya with Semantic Visualitzation

River water pollution is one of the environmental problems that occur in Surabaya. The amount of industrial waste and household waste makes Surabaya River water easily polluted every day, besides that there are also many people who are not aware about the quality of river water in Surabaya. In this paper, we present a new system to classify water quality of river in surabaya. The system involve a semantic visualization of risk-mapping for the river, so that the people of Surabaya are easier to get information about the quality of Surabaya River water. In this paper, we measured the water quality of Surabaya River using Horiba sensor measuring instruments using 5 parameters, namely temperature, PH, DO, Turbidity, TDS. These five parameters are input variables for calculating water quality with the methods applied in this research. We use the Storet Method to determine the quality of Surabaya River water. The results of the Storet Method explained that there were 0.03% of the data on lightly polluted water quality and there were 37.41% of the data being moderately polluted and there were 59.29% of the data heavily polluted. The results of the calculation using the Storet method concluded that the condition of Surabaya River water quality was not good. We also apply the rule of the Storet Method to the Neural Network by using Surabaya River water quality data as learning data and gave performance 70.02% accuracy.

SeDAR: Reading Floorplans Like a Human\u2014Using Deep Learning to Enable Human-Inspired Localisation

The use of human-level semantic information to aid robotic tasks has recently become an important area for both Computer Vision and Robotics. This has been enabled by advances in Deep Learning that allow consistent and robust semantic understanding. Leveraging this semantic vision of the world has allowed human-level understanding to naturally emerge from many different approaches. Particularly, the use of semantic information to aid in localisation and reconstruction has been at the forefront of both fields. Like robots, humans also require the ability to localise within a structure. To aid this, humans have designed high-level semantic maps of our structures called floorplans. We are extremely good at localising in them, even with limited access to the depth information used by robots. This is because we focus on the distribution of semantic elements, rather than geometric ones. Evidence of this is that humans are normally able to localise in a floorplan that has not been scaled properly. In order to grant this ability to robots, it is necessary to use localisation approaches that leverage the same semantic information humans use. In this paper, we present a novel method for semantically enabled global localisation. Our approach relies on the semantic labels present in the floorplan. Deep Learning is leveraged to extract semantic labels from RGB images, which are compared to the floorplan for localisation. While our approach is able to use range measurements if available, we demonstrate that they are unnecessary as we can achieve results comparable to state-of-the-art without them.

Find and Seek

Web designers use visual cues such as layout and typography to make pages easier to navigate and understand. Yet, screen readers generally ignore these features and present page information in a linear audio stream. We investigate whether transcoding the visual semantics of grid-based layouts to tables supports better navigation. In a controlled experiment, participants navigated re-written pages significantly faster when doing data synthesis tasks and more accurately when looking up information meeting multiple criteria. Participants rated their table navigation experience better in terms of effort, memorization, ease of navigation, understanding of page information, and confidence in submitted answers. Participants attributed these gains to the table structure’s support for (1) predictable audio presentation, (2) adopting an appropriate search strategy, and (3) making sense of page content. Contrary to the established belief that tables are inaccessible, our results show that tables can facilitate navigation when users need to synthesize across page content.

The Technology of Media Culture usage in the Educational Environment: Development of Creative Self-Actualization

The article looks at the experience of building a media environment at school as a system of self-education and self-development for modern teenagers. The authors present a model of development of teenagers’ creative self-actualization in the media educational environment, the technology of media culture use in the educational environment and an original diagnostic methodology for determination of the degree of development of school students’ creative self-actualization based on assessment of independent work carried out by teenagers (media projects creation) taking into account emotional and semantic visualization of information and the symbolism of associative-semantic connections to evaluate the quality of the conducted work. As a result of the performed experiment, the authors identified personal characteristics shown by school students in the course of their work on media projects and developed a typology of dominant activities that includes four types of individual characteristics observed in school students: analytical, informational, proactive and synthetic types. Based on this typology the authors have determined the relevance and necessity for differentiation of work with teenagers of different ages in various forms of media educational environment and development of their individual educational paths. Correspondingly, a four-stage mechanism of introduction of a teenager into a media educational environment has been suggested providing for the integration of art into the educational environment. The article presents the results of a pilot project that was carried out in state budgetary educational institutions located in Moscow: School no. 1253, School no. 171, School no. 89 and in the children's creative center Trubetskoy Estate in Khamovniki. The article is intended for scientists, teachers, pedagogues and methodological experts who deal with the issues of media education.

Object manipulation with a variable-stiffness robotic mechanism using deep neural networks for visual semantics and load estimation

In recent years, the computer vision applications in the robotics have been improved to approach human-like visual perception and scene/context understanding. Following this aspiration, in this study, we explored the possibility of better object manipulation performance by connecting the visual recognition of objects to their physical attributes, such as weight and center of gravity (CoG). To develop and test this idea, an object manipulation platform is built comprising a robotic arm, a depth camera fixed at the top center of the workspace, embedded encoders in the robotic arm mechanism, and microcontrollers for position and force control. Since both the visual recognition and force estimation algorithms use deep learning principles, the test set-up was named as Deep-Table. The objects in the manipulation tests are selected from everyday life and are common to be seen on modern office desktops. The visual object localization and recognition processes are performed from two distinct branches by deep convolutional neural network architectures. We present five of the possible cases, having different levels of information availability on the object weight and CoG in the experiments. The results confirm that using our algorithm, the robotic arm can move different types of objects successfully varying from several grams (empty bottle) to around 250 g (ceramic cup) without failure or tipping. The proposed method also shows that connecting the object recognition with load estimation and contact point further improves the performance characterized by a smoother motion.

Synthesis of Image from Text using Generative Adversarial Networks

One of the primary applications of recent conditional generative models is the generation of images from natural language. In addition to the testing ability to these model conditional, extremely dimensional distributions, text-to-image synthesis has several exciting and sensible applications such as photo editing or computer-aided creation of content. Using Generative Adversarial Networks (GANs), recent progress has been made. This project begins with a gentle introduction to these topics and the current state of the art models are discussed. Moreover, here propose a distinct deep architecture and GAN formulation to efficiently bridge the progress of text and image modelling, visual thoughts from characters to pixels. It demonstrates the capability of this model to generate plausible images of birds and flowers from detailed text descriptions. In addition, Wasserstein GAN-CLS is proposing a new model for conditional image generation based on the distance from Wasserstein offering stability guarantees. Then shows how the new loss function of Wasserstein GAN-CLS can be used by a Conditional Progressive Growing GAN. The model, combined with the proposed loss, boosts the model’s best Inception Score (on the Caltech bird’s dataset) by 7.07 %, which uses only the visual semantics at the sentence level. Only the model that performs better than the Conditional Wasserstein Progressive GAN is the newly proposed AttnGAN, which also uses visual semiconductors at the word level.