Representation Of Visual Content Research Articles

Neural image re-exposure

Images and videos often suffer from issues such as motion blur, video discontinuity, or rolling shutter artifacts. Prior studies typically focus on designing specific algorithms to address individual issues. In this paper, we highlight that these issues, albeit differently manifested, fundamentally stem from sub-optimal exposure processes. With this insight, we propose a paradigm termed re-exposure, which resolves the aforementioned issues by performing exposure simulation. Following this paradigm, we design a new architecture, which constructs visual content representation from images and event camera data, and performs exposure simulation in a controllable manner. Experiments demonstrate that, using only a single model, the proposed architecture can effectively address multiple visual issues, including motion blur, video discontinuity, and rolling shutter artifacts, even when these issues co-occur.

Neuronal representation of visual working memory content in the primate primary visual cortex.

The human ability to perceive vivid memories as if they "float" before our eyes, even in the absence of actual visual stimuli, captivates the imagination. To determine the neural substrates underlying visual memories, we investigated the neuronal representation of working memory content in the primary visual cortex of monkeys. Our study revealed that neurons exhibit unique responses to different memory contents, using firing patterns distinct from those observed during the perception of external visual stimuli. Moreover, this neuronal representation evolves with alterations in the recalled content and extends beyond the retinotopic areas typically reserved for processing external visual input. These discoveries shed light on the visual encoding of memories and indicate avenues for understanding the remarkable power of the mind's eye.

Synthetic Media Analysis Using Deep Learning

In an era characterized by the rapid evolution of digital content creation, synthetic media, particularly deepfake videos, present a formidable challenge to the veracity and integrity of online information. Addressing this challenge requires sophisticated analytical techniques capable of discerning between authentic and manipulated media. This research paper presents a comprehensive study on synthetic media analysis leveraging deep learning methodologies. The suggested method integrates advanced deep learning models, including Convolutional Neural Networks (CNNs) like VGG (Visual Geometry Group), alongside recurrent structures such as LSTM (Long Short-Term Memory). These models are trained and evaluated on a meticulously curated dataset, ensuring diversity and relevance in the synthetic media samples. To facilitate experimentation and reproducibility, the dataset is securely hosted on a reliable platform such as Google Drive. Prior to model training, preprocessing steps including frame extraction are employed to extract essential visual features from the video data. The VGG model serves as a feature extractor, capturing high-level representations of visual content, while the LSTM model learns temporal dependencies and contextual information across frames. Following comprehensive experimentation, the proposed method's ability to detect synthetic media is thoroughly assessed, utilizing metrics such as accuracy. This research contributes to the ongoing discourse on digital media forensics by providing insights into the efficacy of deep learning techniques for synthetic media analysis. The findings underscore the importance of continuous research and development in combating the proliferation of synthetic media, thereby safeguarding the authenticity and trustworthiness of online content. Index Terms—CNN, LSTM, VGG

Recent Advances in Synthesis and Interaction of Speech, Text, and Vision

In recent years, there has been increasing interest in the conversion of images into audio descriptions. This is a field that lies at the intersection of Computer Vision (CV) and Natural Language Processing (NLP), and it involves various tasks, including creating textual descriptions of images and converting them directly into auditory representations. Another aspect of this field is the synthesis of natural speech from text. This has significant potential to improve accessibility, user experience, and the applications of Artificial Intelligence (AI). In this article, we reviewed a wide range of image-to-audio conversion techniques. Various aspects of image captioning, speech synthesis, and direct image-to-speech conversion have been explored, from fundamental encoder–decoder architectures to more advanced methods such as transformers and adversarial learning. Although the focus of this review is on synthesizing audio descriptions from visual data, the reverse task of creating visual content from natural language descriptions is also covered. This study provides a comprehensive overview of the techniques and methodologies used in these fields and highlights the strengths and weaknesses of each approach. The study emphasizes the importance of various datasets, such as MS COCO, LibriTTS, and VizWiz Captions, which play a critical role in training models, evaluating them, promoting inclusivity, and solving real-world problems. The implications for the future suggest the potential of generating more natural and contextualized audio descriptions, whereas direct image-to-speech tasks provide opportunities for intuitive auditory representations of visual content.

Learning-based Compression for Noisy Images in the Wild

Digital images in real world applications typically undergo a wide variety of quality degradations before compression or re-compression. Existing learning based codecs are typically data-driven, relying on the predefined compression pipeline with pristine or high quality images as the input. However, the images in the wild may exhibit the substantially different characteristics compared to the high quality images, casting major challenges to the learning based image coding. In this paper, we propose a robust noisy image compression framework with the blind assumption on the specific noise type and level. The specifically designed encoder decomposes the representation of visual content into two types of features, including the Features that represent the Intrinsic Content (FIC) and the Features that account for Additive Degradation (FAD). As such, beyond the philosophy of faithfully reconstructing the given image with high fidelity, only FIC needs to be compactly represented and conveyed. The principled disentanglement strategy facilitates the removal of the redundancy from multiple perspectives (e.g., spatial, channel and content), ensuring the handling of a wide variety of noisy images in the wild. Extensive experimental results show that our model can achieve superior performance in terms of the ultimate quality and exhibit the strong generalizability across images degraded by a variety of means. The proposed scheme also points out a new research avenue on learning based compression for images in the wild, which is technically challenging but desirable in practice.

A national scale web mapping platform for mainstreaming ecosystem services in Greece

This study outlines the conceptualisation and development of a public participation geographic information system (ppGIS) and web GIS (webGIS) platform for hosting, visualizing, and updating ecosystem services (ES) related data at national scale through participatory mapping. The ppGIS/webGIS is developed within the framework of the LIFE-IP 4 Natura, a project has received the most substantial nature-focused funding scheme for Greece approved by the European Commission. One of the project's central goals is the mapping and assessment of ecosystems and their services (MAES) in Greece. This initiative contributes to the implementation of Action 5 of Target 2 of the European Union (EU) Biodiversity Strategy to 2020, following the standard EU MAES guidelines. The LIFE-IP 4 Natura online mapping infrastructure hosts and visualizes spatially explicit maps and information produced earlier in the project at a national level. It also provides a mechanism for updating this baseline information via public participation, thereby fostering increased public awareness of the necessity for ecosystem protection, conservation, and management. Following a multi-stakeholder user requirement analysis, the herein presented ppGIS/webGIS platform is designed and developed. The platform incorporates various geospatial data explorers (GDE), ppGIS services for local and national public participation, geotools and data, visual content representation, and an introductory-level knowledge repository for ES (ES-pedia). An, overview of the platform's preliminary usage is presented, and its potential in promoting the uptake and mainstreaming of the ES concept is discussed.

Digital Image Representation Model Enriched with Semantic Web Technologies: Visual and Non-Visual Information

The types of content of digital images, visual (syntactic and semantic) and non-visual, cause the complexity of their representation. Considering these contents separately hinders digital image retrieval because this creates a gap between the contents of the image and its representation. Therefore, this work aims to present a representation model of visual and non-visual information of digital images, with semantic enrichment through the Semantic Web technologies. For that, a qualitative methodology with a bibliographical approach was used. Theoretical subsidies of the topics addressed were sought, and it has an applied focus since it proposes a model and its exemplification. The developed model depicts the representation image process and allows the semantic enrichment of the data. This enrichment facilitates the retrieval in multiple contexts with technologies that favor the use of the data through inferences. Also, a use case with digital medical images is presented, demonstrating the feasibility of the proposal. It is concluded that the representation of visual and non-visual content aims to improve the way images are retrieved in digital information environments. The junction of the content and the context of images should be considered, even though search mechanisms usually treat this separately due to the disaggregation of image representation itself.

Antagonism between western and Islamic cultures in Iranian English textbooks

Due to its worldwide usage, the necessity of teaching the English language is undeniable even in Islamic and religious countries like Iran. However, in order to avoid ‘cultural invasion’, the Iranian government decided to ban teaching English in the official curriculum of primary schools and asked a team of experts chosen by the Organization for Educational Research and Planning to design localized English textbooks for students in K12. Thus, in response to curriculum changes and the vital role of textbooks in language education, a plethora of Iranian scholars quantitively investigated the locally designed textbooks in language education focusing on objectives, content, culture, gender representation, and improvement of students’ language skills. However, the previous literature does not consider the role of visual content in transmitting the target language culture. Thus, in this study, we investigated the cultural references and cultural dimensions of visual content representation in two series of English language textbooks called ‘English for Schools’ (Prospect 1-3 and Vision 1-3). Due to the nature of the study, we applied qualitative visual content analysis for grouping and categorizing data. The analysis of the visual content of the textbooks revealed that the utmost portion of all six books has been devoted to

Fake news detection for epidemic emergencies via deep correlations between text and images

In recent years, major emergencies have occurred frequently all over the world. When a major global public heath emergency like COVID-19 broke out, an increasing number of fake news in social media networks are exposed to the public. Automatically detecting the veracity of a news article ensures people receive truthful information, which is beneficial to the epidemic prevention and control. However, most of the existing fake news detection methods focus on inferring clues from text-only content, which ignores the semantic correlations across multimodalities. In this work, we propose a novel approach for Fake News Detection by comprehensively mining the Semantic Correlations between Text content and Images attached (FND-SCTI). First, we learn image representations via the pretrained VGG model, and use them to enhance the learning of text representation via hierarchical attention mechanism. Second, a multimodal variational autoencoder is exploited to learn a fused representation of textual and visual content. Third, the image-enhanced text representation and the multimodal fusion eigenvector are combined to train the fake news detector. Experimental results on two real-world fake news datasets, Twitter and Weibo, demonstrate that our model outperforms seven competitive approaches, and is able to capture the semantic correlations among multimodal contents.

Tracking stimulus representation across a 2-back visual working memory task.

How does the neural representation of visual working memory content vary with behavioural priority? To address this, we recorded electroencephalography (EEG) while subjects performed a continuous-performance 2-back working memory task with oriented-grating stimuli. We tracked the transition of the neural representation of an item (n) from its initial encoding, to the status of ‘unprioritized memory item' (UMI), and back to ‘prioritized memory item', with multivariate inverted encoding modelling. Results showed that the representational format was remapped from its initially encoded format into a distinctive ‘opposite' representational format when it became a UMI and then mapped back into its initial format when subsequently prioritized in anticipation of its comparison with item n + 2. Thus, contrary to the default assumption that the activity representing an item in working memory might simply get weaker when it is deprioritized, it may be that a process of priority-based remapping helps to protect remembered information when it is not in the focus of attention.

Limitations of concurrently representing objects within view and in visual working memory

Representing visibly present stimuli is as limited in capacity as representing invisible stimuli in visual working memory (WM). In this study, we explored whether concurrently representing stimuli within view affects representing objects in visual WM, and if so, whether this effect is modulated by the storage states (active and silent state) of memory contents? In experiment 1, participants were asked to perform the change-detect task in a simultaneous-representing condition in which WM content and the continuously-visible stimuli in view were simultaneously represented, as well as a baseline condition in which only the representations of visual WM content were maintained. The results showed that the representations in visual WM would be impaired when the continuously-visible stimuli in view were concurrently represented, revealed by the reduced CDA amplitude and the lower behavior performance. In experiment 2, a dual-serial retro-cue paradigm was adopted to guide participants to maintain memory items in two different storage states, and the results revealed that simultaneously representing the continuously-visible stimuli and the WM content would only impair the WM representations in the active state. These evidences demonstrated that only the visual WM representations that were maintained in the active state would definitely share the limited resources with the representations of continuously-visible information, and further supported the dissociation between the active state and silent state of visual WM storage.

Learning from visual content and style: an image-enhanced recommendation model

Image based platforms are popular in recent years. With a large number of images in these image based platforms, how to properly recommend images that suit each user’s interest is a key problem for recommender systems. While a simple idea is to adopt collaborative filtering for image recommendation, it does not fully utilize the visual information and suffers from the data sparsity issue. Recently, with the huge success of Convolutional Neural Networks (CNN) for image analysis, some researchers proposed to leverage image content information for recommendation. Specifically, Visual Bayesian Personalized Ranking (VBPR) (He and McAuley, in: The association for the advancement of artificial intelligence, 2016) is a state-of-the-art visual based recommendation model, which proposed to learn users’ preferences to items from two spaces: a visual content space learned from CNNs, and a latent space learned from classical collaborative filtering models. VBPR and its variants showed better recommendation performance with image content modeling. In the real-world, when browsing visual images, users not only care the image content, but also concern the matching degree of the image style. Compared to image content, the role of visual styles has been largely ignored in the image recommendation community. Therefore, in this paper, we study the problem of learning both the visual content and style for image recommendation. We leverage advances in computer vision to learn the visual content and style representation, and propose to how to combine visual signals with users’ collaborative data. Finally, experimental results on a real-world dataset clearly show the effectiveness of our proposed model.

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

This work aims to address the problem of video question answering (VideoQA) with a novel model and a new open-ended VideoQA dataset. VideoQA is a challenging field in visual information retrieval, which aims to generate the answer according to the video content and question. Ultimately, VideoQA is a video understanding task. Efficiently combining the multi-grained representations is the key factor in understanding a video. The existing works mostly focus on overall frame-level visual understanding to tackle the problem, which neglects finer-grained and temporal information inside the video, or just combines the multi-grained representations simply by concatenation or addition. Thus, we propose the multi-granularity temporal attention network that enables to search for the specific frames in a video that are holistically and locally related to the answer. We first learn the mutual attention representations of multi-grained visual content and question. Then the mutually attended features are combined hierarchically using a double layer LSTM to generate the answer. Furthermore, we illustrate several different multi-grained fusion configurations to prove the advancement of this hierarchical architecture. The effectiveness of our model is demonstrated on the large-scale video question answering dataset based on ActivityNet dataset.

Visual content representation and retrieval for Cognitive Cyber Physical Systems

Cognitive Cyber Physical Systems (C-CPS) have gained significant attention from academia and industry during the past few years. One of the main reasons behind this interest is the potential of such technologies to revolutionize human life since they intend to work robustly under complex visual scenes, which environmental conditions may vary, adapting to a comprehensive range of unforeseen changes, and exhibiting prospective behavior like predicting possible events based on cognitive capabilities that are able to sense, analyze, and act based on their analysis results. However, perceiving the environment and translating it into knowledge to be useful for the decision making process, still remains a challenge for real time applications due to the complexity of such process. In this paper, we present a multi-domain knowledge structure based on experience, which can be used as a comprehensive embedded knowledge representation for C-CPS, addressing the representation of visual content issue and facilitating its reuse. The implementation of such representation has been tested in a Cognitive Vision Platform for Hazard Control (CVP-HC) which aims to manage of workers’ exposure to risks in industrial environments, facilitating knowledge engineering processes through a flexible and adaptable implementation.

Blind image quality assessment with hierarchy: Degradation from local structure to deep semantics

Though blind image quality assessment (BIQA) is highly desired in perceptual-oriented image processing systems, it is extremely difficult to design a reliable BIQA method. With the help of the prior knowledge, the human visual system (HVS) hierarchically perceives the quality degradation during the visual recognition. Inspired by this, we suggest different levels of distortion generate individual degradations on hierarchical features, and propose to consider the degradations on both low and high level features for quality prediction. By mimicking the orientation selectivity (OS) mechanism in the primary visual cortex, an OS based local structure is designed for low-level visual information representation. At the meantime, the deep residual network, which possesses multiple levels for feature integration, is employed to extract the deep semantics for high-level visual content representation. By fusing the local structure and the deep semantics, a hierarchical feature set is acquired. Next, the correlations between the degradations of image qualities and their corresponding hierarchical feature sets are analyzed, and a novel hierarchical feature degradation (HFD) based BIQA (HFD-BIQA) method is built. Experimental results on the legacy and wild image quality assessment databases demonstrate the prediction accuracy of the proposed HFD-BIQA method, and verify that the HFD-BIQA performs highly consistent with the subjective perception.

Fast content-aware resizing of multi-layer information visualization via adaptive triangulation

Visual graphics and image-based content have become the pervasive modes of interaction with the digital information flow. With the immense proliferation of display systems and devices, visual content representation has become increasingly challenging. Classical static image resizing algorithms are not directly suitable for the current dynamic information visualization of streaming data flows and processes because most of the visual content often consists of superimposed, multi-layered, multi-scale structure. In this paper, we propose a new adaptive method for content-aware resizing of visual information flow. Scaling is performed by deforming a hierarchical triangle mesh that matches the visual saliency map (VSM) of the streaming data. The VSM is generated automatically based on a series of predefined rules operating on a triangular mesh representation of visual features. We present a linear energy function to minimize distortions of the triangular deformations to perceptually preserve informative content. Through multiple experiments on real datasets, we show that the method has both high performance as well as high robustness in the presence of large differences in the visual aspect ratios between target displays.

Enhanced Just Noticeable Difference Model for Images With Pattern Complexity.

The just noticeable difference (JND) in an image, which reveals the visibility limitation of the human visual system (HVS), is widely used for visual redundancy estimation in signal processing. To determine the JND threshold with the current schemes, the spatial masking effect is estimated as the contrast masking, and this cannot accurately account for the complicated interaction among visual contents. Research on cognitive science indicates that the HVS is highly adapted to extract the repeated patterns for visual content representation. Inspired by this, we formulate the pattern complexity as another factor to determine the total masking effect: the interaction is relatively straightforward with a limited masking effect in a regular pattern, and is complicated with a strong masking effect in an irregular pattern. From the orientation selectivity mechanism in the primary visual cortex, the response of each local receptive field can be considered as a pattern; therefore, in this paper, the orientation that each pixel presents is regarded as the fundamental element of a pattern, and the pattern complexity is calculated as the diversity of the orientation in a local region. Finally, considering both pattern complexity and luminance contrast, a novel spatial masking estimation function is deduced, and an improved JND estimation model is built. Experimental results on comparing with the latest JND models demonstrate the effectiveness of the proposed model, which performs highly consistent with the human perception. The source code of the proposed model is publicly available at http://web.xidian.edu.cn/wjj/en/index.html.

Multi-scale local structure patterns histogram for describing visual contents in social image retrieval systems

Content based image retrieval systems rely heavily on the set of features extracted from images. Effective image representation emerges as a crucial step in such systems. A key challenge in visual content representation is to reduce the so called `semantic gap'. It is the inability of existing methods to describe contents in a human-oriented way. Content representation methods inspired by the human vision system have shown promising results in image retrieval. Considerable work has been carried out during the past two decades for developing methods to extract descriptors inspired by the human vision system and attempt to retrieve visual contents efficiently according to the user needs, thereby reducing the semantic gap. Despite the extensive research being conducted in this area, limitations in current image retrieval systems still exist. This paper presents a descriptor for personalized social image collections which utilizes the local structure patterns in salient edge maps of images at multiple scales. The human visual system at the basic level is sensitive to edges, corners, intersections, and other such intensity variations in images generating local structure patterns. Analyzing these patterns at multiple scales allow the most salient fine-grained and coarse-grained features to be captured. The features are accumulated in a local structure patterns histogram to index images allowing flexible querying of visual contents. The retrieval results show that the proposed descriptor ranks well among similar state-of-the-art methods for large social image collections.

Grounding Distributional Semantics in the Visual World

AbstractDistributional semantic models build vector‐based word meaning representations on top of contextual information extracted from large collections of text. Object recognition methods from computer vision derive vector‐based representations of visual content from natural images. This article reviews how methods from computer vision are exploited to tackle the fundamental problem of grounding distributional semantic models, bringing them closer to providing a full‐fledged computational account of meaning.

Representación de imágenes de histopatología utilizada en tareas de análisis automático: estado del arte

<p>This paper presents a review of the state-of-the-art in histopathology image representation used in automatic image analysis tasks. Automatic analysis of histopathology images is important for building computer-assisted diagnosis tools, automatic image enhancing systems and virtual microscopy systems, among other applications. Histopathology images have a rich mix of visual patterns with particularities that make them difficult to analyze. The paper discusses these particularities, the acquisition process and the challenges found when doing automatic analysis. Second an overview of recent works and methods addressed to deal with visual content representation in different automatic image analysis tasks is presented. Third an overview of applications of image representation methods in several medical domains and tasks is presented. Finally, the paper concludes with current trends of automatic analysis of histopathology images like digital pathology.</p>