Visual Concepts Research Articles

An inherently interpretable deep learning model for local explanations using visual concepts.

Over the past decade, deep learning has become the leading approach for various computer vision tasks and decision support systems. However, the opaque nature of deep learning models raises significant concerns about their fairness, reliability, and the underlying inferences they make. Many existing methods attempt to approximate the relationship between low-level input features and outcomes. However, humans tend to understand and reason based on high-level concepts rather than low-level input features. To bridge this gap, several concept-based interpretable methods have been developed. Most of these methods compute the importance of each discovered concept for a specific class. However, they often fail to provide local explanations. Additionally, these approaches typically rely on labeled concepts or learn directly from datasets, leading to the extraction of irrelevant concepts. They also tend to overlook the potential of these concepts to interpret model predictions effectively. This research proposes a two-stream model called the Cross-Attentional Fast/Slow Thinking Network (CA-SoftNet) to address these issues. The model is inspired by dual-process theory and integrates two key components: a shallow convolutional neural network (sCNN) as System-I for rapid, implicit pattern recognition and a cross-attentional concept memory network as System-II for transparent, controllable, and logical reasoning. Our evaluation across diverse datasets demonstrates the model's competitive accuracy, achieving 85.6%, 83.7%, 93.6%, and 90.3% on CUB 200-2011, Stanford Cars, ISIC 2016, and ISIC 2017, respectively. This performance outperforms existing interpretable models and is comparable to non-interpretable counterparts. Furthermore, our novel concept extraction method facilitates identifying and selecting salient concepts. These concepts are then used to generate concept-based local explanations that align with human thinking. Additionally, the model's ability to share similar concepts across distinct classes, such as in fine-grained classification, enhances its scalability for large datasets. This feature also induces human-like cognition and reasoning within the proposed framework.

Focus Group Interview: University-Industrial Collaboration

Business innovations can introduce novel and imaginative concepts, particularly within digitalization. Innovations have the potential to stimulate the advancement of enterprises, fostering market expansion, enhancing competitiveness, and optimizing profitability. Nevertheless, the implementation of innovation may be achievable in collaboration. Collaborations and interactions between academia and industrial entities may yield improved outcomes. Nevertheless, the presence of obstacles between academia and industries could harm the long-term viability of commercial innovation. The concerns and challenges faced by local institutions and enterprises in Malaysia are examined within the framework of a case study. Data was obtained from a random sample of Chief Executive Officers (CEOs) across several industries. A focus group was conducted with 15 representatives from Small and Medium Enterprise (SME) companies and one moderator to lead the discussion. The findings from the focus group have been transcribed and analyzed based on their sharing. The rationale for focusing on CEOs was their role as critical decision-makers responsible for driving corporate innovations. The collection of perspectives from academia and industries was facilitated through face-to-face interviews. The study provided suggestions for potential avenues of future investigation.

Design for shared driverless vehicles of the future

On-demand shared transportation is a major new mobility innovation and potentially the main mode of transport in coming decades. Studies show that driverless vehicles have potential to accelerate uptake of shared vehicles at scale. People perceive sharing positively but do not necessarily translate perception into action, with desire for personal space a major reason for unwillingness to share vehicles. Design research is a powerful tool when creating methods and processes to anticipate future possibilities by visualising detailed features of proposed products. We present a set of design research methods engaging end users in a variety of empathy activities and a design process to translate their needs into visual concepts for future shared driverless vehicles that are attractive and more likely to be adopted.

Ferrimagnet-Based Neuromorphic Device Mimicking the Ventral Visual Pathway for High-Accuracy Target Recognition.

The ventral visual pathway (VVP) of the human brain efficiently implements target recognition by employing a deep hierarchical structure to build complex visual concepts from simple features. Artificial neural networks (ANNs) based on spintronic devices are capable of target recognition, but their poor interpretability and limited network depth hinder ANNs from mimicking the VVP. Hardware implementation of the VVP requires a biorealistic spintronic device as well as the corresponding interpretable and deep network structure, which have not been reported so far. Here, we report a ferrimagnetic neuron with a continuously differentiable exponential linear unit (CeLu) activation function, which is closer to biological neurons and could mitigate the issue of limited network depth. Meanwhile, we also demonstrate that a ferrimagnet can construct artificial synapses with high linearity and symmetry to meet the requirements of weight update algorithms. Based on these neurons and synapses, we propose an all-spin convolutional neural network (CNN) with a high interpretability and deep neural network, to mimic the VVP. Compared to the state-of-the-art spintronic-based neuromorphic computing model, the CNN with bionic function, using experimentally derived device parameters, achieves high recognition accuracies of over 91% and 98% on the CIFAR-10 datasets and the MNIST datasets, respectively, showing improvements of 1.13% and 1.76%. Our work provides a promising method to improve the bionic performance of spintronic device-based neural networks.

Estimation of the Weight and Volume of Lime (Citrus aurantifolia (Christm.) Swingle) Fruit Using Computer Vision Based on Traditional Machine Learning and Deep Learning

The post-harvest process is important to increasing the market value of limes and requires focus. During this process, limes are graded and categorized based on size, weight, and volume. Therefore, identifying efficient means of estimating these properties is very important and remains an open research area. This study applies the concept of computer vision based on traditional machine learning algorithms (partial least square regression (PLS), epsilon-support vector regression (ε-SVR), decision tree (DT), random forest (RF), adaptive boosting (AB), gradient boosting (GB), Bagging meta-estimator (BME), and extremely randomized trees (ERTs)) and pre-trained deep learning (InceptionV3, MoblieNetV2, ResNet50, and VGG-16) for estimating the weight and volume of limes. Our findings showed that the BME and ResNet50 could yield the highest performance for estimating the weight and volume of limes. The BME produced Rtest2 values of 0.954 and 0.882 for weight and volume, respectively, while the Rtest2 values of ResNet50 models were between 0.951 and 0.957 for weight and volume, respectively. This study concluded that computer vision based on both traditional machine learning and deep learning could be used to estimate the weight and volume of limes. The approach proposed in this study can be adopted for applications related to computer vision in the post-harvest process.

Multiexposed Image-Fusion Strategy Using Mutual Image Translation Learning with Multiscale Surround Switching Maps

The dynamic range of an image represents the difference between its darkest and brightest areas, a crucial concept in digital image processing and computer vision. Despite display technology advancements, replicating the broad dynamic range of the human visual system remains challenging, necessitating high dynamic range (HDR) synthesis, combining multiple low dynamic range images captured at contrasting exposure levels to generate a single HDR image that integrates the optimal exposure regions. Recent deep learning advancements have introduced innovative approaches to HDR generation, with the cycle-consistent generative adversarial network (CycleGAN) gaining attention due to its robustness against domain shifts and ability to preserve content style while enhancing image quality. However, traditional CycleGAN methods often rely on unpaired datasets, limiting their capacity for detail preservation. This study proposes an improved model by incorporating a switching map (SMap) as an additional channel in the CycleGAN generator using paired datasets. The SMap focuses on essential regions, guiding weighted learning to minimize the loss of detail during synthesis. Using translated images to estimate the middle exposure integrates these images into HDR synthesis, reducing unnatural transitions and halo artifacts that could occur at boundaries between various exposures. The multilayered application of the retinex algorithm captures exposure variations, achieving natural and detailed tone mapping. The proposed mutual image translation module extends CycleGAN, demonstrating superior performance in multiexposure fusion and image translation, significantly enhancing HDR image quality. The image quality evaluation indices used are CPBDM, JNBM, LPC-SI, S3, JPEG_2000, and SSEQ, and the proposed model exhibits superior performance compared to existing methods, recording average scores of 0.6196, 15.4142, 0.9642, 0.2838, 80.239, and 25.054, respectively. Therefore, based on qualitative and quantitative results, this study demonstrates the superiority of the proposed model.

Advanced Personal Budget Analytics: Combining Optical Character Recognition and Natural Language Processing for Automated Budget Categorization and Insight Extraction

A budget tracking system is a tool designed to help users monitor their financial resources, categorize expenses and control their financial resources. An Intelligent Personal Budget Analysis System is designed to redefine personal finance management. IntelliFinance is a user- friendly app designed for seamless and efficient daily expense management removing the necessity for conventional paper documentation. This project inculcates the concepts of Computer Vision, Machine Learning and Data Analytics, aims to surpass traditional budget tracking by incorporating innovative features such as receipt scanning and analysis. The project is implemented using Android Studio and Firebase, incorporating advanced features such as Optical Character Recognition (OCR) to capture receipt images, extract text, and process it for further analysis. Firebase ML Kit is utilized for scanning and text recognition, while Natural Language Processing (NLP) is applied for categorizing the extracted text. The system provides a personal budget analysis that enhances the accuracy of financial tracking by offering users a comprehensive overview of their spending habits, complemented by detailed reports and visualizations. Ultimately, this project serves as a financial companion, empowering users to take control of their finances, make informed decisions, and achieve their financial goals.

Lifelong Generative Adversarial Autoencoder.

Lifelong learning describes an ability that enables humans to continually acquire and learn new information without forgetting. This capability, common to humans and animals, has lately been identified as an essential function for an artificial intelligence system aiming to learn continuously from a stream of data during a certain period of time. However, modern neural networks suffer from degenerated performance when learning multiple domains sequentially and fail to recognize past learned tasks after being retrained. This corresponds to catastrophic forgetting and is ultimately induced by replacing the parameters associated with previously learned tasks with new values. One approach in lifelong learning is the generative replay mechanism (GRM) that trains a powerful generator as the generative replay network, implemented by a variational autoencoder (VAE) or a generative adversarial network (GAN). In this article, we study the forgetting behavior of GRM-based learning systems by developing a new theoretical framework in which the forgetting process is expressed as an increase in the model's risk during the training. Although many recent attempts have provided high-quality generative replay samples by using GANs, they are limited to mainly downstream tasks due to the lack of inference. Inspired by the theoretical analysis while aiming to address the drawbacks of existing approaches, we propose the lifelong generative adversarial autoencoder (LGAA). LGAA consists of a generative replay network and three inference models, each addressing the inference of a different type of latent variable. The experimental results show that LGAA learns novel visual concepts without forgetting and can be applied to a wide range of downstream tasks.

KRITIK TERHADAP PENGAJARAN VISUALIASI DAN AFIRMASI TEOLOGI KEMAKMURAN BERDASARKAN EKSPOSISI KEJADIAN 15:5-6

What faith is and how to express it are two essential questions that seem easy but are complicated to answer. According to the prosperity gospel adherents, faith must be expressed through imagining (visualization) and saying (affirmation), as written in Genesis 15:5-6. However, for some groups, this teaching deviates from the truth according to the Bible and is more similar to the law of attraction. This article intends to criticize the visualization concept and to explore what is faith according to Genesis 15:5-6 by using the library research method combined with text exposition. The conclusion is that according to Genesis 15:5-6, true faith focuses on God's will entirely without the help of the power of human thoughts and words. Thus, visualization teaching, apart from being unbiblical, is also humanistic and tends to lead to syncretism.

The Semiotics of Place in Saudi Women’s Poetry: Ashjan Hindi as a Model

This research delves into the artistic structures and semiotic connotations of place in the poetry of Ashjan Hindi. In her poetry, place is not merely a tangible geographical patch, but rather it is rich with symbolic meanings and connotations that reflect the poet's philosophical vision of her surroundings. Therefore, it became necessary to explore the semiotic perspectives of place through analysis and interpretation. The research adopted a semiotic approach to investigate the artistic structures of place in Ashjan Hindi's poetry, drawing on the foundational concepts of the metaphorical discourse according to Paul Ricoeur in his book "The Vivid Metaphor," which is rooted in the linguistic theory formulated by Emile Benveniste. The research relies on exploring the semiotics of place in the Collection of Poems, drawing on the procedural concepts of imagination as articulated by the researcher Jilbert Durand in his book "The Anthropological Structures of the Imaginary." It is based on the foundational concepts of this theory, especially the terms of patterns, original models, and symbols, in analyzing the connotations of places in the Collection of Poems.

Multimodality-guided Visual-Caption Semantic Enhancement

Video captions generated with single modality, e.g. video clips, often suffer from insufficient event discovery and inadequate scene description. Therefore, this paper aims to improve the quality of captions by addressing these issues through the integration of multi-modal information. Specifically, We first construct a multi-modal dataset and introduce the triplet annotations of video, audio and text, fostering a comprehensive exploration about the associations between different modalities. Build upon this, We propose to explore the collaborative perception of audio and visual concepts to mitigate inaccuracies and incompleteness in captions in vision-based benchmarks by incorporating audio-visual perception priors. To achieve this, we extract effective semantic features from visual and auditory modalities, bridge the semantic gap between audio-visual modalities and text, and form a more precise knowledge graph multimodal coherence checking and information pruning mechanism. Exhaustive experiments demonstrate that the proposed approach surpasses existing methods and generalizes well with the assistance of ChatGPT.

Related work analysis for determination of fatigue state based on eye movements monitoring

We have reviewed theoretical background of detecting functional state of fatigue based on the strategy of eye movements. Also, modern methods for assessing eye movements were considered. Based on our literature review, we can conclude that nowadays there are multitude numerical characteristics of eye movements, the dynamics of which can hypothetically make it possible to assess degree of fatigue. However, there are still no proposals for a method for determining the degree of fatigue based on an analysis of the strategy of eye movements. In this regard, according to the concepts of static and dynamic vision, it is proposed to consider the shift in the numerical characteristics of eye movements towards characteristics that reflect the strategy of dynamic vision as evidence of fatigue.

A Spatio-Temporal Capsule Neural Network with Self-Correlation Routing for EEG Decoding of Semantic Concepts of Imagination and Perception Tasks.

Decoding semantic concepts for imagination and perception tasks (SCIP) is important for rehabilitation medicine as well as cognitive neuroscience. Electroencephalogram (EEG) is commonly used in the relevant fields, because it is a low-cost noninvasive technique with high temporal resolution. However, as EEG signals contain a high noise level resulting in a low signal-to-noise ratio, it makes decoding EEG-based semantic concepts for imagination and perception tasks (SCIP-EEG) challenging. Currently, neural network algorithms such as CNN, RNN, and LSTM have almost reached their limits in EEG signal decoding due to their own short-comings. The emergence of transformer methods has improved the classification performance of neural networks for EEG signals. However, the transformer model has a large parameter set and high complexity, which is not conducive to the application of BCI. EEG signals have high spatial correlation. The relationship between signals from different electrodes is more complex. Capsule neural networks can effectively model the spatial relationship between electrodes through vector representation and a dynamic routing mechanism. Therefore, it achieves more accurate feature extraction and classification. This paper proposes a spatio-temporal capsule network with a self-correlation routing mechaninsm for the classification of semantic conceptual EEG signals. By improving the feature extraction and routing mechanism, the model is able to more effectively capture the highly variable spatio-temporal features from EEG signals and establish connections between capsules, thereby enhancing classification accuracy and model efficiency. The performance of the proposed model was validated using the publicly accessible semantic concept dataset for imagined and perceived tasks from Bath University. Our model achieved average accuracies of 94.9%, 93.3%, and 78.4% in the three sensory modalities (pictorial, orthographic, and audio), respectively. The overall average accuracy across the three sensory modalities is 88.9%. Compared to existing advanced algorithms, the proposed model achieved state-of-the-art performance, significantly improving classification accuracy. Additionally, the proposed model is more stable and efficient, making it a better decoding solution for SCIP-EEG decoding.

FICE: Text-conditioned fashion-image editing with guided GAN inversion

Fashion-image editing is a challenging computer-vision task where the goal is to incorporate selected apparel into a given input image. Most existing techniques, known as Virtual Try-On methods, deal with this task by first selecting an example image of the desired apparel and then transferring the clothing onto the target person. Conversely, in this paper, we consider editing fashion images with text descriptions. Such an approach has several advantages over example-based virtual try-on techniques: (i) it does not require an image of the target fashion item, and (ii) it allows the expression of a wide variety of visual concepts through the use of natural language. Existing image-editing methods that work with language inputs are heavily constrained by their requirement for training sets with rich attribute annotations or they are only able to handle simple text descriptions. We address these constraints by proposing a novel text-conditioned editing model called FICE (Fashion Image CLIP Editing) that is capable of handling a wide variety of diverse text descriptions to guide the editing procedure. Specifically, with FICE, we extend the common GAN-inversion process by including semantic, pose-related, and image-level constraints when generating images. We leverage the capabilities of the CLIP model to enforce the text-provided semantics, due to its impressive image–text association capabilities. We furthermore propose a latent-code regularization technique that provides the means to better control the fidelity of the synthesized images. We validate the FICE through rigorous experiments on a combination of VITON images and Fashion-Gen text descriptions and in comparison with several state-of-the-art, text-conditioned, image-editing approaches. Experimental results demonstrate that the FICE generates very realistic fashion images and leads to better editing than existing, competing approaches. The source code is publicly available from: https://github.com/MartinPernus/FICE.

Multimodal representations of biomedical knowledge from limited training whole slide images and reports using deep learning

The increasing availability of biomedical data creates valuable resources for developing new deep learning algorithms to support experts, especially in domains where collecting large volumes of annotated data is not trivial. Biomedical data include several modalities containing complementary information, such as medical images and reports: images are often large and encode low-level information, while reports include a summarized high-level description of the findings identified within data and often only concerning a small part of the image. However, only a few methods allow to effectively link the visual content of images with the textual content of reports, preventing medical specialists from properly benefitting from the recent opportunities offered by deep learning models. This paper introduces a multimodal architecture creating a robust biomedical data representation encoding fine-grained text representations within image embeddings. The architecture aims to tackle data scarcity (combining supervised and self-supervised learning) and to create multimodal biomedical ontologies. The architecture is trained on over 6,000 colon whole slide Images (WSI), paired with the corresponding report, collected from two digital pathology workflows. The evaluation of the multimodal architecture involves three tasks: WSI classification (on data from pathology workflow and from public repositories), multimodal data retrieval, and linking between textual and visual concepts. Noticeably, the latter two tasks are available by architectural design without further training, showing that the multimodal architecture that can be adopted as a backbone to solve peculiar tasks. The multimodal data representation outperforms the unimodal one on the classification of colon WSIs and allows to halve the data needed to reach accurate performance, reducing the computational power required and thus the carbon footprint. The combination of images and reports exploiting self-supervised algorithms allows to mine databases without needing new annotations provided by experts, extracting new information. In particular, the multimodal visual ontology, linking semantic concepts to images, may pave the way to advancements in medicine and biomedical analysis domains, not limited to histopathology.

Connecting plant science education in undergraduate life science courses to plant awareness disparity, Vision and Change, and sustainability careers

ABSTRACT Ensuring that botany and plant sciences are being included in undergraduate life science curricula is necessary for developing a future global sustainability workforce. To gain a baseline understanding, we surveyed life science educators in the U.S. about current botanical education. We further evaluated these data to determine connections to the frameworks of plant awareness disparity (PAD), a framework detailing the inability to notice or appreciate the plants in one’s own environment, and Vision and Change, an educational framework in the U.S. for preparing undergraduate biology students for the 21st century. Results from 245 responses revealed that most instructors use botanical examples and implement diverse hands-on botanical experiences. Our data suggest that hands-on botanical experiences are better suited to address PAD, the more science practice-centred framework, while plant examples are better suited to increase the understanding of core concepts of Vision and Change, the more concept-centred framework. We recommend that life science educators should 1) remain diligent in providing plant examples and hands-on botanical experiences in large introductory courses, 2) incorporate more botany education in general education courses (not required by the major but required by the institutions) and non-major courses, and 3) include educators outside of botany in these efforts.

Pemanfaatan Infografis Bagi Pengelola Wisata Alam Pulau Semut

Infographics are a popular form of information today. Infographics use a visual concept approach combined with interesting illustrations because they combine data with colorful designs. Based on these advantages, infographics can be developed as an information and promotion medium for a nature tourism area, namely Ant Island Nature Tourism Pekanbaru. This activity aims to create infographics. The service was carried out at the computer lab of the Faculty of Forestry and Science, Lancang Kuning University, from December 2023 to January 2024. Partners are natural tourism managers, totaling five people. The activities carried out include providing materials, practicing the creation of infographics, assisting the community, and evaluating the partners' knowledge and skills. The evaluation results showed that the partners had increased their knowledge about the meaning, benefits, types, and examples, and how to make infographics of nature tourism on Semut Island, Pekanbaru, by 60%. In addition, the partners were also able to make infographics about the nature tourism area of Semut Island, Pekanbaru, in accordance with the knowledge that the partners had received.

Perancangan Desain Karakter pada Buku Interaktif Pengenalan Literasi Keuangan untuk Anak Usia Dini

The focus of this design is the creation of character designs on interactive financial literacy books media for early childhood. The method used in this design is qualitative and uses qualitative descriptive analysis techniques. Primary data were obtained from interviews, Focus Group Discussions, and observations, while secondary data were obtained from literature and mass media. The design process begins with collecting references, determining concepts, making design alternatives, final designs, and evaluating designs. Verbal and visual concepts refer to the characteristics of the target audience, namely children aged 4-6 years in Indonesia. The drawing style and illustration refers to the book "Sayur Juara" which is used as a study comparator. Characters that have been selected and evaluated, then applied to interactive book media, stickers, and keychains. Introducing financial literacy is important from early childhood. This research is expected to contribute to improving early childhood financial literacy through the development of character designs in interactive books that are interesting and educational.

Understanding 3D seismic data visualization with C++, OpenGL and GLSL

Seismic data visualization in 3D space is a valuable interpretation tool. Several open-source visualization tools are available. However, little explanation is provided about the inner working of visualization process. The current work discusses a “hello world” equivalent source code for 3D seismic data visualization using Graphical Processing Units (GPUs) with OpenGL and the OpenGL Shading Language (GLSL) programming languages. Rendering is the core process generating 2D image that we see on the screen from the 3D data structures being visualized. Texture mapping-based rendering commonly applied to seismic data starts with creating an OpenGL object called texture. The texture is then mapped over a rectangular object to display a seismic line. The work presented here is performed using OpenGL, GLSL, C++ and Qt toolkit. Here Qt provides the application GUI framework, C++ is used for data I/O, filtering, and sorting, and OpenGL and GLSL are used for 3D rendering. This paper describes the data flow through the application, and two implementations of vertex and fragment GLSL shaders. Visualization is critical for seismic data processing and interpretation. The low-level details of the process presented here will hopefully help readers in obtaining better understanding of the visualization concept and inner working principle and facilitate further development.

Multimodal semantic enhanced representation network for micro-video event detection

Currently, micro-videos have gained widespread acceptance as a prominent form of user-generated content across various social media platforms. Accurate event analysis of micro-videos can greatly enhance the many diverse social media platforms applications. Although some studies have shown promising results from a multimodal perspective, there is still a challenge in extracting informative cues from inaccurate modalities, particularly for text modality that is prone to inaccuracies and noise. In this paper, we propose a multimodal semantically enhanced representation network (MSERN) for micro-video event detection. To better address inaccurate and noisy text sentences, we first extract visual concepts in the form of adjective-noun pairs (ANPs), through a fine-grained common representation module, to complement the textual descriptions. To maximize the acquisition of modality-specific cues from both visual and textual modalities, we then implement a coarse-grained private representation module to ensure that private representations encompass unique facets of the modalities beyond the common perspective. Finally, because two modules will collaborate, the fine-grained common and coarse-grained private representations are integrated to ensure a reinforced micro-video representation. We evaluate our proposed method on a micro-video event detection dataset and the experimental results show a superior performance compared to the state-of-the-art methods.