Knowledge-based Image Research Articles

Event-Aware Retrospective Learning for Knowledge-Based Image Captioning

Event-Aware Retrospective Learning for Knowledge-Based Image Captioning

Optimized deep knowledge-based no-reference image quality index for denoised MRI images

The quality of the Magnetic Resonance Imaging (MRI) image influences the disease diagnosis and consequent treatment. However, noise distortion severely impacts these images and tends to interfere with diagnosis during the data acquisition/transmission. This contribution proposes a novel No-reference Image Quality Index (NIQI) method for the intelligent estimation of MRI images and to evaluate its efficacy compared to well-established approaches. A novel Optimized Deep Knowledge-based NIQI (ODK-NIQI) method is developed and tested rigorously. The ODK-NIQI method combines shuffle shepherd optimization and improved deep mish-activated ConvNet approach. The implementation of the projected approach is conducted in MATLAB software. The results demonstrate that the proposed method achieves the best performance and the highest consistency of objectives for both the noisy and denoised MRI brain images investigated. Additionally, the proposed method shows significant improvement over the traditional NIQI techniques using standard performance metrics comprising the Spearman's Rank Correlation Coefficient (SROCC), Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and the Pearson Linear Correlation Coefficient (PLCC). Overall, the proposed ODK-based NIQI strategy performs well in denoising MRI images.

SAR Image Reconstruction and Autofocus Using Complex-Valued Feature Prior and Deep Network Implementation

Synthetic aperture radar (SAR) plays an important role in remote sensing by providing electromagnetic images of the observation scene. The prior knowledge-based SAR image reconstruction method can reduce the requirement of data sampling ratio and improve image quality. The existing prior knowledge-based method usually uses the magnitude information of SAR images while ignoring the phase information. However, since the echo and backscattering coefficients are complex values, the phase information of SAR images will help improve the reconstruction accuracy in the image reconstruction process. To improve the reconstruction performance, this paper proposes a SAR image reconstruction and autofocus method using complex-valued feature prior. In the proposed method, the complex-valued feature prior is learned from data by a complex-valued feature projection operator (CFPO), which can characterize and extract scene features in Range-Doppler domain and 2D frequency domain. The proposed CFPO enables more efficient use of echo data and helps to improve image reconstruction and autofocus performance. In addition, the proposed method is implemented by an unfolded deep network, which enables data-driven feature learning and efficient computation. The proposed method is verified by simulated and measured data.

Dynamic Key-Value Memory Enhanced Multi-Step Graph Reasoning for Knowledge-Based Visual Question Answering

Knowledge-based visual question answering (VQA) is a vision-language task that requires an agent to correctly answer image-related questions using knowledge that is not presented in the given image. It is not only a more challenging task than regular VQA but also a vital step towards building a general VQA system. Most existing knowledge-based VQA systems process knowledge and image information similarly and ignore the fact that the knowledge base (KB) contains complete information about a triplet, while the extracted image information might be incomplete as the relations between two objects are missing or wrongly detected. In this paper, we propose a novel model named dynamic knowledge memory enhanced multi-step graph reasoning (DMMGR), which performs explicit and implicit reasoning over a key-value knowledge memory module and a spatial-aware image graph, respectively. Specifically, the memory module learns a dynamic knowledge representation and generates a knowledge-aware question representation at each reasoning step. Then, this representation is used to guide a graph attention operator over the spatial-aware image graph. Our model achieves new state-of-the-art accuracy on the KRVQR and FVQA datasets. We also conduct ablation experiments to prove the effectiveness of each component of the proposed model.

Underwater Imaging Formation Model-Embedded Multiscale Deep Neural Network for Underwater Image Enhancement

Underwater image enhancement (UIE) plays an essential role in improving the quality of raw images captured in an underwater environment. Existing UIE methods can be categorized into two types: handcraft-designed and deep learning-based methods. Generally, the handcraft-designed methods are more explainable due to the leverage of knowledge-based image priors, while the deep learning-based methods are usually criticized for their weak interpretability. In this study, we address this issue by integrating the merits of both handcraft-designed and deep learning-based methods. Specifically, a physical underwater imaging model-inspired deep CNN for UIE is designed. Instead of estimating a global background light magnitude and a transmission matrix separately in traditional image restoration-based UIE methods, we directly generate a single variable as the joint estimation of these two parameters within a deep CNN and directly recover the enhanced image as an output according to a reformulated physical underwater imaging model. The whole network is trained in an end-to-end manner and more importantly has good interpretability. The proposed method has been validated for the UIE task on a real-world underwater image dataset and the experimental results well demonstrate the superiority of our method over the existing ones for UIE.

Inner Knowledge-based Img2Doc Scheme for Visual Question Answering

Visual Question Answering (VQA) is a research topic of significant interest at the intersection of computer vision and natural language understanding. Recent research indicates that attributes and knowledge can effectively improve performance for both image captioning and VQA. In this article, an inner knowledge-based Img2Doc algorithm for VQA is presented. The inner knowledge is characterized as the inner attribute relationship in visual images. In addition to using an attribute network for inner knowledge-based image representation, VQA scheme is associated with a question-guided Doc2Vec method for question–answering. The attribute network generates inner knowledge-based features for visual images, while a novel question-guided Doc2Vec method aims at converting natural language text to vector features. After the vector features are extracted, they are combined with visual image features into a classifier to provide an answer. Based on our model, the VQA problem is resolved by textual question answering. The experimental results demonstrate that the proposed method achieves superior performance on multiple benchmark datasets.

Faster R-CNN for multi-class fruit detection using a robotic vision system

An accurate and real-time image based multi-class fruit detection system is important for facilitating higher level smart farm tasks such as yield mapping and robotic harvesting. Robotic harvesting can reduce the costs of labour and increase fruit quality. This paper proposes a deep learning framework for multi-class fruits detection based on improved Faster R-CNN. The proposed framework includes fruits image library creation, data argumentation, improved Faster RCNN model generation, and performance evaluation. This work is a pioneer to create a multi-labeled and knowledge-based outdoor orchard image library using 4000 images in the real world. Also, improvement of the convolutional and pooling layers is achieved to have a more accurate and faster detection. The test results show the proposed algorithm has achieved higher detecting accuracy and lower processing time than the traditional detectors, which has excellent potential to build an autonomous and real-time harvesting or yield mapping/estimation system.

A knowledge-based image enhancement and denoising approach

The emergence of computer-aided diagnostic technology has revolutionized the health sector and by use of medical imaging records, health experts are able to get detailed analysis which enable them in precise diagnosis of gliomas tumors. In this paper, we present an approach that uses domain-specific knowledge together with hybrid image enhancement techniques that provides resulting image(s) with more details and lesser noise levels. We did comparison of our KB proposed approach with existing techniques and the experimentation results showed improvement in quality and reduction of arbitrariness of images. The approach is proved to be feasible and effective, thus resulting in better medical diagnosis and evaluation of gliomas problems. Proposed research work recommends a new approach for medical imaging enhancements.

PRINTEPS: An Integrated Intelligent Application Development Platform based on Stream Reasoning and ROS

Although AI and service robot applications have become very popular in many domains recently, many of them are specific applications and it is still difficult to develop integrated intelligent applications such as a robot teahouse and teaching assistant robots. To develop such integrated intelligent applications, we need integrated intelligent application platforms that have AI integration and agile process facilities. From the above background, we are currently developing PRactical INTElligent aPplicationS (PRINTEPS), which is a platform for developing integrated intelligent applications by combining only five types of modules, namely knowledge-based reasoning, spoken dialogue, image sensing, motion management, and machine learning. This paper proposes a workflow editor in PRINETPS based on a service-oriented architecture and a Robot Operating System that enables real-time parallel processing for multiple robots and sensors by integrating the five types of modules. The editor also supports not only developers but also domain experts in updating workflows frequently. This paper also proposes a novel method to integrate signals acquired through image sensing with knowledge (ontologies and business rules) using C-SPARQL and Semantic Web Rule Language. To evaluate PRINTEPS, we developed a robot teahouse application including customer reception and guidance to table services using a humanoid robot with PRINTEPS. Through this case study, we demonstrated that the behaviors of the robot can be modified by changing the workflow, the ontology, and the rules.

Standard dermoscopy and videodermoscopy as tools for medical student dermatologic education.

Background: Standard dermatoscopy improves the medical students’ ability to identify malignant skin lesions by noninvasive inspection of microstructures in the epidermis and superficial dermis. Videodermatoscopy allows for higher magnification, concurrent dermoscopic examination by multiple users. The purpose of this study is to understand the role of standard dermoscopy and videodermoscopy training in medical student dermatologic education. Objectives: To evaluate the effect of instruction using standard dermoscopy and videodermoscopy on the self-reported confidence and objective ability of medical students to identify benign and malignant skin lesions. Methods: Ninety second-year medical students were invited to receive a 45-minute lecture on general dermatologic topics and a 7-minute training session with either a standard dermatoscope or videodermatoscope. A total of 54 students completed an identical pre-test and post-test before and after the training session that included 13 knowledge-based analytic and image identification questions. Results: Both groups reported increased confidence in the ability to identify benign and malignant lesions after the training session. The combined group demonstrated an improved ability to identify dermoscopic features of benign and malignant skin lesions with a mean (SD) pre-test score of 3.4 (1.4) and a mean post-test score of 6.0 (1.6) out of 9 knowledge-based questions (p<0.001); however, there was no difference noted between the two interventions. Conclusions: Instruction using either a standard dermatoscope or a videodermatoscope can increase medical students’ confidence and accuracy in identifying benign and malignant skin lesions.

A User-Centric Platform PRINTEPS to Develop Integrated Intelligent Applications and Application to Robot Teahouse

We are developing PRactical INTEligent aPplicationS (PRINTEPS), which is a user-centric platform to develop integrated intelligent applications only by combining four types of modules such as knowledge-based reasoning, speech dialog, image sensing and motion management. PRINTEPS supports end users to participate in AI applications design (user participation design) and to develop applications easily. This paper introduces the architecture and applications of PRINTEPS for robot teahouse robot where multiple people and robots cooperate.

Implementation of Teacher-Robot Collaboration Lesson Application in PRINTEPS

PRINTEPS is currently being developed as a total intelligent application, which has sub systems for knowledge-based reasoning, speech dialogue, image sensing, motion planning, and machine learning, in order to support end users on easily developing intelligent applications for human-machine collaboration. In this paper, a lesson application for collaborative teaching among a robot, laptop PC, sensor, teachers and students was developed with PRINTEPS. The implementation lesson was performed in a science class for six grade elementary students.

Fuzzy knowledge-based adaptive image watermarking by the method of moments

A novel adaptive image watermarking methodology, robust to common signal processing attacks, is proposed in this paper. The introduced watermarking scheme uses prior knowledge in the form of IF–THEN fuzzy rules to intelligently decide the appropriate embedding strength applied to each coefficient in the moment domain carrying the watermark information. Initially, the prior knowledge regarding the amount of watermark information each moment coefficient of the original image has the capacity to hide is used to build an optimized rule-based fuzzy inference system. The resulted system is able to provide the appropriate embedding strength for each moment coefficient, ensuring high imperceptible watermarked images and high fidelity in watermark extraction. Extensive experiments under several attacking conditions demonstrate that the proposed methodology can improve the robustness of the watermarked images by a significant factor up to 32 %, without affecting their optical quality.

A Knowledge-based Image Retrieval System Integrating Semantic and Visual Features

The main limitations of the existing high level image retrieval approaches concern the high dependance on an external reliable resource (domain ontologies, learning sets, etc.) and a model for mapping semantic and visual information.In this paper, we propose an image retrieval system integrating semantic and visual features. The idea is to automatically build a modular ontology for semantic information and organize visual features in a graph-based model. Both elements are then combined together in a same component called “pattern” used for retrieval. The system has been implemented and the obtained results show that our proposal enables an improvement in the retrieval task.

A knowledge-based multi-layered image annotation system

Major challenge in automatic image annotation is bridging the semantic gap between the computable low-level image features and the human-like interpretation of images. The interpretation includes concepts on different levels of abstraction that cannot be simply mapped to features but require additional reasoning with general and domain-specific knowledge. The problem is even more complex since knowledge in context of image interpretation is often incomplete, imprecise, uncertain and ambiguous in nature. Thus, in this paper we propose a fuzzy-knowledge based intelligent system for image annotation, which is able to deal with uncertain and ambiguous knowledge and can annotate images with concepts on different levels of abstraction that is more human-like. The main contributions are associated with an original approach of using a fuzzy knowledge-representation scheme based on the Fuzzy Petri Net (KRFPN) formalism. The acquisition of knowledge is facilitated in a way that besides the general knowledge provided by the expert, the computable facts and rules about the concepts, as well as their reliability, are produced automatically from data. The reasoning capability of the fuzzy inference engine of the KRFPN is used in a novel way for inconsistency checking of the classified image segments, automatic scene recognition, and the inference of generalized and derived classes.The results of image interpretation of Corel images belonging to the domain of outdoor scenes achieved by the proposed system outperform the published results obtained on the same image base in terms of average precision and recall. Owing to the fuzzy-knowledge representation scheme, the obtained image interpretation is enriched with new, more general and abstract concepts that are close to concepts people use to interpret these images.

A Sign Language Recognition Approach for Human-Robot Symbiosis

This paper introduces a new concept for the establishment of human-robot symbiotic relationship. The system is based on the implementation of knowledge-based image processing methodologies for model based vision and intelligent task scheduling for an autonomous social robot. This paper aims to develop an automatic translation of static gestures of alphabets and signs in American Sign Language (ASL), using neural network with backpropagation algorithm. System deals with images of bare hands to achieve the recognition task. For each individual sign 10 sample images have been considered, which means in total300 samples have been processed. In order to compare between the training set of signs and the considered sample images, are converted into feature vectors. Experimental results reveal that this can recognize selected ASL signs (accuracy of 92.00%). Finally, the system has been implemented issuing hand gesture commands for ASL to a robot car, named “Moto-robo”.

Automatic slice alignment method for cardiac magnetic resonance imaging

Automatic slice alignment is important for easier operation and shorter examination times in cardiac magnetic resonance imaging (MRI) examinations. We propose a new automatic slice alignment method for six cardiac planes (short-axis, vertical long-axis, horizontal long-axis, 4-chamber, 2-chamber, and 3-chamber views). ECG-gated 2D steady-state free precession axial multislice images were acquired using a 1.5-T MRI scanner during a single breath-hold. The scanning time was set to <20s in 23 volumes from 23 healthy volunteers. In this method, the positions of the mitral valve, cardiac apex, left ventricular outflow tract, tricuspid valve, anterior wall of the heart, and right ventricular corner are detected to determine the positions of six reference planes by combining knowledge-based recognition and image processing techniques. In order to evaluate the results of automatic slice alignment for the short-axis, 4-chamber, 2-chamber, and 3-chamber views, the angular and positional errors between the results obtained by our proposed method and by manual annotation were measured. The average angular errors for the short-axis, 4-chamber, 2-chamber, and 3-chamber views were 3.05°, 4.52°, 7.28°, and 5.79°, respectively. The average positional errors for the short-axis (base), short-axis (apex), 4-chamber, 2-chamber, and 3-chamber views were 6.61°, 3.80°, 1.55°, 1.52°, and 1.48°, respectively. The experimental results showed that our proposed method can detect the cardiac planes quickly and accurately. Our method is therefore beneficial to both patients and operators.

Knowledge-based Object-oriented Satellite Cloud Image Classification

Knowledge-based Object-oriented Satellite Cloud Image Classification

Improvement of knowledge-based automatic slice-alignment method for cardiac magnetic resonance imaging

Background Automatic slice alignment allows images of the six standard cardiac planes as defined in the SCMR Image Acquisition Protocols to be obtained by simple and quick operation. Our previously reported method can detect these planes using ECG-gated breath-hold axial multislice images [1]. Achieving higher accuracy and greater robustness for variation in clinical images will lead to improved usability and reliability, resulting in easier cardiac MR examinations. To achieve these goals, we have substantially refined our previously reported automatic slice-alignment method. A combination of knowledge-based recognition and image processing techniques is applied to multiple feature point search to reduce errors in automatic detection. Volunteer and clinical data were used to evaluate of the degree of improvement.

Knowledge-based image segmentation using swarm intelligence techniques

Intelligent techniques such as swarm intelligence techniques rarely have been used for image segmentation or boundary detection. The limited increasing number of agents in the environment and how to find efficiently the right threshold in an image, develop a flexible design, and fully autonomous system that supports different platforms makes the task challenging. Considering challenges this paper presents a swarm-based intelligent technique for image segmentation that is based on a fully agent-based model system, called swarm intelligence-based image segmentation (SIBIS). SIBIS adopts a cellular automata technique where the swarm of agents navigate through the image and operate on their pixels and local regions. Three features such as swarm intelligence, agent-based modelling and cellular automata are integrated to make SIBIS efficient. SIBIS system can find the image segmentation threshold automatically without changing the background or the texture of the image.