Visual Recognition Tasks Research Articles

Two-stage Rule-induction visual reasoning on RPMs with an application to video prediction

Raven’s Progressive Matrices (RPMs) are frequently used in evaluating human’s visual reasoning ability. Researchers have made considerable efforts in developing systems to automatically solve the RPM problem, often through a black-box end-to-end convolutional neural network for both visual recognition and logical reasoning tasks. Based on the intrinsic natures of RPM problem, we propose a Two-stage Rule-Induction Visual Reasoner (TRIVR), which consists of a perception module and a reasoning module, to tackle the challenges of real-world visual recognition and subsequent logical reasoning tasks, respectively. For the reasoning module, we further propose a “2+1” formulation that models human’s thinking in solving RPMs and significantly reduces the model complexity. It derives a reasoning rule from each RPM sample, which is not feasible for existing methods. As a result, the proposed reasoning module is capable of yielding a set of reasoning rules modeling human in solving the RPM problems. To validate the proposed method on real-world applications, an RPM-like Video Prediction (RVP) dataset is constructed, where visual reasoning is conducted on RPMs constructed using real-world video frames. Experimental results on various RPM-like datasets demonstrate that the proposed TRIVR achieves a significant and consistent performance gain compared with state-of-the-art models.

Isokinetic Force Modeling in Sports Based on Embedded System of Motion Recognition Model

With the gradual deepening of sports research, sports training methods integrating more scientific and technological means can greatly improve athletes’ professional ability. Combined with the isokinetic strength training method, it can effectively alleviate muscle recovery and body flexibility, which requires an analysis of the level of sports body promotion, and a more efficient training program for various sports parameters. In this study, taking strength enhancement as an example, the convolutional neural network is innovatively integrated into isokinetic strength training. Using the motion data obtained by wearable sensors in the training, we trained the neural network model by real-time analysis of data samples through an embedded system. Further, we optimized the network parameters to realize the extraction and recognition of EMG depth features. Compared with the traditional isokinetic strength classification model, this model lays a foundation for further constructing personalized strength training guidance methods. After the simulation test, the difference of the model’s accuracy requirement before and after the pre-training is 4.6%, and the difference of the test set before and after the pre-intensive training is [Formula: see text]1%. It is 16.2% higher than the TCAM algorithm, and the gap with STA-LSTM remains between [Formula: see text]. It is verified that the attention concentration control scheme can not only reduce the error rate in visual recognition tasks, but also improve the complexity of the neural network model. The optimized lightweight data structure is especially suitable for embedded systems with limited computing power. All kinds of test projects verify its rationality and make full use of the control ability of the embedded system and the characteristics of parallel computing and programmability, which has great practical application value for the development of the Internet of Things system network. The research results have made a thorough study of the physiological guidance of existing strength training, the identification and evaluation methods of exercise-induced muscle fatigue, and the current situation of the existing isokinetic equipment.

Defective Pennywort Leaf Detection Using Machine Vision and Mask R-CNN Model

Demand and market value for pennywort largely depend on the quality of the leaves, which can be affected by various ambient environment or fertigation variables during cultivation. Although early detection of defects in pennywort leaves would enable growers to take quick action, conventional manual detection is laborious and time consuming as well as subjective. Therefore, the objective of this study was to develop an automatic leaf defect detection algorithm for pennywort plants grown under controlled environment conditions, using machine vision and deep learning techniques. Leaf images were captured from pennywort plants grown in an ebb-and-flow hydroponic system under fluorescent light conditions in a controlled plant factory environment. Physically or biologically damaged leaves (e.g., curled, creased, discolored, misshapen, or brown spotted) were classified as defective leaves. Images were annotated using an online tool, and Mask R-CNN models were implemented with the integrated attention mechanisms, convolutional block attention module (CBAM) and coordinate attention (CA) and compared for improved image feature extraction. Transfer learning was employed to train the model with a smaller dataset, effectively reducing processing time. The improved models demonstrated significant advancements in accuracy and precision, with the CA-augmented model achieving the highest metrics, including a mean average precision (mAP) of 0.931 and an accuracy of 0.937. These enhancements enabled more precise localization and classification of leaf defects, outperforming the baseline Mask R-CNN model in complex visual recognition tasks. The final model was robust, effectively distinguishing defective leaves in challenging scenarios, making it highly suitable for applications in precision agriculture. Future research can build on this modeling framework, exploring additional variables to identify specific leaf abnormalities at earlier growth stages, which is crucial for production quality assurance.

Memory-related brain potentials for visual objects in early AD show impairment and compensatory mechanisms.

Impaired episodic memory is the primary feature of early Alzheimer's disease (AD), but not all memories are equally affected. Patients with AD and amnestic Mild Cognitive Impairment (aMCI) remember pictures better than words, to a greater extent than healthy elderly. We investigated neural mechanisms for visual object recognition in 30 patients (14AD, 16 aMCI) and 36 cognitively unimpaired healthy (19 in the "preclinical" stage of AD). Event-related brain potentials (ERPs) were recorded while participants performed a visual object recognition task. Hippocampal occupancy (integrity), amyloid (florbetapir) PET, and neuropsychological measures of verbal & visual memory, executive function were also collected. A right-frontal ERP recognition effect (500-700ms post-stimulus) was seen in cognitively unimpaired participants only, and significantly correlated with memory and executive function abilities. A later right-posterior negative ERP effect (700-900ms) correlated with visual memory abilities across participants with low verbal memory ability, and may reflect a compensatory mechanism. A correlation of this retrieval-related negativity with right hippocampal occupancy (r = 0.55), implicates the hippocampus in the engagement of compensatory perceptual retrieval mechanisms. Our results suggest that early AD patients are impaired in goal-directed retrieval processing, but may engage compensatory perceptual mechanisms which rely on hippocampal function.

Visual place recognition for aerial imagery: A survey

Aerial imagery and its direct application to visual localization is an essential problem for many Robotics and Computer Vision tasks. While Global Navigation Satellite Systems (GNSS) are the standard default solution for solving the aerial localization problem, it is subject to a number of limitations, such as, signal instability or solution unreliability that make this option not so desirable. Consequently, visual geolocalization is emerging as a viable alternative. However, adapting Visual Place Recognition (VPR) task to aerial imagery presents significant challenges, including weather variations and repetitive patterns. Current VPR reviews largely neglect the specific context of aerial data. This paper introduces a methodology tailored for evaluating VPR techniques specifically in the domain of aerial imagery, providing a comprehensive assessment of various methods and their performance. However, we not only compare various VPR methods, but also demonstrate the importance of selecting appropriate zoom and overlap levels when constructing map tiles to achieve maximum efficiency of VPR algorithms in the case of aerial imagery. The code is available on our GitHub repository — https://github.com/prime-slam/aero-vloc.

HeLP: The Hebrew Lexicon project

Lexicon projects (LPs) are large-scale data resources in different languages that present behavioral results from visual word recognition tasks. Analyses using LP data in multiple languages provide evidence regarding cross-linguistic differences as well as similarities in visual word recognition. Here we present the first LP in a Semitic language—the Hebrew Lexicon Project (HeLP). HeLP assembled lexical decision (LD) responses to 10,000 Hebrew words and nonwords, and naming responses to a subset of 5000 Hebrew words. We used the large-scale HeLP data to estimate the impact of general predictors (lexicality, frequency, word length, orthographic neighborhood density), and Hebrew-specific predictors (Semitic structure, presence of clitics, phonological entropy) of visual word recognition performance. Our results revealed the typical effects of lexicality and frequency obtained in many languages, but more complex impact of word length and neighborhood density. Considering Hebrew-specific characteristics, HeLP data revealed better recognition of words with a Semitic structure than words that do not conform to it, and a drop in performance for words comprising clitics. These effects varied, however, across LD and naming tasks. Lastly, a significant inhibitory effect of phonological ambiguity was found in both naming and LD. The implications of these findings for understanding reading in a Semitic language are discussed.

Probabilistic Contrastive Learning for Long-Tailed Visual Recognition.

Long-tailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Such imbalance issue considerably impairs the performance of standard supervised learning algorithms, which are mainly designed for balanced training sets. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. However, the performance of supervised contrastive learning is plagued by an inherent challenge: it necessitates sufficiently large batches of training data to construct contrastive pairs that cover all categories, yet this requirement is difficult to meet in the context of class-imbalanced data. To overcome this obstacle, we propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space, and samples contrastive pairs accordingly. In fact, estimating the distributions of all classes using features in a small batch, particularly for imbalanced data, is not feasible. Our key idea is to introduce a reasonable and simple assumption that the normalized features in contrastive learning follow a mixture of von Mises-Fisher (vMF) distributions on unit space, which brings two-fold benefits. First, the distribution parameters can be estimated using only the first sample moment, which can be efficiently computed in an online manner across different batches. Second, based on the estimated distribution, the vMF distribution allows us to sample an infinite number of contrastive pairs and derive a closed form of the expected contrastive loss for efficient optimization. Other than long-tailed problems, ProCo can be directly applied to semi-supervised learning by generating pseudo-labels for unlabeled data, which can subsequently be utilized to estimate the distribution of the samples inversely. Theoretically, we analyze the error bound of ProCo. Empirically, extensive experimental results on supervised/semi-supervised visual recognition and object detection tasks demonstrate that ProCo consistently outperforms existing methods across various datasets.

Subword Representations Successfully Decode Brain Responses to Morphologically Complex Written Words

Abstract This study extends the idea of decoding word-evoked brain activations using a corpus-semantic vector space to multimorphemic words in the agglutinative Finnish language. The corpus-semantic models are trained on word segments, and decoding is carried out with word vectors that are composed of these segments. We tested several alternative vector-space models using different segmentations: no segmentation (whole word), linguistic morphemes, statistical morphemes, random segmentation, and character-level 1-, 2- and 3-grams, and paired them with recorded MEG responses to multimorphemic words in a visual word recognition task. For all variants, the decoding accuracy exceeded the standard word-label permutation-based significance thresholds at 350–500 ms after stimulus onset. However, the critical segment-label permutation test revealed that only those segmentations that were morphologically aware reached significance in the brain decoding task. The results suggest that both whole-word forms and morphemes are represented in the brain and show that neural decoding using corpus-semantic word representations derived from compositional subword segments is applicable also for multimorphemic word forms. This is especially relevant for languages with complex morphology, because a large proportion of word forms are rare and it can be difficult to find statistically reliable surface representations for them in any large corpus.

Vision-Language Models for Vision Tasks: A Survey.

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition.

Individual differences in visual pattern completion predict adaptation to degraded speech

Recognizing acoustically degraded speech relies on predictive processing whereby incomplete auditory cues are mapped to stored linguistic representations via pattern recognition processes. While listeners vary in their ability to recognize degraded speech, performance improves when a written transcription is presented, allowing completion of the partial sensory pattern to preexisting representations. Building on work characterizing predictive processing as pattern completion, we examined the relationship between domain-general pattern recognition and individual variation in degraded speech learning. Participants completed a visual pattern recognition task to measure individual-level tendency towards pattern completion. Participants were also trained to recognize noise-vocoded speech with written transcriptions and tested on speech recognition pre- and post-training using a retrieval-based transcription task. Listeners significantly improved in recognizing speech after training, and pattern completion on the visual task predicted improvement for novel items. The results implicate pattern completion as a domain-general learning mechanism that can facilitate speech adaptation in challenging contexts.

Skills and cautiousness predict performance in difficult search.

People differ in how well they search. What are the factors that might contribute to this variability? We tested the contribution of two cognitive abilities: visual working memory (VWM) capacity and object recognition ability. Participants completed three tasks: a difficult inefficient visual search task, where they searched for a target letter T among skewed L distractors; a VWM task, where they memorized a color array and then identified whether a probed color belonged to the previous array; and the Novel Object Memory Test (NOMT), where they learnt complex novel objects and then identified them amongst objects that closely resembled them. Exploratory and confirmatory factor analyses revealed that there are two latent factors that explain the shared variance among these three tasks: a factor indicative of the level of caution participants exercised during the challenging visual search task, and a factor representing their visual cognitive abilities. People who score high on the search cautiousness tend to perform a more accurate but slower search. People who score high on the visual cognitive ability factor tend to have a higher VWM capacity, a better object recognition ability, and a faster search speed. The results reflect two points: (1) Visual search tasks share components with visual working memory and object recognition tasks. (2) Search performance is influenced not only by the search display's properties but also by individual predispositions such as caution and general visual abilities. This study introduces new factors for consideration when interpreting variations in visual search behaviors.

Automatic detection of breast lesions in automated 3D breast ultrasound with cross-organ transfer learning

BackgroundDeep convolutional neural networks have garnered considerable attention in numerous machine learning applications, particularly in visual recognition tasks such as image and video analyses. There is a growing interest in applying this technology to diverse applications in medical image analysis. Automated three-dimensional Breast Ultrasound is a vital tool for detecting breast cancer, and computer-assisted diagnosis software, developed based on deep learning, can effectively assist radiologists in diagnosis. However, the network model is prone to overfitting during training, owing to challenges such as insufficient training data. This study attempts to solve the problem caused by small datasets and improve model detection performance. MethodsWe propose a breast cancer detection framework based on deep learning (a transfer learning method based on cross-organ cancer detection) and a contrastive learning method based on breast imaging reporting and data systems (BI-RADS). ResultsWhen using cross organ transfer learning and BIRADS based contrastive learning, the average sensitivity of the model increased by a maximum of 16.05%. ConclusionOur experiments have demonstrated that the parameters and experiences of cross-organ cancer detection can be mutually referenced, and contrastive learning method based on BI-RADS can improve the detection performance of the model.

WatchCap: Improving Scanning Efficiency in People with Low Vision through Compensatory Head Movement Stimulation

Individuals with low vision (LV) frequently face challenges in scanning performance, which in turn complicates daily activities requiring visual recognition. Although those with PVL can theoretically compensate for these scanning deficiencies through the use of active head movements, few practical applications have sought to capitalize on this potential, especially during visual recognition tasks. In this paper, we present WatchCap, a novel device that leverages the hanger reflex phenomenon to naturally elicit head movements through stimulation feedback. Our user studies, conducted with both sighted individuals in a simulated environment and people with glaucoma-related PVL, demonstrated that WatchCap's scanning-contingent stimulation enhances visual exploration. This improvement is evidenced by the fixation and saccade-related features and positive feedback from participants, which did not cause discomfort to the users. This study highlights the promise of facilitating head movements to aid those with LVs in visual recognition tasks. Critically, since WatchCap functions independently of predefined or task-specific cues, it has a wide scope of applicability, even in ambient task situations. This independence positions WatchCap to complement existing tools aimed at detailed visual information acquisition, allowing integration with existing tools and facilitating a comprehensive approach to assisting individuals with LV.

Predicting the generalization gap in neural networks using topological data analysis

Understanding how neural networks generalize on unseen data is crucial for designing more robust and reliable models. In this paper, we study the generalization gap of neural networks using methods from topological data analysis. For this purpose, we compute homological persistence diagrams of weighted graphs constructed from neuron activation correlations after a training phase, aiming to capture patterns that are linked to the generalization capacity of the network. We compare the usefulness of different numerical summaries from persistence diagrams and show that a combination of some of them can accurately predict and partially explain the generalization gap without the need of a test set. Evaluation on two computer vision recognition tasks (CIFAR10 and SVHN) shows competitive generalization gap prediction when compared against state-of-the-art methods.

Empowering the Blind: An AI Driven Indoor Assistance for Visually Impaired

Abstract: This review paper presents the design and development of a comprehensive mobile application aimed at enhancing the daily lives of visually impaired individuals. The proposed mobile app offers real-time assistance for various visual recognition tasks, including object detection, distance estimation, currency recognition, barcode detection, color recognition, and emotion analysis. The primary objective of this application is to provide visually impaired users with a powerful tool to navigate their surroundings, identify objects and currency, scan barcodes, discern colors, and even gauge the emotions of those they interact with. The app integrates pre-trained deep learning models for object detection and facial emotion analysis, computer vision techniques for distance estimation, and optical character recognition for currency recognition. User feedback and engagement are central to the ongoing improvement of the application, ensuring that it remains a valuable resource for the community it serves. Ethical considerations and privacy concerns are also addressed, with a focus on data security and user privacy. By presenting the development and functionalities of this app, this review paper not only contributes to the field of assistive technology but also underscores the importance of harnessing cutting-edge technology to improve the quality of life for visually impaired individuals.

Classification of Electronics Components using Deep Learning

In this study, we present an advanced electronic component classification system with an exceptional classification accuracy exceeding 99% using state-of-the-art deep learning architectures. We employed EfficientNetV2B3, EfficientNetV2S, EfficientNetB0, InceptionV3, MobileNet, and Vision Transformer (ViT) models for the classification task. The system demonstrates the remarkable potential of these deep learning models in handling complex visual recognition tasks, specifically in the domain of electronic components. Our dataset comprises a diverse set of electronic components, and we meticulously curated and labeled it to ensure high-quality training data. We conducted extensive experiments to fine-tune and optimize the models for the given task, leveraging data augmentation techniques and transfer learning. The high classification accuracy achieved by our system indicates its readiness for real-world deployment, marking a significant step towards advancing automation and efficiency in the electronics industry.

First-sight recognition of touched objects shows that chicks can solve Molyneux's problem.

If a congenitally blind person learns to distinguish between a cube and a sphere by touch, would they immediately recognize these objects by sight once their vision is restored? This question, posed by Molyneux in 1688, has puzzled philosophers and scientists since then. To overcome ethical and practical difficulties in the investigation of cross-modal recognition, we studied inexperienced poultry chicks, which can be reared in darkness until the moment of a visual test with no detrimental consequences. After hatching chicks in darkness, we exposed them to either tactile smooth or tactile bumpy stimuli for 24 h. Immediately after the tactile exposure, chicks were tested in a visual recognition task, during their first experience with light. At first sight, chicks that had been exposed in the tactile modality to smooth stimuli approached the visual smooth stimulus significantly more than those exposed to the tactile bumpy stimuli. These results show that visually inexperienced chicks can solve Molyneux's problem, indicating cross-modal recognition does not require previous multimodal experience. At least in this precocial species, supra-modal brain areas appear functional already at birth. This discovery paves the way for the investigation of predisposed cross-modal cognition that does not depend on visual experience.

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

Self-supervised speech pre-training methods have developed rapidly in recent years, which show to be very effective for many near-field single-channel speech tasks. However, far-field multichannel speech processing is suffering from the scarcity of labeled multichannel data and complex ambient noises. The efficacy of self-supervised learning for far-field multichannel and multi-modal speech processing has not been well explored. Considering that visual information helps to improve speech recognition performance in noisy scenes, in this work we propose the multichannel multi-modal speech self-supervised learning framework AV-wav2vec2, which utilizes video and multichannel audio data as inputs. First, we propose a multi-path structure to process multi-channel audio streams and a visual stream in parallel, with intra-, and inter-channel contrastive as training targets to fully exploit the rich information in multi-channel speech data. Second, based on contrastive learning, we use additional single-channel audio data, which is trained jointly to improve the performance of multichannel multi-modal representation. Finally, we use a Chinese multichannel multi-modal dataset in real scenarios to validate the effectiveness of the proposed method on audio-visual speech recognition (AVSR), automatic speech recognition (ASR), visual speech recognition (VSR) and audio-visual speaker diarization (AVSD) tasks.

Can neural networks benefit from objectives that encourage iterative convergent computations? A case study of ResNets and object classification.

Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful in many domains, so they might provide good solutions for neural networks to learn. However, principled methods for measuring and manipulating iterative convergence in neural networks remain lacking. Here we address this gap by 1) quantifying the degree to which ResNets learn iterative solutions and 2) introducing a regularization approach that encourages the learning of iterative solutions. Iterative methods are characterized by two properties: iteration and convergence. To quantify these properties, we define three indices of iterative convergence. Consistent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer-vision tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across layers using soft gradient coupling. This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a "recurrent" ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time). To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normalization. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and convergent, respectively. To showcase the practicality of our approach, we study how iterative convergence impacts generalization on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or challenging recognition tasks with partial occlusions (Digitclutter). We find that iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets. Importantly, our approach may be useful for investigating other network architectures and tasks as well and we hope that our study provides a useful starting point for investigating the broader question of whether iterative convergence can help neural networks in their generalization.

Electrophysiological analysis of signal detection outcomes emphasizes the role of decisional factors in recognition memory.

Event-related potential (ERP) studies have identified two time windows associated with recognition memory and interpreted them as reflecting two processes: familiarity and recollection. However, using relatively simple stimuli and achieving high recognition rates, most studies focused on hits and correct rejections. This leaves out some information (misses and false alarms) that according to Signal Detection Theory (SDT) is necessary to understand signal processing. We used a difficult visual recognition task with colored pictures of different categories to obtain enough of the four possible SDT outcomes and analyzed them with modern ERP methods. Non-parametric analysis of these outcomes identified a single time window (470 to 670 ms) which reflected activity within fronto-central and posterior-left clusters of electrodes, indicating differential processing. The posterior-left cluster significantly distinguished all STD outcomes. The fronto-central cluster only distinguished ERPs according to the subject's response: yes vs. no. Additionally, only electrophysiological activity within the posterior-left cluster correlated with the discrimination index (d'). We show that when all SDT outcomes are examined, ERPs of recognition memory reflect a single-time window that may reveal a bottom-up factor discriminating the history of items (i.e. memory strength), as well as a top-down factor indicating participants' decision.