Vision Model Research Articles

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines

The rapid release of high-performing computer vision models offers new potential to study the impact of different inductive biases on the emergent brain alignment of learned representations. Here, we perform controlled comparisons among a curated set of 224 diverse models to test the impact of specific model properties on visual brain predictivity – a process requiring over 1.8 billion regressions and 50.3 thousand representational similarity analyses. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and task objectives (e.g. purely visual contrastive learning versus vision- language alignment) achieve near equivalent brain predictivity, when other factors are held constant. Instead, variation across visual training diets yields the largest, most consistent effect on brain predictivity. Many models achieve similarly high brain predictivity, despite clear variation in their underlying representations – suggesting that standard methods used to link models to brains may be too flexible. Broadly, these findings challenge common assumptions about the factors underlying emergent brain alignment, and outline how we can leverage controlled model comparison to probe the common computational principles underlying biological and artificial visual systems.

Knowledge Distillation and Student-Teacher Learning for Weed Detection in Turf

Abstract Machine vision-based herbicide applications relying on object detection or image classification deep convolutional neural network (DCNN) demand high memory and computational resources, resulting in lengthy inference times. To tackle these challenges, this study assessed the effectiveness of three teacher models, each trained on datasets of varying sizes, including D-20k (comprising 10,000 true positive and true negative images) and D-10k (comprising 5,000 true positive and true negative images). Additionally, knowledge distillation was performed on their corresponding student models across a range of temperature settings. After the process of student-teacher learning, the parameters of all student models were reduced. ResNet18 not only achieved higher accuracy (ACC≥0.989) but also maintained higher frames per second (FPS≥742.9) under its optimal temperature condition (T=1). Overall, the results suggest that employing knowledge distillation on the machine vision models enabled accurate and reliable weed detection in turf while reducing the need for extensive computational resources, thereby facilitating real-time weed detection and contributing to the development of smart, machine vision-based sprayers.

Plug-and-play segment anything model improves nnUNet performance.

The automatic segmentation of medical images has widespread applications in modern clinical workflows. The Segment Anything Model (SAM), a recent development of foundational models in computer vision, has become a universal tool for image segmentation without the need for specific domain training. However, SAM's reliance on prompts necessitates human-computer interaction during the inference process. Its performance on specific domains can also be limited without additional adaptation. In contrast, traditional models like nnUNet are designed to perform segmentation tasks automatically during inference and can work well for each specific domain, but they require extensive training on domain-specificdatasets. To leverage the advantages of both foundational and domain-specific models and achieve fully automated segmentation with limited training samples, we propose nnSAM, which combines the robust feature extraction capabilities of SAM with the automatic configuration abilities of nnUNet to enhance the accuracy and robustness of medical image segmentation on smalldatasets. We propose the nnSAM model for small sample medical image segmentation. We made optimizations for this goal via two main approaches: first, we integrated the feature extraction capabilities of SAM with the automatic configuration advantages of nnUNet, which enables robust feature extraction and domain-specific adaptation on small datasets. Second, during the training process, we designed a boundary shape supervision loss based on level set functions and curvature calculations, enabling the model to learn anatomical shape priors from limited annotationdata. We conducted quantitative and qualitative assessments on the performance of our proposed method on four segmentation tasks: brain white matter, liver, lung, and heart segmentation. Our method achieved the best performance across all tasks. Specifically, in brain white matter segmentation using 20 training samples, nnSAM achieved the highest DICE score of 82.77 ( 10.12) % and the lowest average surface distance (ASD) of 1.14 ( 1.03) mm, compared to nnUNet, which had a DICE score of 79.25 ( 17.24) % and an ASD of 1.36 ( 1.63) mm. A sample size study shows that the advantage of nnSAM becomes more prominent under fewer trainingsamples. A comprehensive evaluation of multiple small-sample segmentation tasks demonstrates significant improvements in segmentation performance by nnSAM, highlighting the vast potential of small-samplelearning.

Part-Prototype Models in Medical Imaging: Applications and Current Challenges

Recent developments in Artificial Intelligence have increasingly focused on explainability research. The potential of Explainable Artificial Intelligence (XAI) in producing trustworthy computer-aided diagnosis systems and its usage for knowledge discovery are gaining interest in the medical imaging (MI) community to support the diagnostic process and the discovery of image biomarkers. Most of the existing XAI applications in MI are focused on interpreting the predictions made using deep neural networks, typically including attribution techniques with saliency map approaches and other feature visualization methods. However, these are often criticized for providing incorrect and incomplete representations of the black-box models’ behaviour. This highlights the importance of proposing models intentionally designed to be self-explanatory. In particular, part-prototype (PP) models are interpretable-by-design computer vision (CV) models that base their decision process on learning and identifying representative prototypical parts from input images, and they are gaining increasing interest and results in MI applications. However, the medical field has unique characteristics that could benefit from more advanced implementations of these types of architectures. This narrative review summarizes existing PP networks, their application in MI analysis, and current challenges.

Assessing the utility of computer vision for age determination of Gulf Menhaden

AbstractObjectiveIn this work, we assess the potential of computer vision techniques for age estimation of Gulf Menhaden Brevoortia patronus scales. Scales are the primary structure used for the age determination of Gulf Menhaden, and the ageing process can be labor intensive. Gulf Menhaden is the second‐largest fishery by weight in the United States, with average annual landings from 2018 to 2022 of 449,540 metric tons, and is assessed with age‐structured models that require information about the age structure of the catch and the stock.MethodsWe used convolutional neural networks and deep neural networks to classify the age from images of Gulf Menhaden scales from three different sets of images of scales. The first set of data consists of images of scales from fish at ages 0 and 1 year. The second set of data consists of images of scales from fish at ages 0–4 years. The last set of data consists of images of scales from fish of ages 0, 1, and 2 years and includes only images of scales for which there is agreement by readers of age estimates derived from analyzing sagittal otoliths and scales from the same individual.ResultThe classification of ages was best when using a convolutional neural network model on the first data set. The poorest classification was for the model using deep neural networks with the second data set.ConclusionAlthough we show that computer vision has promise for age determination from fish scale samples, our results indicate that considerable work must be done for wide adoption of the approach. With the continuous enhancements of computer vision models, improvements in the quality of scale images, and the accumulation of larger sets of scale images that can be used to train machine learning models, we believe that using computer vision can serve to reduce processing time and increase the accuracy of age estimates.

Artificial Intelligence and Linguistic Landscape research

Abstract This article explores applications of artificial intelligence (AI) technologies in Linguistic Landscape research. Traditionally, LL research has relied on manual data collection and analysis, often involving photographs of public signage, advertisements, and other visual language displays. However, this manual approach can present challenges, including time-consuming data collection, inconsistent data quality, and potential researcher bias. Two AI technologies in particular hold promise for addressing these challenges in LL research: computer vision (CV) and large language models (LLMs). CV automates the identification and extraction of text from images, improving data accuracy and enabling large-scale image analysis. LLMs, based on natural language processing, can detect, translate, and interpret multilingual text. This article explores the affordances and challenges of using AI technologies in LL research and discusses methods to improve data collection, enhance accuracy, and support the analysis of multilingual environments. It also raises ethical issues and limitations of the technologies.

An attention fused sequence -to-sequence convolutional neural network for accurate solar irradiance forecasting and prediction using sky images

The challenge of accurately forecasting ultra-short-term solar irradiance for photovoltaic systems is complicated by rapidly changing weather, and while ground-based sky images offer potential improvements, effectively extracting spatiotemporal data from these images remains a significant hurdle for current computer vision models. A new hybrid model, "An Attention Fused Sequence-to-Sequence Convolutional Neural Network," is being developed to address this challenge. The model predicts intra-hour GHI, DNI, and DHI with a 10-min lead time by combining a Convolutional Neural Network (for spatial feature extraction from sky images), an attention mechanism (to focus on relevant regions), and a sequence-to-sequence model (for temporal feature extraction from time-series data). The proposed Model is trained using the NREL Solar Radiation Research Laboratory Dataset while evaluating the model with the Mean Bias Error, Mean Absolute Error, Root Mean Squared Error, R Squared, and forecasting skill score. The 10-min and sequence length 2 interval is considered to be the best performing across most of the evaluation metrics with an MBE value is 2.321 W/m2, MAE value of 39.490 W/m2, RMSE value of 62.086 W/m2, R2 value is 0.909 W/m2, FSS value of 23.589 W/m2 and an MBE of 4.876 W/m2, MAE of 56.887 W/m2, RMSE of 85.346 W/m2, R2 of 0.834 and FSS of 24.881 W/m2 respectively. Furthermore, the sensitivity analysis reveals that the proposed model's performance is influenced by both the sequence length and the lead time. The proposed framework outperforms other techniques for ultra-short-term PV generation forecasting, demonstrating its potential for practical deployment in PV systems to improve grid reliability and energy management.

CellSAM: Advancing Pathologic Image Cell Segmentation via Asymmetric Large-Scale Vision Model Feature Distillation Aggregation Network.

Segment anything model (SAM) has attracted extensive interest as a potent large-scale image segmentation model, with prior efforts adapting it for use in medical imaging. However, the precise segmentation of cell nucleus instances remains a formidable challenge in computational pathology, given substantial morphological variations and the dense clustering of nuclei with unclear boundaries. This study presents an innovative cell segmentation algorithm named CellSAM. CellSAM has the potential to improve the effectiveness and precision of disease identification and therapy planning. As a variant of SAM, CellSAM integrates dual-image encoders and employs techniques such as knowledge distillation and mask fusion. This innovative model exhibits promising capabilities in capturing intricate cell structures and ensuring adaptability in resource-constrained scenarios. The experimental results indicate that this structure effectively enhances the quality and precision of cell segmentation. Remarkably, CellSAM demonstrates outstanding results even with minimal training data. In the evaluation of particular cell segmentation tasks, extensive comparative analyzes show that CellSAM outperforms both general fundamental models and state-of-the-art (SOTA) task-specific models. Comprehensive evaluation metrics yield scores of 0.884, 0.876, and 0.768 for mean accuracy, recall, and precision respectively. Extensive experiments show that CellSAM excels in capturing subtle details and complex structures and is capable of segmenting cells in images accurately. Additionally, CellSAM demonstrates excellent performance on clinical data, indicating its potential for robust applications in treatment planning and disease diagnosis, thereby further improving the efficiency of computer-aided medicine.

Plaid masking explained with input-dependent dendritic nonlinearities

A serious obstacle for understanding early spatial vision comes from the failure of the so-called standard model (SM) to predict the perception of plaid masking. But the SM originated from a major oversimplification of single neuron computations, ignoring fundamental properties of dendrites. Here we show that a spatial vision model including computations mimicking the input-dependent nature of dendritic nonlinearities, i.e. including nonlinear neural summation, has the potential to explain plaid masking data.

A novel Tree-augmented Bayesian network for predicting rock weathering degree using incomplete dataset

The precise forecasting of the weathering degree of surrounding rock holds paramount importance for the scientific design and secure execution of tunnel engineering. The apparent features of the surrounding rock serve as critical indicators for evaluating its weathering degree. This paper endeavors to quantify the rock apparent features based on an improved Computer vision model and establish a multi-source heterogeneous dataset encompassing 10 parameters, thereby facilitating data-driven predictions of the weathering degree. Specifically, the rock appearance parameters are quantified and segmented by an improved Tunnel face feature segmentation (TFFSeg) model, which is tailored to the unique characteristics of groundwater, fractures, and interlayers. Concurrently, the TFFSeg model exhibits significantly enhanced performance for these rock features compared to other widely employed Computer vision methods. Subsequently, this multi-source dataset is further enriched by incorporating rock physical and mechanical parameters as well as tunnel design parameters. Nevertheless, the issue of data incompleteness persists within this dataset. To achieve precise prediction of the weathering degree based on this incomplete dataset, a novel Tree-augmented Bayesian network (TAN-BN) is designed, which is capable of learning from incomplete datasets. The predictive outcomes demonstrate that the proposed TAN-BN surpasses other currently utilized meta models and ensemble models, such as ANN, GBRT, and Naive BN. Finally, sensitivity analysis is conducted to determine the importance rankings of the 10 parameters, offering valuable insights for on-site evaluation of the rock weathering degree at the tunnel face.

A Computer Vision Model for Seaweed Foreign Object Detection Using Deep Learning

Seaweed foreign object detection has become crucial for food consumption and industrial use. This process not only can prevent potential health issues, but also maintain the overall marketability of seaweed production in the food industry. Traditional methods of inspecting seaweed foreign objects heavily rely on human judgment, which deals with large volumes with diverse impurities and can be inconsistent and inefficient. An automation system for real-time seaweed foreign object detection in the inspection process should be adopted. However, automated seaweed foreign object detection has several challenges due to its dependency on visual input inspection, such as an uneven surface and undistinguishable impurities. In fact, limited access to advanced technologies and high-cost equipment would also influence visual input acquisition, thereby hindering the advancement of seaweed foreign object detection in this field. Therefore, we introduce a computer vision model utilizing a deep learning-based algorithm to detect seaweed impurities and classify the samples into ‘clean’ and ‘unclean’ categories. In this study, we managed to identify six types of seaweed impurities including sand sticks, shells, discolored seaweed, grass, worm shells, and mixed impurities. We collected 1204 images and our model’s performance was thoroughly evaluated based on comparisons with three pre-trained models, i.e., Yolov8, ResNet, and MobileNet. Our experiment shows that Yolov8 outperforms the other two models with an accuracy of 98.86%. This study also included the development of an Android application to validate the deep learning engine to ensure its optimal performance. Based on our experiments, the mobile application managed to classify 50 pieces of seaweed samples within 0.2 s each, showcasing its potential use in large-scale production lines and factories. This research demonstrates the impact of Artificial Intelligence on food safety by offering a scalable and efficient solution that can be deployed in other food production processes facing similar challenges. Our approach paves the way for broader industry adoption and advancements in automated foreign object detection systems by optimizing detection accuracy and speed.

DCV2I$\text{DCV}^2\text{I}$: Leveraging deep vision models to support geographers' visual interpretation in dune segmentation

AbstractVisual interpretation is extremely important in human geography as the primary technique for geographers to use photograph data in identifying, classifying, and quantifying geographic and topological objects or regions. However, it is also time‐consuming and requires overwhelming manual effort from professional geographers. This paper describes our interdisciplinary team's efforts in integrating computer vision models with geographers' visual image interpretation process to reduce their workload in interpreting images. Focusing on the dune segmentation task, we proposed an approach called featuring a deep dune segmentation model to identify dunes and label their ranges in an automated way. By developing a tool to connect our model with ArcGIS—one of the most popular workbenches for visual interpretation, geographers can further refine the automatically generated dune segmentation on images without learning any CV or deep learning techniques. Our approach thus realized a noninvasive change to geographers' visual interpretation routines, reducing their manual efforts while incurring minimal interruptions to their work routines and tools they are familiar with. Deployment with a leading Chinese geography research institution demonstrated the potential of in supporting geographers in researching and solving drylands desertification.

Enabling Energy-Efficient Deployment of Large Language Models on Memristor Crossbar: A Synergy of Large and Small.

Large language models (LLMs) have garnered substantial attention due to their promising applications in diverse domains. Nevertheless, the increasing size of LLMs comes with a significant surge in the computational requirements for training and deployment. Memristor crossbars have emerged as a promising solution, which demonstrated a small footprint and remarkably high energy efficiency in computer vision (CV) models. Memristors possess higher density compared to conventional memory technologies, making them highly suitable for effectively managing the extreme model size associated with LLMs. However, deploying LLMs on memristor crossbars faces three major challenges. Firstly, the size of LLMs increases rapidly, already surpassing the capabilities of state-of-the-art memristor chips. Secondly, LLMs often incorporate multi-head attention blocks, which involve non-weight stationary multiplications that traditional memristor crossbars cannot support. Third, while memristor crossbars excel at performing linear operations, they are not capable of executing complex nonlinear operations in LLM such as softmax and layer normalization. To address these challenges, we present a novel architecture for the memristor crossbar that enables the deployment of state-of-the-art LLM on a single chip or package, eliminating the energy and time inefficiencies associated with off-chip communication. Our testing on BERT Large showed negligible accuracy loss. Compared to traditional memristor crossbars, our architecture achieves enhancements of up to 39× in area overhead and 18× in energy consumption. Compared to modern TPU/GPU systems, our architecture demonstrates at least a 68× reduction in the area-delay product and a significant 69% energy consumption reduction.

Trends and future directions of artificial intelligence applications in Iranian livestock production systems

Abstract In recent years, the global quest for livestock intensification driven by ever-increasing demands for animal food products raised concerns about animal welfare, environmental sustainability, and public health. Leveraging artificial intelligence (AI) technologies such as remote sensing, Internet of Things (IoT), computer vision, and data-driven modeling has become a hotspot in livestock farming that could facilitate animal monitoring, disease detection, feed optimization, and health management. This review includes an assessment of these topics and research done in Iran so far, proposing future steps for the deployment of AI-powered technologies in farm applications. The Iranian livestock sector already seeing benefits from AI advancements and information technologies, however, most studies focused on model development without applications or deployment for the industry. Significant work is needed to address the limitations and challenges namely lack of data, economic feasibility, ethical concerns, infrastructure issues, and regulatory frameworks. Furthermore, reported AI-based methods and approaches have some inconsistencies in Iran that hinder validation. Looking forward, AI could create a new era in the livestock sector of Iran that not only copes with upcoming challenges but also boosts the circular economy making this country a pioneer in the region. However, tackling some potential limitations accompanying AI application in the Iranian livestock sector warrants the multi-disciplinary collaboration of veterinarians, computer scientists, animal nutritionists, agri-engineers, and governmental organizations.

Fusing remote and social sensing data for flood impact mapping

AbstractThe absence of comprehensive situational awareness information poses a significant challenge for humanitarian organizations during their response efforts. We present Flood Insights, an end‐to‐end system, that ingests data from multiple nontraditional data sources such as remote sensing, social sensing, and geospatial data. We employ state‐of‐the‐art natural language processing and computer vision models to identify flood exposure, ground‐level damage and flood reports, and most importantly, urgent needs of affected people. We deploy and test the system during a recent real‐world catastrophe, the 2022 Pakistan floods, to surface critical situational and damage information at the district level. We validated the system's effectiveness through various statistical analyses using official ground‐truth data, showcasing its strong performance and explanatory power of integrating multiple data sources. Moreover, the system was commended by the United Nations Development Programme stationed in Pakistan, as well as local authorities, for pinpointing hard‐hit districts and enhancing disaster response.

Research and application of intelligent diagnosis method for impeller fault based on centrifugal pump digital twin flow field cloud map

With the development of industrial technology, the demand for health diagnosis and maintenance of centrifugal pumps is becoming increasingly urgent. Combining digital twin and machine vision technology, this paper proposes an intelligent diagnosis method for centrifugal pump impeller machinery fault based on digital twin flow field cloud map. Firstly, the centrifugal pump digital twin model is used to simulate the evolution of random fracture fault of impeller blades, and the impeller flow field pressure and velocity cloud maps with different fault characteristics are generated; secondly, based on the learning and training of Yolov5 algorithm, two types of machine vision models of pressure and velocity cloud maps are obtained, and the preliminary diagnosis of impeller faults is realized by combining statistical analysis; then, considering the complementary advantages of the two types of detection models, the two are integrated based on the idea of stacking integration to improve the accuracy of impeller fault diagnosis. Experimental verification shows that for random fracture faults of impeller blades, the centrifugal spring intelligent fault diagnosis method proposed in this paper can achieve a diagnostic accuracy of more than 0.99, and the developed intelligent diagnosis system for centrifugal pump impeller machinery faults enables the method in this paper to be put into practice.

Remote sensing and computer vision for marine aquaculture.

Aquaculture, the cultivation of aquatic plants and animals, has grown rapidly since the 1990s, but sparse, self-reported, and aggregated production data limit the effective understanding and monitoring of the industry's trends and potential risks. Building on a manual survey of aquaculture production from remote sensing imagery, we train a computer vision model to identify marine aquaculture cages from aerial and satellite imagery and generate a spatially explicit dataset of finfish production locations in the French Mediterranean from 2000 to 2021 including 4010 cages (average cage area, 69 square meters). We demonstrate the value of our method as an easily adaptable, cost-effective approach that can improve the speed and reliability of aquaculture surveys and enables downstream analyses relevant to researchers and regulators. We illustrate its use to compute independent estimates of production and develop a flexible framework to quantify uncertainty in these estimates. Overall, our study presents an efficient, scalable, and adaptable method for monitoring aquaculture production from remote sensing imagery.

Targeted weed management of Palmer amaranth using robotics and deep learning (YOLOv7).

Effective weed management is a significant challenge in agronomic crops which necessitates innovative solutions to reduce negative environmental impacts and minimize crop damage. Traditional methods often rely on indiscriminate herbicide application, which lacks precision and sustainability. To address this critical need, this study demonstrated an AI-enabled robotic system, Weeding robot, designed for targeted weed management. Palmer amaranth (Amaranthus palmeri S. Watson) was selected as it is the most troublesome weed in Nebraska. We developed the full stack (vision, hardware, software, robotic platform, and AI model) for precision spraying using YOLOv7, a state-of-the-art object detection deep learning technique. The Weeding robot achieved an average of 60.4% precision and 62% recall in real-time weed identification and spot spraying with the developed gantry-based sprayer system. The Weeding robot successfully identified Palmer amaranth across diverse growth stages in controlled outdoor conditions. This study demonstrates the potential of AI-enabled robotic systems for targeted weed management, offering a more precise and sustainable alternative to traditional herbicide application methods.

Clinician and Visitor Activity Patterns in an Intensive Care Unit Room: A Study to Examine How Ambient Monitoring Can Inform the Measurement of Delirium Severity and Escalation of Care.

The early detection of the acute deterioration of escalating illness severity is crucial for effective patient management and can significantly impact patient outcomes. Ambient sensing technology, such as computer vision, may provide real-time information that could impact early recognition and response. This study aimed to develop a computer vision model to quantify the number and type (clinician vs. visitor) of people in an intensive care unit (ICU) room, study the trajectory of their movement, and preliminarily explore its relationship with delirium as a marker of illness severity. To quantify the number of people present, we implemented a counting-by-detection supervised strategy using images from ICU rooms. This was accomplished through developing three methods: single-frame, multi-frame, and tracking-to-count. We then explored how the type of person and distribution in the room corresponded to the presence of delirium. Our designed pipeline was tested with a different set of detection models. We report model performance statistics and preliminary insights into the relationship between the number and type of persons in the ICU room and delirium. We evaluated our method and compared it with other approaches, including density estimation, counting by detection, regression methods, and their adaptability to ICU environments.

Bolt loosening assessment using ensemble vision models for automatic localization and feature extraction with target‐free perspective adaptation

AbstractBolt loosening assessment is crucial to identify early warnings of structural degradation and prevent catastrophic events. This paper proposes an automatic bolt loosening assessment methodology. First, a novel end‐to‐end ensemble vision model, Bolt‐FP‐Net, is proposed to reason the locations of bolts and their hexagonal feature patterns concurrently. Second, an adaptive target‐free perspective correction method is proposed to correct perspective distortion and enhance assessment accuracy. Finally, an iterative bolt loosening quantification is developed to estimate and refine the bolt loosening rotation. Experimental parametric studies indicated that the proposed Bolt‐FP‐Net can achieve excellent performance under different environmental conditions. Finally, a case study was conducted on steel bolt connections, which shows the proposed methodology can achieve high accuracy and real‐time speed in bolt loosening assessment.