Objects In Images Research Articles

Shape-Guided Detection: A joint network combining object detection and underwater image enhancement together

Most of the existing underwater image object detection methods involve pre-processing, such as using underwater image enhancement, to improve the accuracy of object detection. However, pre-processing methods are designed to improve the subjective perception of the human eye, which does not necessarily improve the object detection performance and consumes a large amount of computational resources. Therefore, in this paper, we creatively combine these two tasks and propose a Shape-Guided Detection network (SGD) to simultaneously optimize underwater image enhancement and object detection. In the SGD network, we innovatively incorporate the prior shape features as a learnable module embedded in it to fully explore the shape characteristics and structural details of the target object. To ensure that the prior knowledge can be effectively fused into the global network structure, we design a Shape Prior Enhancement module, which aims to realize the deep integration of the prior information with the local details. In order to optimize the stability of model training and enhance its convergence performance, a dual strategy of explicit and implicit constraints is ingeniously proposed in our method. We conduct extensive experiments on public datasets and the results show that the combination of our method with different detectors significantly improves the performance. The object detection performance reaches up to 0.491 mAP for optical images and 0.576 mAP for sonar images, and improves the preprocessing speed by 0.1 s.

Algoritma Deep Learning Untuk Pengenalan Gambar Jenis Daun

Image processing is a branch of informatics that deals with transforming one image into another using certain techniques. Deep learning algorithms have become one of the effective approaches to solving this problem. In this paper, we propose a deep learning algorithm that uses Convolutional Neural Networks (CNN) architecture to recognize leaf types based on a given leaf image. We outline the main steps in model development, including data pre-processing, CNN architecture selection, and model training. The experimental results show that the proposed deep learning algorithm can achieve a high level of accuracy in leaf-type image recognition. In this study, the CNN method is used to identify and classify objects in digital images, specifically leaves. The dataset used consists of 33 leaf classes, with a division of 16,500 data for training, 3,300 for validation, and 1,650 for testing. The training and validation processes were carried out in as many as 150 epochs, which resulted in the highest accuracy of 94% with the lowest loss of 0.28. While in the testing process, the accuracy value obtained reached 84%. The researched method, which integrates CNN with data augmentation and transfer learning, demonstrated superior performance with an accuracy of 94% in leaf type recognition. This outperforms other methods that rely solely on traditional CNN or do not utilize augmentation and transfer learning, which generally achieve lower accuracy rates. The combination of these techniques enables more robust feature extraction and better generalization, leading to more accurate and reliable classification results compared to other approaches.

IMPLEMENTATION OF NATURAL EXPERIMENTS IN PHYSICS USING COMPUTER VISION

Laboratory experiments in physics are a fundamental basis for studying physical phenomena occurring in nature and a methodological tool that provides visibility of the learning process and conducting experiments is important for the formation of students’ scientific worldview, deep understanding of physical laws and increasing interest in the study of physics. Existing in universities and schools, in addition to traditional ones, modern tools, technologies and approaches, such as virtual reality, augmented reality, computer modelling, online laboratories, virtual laboratory and others, are additional tools for improving the quality of the learning process and teaching techniques, which do not replace full-scale experiments, but only supplement them. In our opinion, for better learning, laboratory installations in physics are needed, with the help of which students can carry out real-life experiments and can broadcast them using innovative computer technologies for distance learning. To implement this task, we reviewed and analysed existing laboratory installations, identified their advantages and disadvantages, and then designed and developed alternative digital experimental set-ups for studying physics phenomena in laboratory conditions of educational institutions based on computer vision technology and presented the results of the study in this article. In carrying out the research tasks, effective methods of conducting scientific research were used, such as theoretical substantiation of the issue, experimental testing of the developed hardware and software systems and computer final processing of experimental data. In summary, the research described in the paper presents an innovative mechanism for integrating object tracking based on computer vision to improve the quality of measurements and new ways of conducting physics experiments. The mechanical laboratory complexes we have developed consist of hardware and software parts. The software part consists of server and client parts. The hardware consists of the main part - the scene, where the physical process takes place, i.e. where a physical object is located, such as a mathematical pendulum, an inclined plane, etc., with the help of which many physical phenomena and processes in mechanics can be demonstrated, and an additional part where a microcomputer and a camera are located. The operating principle of the laboratory installation is based on the use of computer vision technology, i.e. a system for monitoring the ongoing physical process, consisting of a digital camera for image processing, object identification and data export, and a microcomputer for processing experimental data. The use of the experimental installations in the process of teaching physics is a new model of teaching with a promising future in secondary and higher education, and the installations themselves will become tools for offline and online learning, due to the use of computer vision technology, revealing new opportunities and approaches to teaching.

Zero-shot autonomous robot manipulation via natural language

Smart manufacturing revolutionizes advanced automation by leveraging deep learning and robotics technology. Nevertheless, deep learning implementation in manufacturing faces several challenges in preparing datasets, training a model, and adapting to various applications. Furthermore, programming languages are required to develop deep learning models and communicate with robots, which is not straightforward to quickly and precisely reflect human intentions in manufacturing processes. To alleviate the issue, this study proposes an autonomous zero-shot robot manipulation framework via natural language. Specifically, the Segmentation Anything Model (SAM), a promptable image segmentation model, first segments objects in images captured by an RGB-D camera, which are passed to Generative Pre-trained Transformers (GPTs), a Large Language Model (LLM), to identify and localize a designated object based on human commands given in natural language. Afterwards, the Robot Operating System (ROS) manipulates a robot to a localized position. With no additional programming or specialized skills in object localization and robot manipulation, the framework demonstrates an autonomous robot automation approach through intuitive communication between humans and machines through natural language.

Zero-Shot Image Caption Inference System Based on Pretrained Models

Recently, zero-shot image captioning (ZSIC) has gained significant attention, given its potential to describe unseen objects in images. This is important for real-world applications such as human–computer interaction, intelligent education, and service robots. However, the zero-shot image captioning method based on large-scale pretrained models may generate descriptions containing objects that are not present in the image, which is a phenomenon termed “object hallucination”. This is because large-scale models tend to predict words or phrases with high frequency, as seen in the training phase. Additionally, the method set a limitation to the description length, which often leads to an improper ending. In this paper, a novel approach is proposed to address and reduce the object hallucination and improper ending problem in the ZSIC task. We introduce additional emotion signals as guidance for sentence generation, and we find that proper emotion will filter words that do not appear in the image. Moreover, we propose a novel strategy that gradually extends the number of words in a sentence to confirm the generated sentence is properly completed. Experimental results show that the proposed method achieves the leading performance on unsupervised metrics. More importantly, the subjective examples illustrate the effect of our method in improving hallucination and generating properly ending sentences.

Using computer vision to classify, locate and segment fire behavior in UAS-captured images

The widely adaptable capabilities of artificial intelligence, in particular deep learning and computer vision have led to significant research output regarding flame and smoke detection. The composition of flame and smoke, also described as fire behavior, can be considerably different depending on factors like weather, fuels, and the specific landscape fire is being observed on. The ability to detect definable classes of fire behavior using computer vision has not been explored and could be helpful given it often dictates how firefighters respond to fire situations. To test whether types of fire behavior could be reliably classified, we collected and labeled a unique unmanned aerial system (UAS) image dataset of fire behavior classifications to be trained and validated using You Only Look Once (YOLO) detection models. Our 960 labeled images were sourced from over 21 h of UAS video collected during prescribed fire operations covering a large region of Texas and Louisiana, United States. National Wildfire Coordinating Group (NWCG) fire behavior observations and descriptions served as a reference for determining fire behavior classes during labeling. YOLOv8 models were trained on NWCG Rank 1–3 fire behavior descriptions in grassland, shrubland, forested, and combined fire regimes within our study area. Models were first trained and validated on classifying isolated image objects of fire behavior, and then separately trained to locate and segment fire behavior classifications in UAS images. Models trained to classify isolated image objects of fire behavior consistently performed at a mAP of 0.808 or higher, with combined fire regimes producing the best results (mAP = 0.897). Most segmentation models performed relatively poorly, except for the forest regime model at a box (locate) and mask (segment) mAP of 0.59 and 0.611, respectively. Our results indicate that classifying fire behavior with computer vision is possible in different fire regimes and fuel models, whereas locating and segmenting fire behavior types around background information is relatively difficult. However, it may be a manageable task with enough data, and when models are developed for a specific fire regime. With an increasing number of destructive wildfires and new challenges confronting fire managers, identifying how new technologies can quickly assess wildfire situations can assist wildfire responder awareness. Our conclusion is that levels of abstraction deeper than just detection of smoke or flame are possible using computer vision and could make even more detailed aerial fire monitoring possible using a UAS.

Transformer-based dual-view X-ray security inspection image analysis

Artificial intelligence technology is rapidly advancing and has been widely applied in the field of intelligent security inspection. Utilizing computer vision technology to detect prohibited items in X-ray images has drawn much attention. Due to the transmission effect of X-rays, single-view security inspection images are prone to object occlusion and poor imaging angles, which seriously affects the performance of object detection models. Dual-view security inspection equipment can simultaneously capture X-ray transmission images of the item under inspection from both horizontal and vertical angles, which can effectively address issues of poor imaging angles and object occlusions that single-view imaging cannot resolve. In this paper, we introduced the artificial intelligence technology in dual-view security inspection image analysis, and proposed the dual-view feature fusion and prohibited item detection model in X-ray security inspection images based on the Vision Transformer framework. The detection model contains two input channels: the main and secondary channel. The main function of the main channel is to detect prohibited items in security inspection images, while the secondary channel is dedicated to providing effective feature information of prohibited items for the main channel. Two feature interaction modules are applied in the proposed model to realize dual channel information exchange and supplement from local and global perspectives respectively. Simulation results based on the public Dualray dataset have demonstrated the state-of-the-art performance of the proposed dual-view X-ray image detection model. Code is available at https://github.com/zhg-SZPT/Trans2Ray.

An approximate method of estimating the depth of objects using wavelet-based transformation

Modern three-dimensional computer vision technologies are actively developing, offering new solutions based on the analysis of environmental details. One of the key tasks is to determine the depth of location of objects in images, which requires efficient and fast methods of their processing. This study proposes a new depth estimation method for 3D computer vision based on wavelet transform optimization. Solutions are offered to simplify the disparity calculation, which is traditionally used for the construction of depth maps, with the possibility of describing contours with adjustable detail based on the wavelet transform. The main advantage of this method is the increase in performance due to the allocation of the Haar wavelet carrier length in the extremum search area. The simulation confirmed the effectiveness of the proposed approach for constructing depth maps, which allows us to recommend it for use in unmanned aerial vehicles (UAVs) in conditions of limited computing and energy resources.

Estimating the Complexity of Objects in Images

Estimating the Complexity of Objects in Images

SOD-YOLOv8-Enhancing YOLOv8 for Small Object Detection in Aerial Imagery and Traffic Scenes.

Object detection, as a crucial aspect of computer vision, plays a vital role in traffic management, emergency response, autonomous vehicles, and smart cities. Despite the significant advancements in object detection, detecting small objects in images captured by high-altitude cameras remains challenging, due to factors such as object size, distance from the camera, varied shapes, and cluttered backgrounds. To address these challenges, we propose small object detection YOLOv8 (SOD-YOLOv8), a novel model specifically designed for scenarios involving numerous small objects. Inspired by efficient generalized feature pyramid networks (GFPNs), we enhance multi-path fusion within YOLOv8 to integrate features across different levels, preserving details from shallower layers and improving small object detection accuracy. Additionally, we introduce a fourth detection layer to effectively utilize high-resolution spatial information. The efficient multi-scale attention module (EMA) in the C2f-EMA module further enhances feature extraction by redistributing weights and prioritizing relevant features. We introduce powerful-IoU (PIoU) as a replacement for CIoU, focusing on moderate quality anchor boxes and adding a penalty based on differences between predicted and ground truth bounding box corners. This approach simplifies calculations, speeds up convergence, and enhances detection accuracy. SOD-YOLOv8 significantly improves small object detection, surpassing widely used models across various metrics, without substantially increasing the computational cost or latency compared to YOLOv8s. Specifically, it increased recall from 40.1% to 43.9%, precision from 51.2% to 53.9%, mAP0.5 from 40.6% to 45.1%, and mAP0.5:0.95 from 24% to 26.6%. Furthermore, experiments conducted in dynamic real-world traffic scenes illustrated SOD-YOLOv8's significant enhancements across diverse environmental conditions, highlighting its reliability and effective object detection capabilities in challenging scenarios.

STORM image denoising and information extraction.

Stochastic optical reconstruction microscopy (STORM) is extensively utilized in the fields of cell and molecular biology as a super-resolution imaging technique for visualizing cells and molecules. Nonetheless, the imaging process of STORM is frequently susceptible to noise, which can significantly impact the subsequent image analysis. Moreover, there is currently a lack of a comprehensive automated processing approach for analyzing protein aggregation states from a large number of STORM images. This paper initially applies our previously proposed denoising algorithm, UNet-Att, in STORM image denoising. This algorithm was constructed based on attention mechanism and multi-scale features, showcasing a remarkably efficient performance in denoising. Subsequently, we propose a collection of automated image processing algorithms for the ultimate feature extractions and data analyses of the STORM images. The information extraction workflow effectively integrates automated methods of image denoising, objective image segmentation and binarization, and object information extraction, and a novel image information clustering algorithm specifically developed for the morphological analysis of the objects in the STORM images. This automated workflow significantly improves the efficiency of the effective data analysis for large-scale original STORM images.

Improving Reference-based Distinctive Image Captioning with Contrastive Rewards

Distinctive Image Captioning (DIC) — generating distinctive captions that describe the unique details of a target image — has received considerable attention over the last few years. A recent DIC method proposes to generate distinctive captions by comparing the target image with a set of semantic-similar reference images, i . e ., reference-based DIC (Ref-DIC). It aims to force the generated captions to distinguish between the target image and the reference image. Unfortunately, reference images used by existing Ref-DIC works are easy to distinguish: these reference images only resemble the target image at scene-level and have few common objects, such that a Ref-DIC model can trivially generate distinctive captions even without considering the reference images. For example, if the target image contains objects “ towel ” and “ toilet ” while all reference images are without them, then a simple caption “ A bathroom with a towel and a toilet ” is distinctive enough to tell apart target and reference images. To ensure Ref-DIC models really perceive the unique objects (or attributes) in target images, we first propose two new Ref-DIC benchmarks. Specifically, we design a two-stage matching mechanism, which strictly controls the similarity between the target and reference images at the object-/attribute- level (v.s. scene-level). Secondly, to generate distinctive captions, we develop a Transformer-based Ref-DIC baseline TransDIC . It not only extracts visual features from the target image, but also encodes the differences between objects in the target and reference images. Taking one step further, we propose a stronger TransDIC++ , which consists of an extra contrastive learning module to make full use of the reference images. This new module is model-agnostic, which can be easily incorporated into various Ref-DIC architectures. Finally, for more trustworthy benchmarking, we propose a new evaluation metric named DisCIDEr for Ref-DIC, which evaluates both the accuracy and distinctiveness of the generated captions. Experimental results demonstrate that our TransDIC++ can generate distinctive captions. Besides, it outperforms several state-of-the-art models on the two new benchmarks over different metrics.

Spatial attention for human-centric visual understanding: An Information Bottleneck method

The selective visual attention mechanism in the Human Visual System (HVS) restricts the amount of information that reaches human visual awareness, allowing the brain to perceive high-fidelity natural scenes in real-time with limited computational cost. This selectivity acts as an “Information Bottleneck (IB)” that balances information compression and predictive accuracy. However, such information constraints are rarely explored in the attention mechanism for deep neural networks (DNNs). This paper introduces an IB-inspired spatial attention module for DNNs, which generates an attention map by minimizing the mutual information (MI) between the attentive content and the input while maximizing that between the attentive content and the output. We develop this IB-inspired attention mechanism based on a novel graphical model and explore various implementations of the framework. We show that our approach can yield attention maps that neatly highlight the regions of interest while suppressing the backgrounds, and are interpretable for the decision-making of the DNNs. To validate the effectiveness of the proposed IB-inspired attention mechanism, we apply it to various computer vision tasks including image classification, fine-grained recognition, cross-domain classification, semantic segmentation, and object detection. Extensive experiments demonstrate that it bootstraps standard DNN structures quantitatively and qualitatively for these tasks.

From Continuous to Discrete to Continuous – Text-to-Image Models as Limit to Indeterminate Phantasy

This essay analyses the interplay of indeterminacy and determinacy in the experience of images generated through text-to-image (T2I) models. Through an interdisciplinary approach, it uncovers three layers of indeterminacy: the computational indeterminacy inherent in text-to-image model processes, the indeterminacy of imagination in Husserl’s concept of protean phantasy, and finally the visual indeterminacy that figures in meaning making in all images. Generated images pass through these stages of indeterminacy, transforming indeterminate phantasy into determined visual objects, resulting in a conflict of consciousness between potential and actual. A distinction emerges between artificial phantasy, characterized by quasi-experience, and artificial imagination, grounded in images both as training data and perceptual image objects. As mediators between indeterminacy and determination, T2I images appear as technical media that mediate multiple forms of indeterminacy, showing the circulation between phantasy and imagination, between continuous and discrete. The generated image marks the limit of the unlimited indeterminate imagination.

Global–Local Query-Support Cross-Attention for Few-Shot Semantic Segmentation

Few-shot semantic segmentation (FSS) models aim to segment unseen target objects in a query image with scarce annotated support samples. This challenging task requires an effective utilization of support information contained in the limited support set. However, the majority of existing FSS methods either compressed support features into several prototype vectors or constructed pixel-wise support-query correlations to guide the segmentation, which failed in effectively utilizing the support information from the global–local perspective. In this paper, we propose Global–Local Query-Support Cross-Attention (GLQSCA), where both global semantics and local details are exploited. Implemented with multi-head attention in a transformer architecture, GLQSCA treats every query pixel as a token, aggregates the segmentation label from the support mask values (weighted by the similarities with all foreground prototypes (global information)), and supports pixels (local information). Experiments show that our GLQSCA significantly surpasses state-of-the-art methods on the standard FSS benchmarks PASCAL-5i and COCO-20i.

Estimating Power, Performance, and Area for On-Sensor Deployment of AR/VR Workloads Using an Analytical Framework

Augmented Reality and Virtual Reality have emerged as the next frontier of intelligent image sensors and computer systems. In these systems, 3D die stacking stands out as a compelling solution, enabling in situ processing capability of the sensory data for tasks such as image classification and object detection at low power, low latency, and a small form factor. These intelligent 3D CMOS Image Sensor (CIS) systems present a wide design space, encompassing multiple domains (e.g., computer vision algorithms, circuit design, system architecture, and semiconductor technology, including 3D stacking) that have not been explored in-depth so far. This article aims to fill this gap. We first present an analytical evaluation framework, STAR-3DSim, dedicated to rapid pre-RTL evaluation of 3D-CIS systems capturing the entire stack from the pixel layer to the on-sensor processor layer. With STAR-3DSim, we then propose several knobs for PPA (power, performance, area) improvement of the Deep Neural Network (DNN) accelerator that can provide up to 53%, 41%, and 63% reduction in energy, latency, and area, respectively, across a broad set of relevant AR/VR workloads. Last, we present full-system evaluation results by taking image sensing, cross-tier data transfer, and off-sensor communication into consideration.

A novel target detection method with dual‐domain multi‐frequency feature in side‐scan sonar images

AbstractSide‐scan sonar (SSS) detection is a key method in underwater environmental security and subsea resource development. However, many detection approaches primarily concentrate on tracking the evolution path of optical image object detection tasks when using acoustic images, resulting in complex structures and limited versatility. To tackle this issue, we introduce a pioneering dual‐domain multi‐frequency network (D2MFNet) meticulously crafted to harness the distinct characteristics of SSS image detection. In D2MFNet, a novel method for optimizing and improving the detection sensitivity in different frequency ranges called multi‐frequency combined attention mechanism (MFCAM) is proposed. This mechanism amplifies the relevance of dual‐domain features across different channels and spaces. Moreover, we introduce a dual‐domain feature pyramid network (D2FPN) significantly augments the depth and breadth of feature information in underwater small datasets. The methods offer plug‐and‐play functionality with substantial performance enhancements. Extensive experiments are conducted to validate the efficacy of the proposed techniques, and the results showcase their state‐of‐the‐art performance. MFCAM improves the mAP by 16.9% in the KLSG dataset and 15.5% in the SCTD dataset. The mAP of D2FPN was improved by 8.4% in the KLSG dataset and by 9.8% in the SCTD dataset. The code and models will be publicly available at https://dagshub.com/estrellaww00/D2MFNet.

A novel multiscale cGAN approach for enhanced salient object detection in single haze images

In computer vision, image dehazing is a low-level task that employs algorithms to analyze and remove haze from images, resulting in haze-free visuals. The aim of Salient Object Detection (SOD) is to locate the most visually prominent areas in images. However, most SOD techniques applied to visible images struggle in complex scenarios characterized by similarities between the foreground and background, cluttered backgrounds, adverse weather conditions, and low lighting. Identifying objects in hazy images is challenging due to the degradation of visibility caused by atmospheric conditions, leading to diminished visibility and reduced contrast. This paper introduces an innovative approach called Dehaze-SOD, a unique integrated model that addresses two vital tasks: dehazing and salient object detection. The key novelty of Dehaze-SOD lies in its dual functionality, seamlessly integrating dehazing and salient object identification into a unified framework. This is achieved using a conditional Generative Adversarial Network (cGAN) comprising two distinct subnetworks: one for image dehazing and another for salient object detection. The first module, designed with residual blocks, Dark Channel Prior (DCP), total variation, and the multiscale Retinex algorithm, processes the input hazy images. The second module employs an enhanced EfficientNet architecture with added attention mechanisms and pixel-wise refinement to further improve the dehazing process. The outputs from these subnetworks are combined to produce dehazed images, which are then fed into our proposed encoder–decoder framework for salient object detection. The cGAN is trained with two modules working together: the generator aims to produce haze-free images, whereas the discriminator distinguishes between the generated haze-free images and real haze-free images. Dehaze-SOD demonstrates superior performance compared to state-of-the-art dehazing methods in terms of color fidelity, visibility enhancement, and haze removal. The proposed method effectively produces high-quality, haze-free images from various hazy inputs and accurately detects salient objects within them. This makes Dehaze-SOD a promising tool for improving salient object detection in challenging hazy conditions. The effectiveness of our approach has been validated using benchmark evaluation metrics such as mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM).

Evolutionary computation-based self-supervised learning for image processing: a big data-driven approach to feature extraction and fusion for multispectral object detection

The image object recognition and detection technology are widely used in many scenarios. In recent years, big data has become increasingly abundant, and big data-driven artificial intelligence models have attracted more and more attention. Evolutionary computation has also provided a powerful driving force for the optimization and improvement of deep learning models. In this paper, we propose an image object detection method based on self-supervised and data-driven learning. Differ from other methods, our approach stands out due to its innovative use of multispectral data fusion and evolutionary computation for model optimization. Specifically, our method uniquely combines visible light images and infrared images to detect and identify image targets. Firstly, we utilize a self-supervised learning method and the AutoEncoder model to perform high-dimensional feature extraction on the two types of images. Secondly, we fuse the extracted features from the visible light and infrared images to detect and identify objects. Thirdly, we introduce a model parameter optimization method using evolutionary learning algorithms to enhance model performance. Validation on public datasets shows that our method achieves comparable or superior performance to existing methods.

Decoding environmental impact with image-based CO2 emission analytics

As per the regulations of Intergovernmental Panel on Climate Change (IPCC), the country’s carbon emission data hold primacy. Nevertheless, establishing and utilizing a database pose a daunting challenge. This study presents the “Carbon Seer System,” a novel software prototype designed to inform and educate users about the carbon footprint during their production and in daily life. The development process involved a three-stage user interview process, ensuring system’s alignment with user needs and preferences. The system employs machine learning and visual recognition technology, including convolutional neural networks (CNNs), feature pyramid networks, and self-attention mechanisms, enabling users to analyze captured images for carbon emission factors. It automatically identifies the carbon emission factors of industrial products, energy producers, household producers, waste treatment, and transportation. A unique “group detection” method allows for the simultaneous analysis of multiple objects in a single image, enhancing user convenience. Additionally, the software features a carbon footprint tracker and a carbon sink dashboard, providing users with insights into their carbon emissions and the efforts needed for offsetting. The study concludes that the “Carbon Seer System” represents a significant step towards individual enablement at understanding and actively participating in a low-carbon lifestyle.