Computer Vision System Research Articles

A computer vision system for apple fruit sizing by means of low-cost depth camera and neural network application

AbstractFruit size is crucial for growers as it influences consumer willingness to buy and the price of the fruit. Fruit size and growth along the seasons are two parameters that can lead to more precise orchard management favoring production sustainability. In this study, a Python-based computer vision system (CVS) for sizing apples directly on the tree was developed to ease fruit sizing tasks. The system is made of a consumer-grade depth camera and was tested at two distances among 17 timings throughout the season, in a Fuji apple orchard. The CVS exploited a specifically trained YOLOv5 detection algorithm, a circle detection algorithm, and a trigonometric approach based on depth information to size the fruits. Comparisons with standard-trained YOLOv5 models and with spherical objects were carried out. The algorithm showed good fruit detection and circle detection performance, with a sizing rate of 92%. Good correlations (r > 0.8) between estimated and actual fruit size were found. The sizing performance showed an overall mean error (mE) and RMSE of + 5.7 mm (9%) and 10 mm (15%). The best results of mE were always found at 1.0 m, compared to 1.5 m. Key factors for the presented methodology were: the fruit detectors customization; the HoughCircle parameters adaptability to object size, camera distance, and color; and the issue of field natural illumination. The study also highlighted the uncertainty of human operators in the reference data collection (5–6%) and the effect of random subsampling on the statistical analysis of fruit size estimation. Despite the high error values, the CVS shows potential for fruit sizing at the orchard scale. Future research will focus on improving and testing the CVS on a large scale, as well as investigating other image analysis methods and the ability to estimate fruit growth.

Intelligent railroad inspection and monitoring

Railways are essential to the global transportation infrastructure, providing eco-friendly and economical solutions for the movement of freight and passengers. Inspecting and maintaining extensive rail networks timely poses significant challenges. My group and collaborators have focused on automated railroad inspection technologies, emphasizing the use of deep learning and computer vision to overcome the limitations of traditional manual inspections. Our research introduces groundbreaking real-time inspection methods, leveraging a specialized dataset of railroad components for enhanced instance segmentation models, achieving unprecedented accuracy and inference speeds. The developed computer vision systems efficiently detect track components and their changes over time, and also quantify rail surface defects. Additionally, our work extends to improving railroad crossing safety, utilizing deep learning frameworks for the detection of unusual pedestrian behaviors and object identification, aimed at reducing crossing incidents and improving emergency response times. Our future research directions aim to further refine the cost-effectiveness and autonomy of railroad inspection systems. Through these innovations, we hope to aid in the inspection and maintenance of railroads, offering practical solutions for railroad and other civil engineering applications.

Development and Evaluation of a Computer Vision System for Robot Navigation and Object Recognition in Real-World Environments

Development and Evaluation of a Computer Vision System for Robot Navigation and Object Recognition in Real-World Environments

LAMA-WAVELET: IMAGE IMPAINTING WITH HIGH QUALITY OF FINE DETAILS AND OBJECT EDGES

Context. The problem of the image impainting in computer graphic and computer vision systems is considered. The subject of the research is deep learning convolutional neural networks for image inpainting.
 Objective. The objective of the research is to improve the image inpainting performance in computer vision and computer graphics systems by applying wavelet transform in the LaMa-Fourier network architecture.
 Method. The basic LaMa-Fourier network decomposes the image into global and local texture. Then it is proposed to improve the network block, processing the global context of the image, namely, the spectral transform block. To improve the block of spectral transform, instead of Fourier Unit Structure the Simple Wavelet Convolution Block elaborated by the authors is used. In this block, 3D wavelet transform of the image on two levels was initially performed using the Daubechies wavelet db4. The obtained coefficients of 3D wavelet transform are splitted so that each subband represents a separate feature of the image. Convolutional layer, batch normalization and ReLU activation function are sequentially applied to the results of splitting of coefficients on each level of wavelet transform. The obtained subbands of wavelet coefficients are concatenated and the inverse wavelet transform is applied to them, the result of which is the output of the block. Note that the wavelet coefficients at different levels were processed separately. This reduces the computational complexity of calculating the network outputs while preserving the influence of the context of each level on image inpainting. The obtained neural network is named LaMa-Wavelet. The FID, PSNR, SSIM indexes and visual analysis were used to estimate the quality of images inpainted with LaMa-Wavelet network.
 Results. The proposed LaMa-Wavelet network has been implemented in software and researched for solving the problem of image inpainting. The PSNR of images inpainted using the LaMa-Wavelet exceeds the results obtained using the LaMa-Fourier network for narrow and medium masks in average by 4.5%, for large masks in average by 6%. The LaMa-Wavelet applying can enhance SSIM by 2–4% depending on a mask size. But it takes 3 times longer to inpaint one image with LaMa-Wavelet than with LaMa-Fourier network. Analysis of specific images demonstrates that both networks show similar results of inpainting of a homogeneous background. On complex backgrounds with repeating elements the LaMa-Wavelet is often more effective in restoring textures.
 Conclusions. The obtained LaMa-Wavelet network allows to improve the image inpainting with large masks due to applying wavelet transform in the LaMa network architecture. Namely, the quality of reconstruction of image edges and fine details is increased.

Tau-Cell-Based Analog Silicon Retina With Spatio- Temporal Filtering and Contrast Gain Control.

Developing precise artificial retinas is crucial because they hold the potential to restore vision, improve visual prosthetics, and enhance computer vision systems. Emulating the luminance and contrast adaption features of the retina is essential to improve visual perception and efficiency to provide an environment realistic representation to the user. In this article, we introduce an artificial retina model that leverages its potent adaptation to luminance and contrast to enhance vision sensing and information processing. The model has the ability to achieve the realization of both tonic and phasic cells in the simplest manner. We have implemented the retina model using 0.18 μm process technology and validated the accuracy of the hardware implementation through circuit simulation that closely matches the software retina model. Additionally, we have characterized a single pixel fabricated using the same 0.18 μm process. This pixel demonstrates an 87.7-% ratio of variance with the temporal software model and operates with a power consumption of 369 nW.

Review of Poultry Monitoring using Computer Vision

Poultry farming is an important unit in global agronomy, contributing immensely to the production of meat and eggs. Safeguarding the health and welfare of poultry is vital for ethical and financial reasons. In recent years, computer vision awareness has gained prominence as a powerful tool for poultry monitoring. This review paper provides an outline of the application of computer vision in poultry monitoring. We explore the different phases of this technology, with real-time image acquisition, object recognition, and behavior analysis. By connecting cameras and sophisticated algorithms, a computer vision system can distinguish unusual behavior, track poultry activities and observe its environmental conditions. Furthermore, this study discovers the benefits of computer vision in poultry farming, including early detection of disease, better production efficiency and improved animal welfare. In conclusion, the application of computer vision in monitoring poultry holds immense potential for the industry; by providing a corridor to a more sustainable and ethical poultry farming system with increased productivity. This paper equally discusses recent developments, challenges, forthcoming prospects in the field and academic research gaps identified.

Evaluating the Performance of Voting Classifier in Multiclass Classification of Dry Bean Varieties

This study explores the application of a voting classifier, integrating Decision Tree, Logistic Regression, and Gaussian Naive Bayes models, for the multiclass classification of dry bean varieties. Utilizing a dataset comprising 13,611 images of dry bean grains, captured through a high-resolution computer vision system, we extracted 16 features to train and test the classifier. Through a rigorous 5-fold cross-validation process, we assessed the model's performance, focusing on accuracy, precision, recall, and F1-score metrics. The results demonstrated significant variability in the classifier's performance across different data subsets, with accuracy rates fluctuating between 31.23% and 96.73%. This variability highlights the classifier's potential under specific conditions while also indicating areas for improvement. The research contributes to the agricultural informatics field by showcasing the effectiveness and challenges of using ensemble learning methods for crop variety classification, a crucial task for enhancing agricultural productivity and food security. Recommendations for future research include exploring additional features to improve model generalization, extending the dataset for broader applicability, and comparing the voting classifier's performance with other ensemble methods or advanced machine learning models. This study underscores the importance of machine learning in advancing agricultural classification tasks, paving the way for more efficient and accurate crop sorting and grading processes.

Design, integration, and field evaluation of a robotic blossom thinning system for tree fruit crops

AbstractThe United States (US) apple industry relies heavily on semi‐skilled manual labor force for essential field operations such as training, pruning, blossom and green fruitlet thinning, and harvesting. Blossom thinning is one of the crucial crop‐load management practices to achieve desired crop load, fruit quality, and return bloom. While several techniques such as chemical and mechanical thinning are available for large‐scale blossom thinning, such approaches often yield unpredictable thinning results and may damage the canopy, spurs, and leaf tissue. Hence, growers still depend on laborious, labor‐intensive, and expensive manual hand blossom thinning for desired thinning outcomes. This research presents a robotic solution for precision blossom thinning in apple orchards using a deep learning‐based computer vision system, a six‐degrees‐of‐freedom UR5e robotic manipulator, and an electrically actuated miniature end‐effector. The integrated robotic system was evaluated in a commercial apple orchard which showed promising results for targeted and selective blossom thinning. Two thinning approaches, center and boundary thinning, were investigated to evaluate the system's ability to remove varying proportions of flowers from apple flower clusters. During boundary thinning, the end‐effector was actuated around the cluster boundary, while center thinning involved end‐effector actuation only at the cluster centroid for a fixed duration of 2 s. Field evaluation results showed that the boundary thinning approach thinned 67.2% of flowers from the targeted clusters with a cycle time of 9.0 s per cluster, whereas the center thinning approach thinned 59.4% of flowers with a cycle time of 7.2 s per cluster. Upon further improvement for commercial adoption, the proposed system could help address problems faced by apple growers with current hand, chemical, and mechanical blossom thinning approaches.

Introducing surgical intelligence in gynecology: Automated identification of key steps in hysterectomy.

The analysis of surgical videos using artificial intelligence holds great promise for the future of surgery by facilitating the development of surgical best practices, identifying key pitfalls, enhancing situational awareness, and disseminating that information via real-time, intraoperative decision-making. The objective of the present study was to examine the feasibility and accuracy of a novel computer vision algorithm for hysterectomy surgical step identification. This was a retrospective study conducted on surgical videos of laparoscopic hysterectomies performed in 277 patients in five medical centers. We used a surgical intelligence platform (Theator Inc.) that employs advanced computer vision and AI technology to automatically capture video data during surgery, deidentify, and upload procedures to a secure cloud infrastructure. Videos were manually annotated with sequential steps of surgery by a team of annotation specialists. Subsequently, a computer vision system was trained to perform automated step detection in hysterectomy. Analyzing automated video annotations in comparison to manual human annotations was used to determine accuracy. The mean duration of the videos was 103 ± 43 min. Accuracy between AI-based predictions and manual human annotations was 93.1% on average. Accuracy was highest for the dissection and mobilization step (96.9%) and lowest for the adhesiolysis step (70.3%). The results of the present study demonstrate that a novel AI-based model achieves high accuracy for automated steps identification in hysterectomy. This lays the foundations for the next phase of AI, focused on real-time clinical decision support and prediction of outcome measures, to optimize surgeon workflow and elevate patient care.

NutriGreen image dataset: a collection of annotated nutrition, organic, and vegan food products.

In this research, we introduce the NutriGreen dataset, which is a collection of images representing branded food products aimed for training segmentation models for detecting various labels on food packaging. Each image in the dataset comes with three distinct labels: one indicating its nutritional quality using the Nutri-Score, another denoting whether it is vegan or vegetarian origin with the V-label, and a third displaying the EU organic certification (BIO) logo. To create the dataset, we have used semi-automatic annotation pipeline that combines domain expert annotation and automatic annotation using a deep learning model. The dataset comprises a total of 10,472 images. Among these, the Nutri-Score label is distributed across five sub-labels: Nutri-Score grade A with 1,250 images, grade B with 1,107 images, grade C with 867 images, grade D with 1,001 images, and grade E with 967 images. Additionally, there are 870 images featuring the V-Label, 2,328 images showcasing the BIO label, and 3,201 images without before-mentioned labels. Furthermore, we have fine-tuned the YOLOv5 segmentation model to demonstrate the practicality of using these annotated datasets, achieving an impressive accuracy of 94.0%. These promising results indicate that this dataset has significant potential for training innovative systems capable of detecting food labels. Moreover, it can serve as a valuable benchmark dataset for emerging computer vision systems.

Towards Robust Visual Understanding: from Recognition to Reasoning

Models that learn from data are widely and rapidly being deployed today for real-world use, but they suffer from unforeseen failures due to distribution shift, adversarial attacks, noise and corruption, and data scarcity. But many failures also occur because many modern AI tasks require reasoning beyond pattern matching -- and such reasoning abilities are difficult to formulate as data-based input-output function fitting. The reliability problem has become increasingly important under the new paradigm of semantic ``multimodal'' learning. My research provides avenues to develop robust and reliable computer vision systems, particularly by leveraging the interactions between vision and language. In this AAAI New Faculty highlights talk, I will cover three thematic areas of my research, ranging from robustness in computer vision, open-domain reliability in visual reasoning, and challenges and opportunities in evaluation of generative models. Readers are encouraged to refer to my website (www.tejasgokhale.com) for more details and updates from my lab's activities towards the goal of robust visual understanding.

A Model for Estimating the Economic Costs of Computer Vision Systems That Use Deep Learning

Deep learning, the most important subfield of machine learning and artificial intelligence (AI) over the last decade, is considered one of the fundamental technologies underpinning the Fourth Industrial Revolution. But despite its record-breaking history, deep learning’s enormous appetite for compute and data means that sometimes it can be too costly to practically use. In this paper, we connect technical insights from deep learning scaling laws and transfer learning with the economics of IT to propose a framework for estimating the cost of deep learning computer vision systems to achieve a desired level of accuracy. Our tool can be of practical use to AI practitioners in industry or academia to guide investment decisions.

VisionDrive: Smart Traffic Analysis with YOLOv7 and DeepSORT

This research introduces a novel vehicle division mechanism intended for dynamic metropolitan settings. Modern technology, including deep learning, computer vision, and real-time warning systems, reduces traffic, enhances road safety, and effectively manages unanticipated situations. A strong vehicle instance segmentation model that can recognize cars in real-time from many sources is first developed using the YOLO technique. In the next round, the cars are tallied and a predetermined threshold is established. If there are more cars in the video or image than a preset threshold, the system will identify this and assume that traffic is heavy. During the last stage, users receive information on congestion analysis and anomaly identification through user-friendly interfaces. This article aims to modernize traffic control by merging cutting-edge technologies with all-encompassing techniques to deliver a responsive, safe, and efficient urban transportation experience. KEYWORDS: YOLOv7, DeepSORT, OpenCV, Object Detection, Multiclass Segmentation, Object Counting.

Using Text Data, Neural Networks are Trained to Develop Emotions that Mimic Human Emotional Understanding

Recent methods of AI have revolutionized the field of computer science. Different sub-sectors of artificial intelligence (AI), like natural language processing (NLP) models, generative AI, computer vision, autonomous and recommendation systems, cybersecurity, quantum computing, etc., have helped automate human tasks, resulting in a tremendous amount of time and energy being saved. Despite the massive development of AI, all AI models lack one major factor, which is emotion. How can emotion be built into AI in order to for it to develop the emotional intelligence of the human brain to interpret and understand emotions so that it could create more human-friendly interactions? In this paper, we hypothesized developing emotions in neural networks as predictive sentiment analysis models using text data in order to replicate the emotional intelligence of the human brain to benefit human relationships. By using the Anaconda Repository, NVIDIA’s CUDA Toolkit, and TensorFlow, we were able to create a sentiment prediction model that achieved an accuracy of 94% and predicted the six basic emotions of joy, sadness, anger, fear, love, and surprise. Concluding this research, we observed that neural networks can develop the habit of recognizing emotions. This can be further fed into complex AI algorithms and systems to fine-tune emotional intelligence, resulting in more natural interactions, benefiting humans in

Artificial Intelligence Methods in Automated Unmanned Aerial Vehicles Control Systems

One of the main problems in ensuring the unmanned aerial systems (UAS) safety and control performance indicators is the operational analysis organization of heterogeneous data coming from on-board sensors and the formation of adequate recommendations and decisions on their basis of flight missions implementation. In recent years, there have been many research papers devoted to solving this problem using artificial intelligence (AI) methods. The article discusses AI methods for using in tasks related to UAS. We have described the sources of information to generate the data necessary for the application of AI methods. We have classified typical tasks of computer vision and navigation systems for solving using AI methods. We have analyzed the generally accepted classification of AI methods within the scope of the research subject. At the same time, special attention is paid to the features of AI methods that allow solving many well-known problems of recognition, approximation, optimization for UAS target and navigation tasks realization and effective operator support. In particular, we have considered neural networks, decision trees, support vector machines, k-nearest neighbors, genetic, ant colony algorithms, artificial immune systems. Currently, the hardware allows integrating complex algorithms based on these methods on board and widely using them in flight missions. The results of the study conducted as part of the review are illustrated by examples from the scientific publications.

Target Detection and Segmentation Technology for Zero-shot Learning

Zero-shot learning (ZSL) in the field of computer vision refers to enabling the model to recognize and understand categories that have not been encountered during the training phase. It is particularly critical for object detection and segmentation tasks, because these tasks require the model to have good generalization capabilities to unknown categories. Object detection requires the model to determine the location of the object, while segmentation further requires the precise demarcation of the object's boundaries. In ZSL research, knowledge representation and transfer are core issues. Researchers have tried to use semantic attributes as a knowledge bridge to connect categories seen during the training phase and categories not seen during the testing phase. These attributes may be color, shape, etc., but this method requires accurate attribute annotation, which is often not easy to achieve in practice. Therefore, researchers have begun to explore the use of non-visual information such as knowledge maps and text descriptions to enrich the recognition capabilities of models, but this also introduces the challenge of information integration and alignment. At present, ZSL has made certain progress in target detection and segmentation tasks, but there is still a significant gap compared with traditional supervised learning. This is mainly due to the limited ability of ZSL models to generalize to new categories. To this end, researchers have begun to explore combining ZSL with other technologies, such as generative adversarial networks (GANs) and reinforcement learning, to enhance the model's detection and segmentation capabilities for new categories. Future research needs to focus on several aspects. The first is how to design a more effective knowledge representation and transfer mechanism so that the model can better utilize existing knowledge. The second step is to develop new algorithms to improve the performance of ZSL in complex environments. In addition, research should focus on how to reduce the dependence on computing resources so that the ZSL method can run effectively in resource-limited environments. In summary, the research on target detection and segmentation technology of zero-shot learning is a cutting-edge topic in the field of computer vision. Despite the challenges, with the deepening of research, we expect these technologies to contribute to improving the generalization ability and intelligence level of computer vision systems.

Advancements in Computer Vision Applications for Traffic Surveillance Systems

With the increasing need for efficient traffic surveillance systems, computer vision emerges as a promising technology to address various challenges in this domain. This paper aims to provide an overview of recent advancements in the application of computer vision in traffic surveillance. It commences by discussing fundamental concepts of computer vision and its role in traffic monitoring, followed by an exploration of various techniques and algorithms used in computer vision systems, including object detection, tracking, and classification. Furthermore, it investigates the integration of computer vision with other technologies such as machine learning and deep learning, emphasizing their impact on enhancing the accuracy and efficiency of traffic surveillance systems. The review explores specific applications of computer vision in traffic surveillance, such as traffic flow monitoring, anomaly detection, and license plate recognition. Additionally, it addresses challenges and limitations associated with the implementation of computer vision in real-world traffic scenarios. Synthesizing existing literature, case studies, and research findings, this review offers a comprehensive insight into the current state-of-the-art in the application of computer vision for traffic surveillance systems. The paper concludes with a discussion of future directions and potential research opportunities in this rapidly evolving field.

Novel computer vision tools applied to marine recreational fisheries spatial planning

Successful marine spatial planning relies on understanding patterns of human use, with accurate, detailed, and up-to-date information about the spatial distribution of fishing effort. In commercial vessels, tracking systems like the vessel monitoring system (VMS) or the automatic identification system (AIS) have helped to maintain and enhance the biodiversity of areas by generating large sources of positional data that served for commercial marine spatial planning. However, there is no regulation regarding location systems such as VMS or AIS for marine recreational fishing boats. Obtaining spatial data on marine recreational fishing can be difficult and time-intensive given the widespread and variable nature of the fleet. Remote cameras and computer vision systems are increasingly used to overcome the cost limitations of these conventional methods. Here we show a novel high-resolution and low-cost tracking system based on photo time-lapses and state-of-the-art computer vision algorithms, including deep learning, to automatically classify and obtain precise trajectories of fishing and cruising boats in coastal areas. Our method contributes to the automatic surveillance of marine protected areas by providing an image-based tool for automatic, real-time monitoring. Our method also allows for determining the intensity and spatial-temporal distribution of recreational fishing effort, important to defining the sustainability of the activity and coastal areas. We finally discuss the opportunities and limitations of computer vision tools applied to marine recreational fisheries spatial planning.

Manufacturing, development and control of a two-way 3D printed soft actuator actuated with SMAs

In this contribution, the development of a novel two-way 3D printed soft actuator actuated with shape memory alloys (SMAs) is presented, considering all the stages from the design, manufacturing, control, and implementation. The SMAs are integrated into the 3D printed composite using thermoplastic polyurethane (TPU). In order to measure the deflection of the soft actuator a computer vision system was implemented. With these measures and using system identification techniques, a mathematical model was developed, which describes the dynamics of the prototype and helps to design of a controller. However, precise control of deflection in systems actuated by SMAs is challenging due to their inherent nonlinearities and hysteretic behavior. To face this challenge, a proportional-integral (PI) controller was designed based on robust stability conditions. The effectiveness of the designed PI controller was validated through experimental results.

A 4-channelled hazy image input generation and deep learning-based single image dehazing

The images in foggy weather are often degraded which may affect the object detection process. Hence, in recent times, several schemes have been designed to eliminate the haze effect and enhance the performance of computer vision systems. Although these methods reduce the haze effect, they also produce over-saturation and over-degradation on most occasions. These problems occur due to inadequate utilization of the regional haze properties during the dehazing process. Therefore, a new dehazing model is proposed in this paper to address the issues faced by existing dehazing algorithms. The residual convolutional neural network (RCNN) with 14 layers is developed in the present work. In this model, features to indicate the extent of haze are extracted using the regional characteristics of a hazy image and are used as the fourth channel along with the three colour channels of a hazy image. The proposed RCNN is trained with 4-channel hazy images as input and their corresponding haze-free images as output. The results of our model are obtained with several images taken from three different datasets. Through the experimentation, quantitative and qualitative improvements are observed when compared to state-of-the-art dehazing methods.