RGB Sensor Research Articles

Acacia dealbata classification from aerial imagery acquired using unmanned aerial vehicles

Non-native plant species can have a negative impact in the ecosystems and in local economies when they spread uncontrollably. Monitoring tools can support their management and spread. In this paper, an exploratory approach is presented for pixelwise detection of Acacia dealbata from UAV-based imagery acquired from RGB and multispectral sensors. Four machine learning algorithms - k-nearest neighbors (KNN), random forest (RF), adaptive boosting (AdaBoost) and a linear kernel SVM (LSVM) - are trained using four datasets (hue, saturation and value - HSV, multispectral - MSP, RGB and a combination of all features) and their classification performance is evaluated. RF classifier obtained the overall best performance, with an accuracy above 86% in all data combinations, with LSVM showing the poorer results. Obtained results are encouraging for monitoring invasive species and can serve as a base for future improvements to detect invasive species.

Multispectral-to-RGB Knowledge Distillation for Remote Sensing Image Scene Classification

Scene classification is a fundamental task in the remote sensing (RS) field, assigning semantic labels to RS images. Multispectral (MS) images play an essential role in scene classification as they contain richer spectral information than RGB images. However, MS images are not always available due to the higher cost and complexity of MS sensors compared to RGB sensors. To improve scene classification performance using only RGB images, in this letter, we propose a novel MS-to-RGB knowledge distillation framework (MS2RGB-KD) that transfers MS knowledge from a teacher model to a student model. Specifically, our MS2RGB-KD drives a student model that requires only an RGB image as input to mimic the feature representations of different modalities extracted by the teacher model. Moreover, we introduce novel loss functions that encourage the student model to preserve intra-modal and inter-modal relationships of the feature representations in the teacher model. Experiments on the EuroSAT dataset demonstrate the effectiveness of MS2RGB-KD compared with other KD baselines.

An Embedded Framework for Fully Autonomous Object Manipulation in Robotic-Empowered Assisted Living

Most of the humanoid social robots currently diffused are designed only for verbal and animated interactions with users, and despite being equipped with two upper arms for interactive animation, they lack object manipulation capabilities. In this paper, we propose the MONOCULAR (eMbeddable autONomous ObjeCt manipULAtion Routines) framework, which implements a set of routines to add manipulation functionalities to social robots by exploiting the functional data fusion of two RGB cameras and a 3D depth sensor placed in the head frame. The framework is designed to: (i) localize specific objects to be manipulated via RGB cameras; (ii) define the characteristics of the shelf on which they are placed; and (iii) autonomously adapt approach and manipulation routines to avoid collisions and maximize grabbing accuracy. To localize the item on the shelf, MONOCULAR exploits an embeddable version of the You Only Look Once (YOLO) object detector. The RGB camera outcomes are also used to estimate the height of the shelf using an edge-detecting algorithm. Based on the item's position and the estimated shelf height, MONOCULAR is designed to select between two possible routines that dynamically optimize the approach and object manipulation parameters according to the real-time analysis of RGB and 3D sensor frames. These two routines are optimized for a central or lateral approach to objects on a shelf. The MONOCULAR procedures are designed to be fully automatic, intrinsically protecting sensitive users' data and stored home or hospital maps. MONOCULAR was optimized for Pepper by SoftBank Robotics. To characterize the proposed system, a case study in which Pepper is used as a drug delivery operator is proposed. The case study is divided into: (i) pharmaceutical package search; (ii) object approach and manipulation; and (iii) delivery operations. Experimental data showed that object manipulation routines for laterally placed objects achieves a best grabbing success rate of 96%, while the routine for centrally placed objects can reach 97% for a wide range of different shelf heights. Finally, a proof of concept is proposed here to demonstrate the applicability of the MONOCULAR framework in a real-life scenario.

Validation of Angle Estimation Based on Body Tracking Data from RGB-D and RGB Cameras for Biomechanical Assessment.

Motion analysis is an area with several applications for health, sports, and entertainment. The high cost of state-of-the-art equipment in the health field makes it unfeasible to apply this technique in the clinics' routines. In this vein, RGB-D and RGB equipment, which have joint tracking tools, are tested with portable and low-cost solutions to enable computational motion analysis. The recent release of Google MediaPipe, a joint inference tracking technique that uses conventional RGB cameras, can be considered a milestone due to its ability to estimate depth coordinates in planar images. In light of this, this work aims to evaluate the measurement of angular variation from RGB-D and RGB sensor data against the Qualisys Tracking Manager gold standard. A total of 60 recordings were performed for each upper and lower limb movement in two different position configurations concerning the sensors. Google's MediaPipe usage obtained close results compared to Kinect V2 sensor in the inherent aspects of absolute error, RMS, and correlation to the gold standard, presenting lower dispersion values and error metrics, which is more positive. In the comparison with equipment commonly used in physical evaluations, MediaPipe had an error within the error range of short- and long-arm goniometers.

Enhancing UAV images to improve the observation of submerged whales using a water column correction method

Enhancing <scp>UAV</scp> images to improve the observation of submerged whales using a water column correction method

Illumination-aware window transformer for RGBT modality fusion

Combination of RGB and thermal sensors has been proven to be useful for many vision applications. However, how to effectively fuse the information of two modalities remains a challenging problem. In this paper, we propose an Illumination-Aware Window Transformer (IAWT) fusion module to handle the RGB and thermal multi-modality fusion. Specifically, the IAWT fusion module adopts a window-based multi-modality attention combined with additional estimated illumination information. The window-based multi-modality attention infers dependency cross modalities within a local window, thus implicitly alleviate the problem caused by weakly spatial misalignment of the RGB and thermal image pairs within specific dataset. The introduction of estimated illumination feature enables the fusion module to adaptively merge the two modalities according to illumination conditions so as to make full use of the complementary characteristics of RGB and thermal images under different environments. Besides, our proposed fusion module is task-agnostic and data-specific, which means it can be used for different tasks with RGBT inputs. To evaluate the advances of the proposed fusion method, we embed the IAWT fusion module into different networks and conduct the experiments on various RGBT tasks, including pedestrian detection, semantic segmentation and crowd counting. Extensive results demonstrate the superior performance of our method.

Estimation of nitrogen content in wheat using indices derived from RGB and thermal infrared imaging

The important period of wheat grain accumulation is from the flowering stage to the filling stage, and the nitrogen content of wheat in this period is of great significance to the yield accumulation. With the rapid development of sensor technology, different sensors have been increasingly used for crop nitrogen status estimation due to their flexibility. This study aimed to investigate the use of a combination of image information from two proximal sensors (RGB and thermal sensors) to assess the nitrogen status of wheat at the reproductive growth stage. Previous studies have focused on estimating leaf N status at the nutritional growth stage of wheat, and the precision of N estimation is not high at the later stages. Considering that the canopy was composed of leaves and spikes in the reproductive stage, we integrated leaf N content and spike N content as plant N content for assessment. A two-year field trial was conducted, and this study used a Sony camera to acquire RGB images from flowering to maturity and obtained thermal images using the handle thermal infrared camera during the same period. Then, these images were further processed to extract the color features (17), the texture features (5) and temperature values (2). Based on these 24 indices, this study used three machine learning algorithms (i.e., Back-Propagation neural network (BP), Random Forest (RF) and Support Vector Regression (SVR)), resulted in nine estimation models based on a single dataset (i.e., c-based BP, te-based BP, t-based BP, c-based RF, te-based RF, t-based RF, c-based SVR, te-based SVR, t-based SVR) and 12 models based on data fusions (i.e., c+te-based BP, c+t-based BP, te+t-based BP, c+te+t-based BP, c+te-based RF, c+t-based RF, te+t-based RF, c+te+t-based RF, c+te-based SVR, c+t-based SVR, te+t-based SVR, c+te+t-based SVR). The performance of the 21 models was evaluated and compared with each other according to the coefficient of determination (R 2 ), root mean square error (RMSE) and residual prediction deviation (RPD) in nitrogen content estimation. The results show that the best model was the c+te+t-based RF, which was a model based on the combination of color features, texture features and temperature values. It achieved high accuracy in estimating plant N content (R 2 = 0.89, RMSE = 3.23 mg g −1 , RPD = 1.90). In conclusion, the combination of information from RGB and thermal images has good potential for application in monitoring crop N content at late reproductive stages, and plant temperature values can be used as effective indicators for assessing crop growth and nitrogen nutrient status. • Wheat temperature and temperature difference have the potential to monitor wheat growth. • Multi-source data (color, texture, and temperature information) based on the source-sink level have a good potential for monitoring nitrogen nutrient status in the later stages of wheat. • The fusion of information from RGB and thermal images has promising applications for monitoring the nitrogen nutrient status of crops. • The machine-learning models incorporating multiple features significantly improved the accuracy of the models for estimating nitrogen content in wheat plants.

PlaneFusion: Real-Time Indoor Scene Reconstruction With Planar Prior

Real-time dense SLAM techniques aim to reconstruct the dense three-dimensional geometry of a scene in real time with an RGB or RGB-D sensor. An indoor scene is an important type of working environment for these techniques. The planar prior can be used in this scenario to improve the reconstruction quality, especially for large low-texture regions that commonly occur in an indoor scene. This article fully explores the planar prior in a dense SLAM pipeline. First, we propose a novel plane detection and segmentation method that runs at 200 Hz on a modern graphics processing unit. Our algorithm for constructing global plane constraints is very efficient; hence, we use it in the process of each input frame for the camera pose estimation while maintaining the real-time performance. Second, we propose herein a plane-based map representation that greatly reduces the memory footprint of plane regions while keeping the geometric details on planes. The experiments reveal that our system yields superior reconstruction results with planar information running at more than 30 fps. Aside from speed and storage improvements, our technique also handles the low-texture problem in plane regions.

THE COFFEE NDVI MODELING USING BUILT-IN RGB PASSIVE SENSOR IN UAS

Studies carried out used different sensors and field applications can be accomplished to obtain information and make decisions related to coffee management. The objective of this work was to train machine learning algorithms to estimate the NDVI based on the GreenseekerTM active optical sensor in 20 arabica coffee cultivars in the experimental area of the National Institute of Coffee Science and Technology (INCT of Coffee) using a passive RGB sensor. in unmanned aerial system (UAS) and its relationship with foliage. With the spectral signatures recorded in the RGB images of the 20 coffee cultivars, the relationships with the foliage and the NDVI data obtained with the Greenseeker™ active optical sensor were analyzed. Through machine learning and digital image processing techniques, it was possible to obtain an NDVI equation using the RGB bands based on the NDVI of the Greenseeker TM active optical sensor. The results were satisfactory when compared to in situ data, providing the use of a simple and effective method of evaluating the vegetative vigor of different coffee cultivars.

A palm-sized wireless device for colorimetric nitrite determination in water

In this work a palm-sized, affordable, and wireless device is presented for colorimetric determination of nitrite in water, based on the Griess method. A RGB sensor APDS 9960 was used, as a detector, to determine the amount of nitrite in water and the color intensity of the dye produced by the reaction between nitrite and Griess reagent was detected. The detection system consisted of a green LED (λmax = 530 nm) used as light source and the signals were monitored by selecting the green channel. An ESP32 board was used for signal processing and this ESP board includes a housing having space for receiving a lithium-ion battery, used for the power source of the circuit. Also, this boards act like a router which was employed to transfer data between the devices. A detection limit of 2 μgL−1 (3σ) was achieved by the proposed procedure with a relative standard deviation value varied from 1.6 to 2.5 for intra-day (and from 2.4 to 2.7 for inter-day precision, when a Green and White LEDs were used, respectively. With a low reagent demand of 20 μL, the proposed device is environmentally friendly, ideal for in situ determination and the resulting battery life of a 7-hours is feasibly for nitrite monitoring in different water samples. A comparison between the presented procedure employing a white and a green LEDs, used as light source, was conducted for nitrite determination and the results were compared with those obtained by UV–vis. The proposed procedure employing the green LED was more sensitive and provides greater accuracy.

Detection of Ecballium elaterium in hedgerow olive orchards using a low-cost uncrewed aerial vehicle and open-source algorithms.

Ecballium elaterium (common name: squirting cucumber) is an emerging weed problem in hedgerow or superintensive olive groves under no tillage. It colonizes the inter-row area infesting the natural or sown cover crops, and is considered a hard-to-control weed. Research in other woody crops has shown E. elaterium has a patchy distribution, which makes this weed susceptible to design a site-specific control strategy only addressed to E. elaterium patches. Therefore, the aim of this work was to develop a methodology based on the analysis of imagery acquired with an uncrewed aerial vehicle (UAV) to detect and map E. elaterium infestations in hedgerow olive orchards. The study was conducted in two superintensive olive orchards, and the images were taken using a UAV equipped with an RGB sensor. Flights were conducted on two dates: in May, when there were various weeds infesting the orchard, and in September, when E. elaterium was the only infesting weed. UAV-orthomosaics in the first scenario were classified using random forest models, and the orthomosaics from September with E. elaterium as the only weed, were analyzed using an unsupervised algorithm. In both cases, the overall accuracies were over 0.85, and the producer's accuracies for E. elaterium ranged between 0.74 and 1.00. These results allow the design of a site-specific and efficient herbicide control protocol which would represent a step forward in sustainable weed management. The development of these algorithms in free and open-source software fosters their application in small and medium farms. © 2022 The Authors. Pest Management Science published by John Wiley & Sons Ltd on behalf of Society of Chemical Industry.

Multispectral Benchmark Dataset and Baseline for Forklift Collision Avoidance

In this paper, multispectral pedestrian detection is mainly discussed, which can contribute to assigning human-aware properties to automated forklifts to prevent accidents, such as collisions, at an early stage. Since there was no multispectral pedestrian detection dataset in an intralogistics domain, we collected a dataset; the dataset employs a method that aligns image pairs with different domains, i.e. RGB and thermal, without the use of a cumbersome device such as a beam splitter, but rather by exploiting the disparity between RGB sensors and camera geometry. In addition, we propose a multispectral pedestrian detector called SSD 2.5D that can not only detect pedestrians but also estimate the distance between an automated forklift and workers. In extensive experiments, the performance of detection and centroid localization is validated with respect to evaluation metrics used in the driving car domain but with distinct categories, such as hazardous zone and warning zone, to make it more applicable to the intralogistics domain.

An Investigation of Winter Wheat Leaf Area Index Fitting Model Using Spectral and Canopy Height Model Data from Unmanned Aerial Vehicle Imagery

The leaf area index (LAI) is critical for the respiration, transpiration, and photosynthesis of crops. Color indices (CIs) and vegetation indices (VIs) extracted from unmanned aerial vehicle (UAV) imagery have been widely applied to the monitoring of the crop LAI. However, when the coverage of the crop canopy is large and only spectral data are used to monitor the LAI of the crop, the LAI tends to be underestimated. The canopy height model (CHM) data obtained from UAV-based point clouds can represent the height and canopy structure of the plant. However, few studies have been conducted on the use of the CHM data in the LAI modelling. Thus, in this study, the feasibility of combining the CHM data and CIs and VIs, respectively, to establish LAI fitting models for winter wheat in four growth stages was investigated, and the impact of image resolution on the extraction of remote sensing variables (the CHM data, CIs, and VIs) and on the accuracy of the LAI models was evaluated. Experiments for acquiring remote sensing images of wheat canopies during the four growth stages from the RGB and multispectral sensors carried by a UAV were carried out. The partial least squares regression (PLSR), random forest regression (RFR), and support vector machine regression (SVR) were used to develop the LAI fitting models. Results showed that the accuracy of the wheat LAI models can be improved in the entire growth stages by the use of the additional CHM data with the increment of 0.020–0.268 in R2 for three regression methods. In addition, the improvement from the Cis-based models was more noticeable than the Vis-based ones. Furthermore, the higher the spatial resolution of the CHM data, the better the improvement made by the use of the additional CHM data. This result provides valuable insights and references for UAV-based LAI monitoring.

A Novel Background Modeling Algorithm for Hyperspectral Ground-Based Surveillance and Through-Foliage Detection.

Foliage penetration is an unsolved important part of border surveillance of remote areas between regular border crossing points. Detecting penetrating objects (e.g., persons and cars) through dense foliage in various climate conditions using visual sensors is prone to high fault rates. Through-foliage scenarios contain an unprecedented amount of occlusion—in fact, they often contain fragmented occlusion (for example, looking through the branches of a tree). Current state-of-the-art detectors based on deep learning perform inadequately under moderate-to-heavy fragmented occlusion. The FOLDOUT project builds a system that combines various sensors and technologies to tackle this problem. Consequently, a hyperspectral sensor was investigated due to its extended spectral bandwidth, beyond the range of typical RGB sensors, where vegetation exhibits pronounced reflectance. Due to the poor performance of deep learning approaches in through-foliage scenarios, a novel background modeling-based detection approach was developed, dedicated to the characteristics of the hyperspectral sensor, namely strong correlations between adjacent spectral bands and high redundancy. The algorithm is based on local dimensional reduction, where the principal subspace of each pixel is maintained and adapted individually over time. The successful application of the proposed algorithm is demonstrated in a through-foliage scenario comprised of heavy fragmented occlusion and a highly dynamical background, where state-of-the-art deep learning detectors perform poorly.

Refining Cotton Replanting Recommendations

The decision to replant suboptimal cotton (Gossypium hirsutum L.) stands has become more challenging in recent years because the suggested retail price per bag of replanting seed has increased by more than 25%. Previous studies have justified replanting when ≥ 50% of planted area is occupied by skips ≥ 0.91 m (3 ft). Revision of replanting recommendations was deemed necessary with the introduction of more advanced and accurate plant and skip detection methods. The objective of this study was to update replanting recommendations using unmanned aerial vehicles (UAVs). The study was conducted at three sites in eastern North Carolina during the 2019 and 2020 growing seasons. Each site had an early- and a late-planted trial. Treatment combinations were produced using various ratios of DP 1646 B2XF and DP 493 cotton seed. Simulated replanted plots were planted with 100% DP 1646 B2XF three to four weeks after initial planting. Following emergence, glyphosate and glufosinate were applied to terminate all conventional seedlings and produce random skips. Cotton skips were detected using a Zenmuse X5 RGB sensor mounted on a UAV. Yield was regressed to the percentage of planted area occupied by skips &gt; 0.91 m. In 2019, the replanted treatment did not yield higher than earlier planted treatments but, in 2020, yields were significantly higher in the replanted treatment compared to most earlier planted treatments. The data suggest that a replant should be triggered when 30 to 40% of the planted area is occupied by skips ≥ 0.91 m.

High-Resolution Flowering Index for Canola Yield Modelling

Canola (Brassica napus), with its prominent yellow flowers, has unique spectral characteristics and necessitates special spectral indices to quantify the flowers. This study investigated four spectral indices for high-resolution RGB images for segmenting yellow flower pixels. The study compared vegetation indices to digitally quantify canola flower area to develop a seed yield prediction model. A small plot (2.75 m × 6 m) experiment was conducted at Kernen Research Farm, Saskatoon, where canola was grown under six row spacings and eight seeding rates with four replicates (192 plots). The flower canopy reflectance was imaged using a high-resolution (0.15 cm ground sampling distance) 100 MP iXU 1000 RGB sensor mounted on an unpiloted aerial vehicle (UAV). The spectral indices were evaluated for their efficiency in identifying canola flower pixels using linear discriminant analysis (LDA). Digitized flower pixel area was used as a predictor of seed yield to develop four models. Seventy percent of the data were used for model training and 30% for testing. Models were compared using performance metrics: coefficient of determination (R2) and root mean squared error (RMSE). The High-resolution Flowering Index (HrFI), a new flower index proposed in this study, was identified as the most accurate in detecting flower pixels, especially in high-resolution imagery containing within-canopy shadow pixels. There were strong, positive associations between digitized flower area and canola seed yield with the peak flowering timing having a greater R2 (0.82) compared to early flowering (0.72). Cumulative flower pixel area predicted 75% of yield. Our results indicate that the HrFI and Modified Yellowness Index (MYI) were better predictors of canola yield compared to the NDYI and RBNI (Red Blue Normalizing Index) as they were able to discriminate between canola petals and within-canopy shadows. We suggest further studies to evaluate the performance of the HrFI and MYI vegetation indices using medium-resolution UAV and satellite imagery.

Accurate classification of frost thickness using visual information in a domestic refrigerator

Due to the intensive use of the domestic refrigerator based on vapor compression and the particular design of the no-frost type, it is essential to propose economic and light strategies to optimize the refrigerator’s energy consumption. When a large amount of frost accumulates on the evaporator, heat exchange decreases, affecting the cooling capacity. The amount of frost accumulation on the evaporator surface plays a critical role in optimizing frost removal. For this reason, this paper presents a novel method that uses visual information to classify the frost accumulated on the evaporator surface accurately. This method has two parts: a data acquisition system based on an RGB sensor and an intelligent model that processes the visual information and classifies the frost accumulated on the evaporator surface. The data acquisition system creates the datasets for the training and validation of the intelligent model. This smart model is based on convolutional neural networks. Many tests were performed on a pair of different domestic refrigerators using typical usage habits for data collection. Each test was performed continuously for 24 h, emulating real-life usage. One of the main contributions of this work is a new, accurate, and low-cost method to classify the amount of frost accumulated on the evaporator surface. Furthermore, one advantage of the proposed method is the ease of integrating into any domestic refrigerator. The results indicate that this proposed method can classify with an accuracy as high as 99.75 % four different levels of frost accumulated on the evaporator surface of a domestic refrigerator.

An Improved LAI Estimation Method Incorporating with Growth Characteristics of Field-Grown Wheat

Leaf area index (LAI), which is an important vegetation structure parameter, plays a crucial role in evaluating crop growth and yield. Generally, it is difficult to accurately estimate LAI only using vegetation index in remote sensing (RS), especially for unmanned aerial vehicle (UAV) based RS, as its high-resolution advantage has not been fully utilized. This study aims to propose an improved LAI estimation method that comprehensively considers spectral information and structural information provided by the UAV-based RS to improve the LAI estimation accuracy of field-grown wheat. Specifically, this method introduces the canopy height model (CHM) to compensate for the lack of structural information in LAI estimation, and then takes canopy coverage (CC) as a correction parameter to alleviate the LAI overestimation. Finally, the performance of this method is verified on RGB and multispectral images, respectively. The results show that canopy structure, namely CHM and CC, can significantly improve the accuracy of LAI estimation. Compared with the traditional method, the proposed method improves the accuracy by 22.6% on multispectral images (R2 = 0.72, RMSE = 0.556) and by 43.6% on RGB images (R2 = 0.742, RMSE = 0.534). This study provides a simple and practical method for UAV-based LAI estimation, especially for the application of low-cost RGB sensors in precision agriculture and other fields.

Mapping the albedo of the active surface at different stages of the growing season using data from various sources

The albedo of active surfaces, such as soil, plant cover, or anthropogenic objects, is one of the most important components of the Earth's energy balance. It is most often determined based on satellite data; however, such data are characterized by relatively low resolution, and the possibility of acquisition is limited to cloudless days. The development of the UAV technology offers new possibilities of albedo measurement. In this study, an attempt was made to use a UAV equipped with a standard RGB sensor for albedo mapping. The results of the measurements and calculations were compared with Sentinel-2 and Landsat 8 satellite data. It has been shown that the commonly used and readily available UAVs can be applied for mapping the broadband albedo of the active surface with the use of appropriate validation algorithms developed by direct measurements on the ground with a precise high spectral resolution spectroradiometer. The quality of collected results can be improved using different conversion algorithms for the individual phases of the growing season.

Monitoring detailed mangrove hurricane damage and early recovery using multisource remote sensing data

Due to their location in tropical latitudes, mangrove forests are susceptible to the impact of hurricanes and can be vastly damaged by their high-speed winds. Given the logistic difficulties regarding field surveys in mangroves, remote sensing approaches have been considered a reliable alternative. We quantified trends in damage and early signs of canopy recovery in a fringe Rhizophora mangle area of Marismas Nacionales, Mexico, following the landfall of Hurricane Willa in October 2018. We monitored (2016–2021) broad canopy defoliation using 21 vegetation indices (VI) from the Google Earth Engine tool (GEE). We also mapped a detailed canopy fragmentation and developed digital surface models (DSM) during five study periods (2018–2021) with a consumer-grade unmanned aerial vehicle (UAV) over an area of 100 ha. Based on optical data from the GEE time series, results indicated an abrupt decline in the overall mangrove canopy. The VARI index was the most reliable VI for the mangrove canopy classification from a standard RGB sensor. The impact of the hurricane caused an overall canopy defoliation of 79%. The series of UAV orthomosaics indicate a gradual recovery in the mangrove canopy, while the linear model predicts at least 8.5 years to reach pre-impact mangrove cover conditions. However, the sequence of DSM estimates that the vertical canopy configuration will require a longer time to achieve its original structure.