Overhead Images Research Articles

UB-FineNet: Urban building fine-grained classification network for open-access satellite images

Fine classification of city-scale buildings using satellite imagery is a crucial research area with significant implications for urban planning, infrastructure development, and population distribution analysis. However, the task faces great challenges due to low-resolution overhead images acquired from high-altitude space-borne platforms and the long-tailed sample distribution of fine-grained urban building categories, leading to a severe class imbalance problem. To address these issues, we propose a deep network approach to the fine-grained classification of urban buildings using open-access satellite images. A Denoising Diffusion Probabilistic Model (DDPM) based super-resolution method is first introduced to enhance the spatial resolution of satellite images, which benefits from domain-adaptive knowledge distillation. Then, a new fine-grained classification network with Category Information Balancing Module (CIBM) and Contrastive Supervision (CS) technique is proposed to mitigate the problem of class imbalance and improve the classification robustness and accuracy. Experiments on Hong Kong data set with 11 distinct building types revealed promising classification results with a mean Top-1 accuracy of 60.45%, which is on par with street-view image based approaches. A comprehensive ablation study demonstrates that the CIBM and CS modules improve Top-1 accuracy by 2.6% and 3.5%, respectively, over the baseline approach. In addition, these modules can be easily integrated into other classification networks, achieving similar performance improvements. This research advances urban analysis by providing an effective solution for detailed classification of buildings in complex mega-city environments using only open-access satellite imagery. The proposed technique can serve as a valuable tool for urban planners, aiding in the understanding of economic, industrial, and population distribution within cities and regions, ultimately facilitating informed decision-making in urban development and infrastructure planning. Data and code will be publicly available at https://github.com/ZhiyiHe1997/UB-FineNet.

Cardboard packaging minimization: Autonomous box selection by RGB-depth vision system and complementary deep learning classification

Amidst the dramatic expansion of online shopping, wasted cardboard packaging has become a significant challenge in modern-day society. In this paper, we propose an innovative approach to address the issue of box packaging waste by efficiently selecting an optimal cardboard box for a set of items. Employing depth vision and deep learning classification techniques, the proposed system analyzes depth images to compute an optimal cuboid for unpackaged items while accounting for fragility characteristics. The system, facilitated by two Intel RealSense RGB-D cameras, first captures the overhead and sideview images to determine the dimensions of an unknown object. Then, the developed vision system utilizes averaging and morphological opening filters to denoise the acquired images, calculates the difference between the background image and object image, and employs thresholding by Otsu’s algorithm to find the minimum cuboid of the irregular object. After, a two-step deep learning model is used to determine the general category and characteristics of an object, considering factors, such as fragility, that may necessitate wrapping or separate packaging. Finally, a modified largest area first fit algorithm uses the outputted dimensions for irregular-shaped objects and packaged objects to determine the optimal fitting box from the standard Amazon box catalog. The computer simulation shows that the proposed method can be implemented in a cardboard packaging warehouse for optimal box selection, effectively minimizing cardboard waste.

A lightweight algorithm for pedestrian detection in overhead images

A lightweight algorithm for pedestrian detection in overhead images

Open-Set Recognition of Individual Cows Based on Spatial Feature Transformation and Metric Learning.

The automated recognition of individual cows is foundational for implementing intelligent farming. Traditional methods of individual cow recognition from an overhead perspective primarily rely on singular back features and perform poorly for cows with diverse orientation distributions and partial body visibility in the frame. This study proposes an open-set method for individual cow recognition based on spatial feature transformation and metric learning to address these issues. Initially, a spatial transformation deep feature extraction module, ResSTN, which incorporates preprocessing techniques, was designed to effectively address the low recognition rate caused by the diverse orientation distribution of individual cows. Subsequently, by constructing an open-set recognition framework that integrates three attention mechanisms, four loss functions, and four distance metric methods and exploring the impact of each component on recognition performance, this study achieves refined and optimized model configurations. Lastly, introducing moderate cropping and random occlusion strategies during the data-loading phase enhances the model's ability to recognize partially visible individuals. The method proposed in this study achieves a recognition accuracy of 94.58% in open-set scenarios for individual cows in overhead images, with an average accuracy improvement of 2.98 percentage points for cows with diverse orientation distributions, and also demonstrates an improved recognition performance for partially visible and randomly occluded individual cows. This validates the effectiveness of the proposed method in open-set recognition, showing significant potential for application in precision cattle farming management.

LidarBoX: a 3D-printed, open-source altimeter system to improve photogrammetric accuracy for off-the-shelf drones

Drones provide a privileged birds’-eye view for collecting high-resolution imagery for morphometric and behavioral sampling of animals. Biologically meaningful measurements extracted from overhead images require an accurate estimate of altitude, but current commercial drones include inaccurate barometer estimates. Recent proposals for coupling altimeter systems to drones have provided customized, open-source solutions, yet assembling such altimeter systems requires advanced technical skills, thereby potentially limiting their use. Here, we built upon recent advances to provide a 3D-printed enclosure for an altimeter system that is inexpensive, self-contained, easy to setup, and transferable across commercial drones. We depart from a published, successful data logger system composed of a GPS and LiDAR sensor and design a more compact and self-powered version (“LidarBoX”) that easily attaches to a variety of commercial drones. We compare flight times with/without LidarBoX attached, test flight maneuverability and performance, and validate the reliability of measurement accuracy. To make LidarBoX accessible, we provide an open-source repository with design code and files and a how-to-assemble guide for non-specialists. We hope this work helps popularize LiDAR altimeter systems on commercial drones to improve the accuracy and reliability of drones as a sampling platform for ecology and wildlife research.

VEPL-Net: A Deep Learning Ensemble for Automatic Segmentation of Vegetation Encroachment in Power Line Corridors Using UAV Imagery

Vegetation encroachment in power line corridors remains a major challenge for modern energy-dependent societies, as it can cause power outages and lead to significant financial losses. Unmanned Aerial Vehicles (UAVs) have emerged as a promising solution for monitoring infrastructure, owing to their ability to acquire high-resolution overhead images of these areas quickly and affordably. However, accurate segmentation of the vegetation encroachment in this imagery is a challenging task, due to the complexity of the scene and the high pixel imbalance between the power lines, the vegetation and the background classes. In this paper, we propose a deep learning-based approach to tackle this problem caused by the original and different geometry of the objects. Specifically, we use DeepLabV3, U-Net and a modified version of the U-Net architecture with VGG-16 weights to train two separate models. One of them segments the dominant classes, the vegetation from the background, achieving an IoU of 0.77. The other one segments power line corridors from the background, obtaining an IoU of 0.64. Finally, ensembling both models into one creates an “encroachment” zone, where power lines and vegetation are intersected. We train our models using the Vegetation Encroachment in Power Line Corridors dataset (VEPL), which includes RGB orthomosaics and multi-label masks for segmentation. Experimental results demonstrate that our approach outperforms individual networks and original prominent architectures when applied to this specific problem. This approach has the potential to significantly improve the efficiency and accuracy of vegetation encroachment monitoring using UAV, thus helping to ensure the reliability and sustainability of power supply.

RoseTracker: A system for automated rose growth monitoring

In cut-flower cultivation, production planning is an important task because demand fluctuates throughout the year. For precise cultivation planning, understanding the cultivation status is necessary by the growing stage. However, manually counting all the roses in the greenhouse to determine the cultivation status is difficult without incurring considerable time and labor. Some studies have engaged in detecting the number of flowers, but these studies used close-up images and could not count flowers without omissions or overlapping in an entire farm. In addition, limited datasets for object detection based on cut-flower blooming stages are available. In this study, we propose the RoseBlooming dataset and an efficient rose-monitoring system called RoseTracker to bridge the gap between computer vision techniques and the horticulture cultivation industry. The RoseBlooming dataset is the innovative dataset of labeled images for cut flowers at the growing stage. RoseTracker can detect small roses from various angles while moving the camera, reduces detection omissions, and achieves an F1 score of 0.950, thereby outperforming conventional models. For application, we used overhead images captured under actual growing conditions. RoseTracker and the RoseBlooming dataset contribute to constructing the rose-growth monitoring system in high demand worldwide.

Tracking Passengers and Baggage Items Using Multiple Overhead Cameras at Security Checkpoints

We introduce a novel framework to track multiple objects in overhead camera videos for airport checkpoint security scenarios where targets correspond to passengers and their baggage items. We propose a self-supervised learning (SSL) technique to provide the model information about instance segmentation uncertainty from overhead images. Our SSL approach improves object detection by employing a test-time data augmentation and a regression-based, rotation-invariant pseudo-label refinement technique. Our pseudo-label generation method provides multiple geometrically transformed images as inputs to a convolutional neural network (CNN), regresses the augmented detections generated by the network to reduce localization errors, and then clusters them using the mean-shift algorithm. The self-supervised detector model is used in a single-camera tracking algorithm to generate temporal identifiers for the targets. Our method also incorporates a multiview trajectory association mechanism to maintain consistent temporal identifiers as passengers travel across camera views. An evaluation of detection, tracking, and association performances on videos obtained from multiple overhead cameras in a realistic airport checkpoint environment demonstrates the effectiveness of the proposed approach. Our results show that self-supervision improves object detection accuracy by up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$42\%$</tex-math> </inline-formula> without increasing the inference time of the model. Our multicamera association method achieves up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$89\%$</tex-math> </inline-formula> multiobject tracking accuracy with an average computation time of less than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$15$</tex-math> </inline-formula> ms.

Blockchain based Agriculture Using the Application of UAV and Deep Learning Technique: Alexnet CNN

Due to the warm and humid environment of Bangladesh, it is highly exposed to occurring perpetuation of various viruses which cause diseases in crops. A huge number of crops are wasted because of these occurring diseases and it directly hurts the production rate and forces import of crops in bulkier amount. Unmanned aerial vehicle usage is one of the smart agriculture technologies being researched for agricultural applications (UAVs) in these days. UAV technology allows farmers to quickly gather information on field conditions by providing overhead images of their agricultural fields or even allowing them to zoom in on a particular area. Using UAV technology, farmers may identify specific areas that need immediate attention and perform the necessary agricultural improvements. Drones collect data that farmers can use to detect crop disease by applying deep learning algorithms to make long-term decisions about planting, land mapping, damage control, and other things. This research uses blockchain technology to establish connection between suppliers and customers by enabling information to be tracked throughout the supply chain and enhances food supply chain safety. It offers a secure method of broadcasting data, focusing on enhancement of supply chain management and prediction of crops which makes it possible to implement and deploy data-driven technologies for smart farming. The research uses UAVs as a means of collecting crop images, implements a prediction model using AlexNet CNN and analyses how it performs with a real Bangladeshi crop disease dataset to help farmers from excessive crop damage. Furthermore, the overall process is carried out using the Blockchain technology to enhance the existing supply chain management process.

Distance-Independent Background Light Estimation Method

A distance-independent background light estimation method is proposed for underwater overhead images. The method addresses the challenge of the absence of the farthest point in underwater overhead images by adopting a global perspective to select the optimal solution and estimate the background light by minimizing the loss function. Moreover, to enhance the information retention in the images, a translation function is employed to adjust the transmission map values within the range of [0.1, 0.95]. Additionally, the method capitalizes on the redundancy of image information and the similarity of adjacent frames, resulting in higher computational efficiency. The comparative experimental results show that the proposed method has better restoration performance on underwater images in various scenarios, especially in handling color bias and preserving information.

Image quality and computer vision performance: assessing the effects of image distortions and modeling performance relationships using the general image quality equation

Substantial research has explored methods to optimize convolutional neural networks (CNNs) for tasks such as image classification and object detection, but research into the image quality drivers of computer vision performance has been limited. Additionally, there are indications that image degradations such as blur and noise affect human visual interpretation and machine interpretation differently. The general image quality equation (GIQE) predicts overhead image quality for human analysis using the National Image Interpretability Rating Scale, but no such model exists to predict image quality for interpretation by CNNs. Here, we assess the relationship between image quality variables and CNN performance. Specifically, we examine the impacts of resolution, blur, and noise on CNN performance for models trained with in-distribution and out-of-distribution distortions. Using two datasets, we observe that while generalization remains a significant challenge for CNNs faced with out-of-distribution image distortions, CNN performance against low visual quality images remains strong with appropriate training, indicating the potential to expand the design trade space for sensors providing data to computer vision systems. Additionally, we find that CNN performance predictions using the functional form of the GIQE can predict CNN performance as a function of image degradation, but we observe that the legacy form of the GIQE (from GIQE versions 3 and 4) does a better job of modeling the impact of blur/relative edge response in our experiments.

Using Neural Networks to Detect Fire from Overhead Images

The use of artificial intelligence (AI) is increasing in our everyday applications. One emerging field within AI is image recognition. Research that has been devoted to predicting fires involves predicting its behaviour. That is, how the fire will spread based on environmental key factors such as moisture, weather condition, and human presence. The result of correctly predicting fire spread can help firefighters to minimise the damage, deciding on possible actions, as well as allocating personnel effectively in potentially fire prone areas to extinguish fires quickly. Using neural networks (NN) for active fire detection has proven to be exceptional in classifying smoke and being able to separate it from similar patterns such as clouds, ground, dust, and ocean. Recent advances in fire detection using NN has proved that aerial imagery including drones as well as satellites has provided great results in detecting and classifying fires. These systems are computationally heavy and require a tremendous amount of data. A NN model is inextricably linked to the dataset on which it is trained. The cornerstone of this study is based on the data dependencieds of these models. The model herein is trained on two separate datasets and tested on three dataset in total in order to investigate the data dependency. When validating the model on their own datasets the model reached an accuracy of 92% respectively 99%. In comparison to previous work where an accuracy of 94% was reached. During evaluation of separate datasets, the model performed around the 60% range in 5 out of 6 cases, with the outlier of 29% in one of the cases.

Point-based metric and topological localisation between lidar and overhead imagery

In this paper, we present a method for solving the localisation of a ground lidar using overhead imagery only. Public overhead imagery such as Google satellite images are readily available resources. They can be used as the map proxy for robot localisation, relaxing the requirement for a prior traversal for mapping as in traditional approaches. While prior approaches have focused on the metric localisation between range sensors and overhead imagery, our method is the first to learn both place recognition and metric localisation of a ground lidar using overhead imagery, and also outperforms prior methods on metric localisation with large initial pose offsets. To bridge the drastic domain gap between lidar data and overhead imagery, our method learns to transform an overhead image into a collection of 2D points, emulating the resulting point-cloud scanned by a lidar sensor situated near the centre of the overhead image. After both modalities are expressed as point sets, point-based machine learning methods for localisation are applied.

Automated Pre-Play Analysis of American Football Formations Using Deep Learning

Annotation and analysis of sports videos is a time-consuming task that, once automated, will provide benefits to coaches, players, and spectators. American football, as the most watched sport in the United States, could especially benefit from this automation. Manual annotation and analysis of recorded videos of American football games is an inefficient and tedious process. Currently, most college football programs focus on annotating offensive formations to help them develop game plans for their upcoming games. As a first step to further research for this unique application, we use computer vision and deep learning to analyze an overhead image of a football play immediately before the play begins. This analysis consists of locating individual football players and labeling their position or roles, as well as identifying the formation of the offensive team. We obtain greater than 90% accuracy on both player detection and labeling, and 84.8% accuracy on formation identification. These results prove the feasibility of building a complete American football strategy analysis system using artificial intelligence. Collecting a larger dataset in real-world situations will enable further improvements. This would likewise enable American football teams to analyze game footage quickly.

Closing the domain gap: blended synthetic imagery for climate object detection

Abstract Accurate geospatial information about the causes and consequences of climate change, including energy systems infrastructure, is critical to planning climate change mitigation and adaptation strategies. When up-to-date spatial data on infrastructure is lacking, one approach to fill this gap is to learn from overhead imagery using deep-learning-based object detection algorithms. However, the performance of these algorithms can suffer when applied to diverse geographies, which is a common case. We propose a technique to generate realistic synthetic overhead images of an object (e.g., a generator) to enhance the ability of these techniques to transfer across diverse geographic domains. Our technique blends example objects into unlabeled images from the target domain using generative adversarial networks. This requires minimal labeled examples of the target object and is computationally efficient such that it can be used to generate a large corpus of synthetic imagery. We show that including these synthetic images in the training of an object detection model improves its ability to generalize to new domains (measured in terms of average precision) when compared to a baseline model and other relevant domain adaptation techniques.

Minimum cuboid estimation of irregular shape products for reducing box packaging waste by using vision-based measurement

Today packaging waste is a prevalent issue due to the increase in deliveries from online shopping. Here a new approach to this issue of cardboard box packaging waste is proposed by adapting an image processing algorithm based on camera vision-based measurement that computes the optimal cuboid bounding algorithm of irregular shape products. The end result is utilized so that packaging workers select the appropriate product box. This approach may also be used as a preliminary process to optimize the packaging of many products into a single cardboard box. The system setup with two cameras is prepared to capture the overhead and sideview images, estimating the box’s width, depth, and height in pixels. This system may then evaluate feasible cuboids that minimize waste, in which the traditional Otsu’s thresholding method, proposed Otsu’s scheme, and 1-D gradient means are utilized to avoid inaccuracies created by shadows. Calibration is performed with a Rubik’s cube to convert the measurement from computer simulations to real-life dimensions. The computer simulations from the overhead and sideview images compared to the actual bounding box of the object show that the proposed algorithm yields superior performance by reducing the area error of the bounding box (%) of an overhead image and the height error (%) of a sideview image than the conventional Otsu’s method.

A comparative experimental study of rill erosion on loess soil and clay loam soil based on a digital close-range photogrammetry technology

Accurately describing rill development is important for building physical process models of soil erosion. The objectives of this study were to investigate the evolution processes of rill erosion morphology, their impacts on runoff and sediment yield, and the relationship between flow hydraulic characteristics and rill erosion dynamic mechanisms on slopes of two soil types, namely, clay loam soil (CLS) and loess soil (LS), under two successive rainfall events at a rainfall intensity of 120 mm h−1. We used digital close range photogrammetry technology to obtain overhead images of two experimental soil bins (5 m long, 1 m wide and 0.5 m deep with a slope gradient of 20°) at 5-minute time intervals to monitor the dynamic processes of rill erosion. The results showed that rill erosion processes exhibited important impacts on sediment and runoff and its hydrodynamic characteristics. Compared with LS, runoff formed and rills developed earlier in the CLS slope. Soil with high clay content was more prone to form rills, but less susceptible to severe soil erosion than LS. Once rills began to develop in LS, soil loss increased and the morphology of rills changed rapidly, as did runoff rate. The flow velocity for the two soils decreased gradually with time and increased along the flow direction. The trend of Froude number (Fr) decreased with time, whereas the Darcy-Weisbach resistance coefficient (f) exhibited an opposite trend. The Reynolds number (Re) of LS increased first and then decreased, while Re of CLS decreased gradually with time. Sediment yield and runoff were positively correlated with f, but significantly negatively correlated with Fr. Moreover, Re was significantly negatively correlated with sediment yield, but not with runoff. These results are of great importance for soil loss prediction.

Force-Bioreactor for Assessing Pharmacological Therapies for Mechanobiological Targets.

Tissue fibrosis is a major health issue that impacts millions of people and is costly to treat. However, few effective anti-fibrotic treatments are available. Due to their central role in fibrotic tissue deposition, fibroblasts and myofibroblasts are the target of many therapeutic strategies centered primarily on either inducing apoptosis or blocking mechanical or biochemical stimulation that leads to excessive collagen production. Part of the development of these drugs for clinical use involves in vitro prescreening. 2D screens, however, are not ideal for discovering mechanobiologically significant compounds that impact functions like force generation and other cell activities related to tissue remodeling that are highly dependent on the conditions of the microenvironment. Thus, higher fidelity models are needed to better simulate in vivo conditions and relate drug activity to quantifiable functional outcomes. To provide guidance on effective drug dosing strategies for mechanoresponsive drugs, we describe a custom force-bioreactor that uses a fibroblast-seeded fibrin gels as a relatively simple mimic of the provisional matrix of a healing wound. As cells generate traction forces, the volume of the gel reduces, and a calibrated and embedded Nitinol wire deflects in proportion to the generated forces over the course of 6 days while overhead images of the gel are acquired hourly. This system is a useful in vitro tool for quantifying myofibroblast dose-dependent responses to candidate biomolecules, such as blebbistatin. Administration of 50 μM blebbistatin reliably reduced fibroblast force generation approximately 40% and lasted at least 40 h, which in turn resulted in qualitatively less collagen production as determined via fluorescent labeling of collagen.

Density Map-based vehicle counting in remote sensing images with limited resolution

Observing traffic flow is of great significance to contemporary urban management. Overhead images, as represented by remote sensing images, provide a major source of information about traffic flow. However, the spatial resolutions of most common high-resolution remote sensing images are often limited to 0.5 m and even below, which makes it unrealistic to count vehicles by means of widely used object detection methods. Therefore, to explore the potential of remote sensing data for studying global urban development and management, this paper introduces a density map-based vehicle counting method for remote sensing imagery with limited resolution. Density map-based models regard the vehicle counting task as estimating the density of vehicle targets in terms of pixel values. We propose an improved CNN-based network, called Congested Scene Recognition Network Minus (CSRNet—), that generates a density map of vehicles from the input remote sensing imagery. A new dataset, RSVC2021, which was generated from the public DOTA and ITCVD datasets, is also introduced for network training and testing. A benchmark on the RSVC2021 dataset is accordingly established and CSRNet— is selected as the baseline model for subsequent experiments. A set of GF-2 time series images with a resolution of 1 m taken before, during and after the COVID-19 epidemic lockdown covering Wuhan city are applied for real-world application testing. The testing results on both the RSVC2021 dataset and real satellite images confirm that, in terms of both the counting values and the visualized density maps, the proposed method achieves good performance and exhibits considerable application potential in this task. The generating codes of RSVC2021 dataset will be publicly available at https://github.com/YinongGuo/RSVC2021-Dataset.

Reconstructing subsurface sandbody connectivity from temporal evolution of surface networks

AbstractCharacterization of groundwater aquifers and hydrocarbon reservoirs requires an understanding of the distribution and connectivity of subsurface sandbodies. In deltaic environments, distributary channel networks serve as the primary conduits for water and sediment. Once these networks are buried and translated into the subsurface, the coarse‐grained channel fills serve as primary conduits for subsurface fluids such as water, oil or gas. The temporal evolution of channels on the surface therefore plays a first‐order role in the 3D permeability and connectivity of subsurface networks. Land surface imagery is more broadly available than topographic or subsurface data, and time‐series imagery of river networks can hold useful information for constraining the shallow subsurface. However, these reconstructions require an understanding of the degree to which channel bathymetry and river kinematics affect connectivity of subsurface sandbodies. Here, we present a novel method for building synthetic cross sections using overhead images of an experimental delta. We use principal components analysis to extract river networks from surface imagery, then couple this with an inverse‐CDF method to estimate channel bathymetry, to generate a time‐series of synthetic delta topography. This synthetic topography is then transformed, accounting for deposition and subsidence, to produce synthetic stratigraphy that differentiates coarse‐grained channel fill from overbank and offshore deposition. We find that large‐scale subsurface architecture is relatively insensitive to details of channel bathymetry, but instead is primarily controlled by channel location and kinematics. We analyse the connectivity of sand bodies and the geometries of barriers to flow and find that periods of rapid sea‐level rise have more variability in sand body connectivity. We also find that barrier width decreases downstream during all sea‐level phases. Our method generates synthetic stratigraphy that closely resembles the large‐scale architecture and 2‐dimensional connectivity of the real stratigraphy built during the experiment it was based on. We anticipate that it will be broadly applicable to other experimental and field‐scale scenarios.