Object-based Features Research Articles

Factors Influencing Movement of the Manila Dunes and Its Impact on Establishing Non-Native Species

Unmanned aerial vehicles (UAVs) are being widely used to monitor microtopographic and vegetation changes in coastal habitats using remote sensing techniques. Sand dune habitats are vital ecosystems along the North coast of Humboldt County in California. This study was conducted at the Manila Dunes, west of the Humboldt Coastal Nature Center, in Manila, California. Various factors influence dune movements, including vegetative stabilization and the creation of social trails. The purpose of this paper is to understand the dune movements in relation to social vs. established trails, vegetation density, topography, and also, mapping invasive vs. native species in the Mal-le’l Dunes area of the Humboldt Bay National Wildlife Refuge. A DJI Mavic Pro multicopter small unmanned aerial vehicle (UAV) was used to fly a 22.5-acre plot of the Manila Dunes. The images from this flight were used to create an orthomosaic image using a photogrammetry process (Structure-from-Motion (SfM)). From our analysis, the installation of trails lessened the impact of dune movements. Social trails digitized within the study site were found to have more local movements than the established trails when compared to movements across the entire site. We compared two methods of classification, viz., the object-based feature extraction method and a pixel-based supervised maximum likelihood classification method, in order to identify the best way to classify dune vegetation. In conclusion, this study is useful for providing baseline dune movement information that can aid in informing how trail and infrastructure constructions can be impacted in land management or in areas with dynamic communities of flora and fauna.

Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video.

In this paper, we tackle the problem of egocentric action anticipation, i.e., predicting what actions the camera wearer will perform in the near future and which objects they will interact with. Specifically, we contribute Rolling-Unrolling LSTM, a learning architecture to anticipate actions from egocentric videos. The method is based on three components: 1) an architecture comprised of two LSTMs to model the sub-tasks of summarizing the past and inferring the future, 2) a Sequence Completion Pre-Training technique which encourages the LSTMs to focus on the different sub-tasks, and 3) a Modality ATTention (MATT) mechanism to efficiently fuse multi-modal predictions performed by processing RGB frames, optical flow fields and object-based features. The proposed approach is validated on EPIC-Kitchens, EGTEA Gaze+ and ActivityNet. The experiments show that the proposed architecture is state-of-the-art in the domain of egocentric videos, achieving top performances in the 2019 EPIC-Kitchens egocentric action anticipation challenge. The approach also achieves competitive performance on ActivityNet with respect to methods not based on unsupervised pre-training and generalizes to the tasks of early action recognition and action recognition. To encourage research on this challenging topic, we made our code, trained models, and pre-extracted features available at our web page: http://iplab.dmi.unict.it/rulstm.

Filtration and Distillation: Enhancing Region Attention for Fine-Grained Visual Categorization

Delicate attention of the discriminative regions plays a critical role in Fine-Grained Visual Categorization (FGVC). Unfortunately, most of the existing attention models perform poorly in FGVC, due to the pivotal limitations in discriminative regions proposing and region-based feature learning. 1) The discriminative regions are predominantly located based on the filter responses over the images, which can not be directly optimized with a performance metric. 2) Existing methods train the region-based feature extractor as a one-hot classification task individually, while neglecting the knowledge from the entire object. To address the above issues, in this paper, we propose a novel “Filtration and Distillation Learning” (FDL) model to enhance the region attention of discriminate parts for FGVC. Firstly, a Filtration Learning (FL) method is put forward for discriminative part regions proposing based on the matchability between proposing and predicting. Specifically, we utilize the proposing-predicting matchability as the performance metric of Region Proposal Network (RPN), thus enable a direct optimization of RPN to filtrate most discriminative regions. Go in detail, the object-based feature learning and region-based feature learning are formulated as “teacher” and “student”, which can furnish better supervision for region-based feature learning. Accordingly, our FDL can enhance the region attention effectively, and the overall framework can be trained end-to-end without neither object nor parts annotations. Extensive experiments verify that FDL yields state-of-the-art performance under the same backbone with the most competitive approaches on several FGVC tasks.

Multi-Spectral Image Classification Based on an Object-Based Active Learning Approach

In remote sensing, active learning (AL) is considered to be an effective solution to the problem of producing sufficient classification accuracy with a limited number of training samples. Though this field has been extensively studied, most papers exist in the pixel-based paradigm. In object-based image analysis (OBIA), AL has been comparatively less studied. This paper aims to propose a new AL method for selecting object-based samples. The proposed AL method solves the problem of how to identify the most informative segment-samples so that classification performance can be optimized. The advantage of this algorithm is that informativeness can be estimated by using various object-based features. The new approach has three key steps. First, a series of one-against-one binary random forest (RF) classifiers are initialized by using a small initial training set. This strategy allows for the estimation of the classification uncertainty in great detail. Second, each tested sample is processed by using the binary RFs, and a classification uncertainty value that can reflect informativeness is derived. Third, the samples with high uncertainty values are selected and then labeled by a supervisor. They are subsequently added into the training set, based on which the binary RFs are re-trained for the next iteration. The whole procedure is iterated until a stopping criterion is met. To validate the proposed method, three pairs of multi-spectral remote sensing images with different landscape patterns were used in this experiment. The results indicate that the proposed method can outperform other state-of-the-art AL methods. To be more specific, the highest overall accuracies for the three datasets were all obtained by using the proposed AL method, and the values were 88.32%, 85.77%, and 93.12% for “T1,” “T2,” and “T3,” respectively. Furthermore, since object-based features have a serious impact on the performance of AL, eight combinations of four feature types are investigated. The results show that the best feature combination is different for the three datasets due to the variation of the feature separability.

Hierarchical detection of abnormal behaviors in video surveillance through modeling normal behaviors based on AUC maximization

Detection of abnormal behaviors is a challenging issue as there are disagreements over how abnormal behaviors should be defined. In this paper, a new method is proposed to estimate a model of normal behaviors and consequently to detect abnormal behaviors. Estimating a model of normal behaviors constitutes a main part of the training phase in detecting abnormal behaviors and is one of the main issues in the field. To estimate such a model, the histogram of the oriented optical flow (HOF) is first extracted as a local-based feature, and then, the proposed features of speed, variance of the trajectory and deviation from the trajectory are extracted as object-based features. Spectral clustering is used to cluster behavioral features that are similar. To design a classifier for estimating the model of normal behaviors in each cluster, a new method is proposed based on maximization of the area under the receiver operating characteristic curve (AUC). This new method does not need manual labeling for estimation of the model, and then, it carries out the estimation process in a semi-supervised fashion. In this paper, a method for hierarchical detection of abnormal behaviors based on the priority of the behavioral features is also proposed in the testing phase of the study. To this purpose, the existence of an abnormal behavior is hierarchically detected through using the HOF feature and the three object-based features of speed, variance of the trajectory and deviation from the trajectory. The distinguishing characteristic of the proposed method is that, as soon as the abnormal behavior is detected along the hierarchical application of the features, the process stops. The experimental results show that the proposed method can effectively detect the abnormal behaviors in several databases and achieve comparable performance to the state-of-the-art methods for detection of abnormal behaviors.

Object-based feature extraction for hyperspectral data using firefly algorithm

Object-based classification methods can improve the accuracy of hyperspectral image classification due to the fact that they incorporate spatial information into the classification procedure by assigning neighboring pixels into the same class. In this paper, a new object-based feature extraction method is proposed which makes use of information theory to reduce the Bayes error. In this way, the proposed method exploits higher order statistics for feature extraction which are very effective for non Gaussian data such as hyperspectral images. The criterion to be minimized is composed of three mutual information terms. The first and second terms, consider the maximal relevance and minimal redundancy, respectively, while the third term takes into account the segmentation map containing disjoint spatial regions. To obtain the segmentation map, we apply the firefly clustering algorithm whose fitness function simultaneously considers the intra-distance between samples and their cluster centroids, and inter-distance between centroids of any two clusters. Our experimental results, performed using a variety of hyperspectral scenes, indicate that the proposed framework gives better classification results than some state-of the-art spectral–spatial feature extraction methods.

Land Cover/Land Use Mapping of LISS IV Imagery Using Object-Based Convolutional Neural Network with Deep Features

The paper proposes a new method for classifying the LISS IV satellite images using deep learning method. Deep learning method is to automatically extract many features without any human intervention. The classification accuracy through deep learning is still improved by including object-based segmentation. The object-based deep feature learning method using CNN is used to accurately classify the remotely sensed images. The method is designed with the technique of extracting the deep features and using it for object-based classification. The proposed system extracts deep features using pre-defined filter values, thus increasing the overall performance of the process compared to randomly initialized filter values. The object-based classification method can preserve edge information in complex satellite images. To improve the classification accuracy and to reduce complexity, object-based deep learning technique is used. The proposed object-based deep learning approach is used to drastically increase the classification accuracy. Here, the remotely sensed images were used to classify the urban areas of Ahmadabad and Madurai cities. Experimental results show a better performance with the object-based classification.

Recurrent convolutional video captioning with global and local attention

Video captioning with encoder–decoder structures has been extensively studied in the recent literature, where a great deal of work focuses on multimodal features and attention mechanisms. Most of the previous work uses only the global temporal features, such as image, motion, and audio features, and ignores the local semantic features existing extensively in the video data. Furthermore, it is difficult to fully utilize the local features due to frame-to-frame redundancy. In this paper, we propose to combine global temporal features and local object-based features in a complementary way to develop a multimodal attention mechanism (global–local attention mechanism). Based on this attention mechanism, we introduce a novel video captioning method, Recurrent Convolutional Video Captioning with Global and Local Attention (RCGL). Further, both LSTM and 1D CNN are incorporated into the decoder to improve the long-range dependency. The experimental results on two standard datasets, MSVD and MSR-VTT, demonstrate that RCGL outperforms the state-of-the-art in four common metrics.

Object-based feature selection using class-pair separability for high-resolution image classification

ABSTRACT With the increasing popularity of object-based image analysis (OBIA) since 2006, numerous classification and mapping tasks were reported to benefit from this evolving paradigm. In these studies, segments are firstly created, followed by classification based on segment-level information. However, the feature space formed by segment-level feature variables can be very large and complex, posing challenges to obtaining satisfactory classification performance. Accordingly, this work attempts to develop a new feature selection approach for segment-level features. Based on the principle of class-pair separability, the segment-level features are grouped according to their types. For each group, the contribution of each segment-level feature to the separation of a pair of classes is quantified. With the information of all feature groups and class pairs, the separability ranking and appearance frequency are considered to compute importance score for each feature. Higher importance score means larger appropriateness to select a feature. By using two Gaofen-2 multi-spectral images, the proposed method is validated. The experimental results show the advantages of the proposed technique over some state-of-the-art feature selection approaches: (1) it can better reduce the number of segment-level features and effectively avoid redundant information; (2) the feature subset obtained by the proposed scheme has good potential to improve classification accuracy.

Evaluation of crop mapping on fragmented and complex slope farmlands through random forest and object-oriented analysis using unmanned aerial vehicles

Conducting field research in Taiwan can be challenging because of the abundance of steep slopes. This study aimed to establish an automatic interpretation procedure applicable to exploring images of large-scale slope land taken using UAVs. The proposed method was compared with traditional field surveying and manual image interpretation techniques to determine the advantages and disadvantages of the proposed procedure in terms of efficiency. The object-based image analysis (OBIA) and texture features were first combined and the random forest (RF) classifier was then employed to interpret crop types. This study selected three sites of slope land and plains for experimentation. The obtained results indicated that the overall accuracy of the proposed classification method exceeded 91%, and the Kappa value was approximately 0.9 for all sites. In addition, interpretation of the proposed method was more efficient than that of the two traditional methods.

UNSUPERVISED WINDOW EXTRACTION FROM PHOTOGRAMMETRIC POINT CLOUDS WITH THERMAL ATTRIBUTES

Abstract. The automatic extraction of windows from photogrammetric data has achieved increasing attention in recent times. An unsupervised windows extraction approach from photogrammetric point clouds with thermal attributes is proposed in this study. First, point cloud segmentation is conducted by a popular workflow: Multiscale supervoxel generation is applied to the image-based 3D point cloud, followed by region growing and energy optimization using spatial positions and thermal attributes of the raw points. Afterwards, an object-based feature (window index) is extracted using the average thermal attribute and the size of the object. Next, thresholding is applied to extract initial window regions. Finally, several criterions are applied to further refine the extraction results. For practical validation, the approach is evaluated on an art nouveau building row façade located at Dresden, Germany.

Object-based fusion for urban tree species classification from hyperspectral, panchromatic and nDSM data

ABSTRACT This study aims at identifying the best object-based fusion strategy that takes advantage of the complementarity of several heterogeneous airborne data sources for improving the classification of 15 tree species in an urban area (Toulouse, France). The airborne data sources are: hyperspectral Visible Near-Infrared (160 spectral bands, spatial resolution of 0.4 m) and Short-Wavelength Infrared (256 spectral bands, 1.6 m), panchromatic (14 cm), and a normalized Digital Surface Model (12.5 cm). Object-based feature and decision level fusion strategies are proposed and compared when applied to a reference site where the species are previously identified during ground truth collection. This allows the best fusion strategy to be selected with a view to introducing the method in an automatic process (tree crown delineation and species classification) on a test site, independent of the reference site used for learning. In particular, a decision level fusion is selected: based on the Support Vector Machine algorithm, Visible Near-Infrared and Short-Wavelength Infrared classifications use Minimum Noise Fraction components at the original spatial resolution, whereas panchromatic and normalized Digital Surface Model classifications use, respectively, Haralick’s and structural features computed at the object scale. After the computation of a decision profile for each source at the object level based on the classification algorithms’ membership probabilities, these decision profiles are combined and a decision rule is applied to predict the species. Focusing on the reference site, the Visible Near-Infrared exhibits the best performances with F-score values higher than 60% for 13 species out of 15. The Short-Wavelength Infrared is the most powerful for three species with F-score greater than 60% for seven common species with the Visible Near-Infrared. The panchromatic and normalized Digital Surface Model contribute marginally. The best fusion strategy (decision fusion) does not improve significantly the overall accuracy with 77% (kappa = 74%) against 75% (kappa = 72%) for the Visible Near-Infrared but in general, it improves the results for cases where complementarities have been observed. When applied to the test site and assessed for the two majority species (Tilia tomentosa and Platanus x hispanica), the selected approach gives consistent results with an overall accuracy of 63% against 55% for the Visible Near-Infrared.

Monitoring the Characteristics of the Bohai Sea Ice Using High-Resolution Geostationary Ocean Color Imager (GOCI) Data

Satellite remote sensing data, such as moderate resolution imaging spectroradiometers (MODIS) and advanced very high-resolution radiometers (AVHRR), are being widely used to monitor sea ice conditions and their variability in the Bohai Sea, the southernmost frozen sea in the Northern Hemisphere. Monitoring the characteristics of the Bohai Sea ice can provide crucial information for ice disaster prevention for marine transportation, oil field operation, and regional climate change studies. Although these satellite data cover the study area with fairly high spatial resolution, their typically limited cloudless images pose serious restrictions for continuous observation of short-term dynamics, such as sub-seasonal changes. In this study, high spatiotemporal resolution (500 m and eight images per day) geostationary ocean color imager (GOCI) data with a high proportion of cloud-free images were used to monitor the characteristics of the Bohai Sea ice, including area and thickness. An object-based feature extraction method and an albedo-based thickness inversion model were used for estimating sea ice area and thickness, respectively. To demonstrate the efficacy of the new dataset, a total of 68 GOCI images were selected to analyze the evolution of sea ice area and thickness during the winter of 2012–2013 with severe sea ice conditions. The extracted sea ice area was validated using Landsat Thematic Mapper (TM) data with higher spatial resolution, and the estimated sea ice thickness was found to be consistent with in situ observation results. The entire sea ice freezing–melting processes, including the key events such as the day with the maximum ice area and the first and last days of the frozen season, were better resolved by the high temporal-resolution GOCI data compared with MODIS or AVHRR data. Both characteristics were found to be closely correlated with cumulative freezing/melting degree days. Our study demonstrates the applicability of the GOCI data as an improved dataset for studying the Bohai Sea ice, particularly for purposes that require high temporal resolution data, such as sea ice disaster monitoring.

Curvelet transform based feature extraction and selection for multimedia event classification

Multimedia event classification has been one of the major endeavors in video event analysis. For event identification, feature extraction plays a critical role, merely distinguishing the right features becomes a challenging job. So, in this paper, feature extraction and selection for video event detection are proposed. For video event detection or classification, identification of object structure and its motion are a basic needs. So, for object detection, curvelet features are considered, due to its high directional selectivity and high anisotropic properties. Next, an algorithm for Shot Boundary Detection (SBD) called Motion based SBD (MSBD) is proposed to identify the shot boundaries. Also, to make the users to access the queried event in less time, an algorithm for representing few representative frames containing whole video content is proposed. To make an event search efficient, object-based features from dominant features are extracted and to enhance the features much more efficient and dominant, feature selection is performed using the ranking method. Lastly, an SVM classifier with RBF kernel is used for event classification. The proposed work is experimented using Columbia Consumer Video (CCV) dataset and evaluated using mean Average Precision (mAP) and find that, it outperforms various other existing methods.

Recent expansion of oil palm plantation in the most eastern part of Indonesia: feature extraction with polarimetric SAR

ABSTRACTTropical forest is important in tackling global climate change. However, large-scale transformation of tropical forest in Indonesia is being done for oil palm plantation. Till date, there is a lack of scientific documentation that shows the most recent situation of oil palm plantation in the most eastern part of the country. This study documents the real condition of oil palm plantation expansion based on Sentinel data of European Space Agency (ESA). Sentinel-1 data sets of Papua Provinces of the year 2017 acquisition are used to avoid cloud cover problems in tropical regions. Object-based geospatial data feature extraction with Polarimetric SAR compared to the multiple classifier methods (that is, Gaussian Mixture Method, Random Forest, Support Vector Machine, and K-Nearest Neighbors) were introduced and employed using GRASS & QGIS of free open source software (FOSS). Google Earth Engine (GEE) was then used to verify the classification result of oil palm plantation. The preprocessing step is very crucial in achieving high accuracy. However, it is time-consuming. This study concluded that the proposed method and SVM classifiers achieved the highest accuracy, 99.7% and 99.12%, respectively. While the KNN, RF, and GMM classifier produced 94.63%, 89.06%, and 72.87%, respectively. We found that vast scale transformation of tropical rainforest in lowland areas is happening in eastern Indonesia. This situation should be strictly monitored and controlled to avoid more significant issues related to socio-economic-environmental implications.

Object-based feature selection for crop classification using multi-temporal high-resolution imagery

ABSTRACTWith high-resolution remote-sensing data, there are numerous possible features for object description, making the selection of optimal features a time-consuming and subjective process. While substantial efforts have been made to compare the utility of feature selection metrics, less attention has been paid to the efficiency of such in the context of object-based image analysis. In this study, the statistical measurement z-score was used to ensure compatibility with objects. We assessed the feasibility of a z-score method, and then ranked and reduced input features using a backward elimination technique. The results showed that separability can be efficiently estimated based on z-score values, and the near-infrared band performed the best for crop classification. A straightforward trend was observed, and the optimal feature set was created, which was a combination of spectral, temporal, texture information and vegetation indices. These features complement one another to help increase crop map accuracy. For the 40% of the entire sample sizes, the optimal feature sets produced the best trade-off between the number of inputs and classification accuracy, with the misclassification error of 7.09%. Additionally, reliable crop maps were obtained, with the overall accuracy of 92.64%, and the z-score method showed great potential for the separability of crops at object scale using remotely sensed multi-temporal data.

EARTHQUAKE BUILDING DAMAGE MAPPING BASED ON FEATURE ANALYZING METHOD FROM SYNTHETIC APERTURE RADAR DATA

Abstract. Playing an important role in gathering information of social infrastructure damage, Synthetic Aperture Radar (SAR) remote sensing is a useful tool for monitoring earthquake disasters. With the wide application of this technique, a standard method, comparing post-seismic to pre-seismic data, become common. However, multi-temporal SAR processes, are not always achievable. To develop a post-seismic data only method for building damage detection, is of great importance. In this paper, the authors are now initiating experimental investigation to establish an object-based feature analysing classification method for building damage recognition.

Synergistic object-based multi-class feature extraction in urban landscape using airborne LiDAR data

This study addresses the potential of geospatial information extraction by using light detection and ranging (LiDAR) data and aerial optical images in an urban landscape. We have adopted an advanced geographic object-based image analysis (GEOBIA) technique consisting of rule-based procedures relying upon the integration of spectral, textural, and spatial characteristics of aerial imagery and roughness of point cloud of LiDAR to fuse aerial imagery and airborne LiDAR for effective urban geospatial information extraction. This study is focused on the extraction of four tangible geospatial features, e.g., buildings, trees, marine vessels, and cars. LiDAR-derived normalized digital surface model (nDSM) was insufficient in delineating the polygon features because of the sparse point cloud density at the edges of features, which greatly affected the accuracy of extracting the polygon features. Therefore aerial imagery was supplemented in order to enhance the quality of extraction. The final feature extraction accuracy was assessed against manual digitization by visual interpretation, statistical analysis, and confusion matrix. The accuracy of feature extraction was found to be ranging from 90 to 95%. The accuracy of buildings class was improved using intensity image generated from LiDAR data and Hough image along with morphological operations. In a nutshell, this study highlights robust improvements in the geospatial extraction of urban features by merging more than one dataset synergistically.

Object-Based Features for House Detection from RGB High-Resolution Images

Automatic building extraction from satellite images, an open research topic in remote sensing, continues to represent a challenge and has received substantial attention for decades. This paper presents an object-based and machine learning-based approach for automatic house detection from RGB high-resolution images. The images are first segmented by an algorithm combing a thresholding watershed transformation and hierarchical merging, and then shadows and vegetation are eliminated from the initial segmented regions to generate building candidates. Subsequently, the candidate regions are subjected to feature extraction to generate training data. In order to capture the characteristics of house regions well, we propose two kinds of new features, namely edge regularity indices (ERI) and shadow line indices (SLI). Finally, three classifiers, namely AdaBoost, random forests, and Support Vector Machine (SVM), are employed to identify houses from test images and quality assessments are conducted. The experiments show that our method is effective and applicable for house identification. The proposed ERI and SLI features can improve the precision and recall by 5.6% and 11.2%, respectively.

Particle Swarm Optimization for Object-Based Feature Selection of VHSR Satellite Images

With the advent of very high spatial resolution (VHSR) satellite imagery, object-based classification (OBC) has attracted a great deal of attention in the remote sensing community. However, due to the numerous spectral, textural, and structural features which can be extracted for each object, feature selection is needed to address the curse-of-dimensionality issue. In this letter, a new wrapper object-based feature (OBF) selection method was applied in order to select the best combination of spectral, textural, and structural OBFs. In the first step, the image is segmented into objects using the multiresolution segmentation method. Subsequently, OBFs are extracted for each object. Last, optimal OBFs are selected using the proposed method based on particle swarm optimization (PSO) combined with the minimum distance classifier. Two VHSR remotely sensed images, acquired by a WorldView-2 sensor, were used to test the proposed method. The results of the experiment indicated that OBC using optimal OBFs led to significant improvement over the case where all extracted features were used. Furthermore, PSO has a better performance than do the three metaheuristic optimization algorithms: genetic algorithm, artificial bee colony, and honey-bee mating, according to the technique of order of preference by similarity to ideal solution analysis.