Object Extraction Research Articles

3D lidar point-cloud projection operator and transfer machine learning for effective road surface features detection and segmentation

The classification and extraction of road markings and lanes are of critical importance to infrastructure assessment, planning and road safety. We present a pipeline for the accurate segmentation and extraction of rural road surface objects in 3D lidar point-cloud, as well as a method to extract geometric parameters belonging to tar seal. To decrease the computational resources needed, the point-clouds were aggregated into a 2D image space before being transformed using affine transformations. The Mask R-CNN algorithm is then applied to the transformed image space to localize, segment and classify the road objects. The segmentation results for road surfaces and markings can then be used for geometric parameter estimation such as road widths estimation, while the segmentation results show that the efficacy of the existing Mask R-CNN to segment needle-type objects is improved by our proposed transformations.

Mathematical Model for Single and Multiple Object Extraction

In the image processing, noise is referred to as the visual distortion. This undesirable by-product may be captured in an image due to unpreventable assorted reasons. The interference of natural phenomena and technical problem, such as small sensor size, long exposure time, low ISO, shadow noise etc., can pollute image. The presence of noise images affects image processing outputs that include segmentation. Segmentation for noisy images is the major concern. To tackle this issue, we propose a modernistic model that is able neutralize the negative effects of outlier using the characteristic of kernel function by different approaches such as linear approach and quadratic approach for global segmentation. Moreover the weight function is used for local segmentation of noisy images. Comparing with classical models, the proposed technique shows robust performance. In comparison with the wellknown models such as Chan-Vese (CV) model , Yongfei Wu and Chuanjiang He (Wu-He) model and Chunming Li (Li) model we conclude that performance of our new model is much better.

Attention Multi-Scale Network for Automatic Layer Extraction of Ice Radar Topological Sequences

Analyzing the surface and bedrock locations in radar imagery enables the computation of ice sheet thickness, which is important for the study of ice sheets, their volume and how they may contribute to global climate change. However, the traditional handcrafted methods cannot quickly provide quantitative, objective and reliable extraction of information from radargrams. Most traditional handcrafted methods, designed to detect ice-surface and ice-bed layers from ice sheet radargrams, require complex human involvement and are difficult to apply to large datasets, while deep learning methods can obtain better results in a generalized way. In this study, an end-to-end multi-scale attention network (MsANet) is proposed to realize the estimation and reconstruction of layers in sequences of ice sheet radar tomographic images. First, we use an improved 3D convolutional network, C3D-M, whose first full connection layer is replaced by a convolution unit to better maintain the spatial relativity of ice layer features, as the backbone. Then, an adjustable multi-scale module uses different scale filters to learn scale information to enhance the feature extraction capabilities of the network. Finally, an attention module extended to 3D space removes a redundant bottleneck unit to better fuse and refine ice layer features. Radar sequential images collected by the Center of Remote Sensing of Ice Sheets in 2014 are used as training and testing data. Compared with state-of-the-art deep learning methods, the MsANet shows a 10% reduction (2.14 pixels) on the measurement of average mean absolute column-wise error for detecting the ice-surface and ice-bottom layers, runs faster and uses approximately 12 million fewer parameters.

Assessing the Visual Esthetics of User Interfaces: A Ten-Year Systematic Mapping

ABSTRACT Visual esthetics is recognized as an essential factor of perceived usability, credibility, and overall quality of software systems user interfaces. Thus, it is crucial to assess this factor as part of the development process to improve graphical user interface (GUI) quality. And, although there exist diverse approaches to assess visual esthetics today, they significantly differ in their definition and techniques for measuring it. Therefore, we performed a systematic mapping to provide an overview comparing the existing approaches from the last ten years (2010–2020). We encountered 27 articles reporting different approach types, including assessments measuring user perception of visual esthetics via human responses, metrics based on GUI elements, extracting handcrafted features from GUI screenshots, and deep-learning-based approaches. Most of them implement objective approaches, either using metrics or feature extraction. The majority of the approaches are aimed at web GUIs, with only three focusing on mobile GUIs, considering their smaller screen size, less use of color, and a more vertical element alignment. Few approaches have been comprehensively evaluated regarding reliability and validity. These results indicate the need for further research considering the importance of visual esthetics as part of software quality. The results of this study present an overview and comparison of these approaches from the last decade providing a systematic basis for researchers to better understand the challenges for GUI visual esthetics assessments and the enhancement of these approaches.

Fast and robust interactive image segmentation in bilateral space with reliable color modeling and higher order potential

We propose an optimization framework for interactive image segmentation (IIS) that operates in bilateral space to achieve robust object extraction and instant visual feedback. More specifically, we first resample an input image using a regular bilateral grid with a resolution that is typically coarser than the input image to reduce the complexity of subsequent IIS tasks. We then design a Markov random field energy on the vertices of the bilateral grid that can be solved efficiently using a standard graph cut label assignment. To achieve this, we introduce reliable color models to distinguish the foreground and background despite the presence of extremely difficult cases and a higher-order potential to encourage spatial consistency in segmentation. We conduct comprehensive experiments on three standard interactive segmentation datasets, MSRA 10K, IIS, and PASCAL VOC 2012 segmentation validation set. The results show that the proposed method achieves competitive performance compared with state-of-the-art methods while making the current system efficient in terms of speed.

Improved Object Matching in Multi-Objects Tracking Based On Zernike Moments and Combination of Multiple Similarity Metrics

In video surveillance, multiple objects tracking (MOT) is a challenging task due to object matching problem in consecutive frames. The present paper aims to propose an improved object matching approach in MOT based on Zernike Moments and combination of multiple similarity distance metrics. In this work, the object is primarily detected using background subtraction method while the Gaussian Mixture Model (GMM) is applied for object extraction in the next frames. Subsequently, the color histogram and the magnitude of Zernike moments of the objects are calculated. In the next step, the objects are matched in the current and the previous frames based on the Hausdorff distance between objects, Earth Mover's (EMD) distance between their color histograms, and Chi-square distance between their Zernike moments. Then, a voting mechanism is designed to find the best consensus object matching from the aforementioned metrics. Eventually, the location of each object is predicted by the Kalman filter to continue tracking in subsequent frames. The results show that the object tracking and matching performance is improved using the proposed method in the video sequences of the multi-camera pedestrian "EPFL" video dataset. Specifically, errors caused by the merging of targets are reduced in the proposed tracking process.

A Grid-based Automated Building Extraction Technique for Low-cost UAV Images

With increasing urbanization, new technology is required to fulfil both human and environmental needs. At present, low-cost UAVs are used in surveying and mapping, and during the past few years, they have reached a level of practical requirements to allow the use of these systems as mapping platforms. Moreover, UAV based mapping provides required accuracy in line with cadastral laws and policies. Extraction of urban objects is a pre-requisite in various applications. In general, detection of buildings plays a major role in the field of remote sensing image processing, and also in urban planning and management. However, there is no ‘proper’ method developed to detect building features automatically from UAV images because there are usually too many details and distortions on the images. This paper presents an effective approach for extracting buildings from UAV images through the incorporation of orthophotographs and dense point clouds, rather than the traditional pixel based classification. In this method, different feature-based conditions are introduced with the help of a grid-based data structure for more accurate and quick extraction of building features. To verify the generality and advantage of the proposed method, the procedure is evaluated by performing experiments with a dataset acquired over the study area, which has a variety of building patterns and styles. The experimental results show an excellent performance in the detection of buildings, with an average overall accuracy greater than 80%. The final overall correctness and quality of building extraction are more than 80% and 65%, respectively. Therefore, there is a need to focus on more advanced conditions for building detection, to obtain optimum results.

Deep learning integral imaging for three-dimensional visualization, object detection, and segmentation

A depth slice image that is computationally reconstructed from an integral imaging system consists of focused and out of focus areas. The unfocused areas affect three-dimensional (3D) image analyses and visualization including 3D object detection, extraction, and tracking. In this work, we present a deep learning integral imaging system that can reconstruct a 3D image without the out of focus areas and can accomplish target detection and segmentation at the same time. A Mask-Regional Convolutional Neural Network (Mask-RCNN) deep learning algorithm was trained using a public dataset and applied to detect and segment multiple targets in two-dimensional (2D) elemental images in the integral imaging system. The 3D images were then reconstructed using segmented elemental images with the target detected. The proposed method works well in the presence of partial occlusions. Experimental results show the performance of the proposed scheme.

A Unified Multi-Task Learning Framework for Joint Extraction of Entities and Relations

Joint extraction of entities and relations focuses on detecting entity pairs and their relations simultaneously with a unified model. Based on the extraction order, previous works mainly solve this task through relation-last, relation-first and relation-middle manner. However, these methods still suffer from the template-dependency, non-entity detection and non-predefined relation prediction problem. To overcome these challenges, in this paper, we propose a unified multi-task learning framework to divide the task into three interacted sub-tasks. Specifically, we first introduce the type-attentional method for subject extraction to provide prior type information explicitly. Then, the subject-aware relation prediction is presented to select useful relations based on the combination of global and local semantics. Third, we propose a question generation based QA method for object extraction to obtain diverse queries automatically. Notably, our method detects subjects or objects without relying on NER models and thus it is capable of dealing with the non-entity scenario. Finally, three sub-tasks are integrated into a unified model through parameter sharing. Extensive experiments demonstrate that the proposed framework outperforms all the baseline methods on two benchmark datasets, and further achieve excellent performance for non-predefined relations.

Video analysis of intelligent teaching based on machine learning and virtual reality technology

In this paper, we study and analyse the teaching video of oil painting art through machine learning combined with virtual reality computing. Since the current oil painting, image acquisition method cannot meet the user's demand for multi-dimensional image description, and at the same time the retrieval method is too simple to perform high flexibility retrieval, we try to adopt the deep learning-based object extraction fusion method. Also, objects with poor performance quality are not suitable for further interception and cutting. We first pre-process the images in the image library to filter out the relatively high-quality images and then filter out the objects whose saliency and clarity do not reach the queue value by judging the saliency and clarity values of the images. Next, a series of aesthetic criteria, such as visual balance, visual triangulation, and centrosymmetric diagonal composition criteria, used to further filter the objects with relatively poor quality and low ratings. Then, we expand the areas with high saliency, match the contours of the segmented image elements with the contours of the user-drawn image to return an optimal matching value, and finally improve the quality and naturalness of the image by learning the deeper features of the image based on the style migration. The experimental framework based on TensorFlow is a new application of deep learning in the field of image synthesis, which has a very good improvement in the implementation efficiency compared with the traditional method. Using virtual reality technology to carry out teaching practice and analyse the effect of teaching practice, students can immerse themselves in art appreciation teaching activities, accept multiculturalism, learn through experience, and improve aesthetic quality.

Cultural Heritage and Obia

The historic centre of a town is its oldest and original core. It needs special protection in order to ensure the conservation of its historical, artistic and environmental heritage. In Italy, the definition of historic centres and the protection of their cultural heritage evolved in time, up to the current special attention for the historical aspects. The main threats to historical centres are real estate speculation and mass tourism. The purpose of this study is to catalog and monitor historic centers over time, in the context of urban planning. High-resolution satellite images and geographic information systems (GIS) offer new tools for urban planning and also for cultural heritage themes. “Real time” evaluation of urban structures, cartographic updating, monitoring of the progress of major works, with particular regard to cultural heritage, are made possible by the use of high-resolution images, which facilitate the identification of changes in urban and non-urban areas. The technique of Object Based Image Analysis (OBIA) has been used for image analysis and interpretation. OBIA allows a good interpretation of the scene captured by sensors thanks to classification-based segmentation and extraction of complete objects and their topological relations. This yields to a classification similar to the output of human photo-interpreter, but with a better reproducibility and homogeneity. In this paper we describe, through an application example, the potentiality and the difficulties of this technique and some results. The whole information obtained from segmented and categorized satellite images has been structured in a proper GIS, so that it can be overlaid with other environmental data. Information structuring and special metaheuristic analyses allow to study and monitor historic centers and cultural heritage. This methodology allows to identify the places at risk that need priority restoration; moreover it allows to keep track of changes that occurred over time

Assessing Shoreline Change using Historical Aerial and RapidEye Satellite Imagery (Cape Jaffa, South Australia)

DaSilva, M.; Miot da Silva, G.; Hesp, P.A.; Bruce, D.; Keane, R., and Moore, C., 2021. Assessing shoreline change using historical aerial and RapidEye satellite imagery (Cape Jaffa, South Australia). Journal of Coastal Research, 37(3), 468–483. Coconut Creek (Florida), ISSN 0749-0208.The coastal zone is a dynamic area which can experience substantial natural change in short time periods, but changes are also associated with human modifications to the coastline. This case study is focused around assessing shoreline change associated with the Cape Jaffa Marina and canal estate in South Australia. The research comprises a GIS based analysis of shoreline change utilising aerial imagery from 1975 to 2005, which provides information of the morphological coastal trends prior to construction in 2008 of the marina/canal estate. In addition, imagery collected by the RapidEye satellite constellation was used to assess shoreline changes in the decade since construction, 2009 to 2019. The shoreline change statistics over the past few decades were calculated using the Digital Shoreline Analysis System extension of ESRI's ArcGIS. The image analysis workflow is based on the objective extraction of shoreline proxies from individual image statistics in a semi-automated process and is used to identify the waterline and edge of vegetation shoreline proxies in a time series analysis. The results provide a case study of a historically progradational sandy coastline experiencing substantial amounts of alongshore sediment transport, the corresponding obstruction from coastal infrastructure, and the resulting morphological changes to the Cape Jaffa shoreline. Shoreline change trends were altered from a predominantly accretional shoreline before the construction of the marina to one oscillating between extremes of erosion and accretion. The results showed significant accretion occurring updrift (Net Shore Movement (NSM) of 106 m) and significant erosion downdrift (NSM of –80 m) of the marina's training walls and entrance. The information derived from Earth observation satellites, such as RapidEye, can provide valuable insights into trend analysis due to their relatively high spatial (5 metre) and temporal resolution (5.5-day revisit).

Automatic Extraction and Labelling of Memorial Objects From 3D Point Clouds

This research addresses the problem of automatic extraction of memorial objects from cultural heritage sites represented as scenes of 3D point clouds. Point clouds provide a fine spatial resolution and accurate proxy of the real world. However, how to use them directly is not always obvious. This is especially true for applications where extensive training data or computational resources are not available. In this paper, we present a methodology for automatic segmentation and labelling of cultural heritage objects from 3D point cloud scenes. The proposed methodology is based on machine learning techniques and, in particular, makes use of the concept of transfer learning. Memorial objects are segmented from the scene based on their geometric shape characteristic through a conditional multi-scale partitioning scheme. Then, high-level latent feature descriptors are extracted by a convolutional neural network pre-trained on different 3D object models from a standard dataset (e.g., ModelNet). Based on these descriptors, a classification model (multilayer perceptron) is trained and applied to obtain semantic labels. Experiments demonstrated that the proposed methodology is effective for the extraction and labelling of grave marker objects from cultural heritage sites.

FourierCam: a camera for video spectrum acquisition in a single shot

The novel camera architecture facilitates the development of machine vision. Instead of capturing frame sequences in the temporal domain as traditional video cameras, FourierCam directly measures the pixel-wise temporal spectrum of the video in a single shot through optical coding. Compared to the classic video cameras and time-frequency transformation pipeline, this programmable frequency-domain sampling strategy has an attractive combination of characteristics for low detection bandwidth, low computational burden, and low data volume. Based on the various temporal filter kernel designed by FourierCam, we demonstrated a series of exciting machine vision functions, such as video compression, background subtraction, object extraction, and trajectory tracking.

Analysis of greenhouse gas emissions in the European Union member states with the use of an agglomeration algorithm

The use of fossil fuels as sources of energy is related to the emission of pollutants into the atmosphere. The implementation of international commitments on reducing emissions requires their continuous monitoring. The main energy resources for electricity production in the world include fossil fuels, i.e. oil, coal and natural gas, and according to projections their dominant role in the market of energy resources will persist for at least the next two decades. The aim of this article is to analyse the level of differentiation of European Union member states in terms of emissions of four greenhouse gases and to identify groups of similar countries based on these criteria. Such studies will provide information that will enrich our knowledge about the contribution of each European Union country to the emissions of greenhouse gases. This article uses a taxonomic method - cluster analysis, namely the agglomerative algorithm, which enables the extraction of objects that are similar to each other from the data and then to merge them into groups. In this way, a number of homogeneous subsets can be obtained from one heterogeneous set of objects. European Union countries make up the objects of segmentation. Each of them are described by their level of greenhouse gas emissions, such as carbon dioxide, methane, nitrogen oxides and nitrous oxides. Groups of homogeneous countries are distinguished due to total emissions and due to the level of their emissions per capita. Analysis is based on annual Eurostat reports concerning greenhouse gas emissions.

Feature Extraction and Selection in Archaeological Images for Automatic Annotation

In this paper, we present two steps in the process of automatic annotation in archeological images. These steps are feature extraction and feature selection. We focus our research on archeological images which are very much studied in our days. It presents the most important steps in the process of automatic annotation in an image. Feature extraction techniques are applied to get the feature that will be used in classifying and recognizing the images. Also, the selection of characteristics reduces the number of unattractive characteristics. However, we reviewed various images of feature extraction techniques to analyze the archaeological images. Each feature represents one or more feature descriptors in the archeological images. We focus on the descriptor shape of the archaeological objects extraction in the images using contour method-based shape recognition of the monuments. So, the feature selection stage serves to acquire the most interesting characteristics to improve the accuracy of the classification. In the feature selection section, we present a comparative study between feature selection techniques. Then we give our proposal of application of methods of selection of the characteristics of the archaeological images. Finally, we calculate the performance of two steps already mentioned: the extraction of characteristics and the selection of characteristics.

A class-independent flexible algorithm to generate region proposals

Generating a sufficient number of regions with high accuracy is an important objective in the region proposal generation techniques. This paper presents a new, robust, and effective approach, which is based on the bottom-up segmentation, to produce a pool of well-quality regions. After image segmentation, the segmented candidates are expanded into the surrounding regions. The suggested algorithm produces some enlarged regions, which better cover objects and stuff. The proposed process can be applied in three different modes, namely fixed_mode, all_mode, and efficient_mode. The fixed_mode extends each region into parts of all the adjacent regions using an extension controller, which considers adjacent sequential pixels for each point on the region boundary. In all_mode, the current region is merged with all the adjacent regions to generate a larger region. The efficient_mode is then implemented using the accumulation of the results from both the fixed_mode and all_mode. Besides, the algorithm can be repeated in the fixed_mode and all_mode by considering a variety of values for the extension controller factor. No features are required to be extracted in the proposed algorithm, except for the image segmentation stage. In this study, four challenging datasets known as MSRC, VOC2007, VOC2012, and COCO 2017 are used to compare the proposed algorithm with other segmentation and region proposal algorithms. As a significant advantage compared to well-known region proposal algorithms, our approach achieves a greater Recall with the desirable number of regions. Furthermore, the algorithm shows a good improvementin extraction of small, medium, and large objects.

Semantic segmentation sample augmentation based on simulated scene generation—case study on dock extraction from high spatial resolution imagery

ABSTRACT Deep learning-based semantic segmentation methods, such as fully convolutional networks (FCNs), are state-of-the-art techniques for object extraction from high spatial resolution images. However, collecting massive scene-formed training samples typically required in FCNs is time-consuming and labour-intensive. A suit of automatic sample augmentation schemes based on simulated scene generation is proposed in this study to reduce the manual workload. Proposed schemes include style transfer, target embedding, and mixed modes by utilizing techniques, such as texture transfer, image inpainting, and region-line primitive association framework, which automatically expand the sample set on the basis of a small number of real samples. Dock extraction experiments using UNet are conducted on China’s GaoFen-2 imagery with expanded sample sets. Results showed that the proposed schemes can successfully generate sufficient simulation samples, increase sample diversity, and subsequently improve semantic segmentation accuracy. Compared with the results that use the original real sample set, measures of F 1-score (F 1) and intersection over union (IoU) of dock extraction accuracy demonstrate a maximum improvement of 20.53% and 23.01%, respectively, after sample augmentation.

Advanced Color Edge Detection Using Clifford Algebra in Satellite Images

Edge detection is widely used for image processing to improve the detection and classification of objects, segmentation, and extraction of other features. Satellite images are rich in information about objects with different color intensity and have a large amount of noise, so it is difficult to achieve recognition, classification, and feature extraction of small objects through traditional edge detection algorithms. The colors in satellite images suffer from a large amount of overlap due to areas or weather conditions that generate a lot of noise. Edge detection provides detailed information about objects in an image by reducing unnecessary feature information. Edge detection in color images is more challenging than edge detection in gray-level images. This paper proposes a method for the edge detection of color images using Clifford algebra and its sub-algebra, quaternions. Quaternion-based Fourier transform is used to process red, green and blue (RGB) images separately in the vector field. A 3×3 quaternion mask is developed to filter out frequencies of the image in multiple directions and only provides details about the edges. The algorithm works on three channels individually; the output is then processed through quaternion Fourier transform (QFT) and inverse QFT with a 3×3 mask to filter high frequencies. The proposed algorithm is compared with traditional edge detection algorithms using a satellite image dataset that has different types of objects and detailed information. Results are validated through entropy, structure similarity, and noise error to prove that our proposed algorithm provides satisfactory performance on different remote sensed images.

Deep Neural Networks for Quantitative Damage Evaluation of Building Losses Using Aerial Oblique Images: Case Study on the Great Wall (China)

Automated damage evaluation is of great importance in the maintenance and preservation of heritage structures. Damage investigation of large cultural buildings is time-consuming and labor-intensive, meaning that many buildings are not repaired in a timely manner. Additionally, some buildings in harsh environments are impossible to reach, increasing the difficulty of damage investigation. Oblique images facilitate damage detection in large buildings, yet quantitative damage information, such as area or volume, is difficult to generate. In this paper, we propose a method for quantitative damage evaluation of large heritage buildings in wild areas with repetitive structures based on drone images. Unlike existing methods that focus on building surfaces, we study the damage of building components and extract hidden linear symmetry information, which is useful for localizing missing parts in architectural restoration. First, we reconstruct a 3D mesh model based on the photogrammetric method using high-resolution oblique images captured by drone. Second, we extract 3D objects by applying advanced deep learning methods to the images and projecting the 2D object segmentation results to 3D mesh models. For accurate 2D object extraction, we propose an edge-enhanced method to improve the segmentation accuracy of object edges. 3D object fragments from multiple views are integrated to build complete individual objects according to the geometric features. Third, the damage condition of objects is estimated in 3D space by calculating the volume reduction. To obtain the damage condition of an entire building, we define the damage degree in three levels: no or slight damage, moderate damage and severe damage, and then collect statistics on the number of damaged objects at each level. Finally, through an analysis of the building structure, we extract the linear symmetry surface from the remaining damaged objects and use the symmetry surface to localize the positions of missing objects. This procedure was tested and validated in a case study (the Jiankou Great Wall in China). The experimental results show that in terms of segmentation accuracy, our method obtains results of 93.23% mAP and 84.21% mIoU on oblique images and 72.45% mIoU on the 3D mesh model. Moreover, the proposed method shows effectiveness in performing damage assessment of objects and missing part localization.