Semantic segmentation of terrestrial whole-sky images using the new W-Net model with the stationary wavelet transform 2D

The advancements in the methods and techniques in the field of computer vision have enabled numerous applications based on understanding and analysis of image data. Moreover, deep learning has brought a massive shift in image analysis, thereby attracting the attention of researchers worldwide. Many real-life application areas used the image segmentation techniques for the identification of different regions in an image and classify them into clusters depending upon the similarity. Many conventional techniques, namely thresh-olding, k-means clustering histogram-based segmentation, and edge detection algorithms were applied for the segmentation; these pre-existing methods were found to be less efficient because of human intervention. But with the turn-up of deep learning, it is considered as a predominant method in image processing. 50In today’s era where computer vision is contemplating image segmentation for various applications, it is of utmost importance to have a detailed review of it. In the past decade, image segmentation has evolved a lot and is defined on two levels of granularity, namely semantic segmentation and instance segmentation. Furthermore, semantic segmentation segments unknown objects or new objects and classify pixels which are semantically together. This approach can lay down the foundations for new models to improve prior existing computer vision methods. It is entrenched as a vigorous implement for the critical analysis of the different areas in given images. Firstly, we elucidate the basic terms and some mandatory concepts related to this particular field for a better understanding of the naive. Then, the chapter focuses on different methods and network structures which are applied in semantic image segmentation for deep analysis of images in different applications in contrast to conventional approaches. Also, we outline the strengths and weaknesses of this approach to present a superior perspective to the individuals. At last, we strive to disclose challenges of semantic image segmentation processes concurrence with deep learning and point out a set of promising future works.

Semantic segmentation is a significant research topic for decades and has been employed in several applications. In recent years, semantic segmentation has been focused on different deep learning approaches in the area of computer vision, which has aimed for getting superior efficiency while analyzing the aerial and remote-sensing images. The main aim of this review is to provide a clear algorithmic categorization and analysis of the diverse contribution of semantic segmentation of aerial images and expects to give the comprehensive details associated with the recent developments. In addition, the emerged deep learning methods demonstrated much improved performance measures on several public datasets and incredible efforts have been dedicated to advancing pixel-level accuracy. Hence, the analysis on diverse datasets of each contribution is studied, and also, the best performance measures achieved by the existing semantic segmentation models are evaluated. Thus, this survey can facilitate researchers in understanding the development of semantic segmentation in a shorter time, simplify understanding of its latest advancements, research gaps, and challenges to be used as a reference for developing the new semantic image segmentation models in the future.

Semantic segmentation forms the foundation for understanding very high resolution (VHR) remote sensing images, with extensive demand and practical application value. The convolutional neural networks (CNNs), known for their prowess in hierarchical feature representation, have dominated the field of semantic image segmentation. Recently, hierarchical vision transformers such as Swin have also shown excellent performance for semantic segmentation tasks. However, the hierarchical structure enlarges the receptive field to accumulate features and inevitably leads to the blurring of object boundaries. We introduce a novel object-aware network, Embedding deep SuperPixel, for VHR image semantic segmentation called ESPNet, which integrates advanced ConvNeXt and the learnable superpixel algorithm. Specifically, the developed task-oriented superpixel generation module can refine the results of the semantic segmentation branch by preserving object boundaries. This study reveals the capability of utilizing deep convolutional neural networks to accomplish both superpixel generation and semantic segmentation of VHR images within an integrated end-to-end framework. The proposed method achieved mIoU scores of 84.32, 90.13, and 55.73 on the Vaihingen, Potsdam, and LoveDA datasets, respectively. These results indicate that our model surpasses the current advanced methods, thus demonstrating the effectiveness of the proposed scheme.

In this paper, we focus on the semantic segmentation of images taken from a camera mounted on the front end of trains for measuring and managing rail-side facilities. Improving the efficiency and perhaps automating such tasks are crucial as they are currently done manually. We aim to realize this by capturing information about the railway environment through the semantic segmentation of train front-view camera images. Specifically, assuming that the lateral movement of trains are smooth, we propose a method to use information from multiple frames to consider temporal continuity during semantic segmentation. Based on the densely estimated optical flow between sequential frames, the weighted mean of class likelihoods of corresponding pixels of the focused frame are calculated. We also construct a new dataset consisting of train front-view camera images and its annotations for semantic segmentation. The proposed method outperforms a conventional single-frame semantic segmentation model, and the use of class likelihoods for the frame combination also proved effective.

Semantic segmentation of images requires an understanding of appearances of objects and their spatial relationships in scenes. The fully convolutional network (FCN) has been successfully applied to recognize objects' appearances, which are represented with RGB channels. Images augmented with depth channels provide more understanding of the geometric information of the scene in an image. In this paper, we present a multiple-branch neural network to utilize depth information to assist in the semantic segmentation of images. Our approach splits the image into layers according to the "scene-scale". We introduce the context-aware receptive field (CARF), which provides better control of the relevant context information of learned features. Each branch of the network is equipped with CARF to adaptively aggregate the context information of image regions, leading to a more focused domain that is easier to learn. Furthermore, we propose a new zig-zag architecture to exchange information between the feature maps at different levels, augmented by the CARFs of the backbone network and decoder network. With the flexible information propagation allowed by our zig-zag network, we enrich the context information of feature maps for the segmentation. We show that the zig-zag network achieves state-of-the-art performances on several public datasets.

Deep learning methods based on convolutional neural networks have shown to give excellent results in semantic segmentation of images, but the inherent irregularity of point cloud data complicates their usage in semantically segmenting 3D laser scanning data. To overcome this problem, point cloud networks particularly specialized for the purpose have been implemented since 2017 but finding the most appropriate way to semantically segment point clouds is still an open research question. In this study we attempted semantic segmentation of point cloud data with convolutional neural networks by using only the raw measurements provided by a multiple echo detection capable profiling laser scanner. We formatted the measurements to a series of 2D rasters, where each raster contains the measurements (range, reflectance, echo deviation) of a single scanner mirror rotation to be able to use the rich research done on semantic segmentation of 2D images with convolutional neural networks. Similar approach for profiling laser scanner in forest context has never been proposed before. A boreal forest in Evo region near Hämeenlinna in Finland was used as experimental study area. The data was collected with FGI Akhka-R3 backpack laser scanning system, georeferenced and then manually labelled to ground, understorey, tree trunk and foliage classes for training and evaluation purposes. The labelled points were then transformed back to 2D rasters and used for training three different neural network architectures. Further, the same georeferenced data in point cloud format was used for training the state-of-the-art point cloud semantic segmentation network RandLA-Net and the results were compared with those of our method. Our best semantic segmentation network reached the mean Intersection-over-Union value of 80.1% and it is comparable to the 80.6% reached by the point cloud -based RandLA-Net. The numerical results and visual analysis of the resulting point clouds show that our method is a valid way of doing semantic segmentation of point clouds at least in the forest context. The labelled datasets were also released to the research community.

Aiming at the problem that the representation ability of traditional features is weakly, this paper proposes a semantic segmentation method based on deep convolutional neural network and conditional random field. The pre-trained VGG-Net-16 model is used to extract more powerful image features, and then the semantic segmentation of images is achieved through the efficient use of multiple features and context information by conditional random fields. The experimental results show that compared with the three methods using traditional classical features, the method achieves the highest overall classification accuracy and Kappa coefficient, indicating that VGG-Net-16 can extract more effective features.

BackgroundThe automatic segmentation of kidneys in medical images is not a trivial task when the subjects undergoing the medical examination are affected by Autosomal Dominant Polycystic Kidney Disease (ADPKD). Several works dealing with the segmentation of Computed Tomography images from pathological subjects were proposed, showing high invasiveness of the examination or requiring interaction by the user for performing the segmentation of the images. In this work, we propose a fully-automated approach for the segmentation of Magnetic Resonance images, both reducing the invasiveness of the acquisition device and not requiring any interaction by the users for the segmentation of the images.MethodsTwo different approaches are proposed based on Deep Learning architectures using Convolutional Neural Networks (CNN) for the semantic segmentation of images, without needing to extract any hand-crafted features. In details, the first approach performs the automatic segmentation of images without any procedure for pre-processing the input. Conversely, the second approach performs a two-steps classification strategy: a first CNN automatically detects Regions Of Interest (ROIs); a subsequent classifier performs the semantic segmentation on the ROIs previously extracted.ResultsResults show that even though the detection of ROIs shows an overall high number of false positives, the subsequent semantic segmentation on the extracted ROIs allows achieving high performance in terms of mean Accuracy. However, the segmentation of the entire images input to the network remains the most accurate and reliable approach showing better performance than the previous approach.ConclusionThe obtained results show that both the investigated approaches are reliable for the semantic segmentation of polycystic kidneys since both the strategies reach an Accuracy higher than 85%. Also, both the investigated methodologies show performances comparable and consistent with other approaches found in literature working on images from different sources, reducing both the invasiveness of the analyses and the interaction needed by the users for performing the segmentation task.

Image segmentation is the first step in image analysis and one of the most important links. Because image segmentation can process images into simpler and more characteristic forms for analysis. With the rapid development of deep learning technology and its wide application in the field of semantic segmentation, the effect of semantic segmentation has been significantly improved. Image semantic segmentation is one of the core tasks of computer vision, and its goal is to efficiently classify each pixel of an input image. In recent years, deep learning is the technology that has the most profound impact on the computer field. With the help of deep learning, image semantic segmentation tasks have achieved many results in the fields of autonomous driving, biomedicine, and augmented reality. Compared with image classification and object detection, semantic segmentation can provide richer image semantic information. However, there are many problems in semantic segmentation based on deep learning. Firstly, it is difficult to make semantic segmentation data sets, which has the problems of difficult training and high production cost; Secondly, the computation and network parameters of most algorithms are huge, which makes them unable to be applied to mobile devices with limited computing resources, which limits the development of semantic segmentation; Moreover, many algorithms do not make full use of the hardware resources of the computing platform to accelerate the running speed of the program. This paper analyzes and summarizes the image semantic segmentation methods based on deep neural network. According to different network training methods, the existing image semantic segmentation is divided into fully supervised learning image semantic segmentation and weakly supervised learning image semantic segmentation. The effects, advantages and disadvantages of representative algorithms in each method are compared and analyzed, it also expounds the contribution of deep neural network to the field of semantic segmentation.

Most progress in semantic segmentation reports on daytime images taken under favorable illumination conditions. We instead address the problem of semantic segmentation of nighttime images and improve the state-of-the-art, by adapting daytime models to nighttime without using nighttime annotations. Moreover, we design a new evaluation framework to address the substantial uncertainty of semantics in nighttime images. Our central contributions are: 1) a curriculum framework to gradually adapt semantic segmentation models from day to night via labeled synthetic images and unlabeled real images, both for progressively darker times of day, which exploits cross-time-of-day correspondences for the real images to guide the inference of their labels; 2) a novel uncertainty-aware annotation and evaluation framework and metric for semantic segmentation, designed for adverse conditions and including image regions beyond human recognition capability in the evaluation in a principled fashion; 3) the Dark Zurich dataset, which comprises 2416 unlabeled nighttime and 2920 unlabeled twilight images with correspondences to their daytime counterparts plus a set of 151 nighttime images with fine pixel-level annotations created with our protocol, which serves as a first benchmark to perform our novel evaluation. Experiments show that our guided curriculum adaptation significantly outperforms state-of-the-art methods on real nighttime sets both for standard metrics and our uncertainty-aware metric. Furthermore, our uncertainty-aware evaluation reveals that selective invalidation of predictions can lead to better results on data with ambiguous content such as our nighttime benchmark and profit safety-oriented applications which involve invalid inputs.

The subject of research in the article is the module of automatic segmentation and subtraction of the background, which is created, based on the sequential application of methods of image preprocessing and modified method of interactive segmentation of images and implemented in the system of optical monitoring of the air situation. The aim of the work is to develop an image segmentation module to increase the efficiency of recognition of an air object type on a video image in the system of visual monitoring of the air environment by means of qualitative automatic segmentation. To solve this problem, a modified interactive algorithm in the mode of automatic selection of an object in the image, which allows more accurately, without the participation of the operator, to determine the foreground pixels of the image for further recognition of the type of airborne object. The following tasks are solved in the article: the analysis of existing methods of binarization of color images for semantic segmentation of images, which are used in image recognition systems; the development of a pipeline of methods for automatic segmentation of images in the system of optical monitoring of the air environment. In the work, the following methods are used: methods of digital image processing, methods of filtering and semantic segmentation of images, methods of graph analysis. The following results are obtained: the results of image processing with the proposed module of segmentation and background subtraction confirm the performance of the module procedures. The developed pipeline of methods included in the module demonstrates correct segmentation in 93% of test images in automatic mode without operator participation, which allows us to conclude about the effectiveness of the proposed module. Conclusions: The implementation of the developed module of segmentation and background subtraction for the system of optical monitoring of the air environment allowed to solve the problem of segmentation of video images for further recognition of aerial objects in the system of optical monitoring of the air environment in automatic mode with a high degree of reliability, thus increasing the operational efficiency of this system.

Semantic image segmentation is one of the fundamental applications of computer vision which can also be called pixel-level classification. Semantic image segmentation is the process of understanding the role of each pixel in an image. Over time, the model for completing Semantic Image Segmentation has developed very rapidly. Due to this rapid growth, many models related to Semantic Image Segmentation have been produced and have also been used or applied in many domains such as medical areas and intelligent transportation. Therefore, our motivation in making this paper is to contribute to the world of research by conducting a review of Semantic Image Segmentation which aims to provide a big picture related to the latest developments related to Semantic Image Segmentation. In addition, we also provide the results of performance measurements on each of the Semantic Image Segmentation methods that we discussed using the Intersectionover-Union (IoU) method. After that, we provide a comparison for each semantic image segmentation model that we discuss using the results of the IoU and then provide conclusions related to a model that has good performance. We hope this review paper can facilitate researchers in understanding the development of Semantic Image Segmentation in a shorter time, simplify understanding of the latest advancements in Semantic Image Segmentation, and can also be used as a reference for developing new Semantic Image Segmentation models in the future

Image semantic segmentation is an essential technique for studying human behavior through image data. This paper proposes an image semantic segmentation method for human behavior research. Firstly, an end-to-end convolutional neural network architecture is proposed, which consists of a depth-separable jump-connected fully convolutional network and a conditional random field network; then jump-connected convolution is used to classify each pixel in the image, and an image semantic segmentation method based on convolutional neural network is proposed; and then a conditional random field network is used to improve the effect of image segmentation of human behavior and a linear modeling and nonlinear modeling method based on the semantic segmentation of conditional random field image is proposed. Finally, using the proposed image segmentation network, the input entrepreneurial image data is semantically segmented to obtain the contour features of the person; and the segmentation of the images in the medical field. The experimental results show that the image semantic segmentation method is effective. It is a new way to use image data to study human behavior and can be extended to other research areas.

Semantic segmentation is an important task in medical-supporting. The purpose of semantic segmentation í to identify pre-defined objects, and its pixel-by-pixel location. The most popular method in semantic segmentation was using Convolutional neural network which has considerably improved semantic image segmentation. This work investigates a Blending loss which incorporates into traditional methods. Three popular algorithms are U-Net, PSPNet and FPN are examined carefully to investigate upgrading performance after combining new objective function. Moreover, we did experiments on two medical datasets to avoid bias and verify performance of the new method.

Food sustainability is still one of the main priorities for many countries as it contributes to the economy and stability of the nation. For government in many countries whose peoples consumes rice as its staple food, food self-sufficiency initiatives highly depend on accurate prediction of paddy field map. Mapping paddy field task is a challenging problem which cannot be handled manually especially when the paddy fields are spread out in very wide geographical areas such as those in Indonesia. Fortunately, wide availability of satellite imagery and the advent of deep learning technology in the past ten years have made it possible to improve efficiency of most parts of those manual works involving image semantic segmentation tasks. However, satellite image-based semantic segmentation is a challenging task. High object complexity, cloud partial occlusion, larger image size than a computer memory can stored can hinder accuracy of the image segmentation results. This paper presents a method for paddy field map generating using semantic image segmentation approach in which Pyramid Scene Parsing Net model is used for segmenting satellite imagery. The generated paddy map can be used as a basis for decision-making, especially in the agricultural sector. Analysis of local land use/land cover dynamics. The results of his experiments using SPOT 6 satellite imagery from the Pahung region of Central Kalimantan achieved average training accuracy, best training accuracy and test accuracy of 0.85, 0.86 and 0.89 respectively. These results indicated that the semantic segmentation model is suitable for addressing the same task in different crops.