Scale-space Concept Research Articles

Multiscale Features Supported DeepLabV3+ Optimization Scheme for Accurate Water Semantic Segmentation

In the task of using deep learning semantic segmentation model to extract water from high-resolution remote sensing images, multiscale feature sensing and extraction have become critical factors that affect the accuracy of image classification tasks. A single-scale training mode will cause one-sided extraction results, which can lead to “reverse” errors and imprecise detail expression. Therefore, fusing multiscale features for pixel-level classification is the key to achieving accurate image segmentation. Based on this concept, this paper proposes a deep learning scheme to achieve fine extraction of image water bodies. The process includes multiscale feature perception splitting of images, a restructured deep learning network model, multiscale joint prediction, and postprocessing optimization performed by a fully connected conditional random field (CRF). According to the scale space concept of remote sensing, we apply hierarchical multiscale splitting processing to images. Then, we improve the structure of the image semantic segmentation model DeepLabV3+, an advanced image semantic segmentation model, and adjust the feature output layer of the model to multiscale features after weighted fusion. At the back end of the deep learning model, the water boundary details are optimized with the fully connected CRF. The proposed multiscale training method is well adapted to feature extraction for the different scale images in the model. In the multiscale output fusion, assigning different weights to the output features of each scale controls the influence of the various scale features on the water extraction results. We carried out a large number of water extraction experiments on GF1 remote sensing images. The results show that the method significantly improves the accuracy of water extraction and demonstrates the effectiveness of the method.

Boundary points based scale invariant 3D point feature

In this paper, we propose a method for encoding scale invariant 3D point features. We extract a set of boundary points from a point cloud. Next, we apply the scale-space concept on the boundary points to detect the scale invariant point border. We confirm three orthometric axes as the local reference frames. Three distribution matrices are generated by implementing the strategy of SPIN image method, and one-row-vector of descriptors are finally calculated. Experimental results on simulated and real scene point clouds demonstrate that the scale-invariant features of 3D point clouds can be effectively encoded by our method.

Temporal Scale Selection in Time-Causal Scale Space

When designing and developing scale selection mechanisms for generating hypotheses about characteristic scales in signals, it is essential that the selected scale levels reflect the extent of the underlying structures in the signal. This paper presents a theory and in-depth theoretical analysis about the scale selection properties of methods for automatically selecting local temporal scales in time-dependent signals based on local extrema over temporal scales of scale-normalized temporal derivative responses. Specifically, this paper develops a novel theoretical framework for performing such temporal scale selection over a time-causal and time-recursive temporal domain as is necessary when processing continuous video or audio streams in real time or when modelling biological perception. For a recently developed time-causal and time-recursive scale-space concept defined by convolution with a scale-invariant limit kernel, we show that it is possible to transfer a large number of the desirable scale selection properties that hold for the Gaussian scale-space concept over a non-causal temporal domain to this temporal scale-space concept over a truly time-causal domain. Specifically, we show that for this temporal scale-space concept, it is possible to achieve true temporal scale invariance although the temporal scale levels have to be discrete, which is a novel theoretical construction. The analysis starts from a detailed comparison of different temporal scale-space concepts and their relative advantages and disadvantages, leading the focus to a class of recently extended time-causal and time-recursive temporal scale-space concepts based on first-order integrators or equivalently truncated exponential kernels coupled in cascade. Specifically, by the discrete nature of the temporal scale levels in this class of time-causal scale-space concepts, we study two special cases of distributing the intermediate temporal scale levels, by using either a uniform distribution in terms of the variance of the composed temporal scale-space kernel or a logarithmic distribution. In the case of a uniform distribution of the temporal scale levels, we show that scale selection based on local extrema of scale-normalized derivatives over temporal scales makes it possible to estimate the temporal duration of sparse local features defined in terms of temporal extrema of first- or second-order temporal derivative responses. For dense features modelled as a sine wave, the lack of temporal scale invariance does, however, constitute a major limitation for handling dense temporal structures of different temporal duration in a uniform manner. In the case of a logarithmic distribution of the temporal scale levels, specifically taken to the limit of a time-causal limit kernel with an infinitely dense distribution of the temporal scale levels towards zero temporal scale, we show that it is possible to achieve true temporal scale invariance to handle dense features modelled as a sine wave in a uniform manner over different temporal durations of the temporal structures as well to achieve more general temporal scale invariance for any signal over any temporal scaling transformation with a scaling factor that is an integer power of the distribution parameter of the time-causal limit kernel. It is shown how these temporal scale selection properties developed for a pure temporal domain carry over to feature detectors defined over time-causal spatio-temporal and spectro-temporal domains.

Distinctive curves features

Curves and lines are geometrical, abstract features of an image. Whereas interest points are more limited, curves and lines provide much more information of the image structure. However, the research done in curve and line detection is very fragmented. The concept of scale space is not yet fused very well into curve and line detection. Keypoint (e.g. SIFT, SURF, ORB) is a successful concept which represent features (e.g. blob, corner etc.) in scale space. Stimulated by the keypoint concept, a method which extracts distinctive curves (DICU) in scale space, including lines as a special form of curve features is proposed. A curve feature can be represented by three keypoints (two end points, and one middle point). A good way to test the quality of detected curves is to analyse the repeatability under various image transformations. DICU using the standard Oxford benchmark is evaluated. The overlap error is calculated by averaging the overlap error of three keypoints on the curve. Experiment results show that DICU achieves good repeatability comparing with other state-of-the-art methods. To match curve features, a relatively uncomplicated way is to combine local descriptors of three keypoints on each curve.

Generalized Gaussian Scale-Space Axiomatics Comprising Linear Scale-Space, Affine Scale-Space and Spatio-Temporal Scale-Space

This paper describes a generalized axiomatic scale-space theory that makes it possible to derive the notions of linear scale-space, affine Gaussian scale-space and linear spatio-temporal scale-space using a similar set of assumptions (scale-space axioms). The notion of non-enhancement of local extrema is generalized from previous application over discrete and rotationally symmetric kernels to continuous and more general non-isotropic kernels over both spatial and spatio-temporal image domains. It is shown how a complete classification can be given of the linear (Gaussian) scale-space concepts that satisfy these conditions on isotropic spatial, non-isotropic spatial and spatio-temporal domains, which results in a general taxonomy of Gaussian scale-spaces for continuous image data. The resulting theory allows filter shapes to be tuned from specific context information and provides a theoretical foundation for the recently exploited mechanisms of shape adaptation and velocity adaptation, with highly useful applications in computer vision. It is also shown how time-causal spatio-temporal scale-spaces can be derived from similar assumptions. The mathematical structure of these scale-spaces is analyzed in detail concerning transformation properties over space and time, the temporal cascade structure they satisfy over time as well as properties of the resulting multi-scale spatio-temporal derivative operators. It is also shown how temporal derivatives with respect to transformed time can be defined, leading to the formulation of a novel analogue of scale normalized derivatives for time-causal scale-spaces. The kernels generated from these two types of theories have interesting relations to biological vision. We show how filter kernels generated from the Gaussian spatio-temporal scale-space as well as the time-causal spatio-temporal scale-space relate to spatio-temporal receptive field profiles registered from mammalian vision. Specifically, we show that there are close analogies to space-time separable cells in the LGN as well as to both space-time separable and non-separable cells in the striate cortex. We do also present a set of plausible models for complex cells using extended quasi-quadrature measures expressed in terms of scale normalized spatio-temporal derivatives. The theories presented as well as their relations to biological vision show that it is possible to describe a general set of Gaussian and/or time-causal scale-spaces using a unified framework, which generalizes and complements previously presented scale-space formulations in this area.

The Hilbert Transform on the Two-Sphere: A Spectral Characterization

The local analysis of signals arising on the sphere is a common task in earth sciences. On the real line the analytic signal turned out to be an important representation in local one-dimensional signal processing. Its generalization to two dimensions is the monogenic signal, and the properties of the analytic and the monogenic signal in the Fourier domain are well known. A generalization to the sphere is given by the Hilbert transform on the sphere known from Clifford analysis. To obtain a spectral characterization, the transform has to be decomposed into spherical harmonic functions. In this paper, we derive the spherical harmonic coefficients of the Hilbert transform on the sphere and give a series expansion. This will show that it acts as a differential operator on the spherical harmonic basis functions of the Laplace equation solution, analogously to the Riesz transform in two dimensions. This allows an interpretation of the Hilbert transform suitable for signal processing of signals naturally arising on the two-sphere. We show that the scale space naturally arising is a Poisson scale space in the unit ball. In addition, the obtained interpretation of the Hilbert transform is used for orientation analysis of plane waves. This representation is justified as a novel signal model on the sphere which can be used to construct intensity and rotation-invariant operators for local signal analysis in a scale-space concept.

Identifying Components in 3D Density Maps of Protein Nanomachines by Multi-scale Segmentation.

Segmentation of density maps obtained using cryo-electron microscopy (cryo-EM) is a challenging task, and is typically accomplished by time-intensive interactive methods. The goal of segmentation is to identify the regions inside the density map that correspond to individual components. We present a multi-scale segmentation method for accomplishing this task that requires very little user interaction. The method uses the concept of scale space, which is created by convolution of the input density map with a Gaussian filter. The latter process smoothes the density map. The standard deviation of the Gaussian filter is varied, with smaller values corresponding to finer scales and larger values to coarser scales. Each of the maps at different scales is segmented using the watershed method, which is very efficient, completely automatic, and does not require the specification of seed points. Some detail is lost in the smoothing process. A sharpening process reintroduces detail into the segmentation at the coarsest scale by using the segmentations at the finer scales. We apply the method to simulated density maps, where the exact segmentation (or ground truth) is known, and rigorously evaluate the accuracy of the resulting segmentations.

The use of automatic scale selection to improve the spatial and spectral resolution of a scintillator-coupled EMCCD

The technology behind the Electron-Multiplying Charge Coupled Device (EMCCD) was successfully exploited by e2v technologies in the late 1990s. Since then, many uses have been found for these low light level (L3) devices including surveillance and many scientific applications. The EMCCD increases or ‘multiplies’ the charge signal by the phenomenon of impact ionisation (or avalanche multiplication) allowing the detection of low signal events of only a few photons. When coupled with a scintillator, this low light capability can be used to image photon flashes from individual X-ray interaction events. The combination of depth of interaction effects in the scintillator, shot noise on the signal and the multiplication noise factor lead to large variations in the profile of the detected signal from a constant energy X-ray source. This variation leads to reduced spectral performance and can have adverse effects on the centering techniques used in photon-counting imagers. The concept of scale-space is similar in many ways to the Fourier or wavelet transforms. Automatic scale selection can be implemented through the scale-space transform as a method of fitting a known profile to the observed photon flash. The process is examined here in the context of the photon-counting EMCCD detector and the results obtained in both simulated and experimental data compared. Through the analysis of the fitting process and the results achieved, the implications on imaging performance and spectral resolution are discussed.

Wavelet transform-based locally orderless images for texture segmentation

Texture analysis is an important issue in many areas like object recognition, image retrieval study, medical imaging, robotics, and remote sensing. Despite the development of a family of techniques over the last couple of decades, there are only a few reliable methods. Multiresolution techniques seem to be attractive for many applications. In this study, we present an approach based on the discrete wavelet transform and scale space concept. We integrate the framework of locally orderless images (LOIs) with the transform coefficients to obtain a flexible method for texture segmentation. Compared to intensity (spatial domain), the wavelet coefficients appear to be more reliable with respect to noise immunity and the ease of feature formation. Hence, we represent each discrete coefficient value with a probability density function to form isophote images. Each isophote image is then convolved with a Gaussian to form LOIs, which specify a local histogram in each transform point. These LOIs, or statistical moments computed from LOIs, can be regarded as texture features. An experiment with the standard Brodatz's and VisTex texture databases demonstrates the superior performance of the wavelet-based LOIs compared to conventional LOI-based moments or wavelet and Gabor energy features. The elegance of the approach is in the relatively greater flexibility in producing segmentation results. A simple minimum distance classifier and confusion matrix analyses confirm the above attributes.

Scale-Space Properties of Nonstationary Iterative Regularization Methods

Most scale-space concepts have been expressed as parabolic or hyperbolic partial differential equations (PDEs). In this paper we extend our work on scale-space properties of elliptic PDEs arising from regularization methods: we study linear and nonlinear regularization methods that are applied iteratively and with different regularization parameters. For these so-called nonstationary iterative regularization techniques we clarify their relations to both isotropic diffusion filters with a scalar-valued diffusivity and anisotropic diffusion filters with a diffusion tensor. We establish scale-space properties for iterative regularization methods that are in complete accordance with those for diffusion filtering. In particular, we show that nonstationary iterative regularization satisfies a causality property in terms of a maximum–minimum principle, possesses a large class of Lyapunov functionals, and converges to a constant image as the regularization parameters tend to infinity. We also establish continuous dependence of the result with respect to the sequence of regularization parameters. Numerical experiments in two and three space dimensions are presented that illustrate the scale-space behavior of regularization methods.

Shape-adapted smoothing in estimation of 3-D shape cues from affine deformations of local 2-D brightness structure

This article describes a method for reducing the shape distortions due to scale-space smoothing that arise in the computation of 3-D shape cues using operators (derivatives) defined from scale-space representation. More precisely, we are concerned with a general class of methods for deriving 3-D shape cues from a 2-D image data based on the estimation of locally linearized deformations of brightness patterns. This class constitutes a common framework for describing several problems in computer vision (such as shape-from-texture, shape-from disparity-gradients, and motion estimation) and for expressing different algorithms in terms of similar types of visual front-end-operations. It is explained how surface orientation estimates will be biased due to the use of rotationally symmetric smoothing in the image domain. These effects can be reduced by extending the linear scale-space concept into an affine Gaussian scalespace representation and by performing affine shape adaptation of the smoothing kernels. This improves the accuracy of the surface orientation estimates, since the image descriptors, on which the methods are based, will be relative invariant under affine transformations, and the error thus confined to the higher-order terms in the locally linearized perspective transformation. A straightforward algorithm is presented for performing shape adaptation in practice. Experiments on real and synthetic images with known orientation demonstrate that in the presence of moderately high noise levels the accuracy is improved by typically one order of magnitude.

Higher order differential structure of images

This paper is meant as a tutorial on the basic concepts for vision in the ‘Koenderink’ school. The concept of scale-space is a necessity, if the extraction of structure from measured physical signals (i.e. images) is at stage. The Gaussian derivative kernels for physical signals are then the natural analogues of the mathematical differential operators. This paper discusses some interesting properties of the Gaussian derivative kernels, like their orthogonality and behaviour with noisy input data. Geometrical structure to extract is expressed in terms of differential invariants, in this paper limited to invariants under orthogonal transformations. Three representations are summarized: Cartesian, gauge and manifest invariant notation. Many explicit examples are given. A section is included about the computer implementation of the calculation of higher order invariant structure.

Adaptive determination of filter scales for edge detection

The authors suggest a regularization method for determining scales for edge detection adaptively for each site in the image plane. Specifically, they extend the optimal filter concept of T. Poggio et al. (1984) and the scale-space concept of A. Witkin (1983) to an adaptive scale parameter. To avoid an ill-posed feature synthesis problem, the scheme automatically finds optimal scales adaptively for each pixel before detecting final edge maps. The authors introduce an energy function defined as a functional over continuous scale space. Natural constraints for edge detection are incorporated into the energy function. To obtain a set of optimal scales that can minimize the energy function, a parallel relaxation algorithm is introduced. Experiments for synthetic and natural scenes show the advantages of the algorithm. In particular, it is shown that this system can detect both step and diffuse edges while drastically filtering out the random noise. >