Linear Scale Space Research Articles

DV Emotion Net: An Integrated Multimodal Approach for Emotion Recognition

This study introduces a novel approach to emotion recognition by amalgamating information from heterogeneous modalities, specifically audio and video. We employed techniques such as energy, zero crossing rate, and Mel-Frequency Cepstral Coefficients (MFCC) for audio feature extraction, which showed promising results. For video feature extraction, spatialtemporal Gaussian kernels were used to organize video frames within a linear scale space, followed by the application of a Gaussian-weighted function to the second momentum matrix for further feature extraction. The Multimodal Feature Aggregation (MFA) fusion method was employed to unify audio and video features, resulting in a comprehensive dataset. Evaluation through the Fusion of Emotion Recognition Convolutional Neural Network (FERCNN) model, supported by the "TPU VM v3-8" accelerator TPU is a Tensor Processing Unit, showcased notable performance improvements. Using the RAVDESS and CREMAD datasets, accuracies of 94.66%, 95.82%, and 94.36% in the RAVDESS dataset and 79.45%, 96.62%, and 70.14% in the CREMAD dataset for audio, video, and multimodal modalities, respectively, were achieved. These outcomes surpass the capabilities of existing multimodal systems, underscoring the efficacy of our proposed approach. Emotion recognition, particularly through multimodal means, plays a critical role in various domains, including human-computer interfaces, healthcare, legal proceedings, and entertainment. Fusing Audio and Video Modalities to Elevate Human-Computer Interaction and Intelligent System Performance is essential for enhancing communication within these domains. The proposed model is termed "DualVision EmotionNet: DV EmotionNet".

Optical and SAR Image Registration Based on Multi-Scale Orientated Map of Phase Congruency

Optical and Synthetic Aperture Radar (SAR) images are highly complementary, and their registrations are a fundamental task for other remote sensing applications. Traditional feature-matching algorithms fail to solve the significant nonlinear radiation difference (NRD) caused by different sensors. To address this problem, a robust registration algorithm with the multi-scale orientated map of phase congruency (MSPCO) is proposed. First, a nonlinear diffusion scale space is established to obtain the scale invariance of feature points. Compared with the linear Gaussian scale space, the nonlinear diffusion scale space can better preserve the edge and texture information. Second, to ensure the quantity and repeatability of features, corner points and edge points are detected on the moment map of phase congruency, respectively, which is the foundation to the next feature matching. Third, the MSPCO descriptor is constructed via the orientation of phase congruency (PCO). PCO is highly robust to NRD, and the different scales of PCOs enhance the robustness of the descriptor. Finally, a feature-matching strategy based on an effective scale ratio is proposed, which reduces the number of comparisons among features and improves computational efficiency. The experimental results show that the proposed method is better than the existing feature-based methods in terms of the number of correct matches and registration accuracy. The registration accuracy is only inferior to that of the most advanced template matching method, and the accuracy difference is within 0.3 pixels, which fully demonstrates the robustness and accuracy of the proposed method in optical and SAR image registration.

Cloning detection scheme based on linear and curvature scale space with new false positive removal filters

Cloning detection scheme based on linear and curvature scale space with new false positive removal filters

Human Emotion Recognition by Integrating Facial and Speech Features: An Implementation of Multimodal Framework using CNN

This Emotion recognition plays a prominent role in today's intelligent system applications. Human computer interface, health care, law, and entertainment are a few of the applications where emotion recognition is used. Humans convey their emotions in the form of text, voice, and facial expressions, thus developing a multimodal emotional recognition system playing a crucial role in human-computer or intelligent system communication. The majority of established emotional recognition algorithms only identify emotions in unique data, such as text, audio, or image data. A multimodal system uses information from a variety of sources and fuses the information by using fusion techniques and categories to improve recognition accuracy. In this paper, a multimodal system to recognise emotions was presented that fuses the features from information obtained from heterogenous modalities like audio and video. For audio feature extraction energy, zero crossing rate and Mel-Frequency Cepstral Coefficients (MFCC) techniques are considered. Of these, MFCC produced promising results. For video feature extraction, first the videos are converted to frames and stored in a linear scale space by using a spatial temporal Gaussian Kernel. The features from the images are further extracted by applying a Gaussian weighted function to the second momentum matrix of linear scale space data. The Marginal Fisher Analysis (MFA) fusion method is used to fuse both the audio and video features, and the resulted features are given to the FERCNN model for evaluation. For experimentation, the RAVDESS and CREMAD datasets, which contain audio and video data, are used. Accuracy levels of 95.56, 96.28, and 95.07 on the RAVDESS dataset and accuracies of 80.50, 97.88, and 69.66 on the CREMAD dataset in audio, video, and multimodal modalities are achieved, whose performance is better than the existing multimodal systems.

Additive White Gaussian Noise Level Estimation for Natural Images Using Linear Scale-Space Features

Noise in images is often modelled with additive white Gaussian noise (AWGN). An accurate estimation of noise level without any prior knowledge of noisy input image leads to effective blind image denoising methods. The performance of certain image denoising methods under AWGN model is dependent on the accuracy of noise level estimation (NLE). Hence, there is a need to develop an effective NLE method in order to achieve better performance in image denoising. Even though the existing NLE methods perform well on natural images, these methods involve complex segmentation tasks such as homogeneous regions extraction and super-pixel decomposition. Hence, a simple, fast, and accurate NLE method for AWGN is proposed in this paper. In the presented NLE method, the statistical features of high-frequency details of noisy input image are obtained at multiple linear (Gaussian) scale-space which are used to construct a feature vector. It is perceived that the features obtained are almost linear and separable. Hence, supervised linear regression (LR) models that are trained globally and locally are suggested for NLE. The proposed method estimates the noise level in two stages. In stage-1, a globally trained LR model is used to estimate the noise level. It is observed that the accuracy of the noise level obtained through stage-1 can be further improved in stage-2 by adopting the proposed locally trained LR model. The proposed NLE method is evaluated with artificially generated noisy natural images using AWGN model. The high-quality natural images from Waterloo and BSD500 datasets are selected using image quality selection module and then used in training and testing phases. The average absolute deviation (AAD) is evaluated from each selected image in the datasets over a wide range of noise levels ([0 100]). The average AAD for selected images in Waterloo (BSD500) dataset is 0.21 (0.18), and execution time required to estimate the noise level is 0.04 s per image. From the obtained results, it is clear that the proposed method is simple, fast, and accurate as compared to several existing NLE methods. The effectiveness of the proposed NLE method is illustrated with fast and flexible denoising convolutional neural network using standard test images at randomly selected noise levels.

Sea Ice Drift Tracking From Sequential SAR Images Using Accelerated-KAZE Features

In this paper, we propose a feature-tracking algorithm for sea ice drift retrieval from a pair of sequential satellite synthetic aperture radar (SAR) images. The method is based on feature tracking comprising feature detection, description, and matching steps. The approach exploits the benefits of nonlinear multiscale image representations using accelerated-KAZE (A-KAZE) features, a method that detects and describes image features in an anisotropic scale space. We evaluated several state-of-the-art feature-based algorithms, including A-KAZE, Scale Invariant Feature Transform (SIFT), and a very fast feature extractor that computes binary descriptors known as Oriented FAST and Rotated BRIEF (ORB) on dual polarized Sentinel-1A C-SAR extra wide swath mode data over the Arctic. The A-KAZE approach outperforms both ORB and SIFT up to an order of magnitude in ice drift. The experimental results showed high relevance of the proposed algorithm for retrieval of ice drift at subkilometre resolution from a pair of SAR images with 100-m pixel size. From this paper, we found that feature tracking using nonlinear scale-spaces is preferable due to its high efficiency against noise with respect to image features compared with other existing feature tracking alternatives that make use of Gaussian or linear scale spaces.

A new generalised α scale spaces quadrature filters

The α scale spaces is a recent theory that open new possibilities of phase-based image processing. It is a parameterised class (α∈]0,1]) of linear scale space representations, which allows a continuous connection beyond the well-known Gaussian scale space (α=1). In this paper, we make use of this unified representation to derive new families of band pass quadrature filters, built from derivatives and difference of the α scale space generating kernels. This construction leads to generalised α kernel filters including the commonly known families derived from the Gaussian and the Poisson kernels. The properties of each family are first studied and then experiments on one- and two-dimensional signals are shown to exemplify how the suggested filters can be used for edge detection. This work is complemented by an experimental evaluation, which demonstrates that the new proposed filters are a good alternative to the commonly used Log-Gabor filter.

A novel marker enhancement filter (MEF) for fluoroscopic images

To enhance the measurements of radio-opaque cylindrical fiducial markers in low contrast x-ray and fluoroscopic images, a novel nonlinear marker enhancement filter (MEF) has been designed. It was primarily developed to assist in automatic initialization of a tracking procedure for intra-fraction organ motion analysis in fluoroscopic sequences. Conventional procedures were not able to provide sufficient improvement due to the complications of noise, small marker size, cylindrical shape and multiple orientations, intensity variations of the background, and the presence of overlaying anatomical measurements in this application. The proposed MEF design is based on the principles of linear scale space. It includes measures that assess the probability of each pixel to belong to a marker measurement, morphological operations, and a novel contrast enhancement function for standardization of the filter output. The MEF was tested on fluoroscopic images of two phantoms and three prostate patients, and was shown to perform better or comparable to the existing filters in terms of marker enhancement and background suppression, while performing significantly better in marker shape preservation.

On Covariant Derivatives and Their Applications to Image Regularization

We present a generalization of the Euclidean and Riemannian gradient operators to a vector bundle, a geometric structure generalizing the concept of a manifold. One of the key ideas is to replace the standard differentiation of a function by the covariant differentiation of a section. Dealing with covariant derivatives satisfying the property of compatibility with vector bundle metrics, we construct generalizations of existing mathematical models for image regularization that involve the Euclidean gradient operator, namely, the linear scale-space and the Rudin--Osher--Fatemi denoising models. For well-chosen covariant derivatives, we show that our denoising model outperforms state-of-the-art variational denoising methods of the same type both in terms of peak signal-to-noise ratio (PSNR) and Q-index [Z. Wang and A. Bovik, IEEE Signal Process. Lett., 9 (2002), pp. 81--84].

Accurate Junction Detection and Characterization in Natural Images

Accurate junction detection and characterization are of primary importance for several aspects of scene analysis, including depth recovery and motion analysis. In this work, we introduce a generic junction analysis scheme. The first asset of the proposed procedure is an automatic criterion for the detection of junctions, permitting to deal with textured parts in which no detection is expected. Second, the method yields a characterization of L-, Y- and X- junctions, including a precise computation of their type, localization and scale. Contrary to classical approaches, scale characterization does not rely on the linear scale-space. First, an a contrario approach is used to compute the meaningfulness of a junction. This approach relies on a statistical modeling of suitably normalized gray level gradients. Then, exclusion principles between junctions permit their precise characterization. We give implementation details for this procedure and evaluate its efficiency through various experiments.

Erratum to: Morphological and Linear Scale Spaces for Fiber Enhancement in DW-MRI

The online version of the original article can be found under doi: http://dx.doi.org/10.1007/s10851-012-0387-2

Grid services trusted authentication algorithm based on image trusted root in scale space

A new authentication algorithm for grid identity trusted computing unlimited by hardware is presented; the trusted root is made as an image data. The grid entity is trusted in the soft platform when its feature of image root is entirely matched with that from the other entities’ feature database in a scale space process. To recognize and detect the stable image root feature, the non-homogeneous linear expandable scale space is proposed. Focusing on relations between the scale parameter of the inhomogeneous Gaussian function terms and the space evolution of thermal diffusion homogeneous equations, three space evolution operators are constructed to exact and mark the feature from image root. Analysis and verification are carried on the new scale space, operators and the core of making decisions for grid entities certifications.

Morphological and Linear Scale Spaces for Fiber Enhancement in DW-MRI

We consider morphological and linear scale spaces on the space ℝ3⋊S 2 of 3D positions and orientations naturally embedded in the group SE(3) of 3D rigid body movements. The general motivation for these (convection-)diffusions and erosions is to obtain crossing-preserving fiber enhancement on probability densities defined on the space of positions and orientations. The strength of these enhancements is that they are expressed in a moving frame of reference attached to fiber fragments, allowing us to diffuse along the fibers and to erode orthogonal to them. The linear scale spaces are described by forward Kolmogorov equations of Brownian motions on ℝ3⋊S 2 and can be solved by convolution with the corresponding Green’s functions. The morphological scale spaces are Bellman equations of cost processes on ℝ3⋊S 2 and we show that their viscosity solutions are given by a morphological convolution with the corresponding morphological Green’s function. For theoretical underpinning of our scale spaces on ℝ3⋊S 2 we introduce Lagrangians and Hamiltonians on ℝ3⋊S 2 indexed by a parameter η∈[1,∞). The Hamiltonian induces a Hamilton-Jacobi-Bellman system that coincides with our morphological scale spaces on ℝ3⋊S 2. By means of the logarithm on SE(3) we provide tangible estimates for both the linear- and the morphological Green’s functions. We also discuss numerical finite difference upwind schemes for morphological scale spaces (erosions) of Diffusion-Weighted Magnetic Resonance Imaging (DW-MRI), which allow extensions to data-adaptive erosions of DW-MRI. We apply our theory to the enhancement of (crossing) fibres in DW-MRI for imaging water diffusion processes in brain white matter.

Multiscale roughness measure for color image segmentation

Color image segmentation is always an important technique in image processing system. Highly precise segmentation with low computation complexity can be achieved through roughness measurement which approximate the color histogram based on rough set theory. However, due to the imprecise description of neighborhood similarity, the existing roughness measure tends to over-focus on the trivial homogeneous regions but is not accurate enough to measure the color homogeneity. This paper aims to construct a multiscale roughness measure through simulating the human vision. We apply the theories of linear scale-space and rough sets to generate the hierarchical roughness of color distribution under multiple scales. This multiscale roughness can tolerate the disturbance of trivial regions and also can provide the multilevel homogeneity representation in vision, which therefore produces precise and intuitive segmentation results. Furthermore, we propose roughness entropy for scale selection. The optimal scale for segmentation is decided by the entropy variation. The proposed method shows the encouraging performance in the experiments based on Berkeley segmentation database.

A Sampling Method Based on Gauss Kernel Learning and the Expanding Research

In this paper, the expansion of feature points of the linear scale space is transformed into the classification of multi-scale data set within the same scale, which belongs to the classification of scale invariant non-equilibrium .The paper presents a sample approach based on kernel learning to solve classification on imbalance dataset by Support Vector Machine (SVM). The method first preprocesses the data by oversampling the minority class in kernel space, and then the pre-images of the synthetic samples are found based on a distance relation between kernel space and input space. Finally, these pre-images are appended to the original dataset to train a SVM. As a result, the inconsistency which is brought about by processing samples in different spaces is overcome. On the other hand, the sampling strategies of the method not only can decrease imbalanced rate of training dataset, but also can enlarge convex hull of the minority class. Consequently, the problem of the boundary skew can be amended more effectively. Experiments on real dataset indicate that the generalization performance of the result classifier is improved and the algorithm can work well on expanding the feature points stably for a certain scale .

Generalized Gaussian Scale-Space Axiomatics Comprising Linear Scale-Space, Affine Scale-Space and Spatio-Temporal Scale-Space

This paper describes a generalized axiomatic scale-space theory that makes it possible to derive the notions of linear scale-space, affine Gaussian scale-space and linear spatio-temporal scale-space using a similar set of assumptions (scale-space axioms). The notion of non-enhancement of local extrema is generalized from previous application over discrete and rotationally symmetric kernels to continuous and more general non-isotropic kernels over both spatial and spatio-temporal image domains. It is shown how a complete classification can be given of the linear (Gaussian) scale-space concepts that satisfy these conditions on isotropic spatial, non-isotropic spatial and spatio-temporal domains, which results in a general taxonomy of Gaussian scale-spaces for continuous image data. The resulting theory allows filter shapes to be tuned from specific context information and provides a theoretical foundation for the recently exploited mechanisms of shape adaptation and velocity adaptation, with highly useful applications in computer vision. It is also shown how time-causal spatio-temporal scale-spaces can be derived from similar assumptions. The mathematical structure of these scale-spaces is analyzed in detail concerning transformation properties over space and time, the temporal cascade structure they satisfy over time as well as properties of the resulting multi-scale spatio-temporal derivative operators. It is also shown how temporal derivatives with respect to transformed time can be defined, leading to the formulation of a novel analogue of scale normalized derivatives for time-causal scale-spaces. The kernels generated from these two types of theories have interesting relations to biological vision. We show how filter kernels generated from the Gaussian spatio-temporal scale-space as well as the time-causal spatio-temporal scale-space relate to spatio-temporal receptive field profiles registered from mammalian vision. Specifically, we show that there are close analogies to space-time separable cells in the LGN as well as to both space-time separable and non-separable cells in the striate cortex. We do also present a set of plausible models for complex cells using extended quasi-quadrature measures expressed in terms of scale normalized spatio-temporal derivatives. The theories presented as well as their relations to biological vision show that it is possible to describe a general set of Gaussian and/or time-causal scale-spaces using a unified framework, which generalizes and complements previously presented scale-space formulations in this area.

Linear Gaussian blur evolution for detection of blurry images

Even though state-of-the-art digital cameras are equipped with auto-focusing and motion compensation functions, several other factors including limited contrast, inappropriate exposure time and improper device handling can still lead to unsatisfactory image quality such as blurriness. Indeed, blurry images make up a significant percentage of anyone's picture collections. Consequently, an efficient tool to detect blurry images and label or separate them for automatic deletion in order to preserve storage capacity and the quality of image collections is needed. A new technique for automatic detection and removal of blurry pictures is presented. Initially, a set of interest points and local image areas is extracted. These areas are then evolved in time according to the conventional linear scale space. The gradient of the evolution curve through scale is then used to produce a ‘blur graph’ representing the probability of a picture being blurred or not. Complexity is kept low by applying a Monte-Carlo like technique for the selection of representative image areas and interest points and by implicitly estimating the gradient of the scale-space curve evolution. An exhaustive evaluation of the proposed technique is conducted to validate its performance in terms of detection accuracy and efficiency.

Local Scale Measure from the Topographic Map and Application to Remote Sensing Images

In this paper, we propose a spatially accurate definition of scale for images. The proposed definition of local scale relies on the hierarchical structure of the topographic map, the set of all level lines of an image. Namely, a connected component of a level line is associated with each point of the image and the scale at this point is then computed from the corresponding shape. Level lines are selected under the assumption that the blur is uniform over the image, a realistic assumption for satellite images. First, we discuss the links between the proposed approach and recent scale definitions making use of the total variation flow or the Rudin–Osher–Fatemi model. This comparison sheds some light on the relationships between morphological and variational approaches to scale definitions. Then we perform several experiments on synthetic and satellite images and numerically compare our local scale definition with results using the linear scale space, variational, or morphological approaches. These experimen...

Functional Characterization of Three G Protein-coupled Receptors for Pigment Dispersing Factors in Caenorhabditis elegans

Here, we report the identification, cloning, and functional characterization of three Caenorhabditis elegans G protein-coupled pigment dispersing factor (PDF) receptors, which we designated as Ce_PDFR-1a, -b, and -c. They represent three splice isoforms of the same gene (C13B9.4), which share a high degree of similarity with the Drosophila PDF receptor and are distantly related to the mammalian vasoactive intestinal peptide receptors (VPAC2) and calcitonin receptors. In a reverse pharmacological screen, three bioactive C. elegans neuropeptides, which were recently identified as the Drosophila PDF orthologues, were able to activate these receptors in a dose-dependent manner with nanomolar potency (isoforms a and b). Integrated green fluorescent protein reporter constructs reveal the expression of these PDF receptors in all body wall muscle cells and many head and tail neurons involved in the integration of environmental stimuli and the control of locomotion. Using a custom data analysis system, we demonstrate the involvement of this newly discovered neuropeptide signaling system in the regulation of locomotor behavior. Overexpression of PDF-2 phenocopies the locomotor defects of a PDF-1 null mutant, suggesting that they elicit opposite effects on locomotion through the identified PDF receptors. Our findings strengthen the hypothesis that the PDF signaling system, which imposes the circadian clock rhythm on behavior in Drosophila, has been functionally conserved throughout the protostomian evolutionary lineage.

Second Order Structure of Scale-Space Measurements

The second-order structure of random images f :? d ?? N is studied under the assumption of stationarity of increments, isotropy and scale invariance. Scale invariance is defined via linear scale space theory. The results are formulated in terms of the covariance structure of the jet consisting of the scale space derivatives at a single point. Operators describing the effect in jet space of blurring and scaling are investigated. The theory developed is applicable in the analysis of naturally occurring images of which examples are provided.