Neural Network Retrieval Research Articles

Context-dependent retrieval of information by neural-network dynamics with continuous attractors

Memory retrieval in neural networks has traditionally been described by dynamic systems with discrete attractors. However, recent neurophysiological findings of graded persistent activity suggest that memory retrieval in the brain is more likely to be described by dynamic systems with continuous attractors. To explore what sort of information processing is achieved by continuous-attractor dynamics, keyword extraction from documents by a network of bistable neurons, which gives robust continuous attractors, is examined. Given an associative network of terms, a continuous attractor led by propagation of neuronal activation in this network appears to represent keywords that express underlying meaning of a document encoded in the initial state of the network-activation pattern. A dominant hypothesis in cognitive psychology is that long-term memory is archived in the network structure, which resembles associative networks of terms. Our results suggest that keyword extraction by the neural-network dynamics with continuous attractors might symbolically represent context-dependent retrieval of short-term memory from long-term memory in the brain.

Neural-network approach to ground-based passive microwave estimation of precipitation intensity and extinction

Summary A physically-based passive microwave technique is proposed to estimate precipitation intensity and extinction from ground. Multi-frequency radiometric measurements are inverted to retrieve surface rain rate, columnar precipitation contents and rainfall microwave extinction. A new inversion methodology, based on an artificial neural-network feed-forward algorithm, is evaluated and compared against a previously developed regression technique. Both retrieval techniques are trained by numerical simulations of a radiative transfer model applied to microphysically-consistent precipitating cloud structures. Cloud microphysics is characterized by using parameterized hydrometeor drop size distribution, spherical particle shape and dielectric composition. The radiative transfer equation is solved for plane-parallel seven-layer structures, including liquid, melted, and ice spherical hydrometeors. The proposed neural-network inversion technique is tested and compared with the regression algorithm on synthetic data in order to understand their potential and to select the best frequency set for rainfall rate, columnar contents and extinction estimation. Available ground-based radiometric measurements at 13.0, 23.8, and 31.6 GHz are used for experimentally testing and comparing the neural-network retrieval algorithm. Comparison with rain gauge data and rain extinction measurements, derived from three satellite beacon channels at 18.7, 39.6, and 49.5 GHz acquired at Pomezia (Rome, Italy), are performed and discussed for a selected case of light-to-moderate rainfall.

A neural-network technique for the retrieval of atmospheric temperature and moisture profiles from high spectral resolution sounding data

A novel statistical method for the retrieval of atmospheric temperature and moisture profiles has been developed and evaluated with simulated clear-air and observed partially cloudy sounding data from the Atmospheric InfraRed Sounder (AIRS) and the Advanced Microwave Sounding Unit (AMSU). The algorithm is implemented in two stages. First, a projected principal components (PPC) transform is used to reduce the dimensionality of and optimally extract geophysical profile information from the cloud-cleared infrared radiance data. Second, a multilayer feedforward neural network (NN) is used to estimate the desired geophysical parameters from the PPCs. For the first time, NN temperature and moisture retrievals are presented using actual microwave and hyperspectral infrared observations of cloudy atmospheres, over both ocean and land (with variable terrain elevation), and at all sensor scan angles. The performance of the NN retrieval method (henceforth referred to as the PPC/NN method) was evaluated using global Earth Observing System Aqua orbits colocated with European Center for Medium-range Weather Forecasting fields for seven days throughout 2002 and 2003. Over 350,000 partially cloudy footprints were used in the study, and retrieval performance was compared with the AIRS Science Team Level-2 retrieval algorithm (version 3). Performance compares favorably with that obtained with simulated clear-air observations from the NOAA88b radiosonde set of approximately 7500 profiles. The PPC/NN method requires significantly less computation than traditional variational retrieval methods, while achieving comparable performance.

Retrieving atmospheric temperature profiles from AMSU-A data with neural networks

Back propagation neural networks are used to retrieve atmospheric temperature profiles from NOAA-16 Advanced Microwave Sounding Unit-A (AMSU-A) measurements over East Asia. The collocated radiosonde observation and AMSU-A data over land in 2002–2003 are used to train the network, and the data over land in 2004 are used to test the network. A comparison with the multi-linear regression method shows that the neural network retrieval method can significantly improve the results in all weather conditions. When an offset of 0.5 K or a noise level of ±0.2 K is added to all channels simultaneously, the increase in the overall root mean square (RMS) error is less than 0.1 K. Furthermore, an experiment is conducted to investigate the effects of the window channels on the retrieval. The results indicate that the brightness temperatures of window channels can provide significantly useful information on the temperature retrieval near the surface. Additionally, the RMS errors of the profiles retrieved with the trained neural network are compared with the errors from the International Advanced TOVS (ATOVS) Processing Package (IAPP). It is shown that the network-based algorithm can provide much better results in the experiment region and comparable results in other regions. It is also noted that the network can yield remarkably better results than IAPP at the low levels and at about the 250-hPa level in summer skies over ocean. Finally, the network-based retrieval algorithm developed herein is applied in retrieving the temperature anomalies of Typhoon Rananim from AMSU-A data.

Potential of Advanced Microwave Sounding Unit‐A (AMSU‐A) and AMSU‐B measurements for atmospheric temperature and humidity profiling over land

A neural network retrieval method has been applied to investigate AMSU‐A/AMSU‐B atmospheric temperature and humidity profiling capabilities over land. The retrieval method benefits from a reliable estimate of the land emissivity and skin temperature as well as first guess information regarding the temperature‐humidity profiles. It has been applied on a large geographic area (60°W–60°E, 60°S–60°N) and atmospheric situations (winter and summer). The retrieved RMS errors are within 2 K and 9% in temperature and relative humidity, respectively. Regardless of scanning conditions, vegetation types, and atmospheric situations, the algorithm retrieval results are satisfactory for both temperature and relative humidity. The retrieval approach has been evaluated by comparison with available in situ measurements.

Synaptic value bounds for optimizing retrieval in recurrent neural networks

We present an analysis of the range of values of synaptic connections to enhance the storage properties of neural networks. Random patterns are shown to a random connected system, imposing specific activities over neuron pairs that might be connected evoking long-term synapse modifications. Two approaches are given to quantify quality retrieval. The first focuses on the noise context and addresses a simple instrument to directly quantify the retrieval, while the second one focuses on the learning rule considering the information content of the network. We find that increasing the variability among the synapse values within a given range is followed by an increase of the retreival quality.

Synaptic value bounds for optimizing retrieval in recurrent neural networks

We present an analysis of the range of values of synaptic connections to enhance the storage properties of neural networks. Random patterns are shown to a random connected system, imposing specific activities over neuron pairs that might be connected evoking long-term synapse modifications. Two approaches are given to quantify quality retrieval. The first focuses on the noise context and addresses a simple instrument to directly quantify the retrieval, while the second one focuses on the learning rule considering the information content of the network. We find that increasing the variability among the synapse values within a given range is followed by an increase of the retreival quality.

First inversions of observed submillimeter limb sounding radiances by neural networks

First inversions of observed limb sounding data by a very fast neural network technique are presented. The technique is based on training a set of multilayer perceptrons. The training set was built by random generation of atmospheric states followed by forward model calculations to derive the corresponding radiances. A novel spectral reduction technique was used to reduce the dimensionality of the spectral space, allowing the setting up of multilayer perceptrons with a practical topology. For training, an algorithm that relies on Bayesian techniques was applied in order to favor generalization. The technique was tested by inverting measured radiances from the Odin‐SMR submillimeter radiometer and judged by comparing with similar inversions from an optimal estimation code. The neural network technique resulted in much faster inversions than the optimal estimation code, but biases between the optimal estimation and neural network retrievals were found. The problem was tracked down to the presence of spectral artifacts in the measured radiances. The optimal estimation inversions were also affected but seemed less sensitive. Nevertheless, the results were promising and opened the possibility of implementing a very fast processing algorithm for the Odin‐SMR data.

Ozone profile retrieval from Global Ozone Monitoring Experiment (GOME) data using a neural network approach (Neural Network Ozone Retrieval System (NNORSY))

The inverse radiative transfer equation to retrieve atmospheric ozone distribution from the UV‐visible satellite spectrometer Global Ozone Monitoring Experiment (GOME) has been modeled by means of a feed forward neural network. This Neural Network Ozone Retrieval System (NNORSY) was trained exclusively on a data set of GOME radiances collocated with ozone measurements from ozonesondes, Halogen Occultation Experiment, Stratospheric Aerosol and Gas Experiment II, and Polar Ozone and Aerosol Measurement III. Network input consists of a combination of spectral, geolocation, and climatological information (time and latitude). In the stratosphere the method globally reduces standard deviation with respect to an ozone climatology by around 40%. Tropospheric ozone can also be retrieved in many cases with corresponding reduction of 10–30%. All GOME data from January 1996 to July 2001 were processed. In a number of case studies involving comparisons with ozonesondes from Hohenpeissenberg, Syowa, and results from the classical Full Retrieval Method, we found good agreement with our results. The neural network was found capable of implicitly correcting for instrument degradation, pixel cloudiness, and scan angle effects. Integrated profiles generally agree to within ±5% with the monthly Total Ozone Mapping Spectrometer version 7 total ozone field. However, some problems remain at high solar zenith angles and very low ozone values, where local deviations of 10–20% have been observed in some cases. In order to better characterize individual ozone profiles, two local error estimation methods are presented. Vertical resolution of the profiles was assessed empirically and seems to be of the order of 4–6 km. Since neural network retrieval is a mathematically simple, one‐step procedure, NNORSY is about 103–105 times faster than classical retrieval techniques based upon optimal estimation.

Hebbian imprinting and retrieval in oscillatory neural networks.

We introduce a model of generalized Hebbian learning and retrieval in oscillatory neural networks modeling cortical areas such as hippocampus and olfactory cortex. Recent experiments have shown that synaptic plasticity depends on spike timing, especially on synapses from excitatory pyramidal cells, in hippocampus, and in sensory and cerebellar cortex. Here we study how such plasticity can be used to form memories and input representations when the neural dynamics are oscillatory, as is common in the brain (particularly in the hippocampus and olfactory cortex). Learning is assumed to occur in a phase of neural plasticity, in which the network is clamped to external teaching signals. By suitable manipulation of the nonlinearity of the neurons or the oscillation frequencies during learning, the model can be made, in a retrieval phase, either to categorize new inputs or to map them, in a continuous fashion, onto the space spanned by the imprinted patterns. We identify the first of these possibilities with the function of olfactory cortex and the second with the observed response characteristics of place cells in hippocampus. We investigate both kinds of networks analytically and by computer simulations, and we link the models with experimental findings, exploring, in particular, how the spike timing dependence of the synaptic plasticity constrains the computational function of the network and vice versa.

Neural network retrieval of wind velocity profiles from satellite data

AbstractA method has been developed for retrieval of wind velocity profiles from satellite sounder radiances using a neural network technique. Each wind velocity represents a mean value for a layer approximately centred at that standard height or pressure level. The neural network input vector includes sounder radiances from the Geostationary Operational Environmental Satellite (GOES), plus ancillary information such as latitude and longitude. A set of ‘co‐incident’ rawinsonde soundings provide the ‘truth’ data set. In the initial results root mean square errors for the standard levels from 850 to 300 hPa have about the same magnitude as those for current methods such as tracking of moisture features in GOES imagery. As with more traditional methods, the largest root mean square error occurs near the tropopause. Although the results to date appear highly promising, considerable further work is needed before we have an operationally capable technique. Copyright © 2000 Royal Meteorological Society

Retrieval of CO columns from IMG/ADEOS spectra

The high-resolution atmospheric spectra recorded by the Interferometric Monitor for Greenhouse gases instrument on the ADvanced Earth Observing System (IMG/ADEOS) in June, 1997, were analyzed by two different and complementary inversion algorithms to retrieve CO total columns. Spectral fits of high quality were obtained using a global fit algorithm on selected cloud-free measurements. Global distribution maps of CO obtained using a fast neural network retrieval algorithm are compared to the results provided by current chemistry-transport models. These studies highlight the need for complementary information (temperature, emissivity, instrumental function, and cloud content) to retrieve accurate CO vertical columns.

Application of multilayer feedforward neural networks to precipitation cell-top altitude estimation

The use of passive 118-GHz O/sub 2/ observations of rain cells for precipitation cell-top altitude estimation is demonstrated by using a multilayer feedforward neural network retrieval system. Rain cell observations at 118 GHz were compared with estimates of the cell-top altitude obtained by optical stereoscopy. The observations were made with 2-4-km horizontal spatial resolution by using the millimeter-wave temperature sounder (MTS) scanning spectrometer aboard the NASA ER-2 research aircraft during the Genesis of Atlantic Lows Experiment (GALE) and the Cooperative Huntsville Meteorological Experiment (COHMEX) in 1986. The neural network estimator applied to MTS spectral differences between clouds, and nearby clear air yielded an rms discrepancy of 1.76 km for a combined cumulus, mature, and dissipating cell set and 1.44 km for the cumulus-only set. An improvement in rms discrepancy to 1.36 km was achieved by including additional MTS information on the absolute atmospheric temperature profile. An incremental method for training neural networks was developed that yielded robust results, despite the use of as few as 56 training spectra. Comparison of these results with a nonlinear statistical estimator shows that superior results can be obtained with a neural network retrieval system. Imagery of estimated cell-top altitudes was created from 118-GHz spectral imagery gathered from CAMEX, September through October 1993, and from cyclone Oliver, February 7, 1993.

Learning and retrieval in attractor neural networks with noise

A recent study on noiseless learning and retrieval in attractor neural networks above saturation, by Griniasty and Gutfreund, is extended to take account for imperfect learning by means of a temperature β −1 = T. Violations of the constraint imposed on the local stabilities are taken into account by various cost functions. The distribution of local stabilities and the fraction of errors during the learning stage are analysed. The retrieval dynamics for the sparsely connected network is studied showing high retrieval overlap in reduced retrieval regions for finite T.

Controlling chaos by pinning neurons in a neural network.

Chaos control in an artificial neural network is achieved by ``pinning'' the state of a few neurons and adjusting the threshold value. The activity of the network, when the control is on, can be constant or periodic. The mechanism used is compared with other well-known algorithms of chaos control in low-dimensional maps. Some comments are made on possible similarities with memory retrieval in natural neural networks.

Attractor properties of dynamical systems: neural network models

We propose tools to probe the nature of attractors in dynamical systems. These include the activity distribution, the evolutions of the state damage, activity damage and temporal correlation damage. When they are used to study the retrieval attractors in dilute asymmetric neural networks, a transition from a partially frozen phase to an unfrozen phase is found for networks trained with sufficiently noisy data near storage saturation, and this confirms that the retrieval attractors are more chaotic in this case. We are also able to demonstrate that the retrieval attractors in dilute asymmetric neural networks are not clouds of attractors, but consist of a single chaotic attractor for each stored pattern. Furthermore, they facilitate the device of effective freezing procedures, which significantly improve the quality of retrieval in neural networks.

Retrieval and chaos in layeredQ-Ising neural networks

Using a probabilistic approach, we study the parallel dynamics of theQ-Ising layered networks for arbitraryQ. By introducing auxiliary thermal fields, we express the stochastic dynamics within the gain function formulation of the deterministic dynamics. Evolution equations are derived for arbitraryQ at both zero and finite temperatures. An explicit analysis of the fixed-point equations is carried out for bothQ=3 andQ→∞. The retrieval properties are discussed in terms of the gain parameter, the storage capacity, and the temperature. Using the time evolution of the distance between two network configurations, we investigate the possibility of microscopic chaos. Chaotic behavior is always present for arbitrary finiteQ. However, in the limitQ→∞ the existence of chaos depends on the parameters of the system.

Probing the attractors in neural networks

I propose tools to probe the nature of the retrieval attractors in neural networks. These include the activity distribution, the evolutions of the state damage, activity damage and temporal correlation damage. They enable us to demonstrate that the retrieval attractors in dillute asymmetric neutral networks are not clouds of attractors, but consists of a single chaotic attractor for each stored pattern. Furthermore, they facilitate the devise of effective freezing procedures, which significantly improve the quality of retrieval in dilute asymmetric neural networks.

Learning and retrieval in attractor neural networks above saturation

The authors investigate neural networks in the range of parameters when the ground-state energy is positive; namely, when a synaptic matrix which satisfies all the desired constraints cannot be found by the learning algorithm. In particular, they calculate the typical distribution functions of local stabilities obtained for a number of algorithms in this region. These functions are used to investigate the retrieval properties as reflected by the size of the basins of attraction. This is done analytically in sparsely connected networks, and numerically in fully connected networks. The main conclusion is that the retrieval behaviour of attractor neural networks can be improved by learning above saturation.

Chaotic Image Retrieval in Markovian Asymmetric Neural Networks with Sign-Constrained Synaptic Couplings

We propose a simple asymmetric neural network which exhibits chaotic motions in retrieval dynamics with a finite number of memory patterns. The characteristic feature of the model is that the synaptic couplings are designed in such a way that each neuron is given an exclusively excitatory or inhibitory function, i.e., a physiological constraint of the Dale hypothesis is taken into account. The updating rule of the neurons is assumed to be simple Markovian stochastic dynamics of the Little type (without time delay) in which the threshold for neuron firing is incorporated. Our analysis is based on the exact time evolution equations derived in the thermodynamic limit for the macroscopic pattern overlaps. It is shown that chaotic image retrieval can take place only when a finite amount of stochastic noise exists.