Input Encoding Research Articles

Dynamic Upsampling of Smoke through Dictionary-based Learning

Simulating turbulent smoke flows with fine details is computationally intensive. For iterative editing or simply faster generation, efficiently upsampling a low-resolution numerical simulation is an attractive alternative. We propose a novel learning approach to the dynamic upsampling of smoke flows based on a training set of flows at coarse and fine resolutions. Our multiscale neural network turns an input coarse animation into a sparse linear combination of small velocity patches present in a precomputed over-complete dictionary. These sparse coefficients are then used to generate a high-resolution smoke animation sequence by blending the fine counterparts of the coarse patches. Our network is initially trained from a sequence of example simulations to both construct the dictionary of corresponding coarse and fine patches and allow for the fast evaluation of a sparse patch encoding of any coarse input. The resulting network provides an accurate upsampling when the coarse input simulation is well approximated by patches present in the training set (e.g., for re-simulation), or simply visually plausible upsampling when input and training sets differ significantly. We show a variety of examples to ascertain the strengths and limitations of our approach and offer comparisons to existing approaches to demonstrate its quality and effectiveness.

Cooperative Resolvability and Secrecy in the Cribbing Multiple-Access Channel

We study channel resolvability for the discrete memoryless multiple-access channel with cribbing, i.e., the characterization of the amount of randomness required at the inputs to approximately produce a chosen i.i.d. output distribution according to Kullback-Leibler divergence. We analyze resolvability rates when one encoder cribs (i) the input of the other encoder; or the output of the other encoder, (ii) non-causally, (iii) causally, or (iv) strictly-causally. For scenarios (i)-(iii), we exactly characterize the channel resolvability region. For (iv), we provide inner and outer bounds for the channel resolvability region; the crux of our achievability result is to handle the strict causality constraint with a block-Markov coding scheme in which dependencies across blocks are suitably hidden. Finally, we leverage the channel resolvability results to derive achievable secrecy rate regions for each of the cribbing scenarios under strong secrecy constraints.

Input Redundancy for Parameterized Quantum Circuits

In this paper we deal with a special type of parameterized quantum circuits, the so-called quantum neural networks or QNN's, which are trained to estimate a given function, specifically the type of circuits proposed by Mitarai et al.\ (Phys.\ Rev.\ A, 2018; see below). The input is encoded into amplitudes of states of qubits. The no-cloning principle of quantum mechanics suggests that there is an advantage in redundantly encoding the input value several times. We follow this suggestion, and prove lower bounds on the number of redundant copies for two types of input encoding. We draw conclusions for the architecture design of QNNs.

HybridGAN: hybrid generative adversarial networks for MR image synthesis

In this paper, we propose HybridGAN – a new medical MR image synthesis methods via generative adversarial learning. Specifically, our synthesizer generates MRI data in a sequential manner: first in order to improve the robustness of image synthesis, an input full-size real MR image is divided into an array of sub-images. Then, to avoid overfitting limited MRI encodings, these sub-images and an unlimited amount of random latent noise vectors become the input of automatic encoder for learning the marginal image distributions of real images. Finally, pseudo patches with constrained noise vectors are put into RU-NET which is a component of our HybridGAN to generate a large number of synthetic MR images. In RU-NET, A SpliceLayer is then employed to fuse sub-images together in an interlaced manner into a full-size image. The experimental results show that HybridGAN can effectively synthesize a large variety of MR images and display a good visual quality. Compared to the state-of-the-art synthesis methods, our method achieves a significant improvement in terms of both visual and quantitative evaluation metrics.

Neurolight: A Deep Learning Neural Interface for Cortical Visual Prostheses.

Visual neuroprosthesis, that provide electrical stimulation along several sites of the human visual system, constitute a potential tool for vision restoration for the blind. Scientific and technological progress in the fields of neural engineering and artificial vision comes with new theories and tools that, along with the dawn of modern artificial intelligence, constitute a promising framework for the further development of neurotechnology. In the framework of the development of a Cortical Visual Neuroprosthesis for the blind (CORTIVIS), we are now facing the challenge of developing not only computationally powerful tools and flexible approaches that will allow us to provide some degree of functional vision to individuals who are profoundly blind. In this work, we propose a general neuroprosthesis framework composed of several task-oriented and visual encoding modules. We address the development and implementation of computational models of the firing rates of retinal ganglion cells and design a tool - Neurolight - that allows these models to be interfaced with intracortical microelectrodes in order to create electrical stimulation patterns that can evoke useful perceptions. In addition, the developed framework allows the deployment of a diverse array of state-of-the-art deep-learning techniques for task-oriented and general image pre-processing, such as semantic segmentation and object detection in our system's pipeline. To the best of our knowledge, this constitutes the first deep-learning-based system designed to directly interface with the visual brain through an intracortical microelectrode array. We implement the complete pipeline, from obtaining a video stream to developing and deploying task-oriented deep-learning models and predictive models of retinal ganglion cells' encoding of visual inputs under the control of a neurostimulation device able to send electrical train pulses to a microelectrode array implanted at the visual cortex.

Integral Control Barrier Functions for Dynamically Defined Control Laws

This letter introduces integral control barrier functions (I-CBFs) as a means to enable the safety-critical integral control of nonlinear systems. Importantly, I-CBFs allow for the holistic encoding of both state constraints and input bounds in a single framework. We demonstrate this by applying them to a dynamically defined tracking controller, thereby enforcing safety in state and input through a minimally invasive I-CBF controller framed as a quadratic program.

Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation

End-to-end speech translation, a hot topic in recent years, aims to translate a segment of audio into a specific language with an end-to-end model. Conventional approaches employ multi-task learning and pre-training methods for this task, but they suffer from the huge gap between pre-training and fine-tuning. To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN) which bridges the gap by reusing all subnets in fine-tuning, keeping the roles of subnets consistent, and pre-training the attention module. Furthermore, we propose two simple but effective methods to guarantee the speech encoder outputs and the MT encoder inputs are consistent in terms of semantic representation and sequence length. Experimental results show that our model leads to significant improvements in En-De and En-Fr translation irrespective of the backbones.

Solving a classification task by spiking neural network with STDP based on rate and temporal input encoding

This paper develops local learning algorithms to solve a classification task with the help of biologically inspired mathematical models of spiking neural networks involving the mechanism of spike‐timing‐dependent plasticity (STDP). The advantages of the models are their simplicity and, hence, the potential ability to be hardware‐implemented in low‐energy‐consuming biomorphic computing devices. The methods developed are based on two key effects observed in neurons with STDP: mean firing rate stabilization and memorizing repeating spike patterns. As the result, two algorithms to solve a classification task with a spiking neural network are proposed: the first based on rate encoding of the input data and the second based on temporal encoding. The accuracy of the algorithms is tested on the benchmark classification tasks of Fisher's Iris and Wisconsin breast cancer, with several combinations of input data normalization and preprocessing. The respective accuracies are 99% and 94% by F1‐score.

SPSNN: nth Order Sequence-Predicting Spiking Neural Network

We introduce a means of harnessing spiking neural networks (SNNs) with rich dynamics as a dynamic hypothesis to learn complex sequences. The proposed SNN is referred to as nth order sequence-predicting SNN (n-SPSNN), which is capable of single-step prediction and sequence-to-sequence prediction, i.e., associative recall. As a key to these capabilities, we propose a new learning algorithm, named the learning by backpropagating action potential (LbAP) algorithm, which features (i) postsynaptic event-driven learning, (ii) access to topological and temporal local data only, (iii) competition-induced weight normalization effect, and (iv) fast learning. Most importantly, the LbAP algorithm offers a unified learning framework over the entire SPSNN based on local data only. The learning capacity of the SPSNN is mainly dictated by the number of hidden neurons h; its prediction accuracy reaches its maximum value (~1) when the hidden neuron number h is larger than twice training sequence length l, i.e., h ≥ 2l. Another advantage is its high tolerance to errors in input encoding compared to the state-of-the-art sequence learning networks, namely long short-term memory (LSTM) and gated recurrent unit (GRU). Additionally, its efficiency in learning is approximately 100 times that of LSTM and GRU when measured in terms of the number of synaptic operations until successful training, which corresponds to multiply-accumulate operations for LSTM and GRU. This high efficiency arises from the higher learning rate of the SPSNN, which is attributed to the LbAP algorithm. The code is available on-line (https://github.com/galactico7/SPSNN).

Rate Matching for Polar Codes Based on Binary Domination

In this paper, we investigate the fundamentals of puncturing and shortening for polar codes, based on binary domination which plays a key role in polar code construction. We first prove that the orders of encoder input bits to be made incapable (by puncturing) or to be shortened are governed by binary domination. In particular, we show that binary domination completely determines incapable or shortened bit patterns for polar codes, and that all the possible incapable or shortened bit patterns can be identified. We then present the patterns of the corresponding encoder output bits to be punctured or fixed, when the incapable or shortened bits are given. We also demonstrate that the order and the pattern of puncturing and shortening for polar codes can be aligned. In the previous work on the rate matching for polar codes, puncturing of encoder output bits begins from a low-indexed bit, while shortening starts from a high-indexed bit. Unlike such a conventional approach, we show that encoder output bits can be punctured from high-indexed bits, while keeping the incapable bit pattern exactly the same. This makes it possible to design a unified circular-buffer rate matching (CB-RM) scheme that includes puncturing, shortening, and repetition.

Unsupervised Learning on Resistive Memory Array Based Spiking Neural Networks.

Spiking Neural Networks (SNNs) offer great potential to promote both the performance and efficiency of real-world computing systems, considering the biological plausibility of SNNs. The emerging analog Resistive Random Access Memory (RRAM) devices have drawn increasing interest as potential neuromorphic hardware for implementing practical SNNs. In this article, we propose a novel training approach (called greedy training) for SNNs by diluting spike events on the temporal dimension with necessary controls on input encoding phase switching, endowing SNNs with the ability to cooperate with the inevitable conductance variations of RRAM devices. The SNNs could utilize Spike-Timing-Dependent Plasticity (STDP) as the unsupervised learning rule, and this plasticity has been observed on our one-transistor-one-resistor (1T1R) RRAM devices under voltage pulses with designed waveforms. We have also conducted handwritten digit recognition task simulations on MNIST dataset. The results show that the unsupervised SNNs trained by the proposed method could mitigate the requirement for the number of gradual levels of RRAM devices, and also have immunity to both cycle-to-cycle and device-to-device RRAM conductance variations. Unsupervised SNNs trained by the proposed methods could cooperate with real RRAM devices with non-ideal behaviors better, promising high feasibility of RRAM array based neuromorphic systems for online training.

Generating Style-Specific Chinese Tang Poetry With a Simple Actor-Critic Model

Recent studies in sequence-to-sequence learning demonstrate that recurrent neural network (RNN) encoder-decoder structure can do well in Chinese classical poetry generation. With topic words or first line as the encoder input, an entire poem is then incrementally generated by the decoder from left to right with the highest probability. However, this locally incremental nature of decoding model can lead to the incongruity of style between the front and back of the poem generated. Inspired by the behavior of people tending to plan and associate the following parts in advance when they are writing poems, this paper employs a simple actor-critic method to generate style-specific Chinese poems. We design a style matching reward function and employ a value network with Monte Carlo search as the critic to estimate the future rewards of the desired style for poem generation. This approach makes the generation process more flexible and controllable. The experimental results demonstrate that our approach can generate three specific styles of high-quality poetry, and enhance the consistency of style of generated poems.

Exploring correlation information for image compression of four-dimensional computed tomography.

Nowadays four-dimensional computed tomography (4DCT) is popularly used in evaluating respiration-related organ motion for patients under radiotherapy. As consisting of multiple subsets of CT images, a larger storage space is needed for 4DCT. In this study, the correlations information within these subsets was explored and the popular video encoders were used for 4DCT image compression. The images of 4DCT subsets were arranged in an order and put into a sequence for the input of video encoder. The effects of two ordering methods on the compression performance of video encoder were compared. One ordering method, phase-prioritized (PP) sequence, arranged 4DCT images according to their respiration phases. Another ordering method, location-prioritized (LP) sequence, arranged 4DCT images by their slice locations. Three popular video encoders were selected including one lossless compression algorithm and two lossy compression algorithms. Based on a publicly available database consisting of 82 4DCT datasets of 20 lung cancer patients, the performance of two ordering methods and three video encoders was quantitatively assessed. The highest compression ratios were 310 and 260 for LP and PP ordering methods respectively and achieved by one video encoder employing inter-frame prediction coding algorithm. The highest compression ratios for the other two video encoders employing intra-frame coding algorithms were 16 and 7. The LP ordering method showed less inter-frame variation and higher inter-frame similarity over the PP ordering method. The LP ordering method would result in higher compression ratio than the PP ordering method for 4DCT image compression. The compression performance of video encoder employing inter-frame prediction coding algorithm is superior to those of video encoders employing intra-frame coding algorithms. The video encoder with inter-frame prediction coding algorithm and LP ordering method would be highly suitable for 4DCT image compression.

Two mechanisms underlying auditory dominance: Overshadowing and response competition

There are occasions when infants and children have difficulty in processing arbitrary auditory–visual pairings, with auditory input sometimes attenuating visual processing (i.e., auditory dominance). The current research examined possible mechanisms underlying these auditory dominance effects in infants and 4-year-olds. Do auditory dominance effects stem from auditory input attenuating encoding of visual input, from the difficulty of inhibiting auditory-based responses, or from a combination of these factors? In five reported experiments, 4-year-olds (Experiments 1A, 1B, 2A, and 2B) and 14- and 22-month-olds (Experiment 3) were presented with a variety of tasks that required simultaneous processing of auditory and visual input, and then we assessed memory for the visual items at test. Auditory dominance in young children resulted from response competition that children could not resolve. Infants’ results were not as robust, but they provided some evidence that nonlinguistic sounds and possibly spoken words may attenuate encoding of visual input. The current findings shed light on mechanisms underlying cross-modal processing and auditory dominance and have implications for many tasks that hinge on the processing of arbitrary auditory–visual pairings.

Serotonin regulates dynamics of cerebellar granule cell activity by modulating tonic inhibition.

Understanding how afferent information is integrated by cortical structures requires identifying the factors shaping excitation and inhibition within their input layers. The input layer of the cerebellar cortex integrates diverse sensorimotor information to enable learned associations that refine the dynamics of movement. Specifically, mossy fiber afferents relay sensorimotor input into the cerebellum to excite granule cells, whose activity is regulated by inhibitory Golgi cells. To test how this integration can be modulated, we have used an acute brain slice preparation from young adult rats and found that encoding of mossy fiber input in the cerebellar granule cell layer can be regulated by serotonin (5-hydroxytryptamine, 5-HT) via a specific action on Golgi cells. We find that 5-HT depolarizes Golgi cells, likely by activating 5-HT2A receptors, but does not directly act on either granule cells or mossy fibers. As a result of Golgi cell depolarization, 5-HT significantly increases tonic inhibition onto both granule cells and Golgi cells. 5-HT-mediated Golgi cell depolarization is not sufficient, however, to alter the probability or timing of mossy fiber-evoked feed-forward inhibition onto granule cells. Together, increased granule cell tonic inhibition paired with normal feed-forward inhibition acts to reduce granule cell spike probability without altering spike timing. Hence, these data provide a circuit mechanism by which 5-HT can reduce granule cell activity without altering temporal representations of mossy fiber input. Such changes in network integration could enable flexible, state-specific suppression of cerebellar sensorimotor input that should not be learned or enable reversal learning for unwanted associations. NEW & NOTEWORTHY Serotonin (5-hydroxytryptamine, 5-HT) regulates synaptic integration at the input stage of cerebellar processing by increasing tonic inhibition of granule cells. This circuit mechanism reduces the probability of granule cell spiking without altering spike timing, thus suppressing cerebellar input without altering its temporal representation in the granule cell layer.

Foreground segmentation using convolutional neural networks for multiscale feature encoding

Several methods have been proposed to solve moving objects segmentation problem accurately in different scenes. However, many of them lack the ability of handling various difficult scenarios such as illumination changes, background or camera motion, camouflage effect, shadow etc. To address these issues, we propose two robust encoder-decoder type neural networks that generate multi-scale feature encodings in different ways and can be trained end-to-end using only a few training samples. Using the same encoder-decoder configurations, in the first model, a triplet of encoders take the inputs in three scales to embed an image in a multi-scale feature space; in the second model, a Feature Pooling Module (FPM) is plugged on top of a single input encoder to extract multi-scale features in the middle layers. Both models use a transposed convolutional network in the decoder part to learn a mapping from feature space to image space. In order to evaluate our models, we entered the Change Detection 2014 Challenge (changedetection.net) and our models, namely FgSegNet_M and FgSegNet_S, outperformed all the existing state-of-the-art methods by an average F-Measure of 0.9770 and 0.9804, respectively. We also evaluate our models on SBI2015 and UCSD Background Subtraction datasets. Our source code is made publicly available at https://github.com/lim-anggun/FgSegNet.

Controlling an organic synthesis robot with machine learning to search for new reactivity.

The discovery of chemical reactions is an inherently unpredictable and time-consuming process1. An attractive alternative is to predict reactivity, although relevant approaches, such as computer-aided reaction design, are still in their infancy2. Reaction prediction based on high-level quantum chemical methods is complex3, even for simple molecules. Althoughmachine learning is powerful for data analysis4,5, its applications in chemistry are still being developed6. Inspired by strategies based on chemists' intuition7, we propose that a reaction system controlled by a machine learning algorithm may be able to explore the space of chemical reactions quickly, especially if trained by an expert8. Here we present an organic synthesis robot that can perform chemical reactions and analysis faster than they can be performed manually, as well as predict the reactivity of possible reagent combinations after conducting a small number of experiments, thus effectively navigating chemical reaction space. By using machine learning for decision making, enabled by binary encoding of the chemical inputs, the reactions can be assessed in real time using nuclear magnetic resonance and infrared spectroscopy. The machine learning system was able to predict the reactivity of about 1,000 reaction combinations with accuracy greater than 80 per cent after considering the outcomes of slightly over 10 per cent of the dataset. This approach was also used to calculate the reactivity of published datasets. Further, by using real-time data from our robot, these predictions were followed up manually by a chemist, leading to the discovery of four reactions.

A computational study of the role of spatial receptive field structure in processing natural and non-natural scenes

The center-surround receptive field structure, ubiquitous in the visual system, is hypothesized to be evolutionarily advantageous in image processing tasks. We address the potential functional benefits and shortcomings of spatial localization and center-surround antagonism in the context of an integrate-and-fire neuronal network model with image-based forcing. Utilizing the sparsity of natural scenes, we derive a compressive-sensing framework for input image reconstruction utilizing evoked neuronal firing rates. We investigate how the accuracy of input encoding depends on the receptive field architecture, and demonstrate that spatial localization in visual stimulus sampling facilitates marked improvements in natural scene processing beyond uniformly-random excitatory connectivity. However, for specific classes of images, we show that spatial localization inherent in physiological receptive fields combined with information loss through nonlinear neuronal network dynamics may underlie common optical illusions, giving a novel explanation for their manifestation. In the context of signal processing, we expect this work may suggest new sampling protocols useful for extending conventional compressive sensing theory.

Code-Division Multiplex Encoding Analog Front End for Current-Mode Sensor Arrays

This letter presents a new approach for applying orthogonal code-division multiplex encoding to current-mode sensor arrays with analog-domain encoding directly in the column readout circuitry, which enables simultaneous readout for multiple sensors. The approach is applied to a $8\,\,\boldsymbol \times16$ array of CMOS-integrated photodetectors and implemented in a general purpose 180-nm CMOS process. The sensor array demonstrates 63-dB signal-to-noise ratio with an array-wide sampling rate of 100 Hz with 16-channel encoding, an improvement of more than 12 dB compared with time-division multiplexing at the same sampling rate. The complete photodetector array consumes 5 mW total power. This letter represents the first reported code-division multiplexed sensing architecture with analog-encoding front end, and the orthogonal coding scheme is broadly applicable to electronic current-mode sensor arrays for physical, chemical, and biological sensing applications without requiring sensor input encoding.

The Magnitude, But Not the Sign, of MT Single-Trial Spike-Time Correlations Predicts Motion Detection Performance

Spike-time correlations capture the short timescale covariance between the activity of neurons on a single trial. These correlations can significantly vary in magnitude and sign from trial to trial, and have been proposed to contribute to information encoding in visual cortex. While monkeys performed a motion-pulse detection task, we examined the behavioral impact of both the magnitude and sign of single-trial spike-time correlations between two nonoverlapping pools of middle temporal (MT) neurons. We applied three single-trial measures of spike-time correlation between our multiunit MT spike trains (Pearson's, absolute value of Pearson's, and mutual information), and examined the degree to which they predicted a subject's performance on a trial-by-trial basis. We found that on each trial, positive and negative spike-time correlations were almost equally likely, and, once the correlational sign was accounted for, all three measures were similarly predictive of behavior. Importantly, just before the behaviorally relevant motion pulse occurred, single-trial spike-time correlations were as predictive of the performance of the animal as single-trial firing rates. While firing rates were positively associated with behavioral outcomes, the presence of either strong positive or negative correlations had a detrimental effect on behavior. These correlations occurred on short timescales, and the strongest positive and negative correlations modulated behavioral performance by ∼9%, compared with trials with no correlations. We suggest a model where spike-time correlations are associated with a common noise source for the two MT pools, which in turn decreases the signal-to-noise ratio of the integrated signals that drive motion detection.SIGNIFICANCE STATEMENT Previous work has shown that spike-time correlations occurring on short timescales can affect the encoding of visual inputs. Although spike-time correlations significantly vary in both magnitude and sign across trials, their impact on trial-by-trial behavior is not fully understood. Using neural recordings from area MT (middle temporal) in monkeys performing a motion-detection task using a brief stimulus, we found that both positive and negative spike-time correlations predicted behavioral responses as well as firing rate on a trial-by-trial basis. We propose that strong positive and negative spike-time correlations decreased behavioral performance by reducing the signal-to-noise ratio of integrated MT neural signals.