Neural Signal Processing Research Articles

MFF-Net: Deepfake Detection Network Based on Multi-Feature Fusion.

Significant progress has been made in generating counterfeit images and videos. Forged videos generated by deepfaking have been widely spread and have caused severe societal impacts, which stir up public concern about automatic deepfake detection technology. Recently, many deepfake detection methods based on forged features have been proposed. Among the popular forged features, textural features are widely used. However, most of the current texture-based detection methods extract textures directly from RGB images, ignoring the mature spectral analysis methods. Therefore, this research proposes a deepfake detection network fusing RGB features and textural information extracted by neural networks and signal processing methods, namely, MFF-Net. Specifically, it consists of four key components: (1) a feature extraction module to further extract textural and frequency information using the Gabor convolution and residual attention blocks; (2) a texture enhancement module to zoom into the subtle textural features in shallow layers; (3) an attention module to force the classifier to focus on the forged part; (4) two instances of feature fusion to firstly fuse textural features from the shallow RGB branch and feature extraction module and then to fuse the textural features and semantic information. Moreover, we further introduce a new diversity loss to force the feature extraction module to learn features of different scales and directions. The experimental results show that MFF-Net has excellent generalization and has achieved state-of-the-art performance on various deepfake datasets.

Associative anticipatory learning and control of the cerebellar cortex based on the spike-timing-dependent plasticity of the parallel fiber-Purkinje cell synapses

Time delays are inevitable in the neural processing of sensorimotor systems; small delays can cause severe damage to movement accuracy and stability. It is strongly suggested that the cerebellum compensates for delays in neural signal processing and performs predictive control. Neural computational theories have explored concepts of the internal models of control objects—believed to avoid delays by providing internal feedback information—although there has been no clear relevance to neural processing. The timing-dependent plasticity of parallel fiber-Purkinje cell synapses is well known. The long-term depression of the synapse is observed when parallel fiber activation precedes climbing fiber activation within −50–300 ms, and is the greatest within 50–200 ms. This paper presents a theory that this temporal difference of 50–200 ms is the basis for an associative anticipation of as many milliseconds. Associative learning can theoretically connect an input signal to a desired signal; therefore, a 50–200 ms earlier input signal can be connected to a desired output signal through temporary asymmetric plasticity. After learning is completed, an input signal generates a desired output signal that appears 50–200 ms later. For the associative learning of temporally continuous signals, this study integrates the universal function approximation capability of the cerebellar cortex model and temporally asymmetric synaptic plasticity to create the theory of associative anticipatory learning of the cerebellum. The effective motor control of this learning is demonstrated by adaptively stabilizing an inverted pendulum with a delay similar to that done by humans.

Editorial: Overview and Some New Directions

The fractional derivative holds historical dependence or memory effects. But it also brings error accumulation of the numerical solutions as well as the theoretical analysis since many properties from the integer order case cannot hold. The fractional difference is discrete counterpart of the fractional derivative. The memory kernel is defined by applying the discrete functions on time scale and avoids the errors from the numerical discretization. It is particularly suitable for fractional modeling with computer implementations. Many applications of fractional dynamics to neural networks, signal processing, time series and big data become possible now. This issue of work presents the latest progress in the neighborhood related to fractional calculus.

A Hybrid Vision Processing Unit with a Pipelined Workflow for Convolutional Neural Network Accelerating and Image Signal Processing

Vision processing chips have been widely used in image processing and recognition tasks. They are conventionally designed based on the image signal processing (ISP) units directly connected with the sensors. In recent years, convolutional neural networks (CNNs) have become the dominant tools for many state-of-the-art vision processing tasks. However, CNNs cannot be processed by a conventional vision processing unit (VPU) with a high speed. On the other side, the CNN processing units cannot process the RAW images from the sensors directly and an ISP unit is required. This makes a vision system inefficient with a lot of data transmission and redundant hardware resources. Additionally, many CNN processing units suffer from a low flexibility for various CNN operations. To solve this problem, this paper proposed an efficient vision processing unit based on a hybrid processing elements array for both CNN accelerating and ISP. Resources are highly shared in this VPU, and a pipelined workflow is introduced to accelerate the vision tasks. We implement the proposed VPU on the Field-Programmable Gate Array (FPGA) platform and various vision tasks are tested on it. The results show that this VPU achieves a high efficiency for both CNN processing and ISP and shows a significant reduction in energy consumption for vision tasks consisting of CNNs and ISP. For various CNN tasks, it maintains an average multiply accumulator utilization of over 94% and achieves a performance of 163.2 GOPS with a frequency of 200 MHz.

Online Decoding System with Calcium Image From Mice Primary Motor Cortex.

With the development of calcium imaging, neuroscientists have been able to study neural activity with a higher spatial resolution. However, the real-time processing of calcium imaging is still a big challenge for future experiments and applications. Most neuroscientists have to process their imaging data offline due to the time-consuming of most existing calcium imaging analysis methods. We proposed a novel online neural signal processing framework for calcium imaging and established an Optical Brain-Computer Interface System (OBCIs) for decoding neural signals in real-time. We tested and evaluated this system by classifying the calcium signals obtained from the primary motor cortex of mice when the mice were performing a lever-pressing task. The performance of our online system could achieve above 80% in the average decoding accuracy. Our preliminary results show that the online neural processing framework could be applied to future closed-loop OBCIs studies.

Neural network-assisted signal processing in Brillouin optical correlation-domain sensing for potential high-speed implementation.

The general neural networks (NNs) based on classification convert the Brillouin frequency shift (BFS) extraction in Brillouin-based distributed sensing to a problem in which the possible BFS output of the sensing system belongs to a finite number of discrete values. In this paper, we demonstrate a method of applying NNs with adaptive BFS incremental steps to signal processing for Brillouin optical correlation-domain sensing and achieve higher accuracy and operation speed. The comparison with the conventional curving fitting method shows that the NN improves the BFS measurement accuracy by 2-3 times and accelerates the signal processing speed by 1000 times for simulated signals. The experimental results demonstrate the NN provides 1.6-2.7 times enhancement for BFS measurement accuracy and 5000 times acceleration for the BFS extraction speed. This method supplies a potential solution to online signal processing for real-time Brillouin sensing.

Polymer-Based Biocompatible Packaging for Implantable Devices: Packaging Method, Materials, and Reliability Simulation.

Polymer materials attract more and more interests for a biocompatible package of novel implantable medical devices. Medical implants need to be packaged in a biocompatible way to minimize FBR (Foreign Body Reaction) of the implant. One of the most advanced implantable devices is neural prosthesis device, which consists of polymeric neural electrode and silicon neural signal processing integrated circuit (IC). The overall neural interface system should be packaged in a biocompatible way to be implanted in a patient. The biocompatible packaging is being mainly achieved in two approaches; (1) polymer encapsulation of conventional package based on die attach, wire bond, solder bump, etc. (2) chip-level integrated interconnect, which integrates Si chip with metal thin film deposition through sacrificial release technique. The polymer encapsulation must cover different materials, creating a multitude of interface, which is of much importance in long-term reliability of the implanted biocompatible package. Another failure mode is bio-fluid penetration through the polymer encapsulation layer. To prevent bio-fluid leakage, a diffusion barrier is frequently added to the polymer packaging layer. Such a diffusion barrier is also used in polymer-based neural electrodes. This review paper presents the summary of biocompatible packaging techniques, packaging materials focusing on encapsulation polymer materials and diffusion barrier, and a FEM-based modeling and simulation to study the biocompatible package reliability.

Stochastic Memristive Interface for Neural Signal Processing.

We propose a memristive interface consisting of two FitzHugh–Nagumo electronic neurons connected via a metal–oxide (Au/Zr/ZrO2(Y)/TiN/Ti) memristive synaptic device. We create a hardware–software complex based on a commercial data acquisition system, which records a signal generated by a presynaptic electronic neuron and transmits it to a postsynaptic neuron through the memristive device. We demonstrate, numerically and experimentally, complex dynamics, including chaos and different types of neural synchronization. The main advantages of our system over similar devices are its simplicity and real-time performance. A change in the amplitude of the presynaptic neurogenerator leads to the potentiation of the memristive device due to the self-tuning of its parameters. This provides an adaptive modulation of the postsynaptic neuron output. The developed memristive interface, due to its stochastic nature, simulates a real synaptic connection, which is very promising for neuroprosthetic applications.

Application and Evolution for Neural Network and Signal Processing in Large-Scale Systems

Low frequency oscillation is an important attribute of human brain activity, and the amplitude of low frequency fluctuation (ALFF) is an effective method to reflect the characteristics of low frequency oscillation, which has been widely used in the treatment of brain diseases and other fields. However, due to the low accuracy of the current analysis methods for low frequency signal extraction of ALFF, we propose the Fourier-based synchrosqueezing transform (FSST), which is often used in the field of signal processing to extract the ALFF of the low frequency power spectrum of the whole-time dimension. The low frequency characteristics of the extracted signal are compared with those of FSST and fast Fourier transform (FFT) through the resting-state data. It is clear that the signal extracted by FSST has more low frequency characteristics, which is significantly different from FFT.

A 10.8 µW Neural Signal Recorder and Processor With Unsupervised Analog Classifier for Spike Sorting.

Implantable brain machine interfaces for treatment of neurological disorders require on-chip, real-time signal processing of action potentials (spikes). In this work, we present the first spike sorting SoC with integrated neural recording front-end and analog unsupervised classifier. The event-driven, low power spike sorter features a novel hardware-optimized, K-means based algorithm that effectively eliminates duplicate clusters and is implemented using a novel clockless and ADC-less analog architecture. The 1.4 mm2 chip is fabricated in a 180-nm CMOS SOI process. The analog front-end achieves a 3.3 μVrms noise floor over the spike bandwidth (400 - 5000 Hz) and consumes 6.42 μW from a 1.5 V supply. The analog spike sorter consumes 4.35 μW and achieves 93.2% classification accuracy on a widely used synthetic test dataset. In addition, higher than 93% agreement between the chip classification result and that of a standard spike sorting software is observed using pre-recorded real neural signals. Simulations of the implemented spike sorter show robust performance under process-voltage-temperature variations.

Catecholaminergic Innervation of the Lateral Nucleus of the Cerebellum Modulates Cognitive Behaviors.

The cerebellum processes neural signals related to rewarding and aversive stimuli, suggesting that the cerebellum supports nonmotor functions in cognitive and emotional domains. Catecholamines are a class of neuromodulatory neurotransmitters well known for encoding such salient stimuli. Catecholaminergic modulation of classical cerebellar functions have been demonstrated. However, a role for cerebellar catecholamines in modulating cerebellar nonmotor functions is unknown. Using biochemical methods in male mice, we comprehensively mapped TH+ fibers throughout the entire cerebellum and known precerebellar nuclei. Using electrochemical (fast scan cyclic voltammetry), and viral/genetic methods to selectively delete Th in fibers innervating the lateral cerebellar nucleus (LCN), we interrogated sources and functional roles of catecholamines innervating the LCN, which is known for its role in supporting cognition. The LCN has the most TH+ fibers in cerebellum, as well as the most change in rostrocaudal expression among the cerebellar nuclei. Norepinephrine is the major catecholamine measured in LCN. Distinct catecholaminergic projections to LCN arise only from locus coeruleus, and a subset of Purkinje cells that are positive for staining of TH. LC stimulation was sufficient to produce catecholamine release in LCN. Deletion of Th in fibers innervating LCN (LCN-Th-cKO) resulted in impaired sensorimotor integration, associative fear learning, response inhibition, and working memory in LCN-Th-cKO mice. Strikingly, selective inhibition of excitatory LCN output neurons with inhibitory designer receptor exclusively activated by designer drugs led to facilitation of learning on the same working memory task impaired in LCN-Th-cKO mice. Collectively, these data demonstrate a role for LCN catecholamines in cognitive behaviors.SIGNIFICANCE STATEMENT Here, we report on interrogating sources and functional roles of catecholamines innervating the lateral nucleus of the cerebellum (LCN). We map and quantify expression of TH, the rate-limiting enzyme in catecholamine synthesis, in the entire cerebellar system, including several precerebellar nuclei. We used cyclic voltammetry and pharmacology to demonstrate sufficiency of LC stimulation to produce catecholamine release in LCN. We used advanced viral techniques to map and selectively KO catecholaminergic neurotransmission to the LCN, and characterized significant cognitive deficits related to this manipulation. Finally, we show that inhibition of excitatory LCN neurons with designer receptor exclusively activated by designer drugs, designed to mimic Gi-coupled catecholamine GPCR signaling, results in facilitation of a working memory task impaired in LCN-specific TH KO mice.

Restoring upper extremity function with brain-machine interfaces.

One of the most exciting advances to emerge in neural interface technologies has been the development of real-time brain-machine interface (BMI) neuroprosthetic devices to restore upper extremity function. BMI neuroprostheses, made possible by synergistic advances in neural recording technologies, high-speed computation and signal processing, and neuroscience, have permitted the restoration of volitional movement to patients suffering the loss of upper-extremity function. In this chapter, we review the scientific and technological advances underlying these remarkable devices. After presenting an introduction to the current state of the field, we provide an accessible technical discussion of the two fundamental requirements of a successful neuroprosthesis: signal extraction from the brain and signal decoding that results in robust prosthetic control. We close with a presentation of emerging technologies that are likely to substantially advance the field.

Novel Arithmetics in Deep Neural Networks Signal Processing for Autonomous Driving: Challenges and Opportunities

This article focuses on the trends, opportunities, and challenges of novel arithmetic for deep neural network (DNN) signal processing, with particular reference to assisted- and autonomous driving applications. Due to strict constraints in terms of the latency, dependability, and security of autonomous driving, machine perception (i.e., detection and decision tasks) based on DNNs cannot be implemented by relying on remote cloud access. These tasks must be performed in real time in embedded systems on board the vehicle, particularly for the inference phase (considering the use of DNNs pretrained during an offline step). When developing a DNN computing platform, the choice of the computing arithmetic matters. Moreover, functional safe applications, such as autonomous driving, impose severe constraints on the effect that signal processing accuracy has on the final rate of wrong detection/decisions. Hence, after reviewing the different choices and tradeoffs concerning arithmetic, both in academia and industry, we highlight the issues in implementing DNN accelerators to achieve accurate and lowcomplexity processing of automotive sensor signals (the latter coming from diverse sources, such as cameras, radar, lidar, and ultrasonics). The focus is on both general-purpose operations massively used in DNNs, such as multiplying, accumulating, and comparing, and on specific functions, including, for example, sigmoid or hyperbolic tangents used for neuron activation.

Systematic analysis of wavelet denoising methods for neural signal processing

Objective. Among the different approaches for denoising neural signals, wavelet-based methods are widely used due to their ability to reduce in-band noise. All wavelet denoising algorithms have a common structure, but their effectiveness strongly depends on several implementation choices, including the mother wavelet, the decomposition level, the threshold definition, and the way it is applied (i.e. the thresholding). In this work, we investigated these factors to quantitatively assess their effects on neural signals in terms of noise reduction and morphology preservation, which are important when spike sorting is required downstream. Approach. Based on the spectral characteristics of the neural signal, according to the sampling rate of the signals, we considered two possible decomposition levels and identified the best-performing mother wavelet. Then, we compared different threshold estimation and thresholding methods and, for the best ones, we also evaluated their effect on clearing the approximation coefficients. The assessments were performed on synthetic signals that had been corrupted by different types of noise and on a murine peripheral nervous system dataset, both of which were sampled at about 16 kHz. The results were statistically analysed in terms of their Pearson’s correlation coefficients, root-mean-square errors, and signal-to-noise ratios. Main results. As expected, the wavelet implementation choices greatly influenced the processing performance. Overall, the Haar wavelet with a five-level decomposition, hard thresholding method, and the threshold proposed by Han et al (2007) achieved the best outcomes. Based on the adopted performance metrics, wavelet denoising with these parametrizations outperformed conventional 300–3000 Hz linear bandpass filtering. Significance. These results can be used to guide the reasoned and accurate selection of wavelet denoising implementation choices in the context of neural signal processing, particularly when spike-morphology preservation is required.

(Invited) In-Memory Computing with Memristor Circuit Primitives

The dramatic rise of data-intensive workloads has revived special-purpose hardware and architectures for continuing improvements in computational speed and energy efficiency. While traditional CMOS ASICs deliver some performance gains, typically by limiting data movement or implementing “in-memory computation”, such approaches still suffer from low power efficiency. Therefore, new proposals leveraging non-volatile memristive devices for in-memory computation are highly attractive in a variety of application domains. While originally developed for as digital (binary) high density non-volatile memories, memristive devices have demonstrated a wide range of behaviors and properties – such as a wide range of tunable analog resistance and non-linear dynamics – which motivate their use in novel functions and new computational models. Many recent in-memory compute studies have focused on crossbar circuit architectures, demonstrating their application for neural networks, scientific computing and signal processing. However, other circuit primitives – such as content addressable memories (CAMs) and combined systems such as crossbar arrays and non-linear elements– have shown further promise for mapping a diverse range of complimentary computational models such as finite state machines, pattern matching, hashing algorithms and Hopfield networks for tackling optimization problems. In this talk, I will review the different exciting opportunities for in-memory computational primitives levering non-volatile memristive devices and their circuits and architectures for enabling low power, high-throughput computation in a variety of application domains. I will also highlight the different materials and device requirements which steer material stack design and fabrication for these diverse circuits and application domains. Finally, I will show recent lab demonstrations of various applications mapped to these in-memory computational circuit primitives based on memristor devices, and give an outlook on performance.

An FPGA-Based Neuron Activity Extraction Unit for a Wireless Neural Interface

As computational and functional brain model development are solely dependent upon the data acquired from the neural interface, this device plays a vital role in both prosthetic developments and neurological experiments. A wireless neural interface is preferred over a traditional wired one because it can maximize the comfort of the subject and ensure the freedom of movement while implemented. This paper describes the field programmable gate array (FPGA) prototype design of a low-power multichannel neuron activity extraction unit suitable for a wireless neural interface. To achieve the low-power requirement, we proposed a novel neural signal extraction algorithm which can provide an up to 6000X transmission rate reduction considering the input signal. Consequently, this technique offers at least 2X power reduction compared to the state-of-the-art systems. We implemented this scheme in Xilinx Zynq-7000 FPGA, which can be used as an intermediate transition towards the application specific integrated circuit (ASIC) design for on-chip neural signal processing. The proposed FPGA prototype offers reconfigurable computability, which means the model can be modified and verified according to prerequisites before the final ASIC design. This prototype consists of a signal filtering unit and a signal extraction unit which can be used either as stand-alone units or combined as a complete system. Our proposed scheme also provides a provision to work as a single-channel or a scalable multichannel interface based on user’s demands. We collected practical neural signals from rat brains and validated the efficacy of the implemented system using in-silico signal processing.

Multichannel parallel processing of neural signals in memristor arrays.

Fully implantable neural interfaces with massive recording channels bring the gospel to patients with motor or speech function loss. As the number of recording channels rapidly increases, conventional complementary metal-oxide semiconductor (CMOS) chips for neural signal processing face severe challenges on parallelism scalability, computational cost, and power consumption. In this work, we propose a previously unexplored approach for parallel processing of multichannel neural signals in memristor arrays, taking advantage of their rich dynamic characteristics. The critical information of neural signal waveform is extracted and encoded in the memristor conductance modulation. A signal segmentation scheme is developed to adapt to device variations. To verify the fidelity of the processed results, seizure prediction is further demonstrated, with high accuracy above 95% and also more than 1000× improvement in power efficiency compared with CMOS counterparts. This work suggests that memristor arrays could be a promising multichannel signal processing module for future implantable neural interfaces.

Array processing of neural signals recorded from the peripheral nervous system for the classification of action potentials

BackgroundRecording from the peripheral nervous system is key in the development of implantable neural interfaces. Despite a long history of using implantable electrodes for neuro-stimulation, it is difficult to make recordings from the nerves as signal amplitudes are often too small to be detected. Methods exist that are suitable for recording evoked potentials, but these require artificial stimulation of the nerve and thus have limited use in implanted neural interfaces. New methodIn order to address these issues new methods are developed to analyse spontaneously occurring action potentials by extending an approach called velocity selective recording, which uses longitudinally spaced electrodes to record action potentials as they propagate. The new methods using image processing techniques to automatically identify and classify action potentials without any prior knowledge of their morphology. ResultsSimulations are developed to test the methods, and a detailed experimental validation is performed using in-vivo recordings from the L5 dorsal rootlet of rat. Results show that this new approach can discriminate action potentials from both simulated and real recordings and the experimental validation demonstrates an ability to detect dermal stimulation by changes in the firing patterns of different axons. Comparison to existing methodsThis framework, unlike existing methods, is intrinsically suitable for recordings of spontaneous neural activity. Further it improves upon both the computational complexity and the overall performance of existing methods. ConclusionIt is possible to perform on-line discrimination and identification of action potentials without any prior knowledge of their morphology using new image processing inspired methods.

Multidimensional fiber echo state network analogue

Optical neuoromorphic technologies enable neural network-based signal processing through a specifically designed hardware and may confer advantages in speed and energy. However, the advances of such technologies in bandwidth and/or dimensionality are often limited by the constraints of the underlying material. Optical fiber presents a well-studied low-cost solution with unique advantages for low-loss high-speed signal processing. The fiber echo state network analogue (FESNA), fiber-based neuromorphic processor, has been the first technology suitable for multichannel high bandwidth (including THz) and dual-quadrature signal processing. Here we propose the multidimensional FESNA (MD-FESNA) processing by utilizing multi-mode fiber non-linearity. Thus, the developed MD-FESNA is the first neuromorphic technology which augments all aforementioned advantages of FESNA with multidimensional spatio-temporal processing. We demonstrate the performance and flexibility of the technology on the example of prediction tasks for hyperchaotic systems. These results will pave the way for a high-speed neuromorphic processing of multidimensional tasks, hardware for spatio-temporal neural networks and open new application venues for fiber-based spatio-temporal multiplexing.

Plasticity and Adaptation in Neuromorphic Biohybrid Systems

SummaryNeuromorphic systems take inspiration from the principles of biological information processing to form hardware platforms that enable the large-scale implementation of neural networks. The recent years have seen both advances in the theoretical aspects of spiking neural networks for their use in classification and control tasks and a progress in electrophysiological methods that is pushing the frontiers of intelligent neural interfacing and signal processing technologies. At the forefront of these new technologies, artificial and biological neural networks are tightly coupled, offering a novel “biohybrid” experimental framework for engineers and neurophysiologists. Indeed, biohybrid systems can constitute a new class of neuroprostheses opening important perspectives in the treatment of neurological disorders. Moreover, the use of biologically plausible learning rules allows forming an overall fault-tolerant system of co-developing subsystems. To identify opportunities and challenges in neuromorphic biohybrid systems, we discuss the field from the perspectives of neurobiology, computational neuroscience, and neuromorphic engineering.