Research on image recognition using memristor crossbar

In deep neural network (DNN) models, the weighted summation, or multiply-and-accumulate (MAC) operation, is an essential and heavy calculation task, which leads to high power consumption in current digital processors. The use of analog operation in complementary metal-oxide-semiconductor (CMOS) very-large-scale integration (VLSI) circuits is a promising method for achieving extremely low power-consumption operation for such calculation tasks. In this paper, a time-domain analog weighted-sum calculation model is proposed based on an integrate-and-fire-type spiking neuron model. The proposed calculation model is applied to multi-layer feedforward networks, in which weighted summations with positive and negative weights are separately performed, and two timings proportional to the positive and negative ones are produced, respectively, in each layer. The timings are then fed into the next layers without their subtraction operation. We also propose VLSI circuits to implement the proposed model. Unlike conventional analog voltage or current mode circuits, the time-domain analog circuits use transient operation in charging/discharging processes to capacitors. Since the circuits can be designed without operational amplifiers, they can operate with extremely low power consumption. We designed a proof-of-concept (PoC) CMOS circuit to verify weighted-sum operation with the same weights. Simulation results showed that the precision was above 4-bit, and the energy efficiency for the weighted-sum calculation was 237.7 Tera Operations Per Second Per Watt (TOPS/W), more than one order of magnitude higher than that in state-of-the-art digital AI processors. Our model promises to be a suitable approach for performing intensive in-memory computing (IMC) of DNNs with moderate precision very energy-efficiently while reducing the cost of analog-digital-converter (ADC) overhead.

With the rapid development of electronic technology, network technology and cloud computing technology, the current data is increasing in the way of mass, has entered the era of big data. Based on cloud computing clusters, this paper proposes a novel method of parallel implementation of multilayered neural networks based on Map-Reduce. Namely in order to meet the requirements of big data processing, this paper presents an efficient mapping scheme for a fully connected multi-layered neural network, which is trained by using error back propagation (BP) algorithm based on Map-Reduce on cloud computing clusters (MRBP). The batch-training (or epoch-training) regimes are used by effective segmentation of samples on the clusters, and are adopted in the separated training method, weight summary to achieve convergence by iterating. For a parallel BP algorithm on the clusters and a serial BP algorithm on uniprocessor, the required time for implementing the algorithms is derived. The performance parameters, such as speed-up, optimal number and minimum of data nodes are evaluated for the parallel BP algorithm on the clusters. Experiment results demonstrate that the proposed parallel BP algorithm in this paper has better speed-up, faster convergence rate, less iterations than that of the existed algorithms.

Hybrid no-propagation learning for multilayer neural networks

In this paper, the implementation of new digital architecture for a multilayer neural network (MNN) with on-chip learning is discussed. The advantage of using the digital approach is that it can use state-of-the-art VLSI and ULSI implementation techniques. One of the major hard-ware problems in implementing a neural network is the activating function of the neurons. The proposed MNN uses a simple function as the neuron's activating function to reduce the circuit size. Moreover, the proposed MNN has an on-chip learning capability. As the learning algorithm, a backpropagation algorithm is modified for effective hard-wave implementation. The proposed MNN is implemented on a field-programmable gate array (FPGA) to evaluate the learning performance and circuit size.

This paper presents a novel architecture for the FPGA-based implementation of multilayer neural network (NN), which integrates the layer-multiplexing and pipeline architecture together. The proposed method is aimed at enhancing the efficiency of resource usage and improving the forward speed at the module level, so that a larger NN can be implemented on commercial FPGAs. We developed a mapping method from NN schematic to physical architecture in FPGA by using the hybrid architecture, and also developed an algorithm to automatically determine the architecture by optimizing the application specific neural network topology. The experimental results with several different network topologies show that the proposed architecture can produce a very compact circuit with higher speed, compared with conventional methods.

We describe a modular architecture for the VLSI implementation of multilayer neural networks using a universal hybrid building block. Based on this approach, a programmable smart photosensor is designed which is in fact a VLSI realization of a multilayer feedforward neural network with an integrated photoreceptor array using 1.2 /spl mu/m CMOS technology. Each universal building block in this architecture comprises a multiplying DAC synapse, a portion of a nonlinear distributed neuron and compact digital registers for programming and storing a synaptic weight. The proposed modular neural network architecture features design simplicity and scalability, area efficiency, reduced interconnection problems and increased robustness. Based on this architecture and using cell-level optimization, the synaptic density in this version of the neural-based smart sensor has been increased by a factor of two. This has lead to an increase in the area available for a larger and higher resolution optical input array.

This paper deals with facility location problems on graphs with positive and negative vertex weights. We consider two different objective functions: In the first one (MWD) vertices with positive weight are assigned to the closest facility, whereas vertices with negative weight are assigned to the farthest facility. In the second one (WMD) all the vertices are assigned to the nearest facility. For the MWD model it is shown that there exists a finite set of points in the graph which contains the locations of facilities in an optimal solution. Furthermore, algorithms for both models for the 2-median problem on a cycle are developed. The algorithm for the MWD model runs in linear time, whereas the algorithm for the WMD model has a time complexity of O(n 2 ).

An architecture for VLSI implementation of multi-layer neural networks is presented in this paper. It is based on direct utilization of external digital weight memory. The architecture lends itself easily to a chip-in-loop training scheme which is controlled by a digital host computer. This architecture has been applied for VLSI implementation of the classifier unit of a moment-invariant contour-shape recognition system. >

Objective: Implementation of multilayer neural network (MLNN) with sigmoid activation function for the diagnosis of hepatitis disease. Methods: Artificial neural networks (ANNs) are efficient tools currently in common use for medical diagnosis. In hardware based architectures activation functions play an important role in ANN behavior. Sigmoid function is the most frequently used activation function because of its smooth response. Thus, sigmoid function and its close approximations were implemented as activation function. The dataset is taken from the UCI machine learning database. Results: For the diagnosis of hepatitis disease, MLNN structure was implemented and Levenberg Morquardt (LM) algorithm was used for learning. Our method of classifying hepatitis disease produced an accuracy of 91.9% to 93.8% via 10 fold cross validation. Conclusion: When compared to previous work that diagnosed hepatitis disease using artificial neural networks and the identical data set, our results are promising in order to reduce the size and cost of neural network based hardware. Thus, hardware based diagnosis systems can be developed effectively by using approximations of sigmoid function. Key words: Hepatitis disease diagnosis, multilayer neural network, 10-fold cross validation, approximations of sigmoid activation function

Two methods for pipelining analog or hybrid neural networks with analog outputs are presented. These methods provide concurrent operation of various stages on different sets of the network input stream. A new analog neuron with an embedded latch for implementation of one of the architectures is also presented. These methods are particularly attractive for time-multiplexed implementation of multi-layer neural networks. It is shown that significant speed improvement can be achieved by these methods. >

Center problems with pos/neg weights on trees

ABSTRACTThe present paper aims to extend the potential learning method to overcome the problem of collective interpretation, which aims to interpret multi-layered neural networks by compressing them into the simplest ones. In the process of compression, positive, negative, and complicated weights have had unfavourable effects for interpretation. To deal with the problems of collective interpretation, the potential learning is extended only to use positive weights. In addition, to obtain more appropriate weights for interpretation, the number of candidate weights for higher potentialities is first increased as much as possible. Then, from among many candidates, more appropriate weights are selected as more important ones. This extended potentiality learning is expected to produce more stable and more simple representations for easy interpretation. The extended method was applied to three datasets, namely, an artificial dataset, a real eye-tracking dataset, and a student evaluation dataset. In all cases, it was observed that the selectivity of connection weights could be increased. Correspondingly, the majority of connection weights became positive, and the collective weights were quite similar to the regression coefficients of the logistic regression analysis. Finally, for the third dataset (student evaluations), the extended method could extract more explicit input-output relations, compared with the logistic regression analysis, while improving generalisation performance.

On-chip training of memristor crossbar based multi-layer neural networks

One of the most basic graph problems, All-Pairs Shortest Paths (APSP) is known to be solvable in n^{3-o(1)} time, and it is widely open whether it has an O(n^{3-e}) time algorithm for e > 0. To better understand APSP, one often strives to obtain subcubic time algorithms for structured instances of APSP and problems equivalent to it, such as the Min-Plus matrix product. A natural structured version of Min-Plus product is Monotone Min-Plus product which has been studied in the context of the Batch Range Mode [SODA'20] and Dynamic Range Mode [ICALP'20] problems. This paper improves the known algorithms for Monotone Min-Plus Product and for Batch and Dynamic Range Mode, and establishes a connection between Monotone Min-Plus Product and the Single Source Replacement Paths (SSRP) problem on an n-vertex graph with potentially negative edge weights in {-M, …, M}. SSRP with positive integer edge weights bounded by M can be solved in O(Mn^ω) time, whereas the prior fastest algorithm for graphs with possibly negative weights [FOCS'12] runs in O(M^{0.7519} n^{2.5286}) time, the current best running time for directed APSP with small integer weights. Using Monotone Min-Plus Product, we obtain an improved O(M^{0.8043} n^{2.4957}) time SSRP algorithm, showing that SSRP with constant negative integer weights is likely easier than directed unweighted APSP, a problem that is believed to require n^{2.5-o(1)} time. Complementing our algorithm for SSRP, we give a reduction from the Bounded-Difference Min-Plus Product problem studied by Bringmann et al. [FOCS'16] to negative weight SSRP. This reduction shows that it might be difficult to obtain an O(M n^{ω}) time algorithm for SSRP with negative weight edges, thus separating the problem from SSRP with only positive weight edges.

Introduction: Synapse based on two successive memristors builds the synaptic weights of the artificial neural network for training three-bit parity problem and five-character recognition. Methods: The proposed memristor synapse circuit creates positive weights in the range [0;1], and maps it to range [-1;1] to program both the positive and negative weights. The proposed scheme achieves the same accuracy rate as the conventional bridge synapse schemes which consist of four memristors. Results and Conclusion: However, proposed synapse circuit decreases 50% the number of memristors and 76.88% power consumption compared to the conventional bridge memristor synapse.