Handwritten digit feature extraction and classification using neural networks

Abstract

Presents results obtained on a handwritten numeral classification problem using neural networks. The database of handwritten digits used for this study was also used by a group at the University of Windsor. The neural network that receives the most attention is a hierarchical multi-layered network. The architecture of this network was originally proposed by a group at ATT the networks learned to extract the relevant features of the input. With the hierarchical network, the training data (about 2000 digits) were consistently learned almost perfectly. The percentage of test patterns (another 2000 digits) rejected as unclassifiable (uncertain) to obtain 1% error on the remaining (classifiable) test patterns ranged from 10.6 to 11.8%. These generalization performances compare favorably to Bell Labs' result on a different dataset, of 12.0% rejections. It was found that an elaborate conjugate gradient minimization technique yielded little improvement in generalization performance and resulted in six times longer training time than ordinary backpropagation. The authors show that the neural networks were also able to extract meaningful features of the digits, such as edges. Some additional simulation results are reported that show the importance of using a hierarchical network: both a simple two-layered network and a cascade correlation network yielded inferior results. The cascade correlation network was the least successful, possibly because the network was committing itself to poor results early on in training when few hidden units were present. >