Multilayer cellular neural networks (CNNs) with multiple state variables in each cell associated with multiple dynamic rules are believed to possess more powerful data computing and signal processing capabilities than single-layer CNNs and are specially suitable for solving complex problems. However, at present, their large scale integrated hardware implementation is still quite challenging based on traditional CMOS-based technology due to high circuity complexity and their applications are thus limited in practice. In this paper, a novel compact multilayer CNN model based on nanometer scale resonant tunneling diodes (RTDs) and memristors is presented. More specifically, in this model, one multilayer CNN cell consists of several sub-cells located in different layers. The resonant tunneling diode with quantum tunneling induced nonlinearity and uniquely folded current–voltage characteristics is used to implement the compact and high-speed cell via replacing the original linear resistor and removing the output function of conventional CNN cells. Furthermore, the interactions between these cells are determined by a pair of multi-dimensional cloning templates. And a compact synaptic circuit based on memristors is designed to realize the cloning template parameter (weight strength) and the multiplication (weighting) operation, by leveraging its nonvolatility, good scalability, and variable conductance. The combination of these desirable elements equips the proposed multilayer CNN with advantages of powerful processing capability as well as high compactness, versatility, and possibility of very large scale integration (VLSI) circuit implementations. Finally, the performance of the proposed multilayer CNN is validated by five illustrative examples in color image processing with each layer dealing with each primary color (red, green or blue) plane.