Image recognition and motion tracking are widely utilized in the field of Augmented Reality (AR). Although their computational cost is huge, they enable to extend the practicality and the range of applications if all computation is processed within real time. Toward this goal, in this paper, we propose a handheld AR system optimized for direct hardware computation. It includes a subspace method for image recognition and a KLT tracking algorithm for motion tracking. The AR system is composed of one two-million-pixel-CCD-image sensor, one head-mounted display, one reconfigurable device called DAPDNA-2, and so on. DAPDNA-2 is a coarse-grained and dynamic-reconfigurable device which is produced by Tokyo Keiki Inc. The merit of DAPDNA-2 is its short-reconfiguration time and it is utilised to full for not only high performance but also the reduction of power consumption. The experimental result through a real Japanese-English translation system shows image recognition and motion tracking are computed within real-time; the computation time is less than 0.741 milliseconds per a VGA-resolution (640 x 480 pixels) frame. Thus, we are able to find a highly efficient computation using a coarse-grained architecture compared with general-purpose processors and embedded processors.