Efficient typing in a mobile device is a long-standing problem because of its small-size touch screen. Methods for contactless solutions using acoustic sensors have been proposed recently, which are device-free and low cost. However, these methods are not practical enough as they use either the microphone of the mobile device for recording the sounds generated during writing or the speaker of the device for playing a frequency modulated ultrasonic signal and then the signal reflected from the hand of the user is analyzed, where the former process requires a quiet environment and the latter process requires the user to write a letter in a single stroke and large size. In order to overcome those challenges, we present DMHC, which is an effective and device-free multi-modal handwritten character recognition system. DMHC fuses two different acoustic modalities, which are ultrasonic and general audio signals. The main feature of DMHC is that we extract the latent interactions between the ultrasonic and general acoustic signals. Specifically, DMHC implements the noise-resistant handwritten character recognition system for reliably recognizing handwritten characters in real life. We implement our proposed module based on the self-attention method to multiple channel fusion. Extensive experimental results demonstrate the effectiveness and robustness of DMHC under various conditions, where the recognition accuracies of letters and words are found to be 97.4% and 95.3%, respectively.