Currently, various modalities for in-vehicle information systems have gained popularity. However, the secondary tasks of an in-vehicle information system (IVIS) may result in driver distraction while driving. This study aimed to evaluate the effects of classified secondary tasks performed in four modalities on driving performance and visual distraction. A driving simulator capable of connecting mass-produced cars to a virtual driving scenario was proposed. Ten secondary tasks were classified based on complexity and difficulty into three classes (basic, medium, and advanced). Thereafter, these tasks were tested using four modalities: knobs and buttons on the center stack, touch screen, steering wheel (SW) buttons, and voice control. For dependent measures, the standard deviation of speed, the standard deviation of lane position, fixation counts, and average fixation duration were collected. Consequently, significant primary effects of modality and task class were found in terms of each dependent measure. The analysis revealed that SW buttons and voice control were more suitable for basic tasks. Moreover, voice control was best for tasks in the medium class. In the advanced class, the voice control and touch screens outperformed the knobs and buttons. However, considering the efficiency and full interaction process, voice control is unsuitable for all advanced secondary tasks. Hence, cross-mode interaction was recommended for advanced class tasks with complicated input and feedback, owing to the trade-off between different interaction modes. The results of this study are significant for selecting the interaction modes and task design of IVIS for enhancing driving safety.