In this human subject study about interaction with head-up displays (HUDs), the impacts of three types of HUDs (flag, perspective, and flag and perspective) on experienced and inexperienced drivers were subjectively and objectively measured. The driving experiment divided 60 participants into two groups (experienced and inexperienced) and assessed their driving experiences in real high- and low-level driving tasks using the three types of HUDs. During the subjective performance evaluation, the participants executed secondary display tasks and performed a primary driving task with eye-tracking and physiological monitoring. The flag and perspective display (display FP) and the perspective display (display P) were more effective in helping drivers execute simple tasks and complex tasks, and both experienced and inexperienced drivers were HUD-dependent. Based on the study, we provided an optimized design strategy for the HUD interface. Also, we suggest the HUD design should give special consideration for inexperienced drivers.