In challenging work environments, teleoperation has been adopted for executing tasks over long distances, ensuring worker safety off-site. The physical distance from jobsites typically poses challenges to teleoperators, limiting their cognition and perception of sites. Particularly, these human-machine interface challenges vary depending on the work environment conditions. This paper analyzes the impact of different visual interfaces on teleoperators and their work performance in challenging jobsite environments, using interactive workplace modeling and simulation to execute teleoperation tasks. Based on experiments with 33 subjects in construction-related fields with industrial experience, the study observed data overload problems and distraction with effects varying by work environment. Additionally, the analysis includes how their eye gaze patterns could demonstrate these effects. The findings contribute valuable insights into human factors crucial for designing user-centered teleoperation interfaces in construction, forming the basis for an intuitive and informative interface design in task-specific settings.