Helmet-mounted Display Research Articles

Burden and Risk Factors of Cervical Spine Conditions in Military Aircrew From 1997 to 2015: A Retrospective Cohort Study.

Military aviators may have increased risk of cervical spine injuries because of exposure to supraphysiologic forces and vibration during dynamic flight. Aviator medical attrition impairs mission readiness, decreases operational capabilities, increases overall DoDcosts, and decreases retention of seasoned aviators. This study evaluated incidence and risk factors for cervical spine conditions in U.S. military aviators from 1997 to 2015. The Defense Medical Epidemiological Database was queried for aviators with a diagnosis of cervical spine conditions. Pertinent ICD-9 codes for cervical spine pathology were ascertained from U.S. Army, Air Force, and Naval Air Forces aeromedical references. Negative binomial regressions assessed sex, age, service, aircraft, and year on incidence of conditions. Rates were compared to non-aviator controls. The study was approved by the Institutional Review Board at the Naval Health Research Center (NHRC.2020.0205-NHSR). Incidence rates were 9.78 to 12.57/1,000 person-years for neck pain, 2.04 to 3.89/1,000 person-years for degenerative conditions without neurological involvement, and 0.94 to 1.36/1,000 person-years for degenerative conditions with neurological involvement. Aviation occupation (relative risk [RR] 1.41-2.05), female sex (RR 3.32-7.89), age over 40 (RR 2.39-4.62), and service in the Army or Marine Corps (RR 1.62-2.14) were risk factors. Military aviators had a statistically significant increase in risk of neck pain and medically disqualifying degenerative cervical spine conditions compared to non-aviator controls. Rates of neck pain increased in all aviators over the study epoch. Possible explanations could be related to the operational demands and the increased use of forward helmet-mounted display systems during the study period, a supposition that requires further investigation. There was no significant difference in rates of neck pain or degenerative cervical conditions between aircraft platforms (fighter/bomber, other fixed wing, and rotary wing). Female sex, age over 40 years, and Army/Marine Corps service were the greatest risk factors for neck pain and degenerative cervical spine conditions. Targeted prevention programs and expanded treatment modalities are necessary to reduce aviator attrition and Department of Defense cost burden.

Attentional Tunneling in Pilots During a Visual Tracking Task With a Head Mounted Display.

We examined whether active head aiming with a Helmet Mounted Display (HMD) can draw the pilot's attention away from a primary flight task. Furthermore, we examined whether visual clutter increases this effect. Head up display symbology can result in attentional tunneling, and clutter makes it difficult to identify objects. Eighteen military pilots had to simultaneously perform an attitude control task while flying in clouds and a head aiming task in a fixed-base flight simulator. The former consisted of manual compensation for roll disturbances of the aircraft, while the latter consisted of keeping a moving visual target inside a small or large head-referenced circle. A "no head aiming" condition served as a baseline. Furthermore, all conditions were performed with or without visual clutter. Head aiming led to deterioration of the attitude control task performance and an increase of the amount of roll-reversal errors (RREs). This was even the case when head aiming required minimal effort. Head aiming accuracy was significantly lower when the roll disturbances in the attitude control task were large compared to when they were small. Visual clutter had no effect on both tasks. We suggest that active head aiming of HMD symbology can cause attentional tunneling, as expressed by an increased number of RREs and less accuracy on a simultaneously performed attitude control task. This study improves our understanding in the perceptual and cognitive effects of (military) HMDs, and has implications for operational use and possibly (re)design of HMDs.

Helicopter Pilot Performance and Workload in a Following Task in a Degraded Visual Environment.

BACKGROUND: In this study, we investigated the impact of a loss of horizon due to atmospheric conditions on flight performance and workload of helicopter pilots during a low-altitude, dynamic flight task in windy conditions at sea. We also examined the potential benefits of a helmet-mounted display (HMD) for this specific task.METHODS: In a fixed-based helicopter simulator, 16 military helicopter pilots were asked to follow a maneuvering go-fast vessel in a good visual environment (GVE) and in a degraded visual environment (DVE). DVE was simulated by fog, obscuring the horizon and reducing contrast. Both visual conditions were performed once with and once without an HMD, which was simulated by projecting head-slaved symbology in the outside visuals. Objective measures included flight performance, control inputs, gaze direction, and relative positioning. Subjective measures included self-ratings on performance, situation awareness, and workload.RESULTS: The results showed that in DVE the pilots perceived higher workload and were flying closer to the go-fast vessel than in GVE. Consequently, they responded with larger control inputs to maneuvers of the vessel. The availability of an HMD hardly improved flight performance but did allow the pilots to focus their attention more outside, significantly improving their situation awareness and reducing workload. These benefits were found in DVE as well as GVE conditions.DISCUSSION: DVE negatively affects workload and flight performance of helicopter pilots in a dynamic, low-altitude following task. An HMD can help improve situation awareness and lower the workload during such a task, irrespective of the visual conditions.Ledegang WD, van der Burg E, Valk PJL, Houben MMJ, Groen EL. Helicopter pilot performance and workload in a following task in a degraded visual environment. Aerosp Med Hum Perform. 2024; 95(1):16-24.

Hypoxia impairs reaction time but not response accuracy in a visual choice reaction task

We investigated the effect of hypoxia on the reaction time (RT) and response accuracy of pilots performing a visual choice reaction task that corresponded to the scanning of helmet mounted display (HMD) symbology. Eighteen male military pilots performed the task in a hypobaric chamber at two simulated altitudes (92 m and 4572 m) in a single-blinded repeated measures and counter-balanced design. The visual stimuli were displayed in low and high contrast and at a 30- and 50-degree field of view (FoV). We measured the pilots' RT and response accuracy. Using an eye tracker, we measured the pilot's glance time at each stimulus location. Finally, we collected subjective ratings of alertness. The results show that hypoxia increased the RT and glance time. Lowering the stimulus contrast and increasing the FoV further increased the RT, independent of hypoxia. These findings provide no evidence for hypoxia-induced changes in visual contrast sensitivity or visual field. Instead, hypoxia seemed to affect RT and glance time by reducing alertness. Despite the increased RT, the pilots maintained their accuracy on the visual task, suggesting that visual scanning of HMD symbology may be resistant to the effects of acute hypoxia.

Design of Optical System for Dual-Channel Helmet-Mounted Display Based on Dual-Prism Structure

Design of Optical System for Dual-Channel Helmet-Mounted Display Based on Dual-Prism Structure

A Digital Alternative to the TNO Stereo Test to Qualify Military Aircrew

INTRODUCTION: Stereopsis is usually required in military aviators and may become increasingly important with reliance on newer technologies such as binocular Helmet-Mounted Displays (HMDs) and stereo displays. The current stereo test used to qualify UK military aircrew (TNO test) has many limitations. To address these limitations, two computer-based digital versions of a random dot stereogram (RDS) were developed: a static version (dRDS-S), and a version in which the dots appear to move dynamically within the depth plane (dRDS-D), both capable of measuring stereo acuity to threshold.METHODS: There were 41 participants who performed all 3 stereo tests, TNO and both digital dRDS tests, on two separate occasions.RESULTS: The best (lowest) mean stereo acuity threshold was measured with dRDS-S (33.79 arcseconds, range 12.64-173) and the worst mean stereo acuity thresholds were measured with the TNO test (91 arcseconds, range 60-240). Both dRDS tests were strongly correlated, but neither correlated with the TNO test. Both dRDS tests were more reliable, as indicated with tighter limits of agreement.DISCUSSION: With a large floor effect at 60 arcseconds, the TNO test was unable to characterize any finer degree of stereo acuity. Both dRDS tests demonstrated better test-retest reliability and addressed many of the limitations seen with the TNO test. The dRDS tests were not correlated with the TNO test, which suggests that the TNO test does not provide the accuracy or reliability for use as a meaningful aeromedical screening test. The dRDS tests will enable research to investigate the relationship between stereo acuity and operational performance.Posselt BN, Seemiller E, Winterbottom M, Baber C, Hadley S. A digital alternative to the TNO stereo test to qualify military aircrew. Aerosp Med Hum Perform. 2022; 93(12):846-854.

Helmet-Mounted Display System Based on IoT

Helmet-Mounted Display System Based on IoT

Eye Gaze Estimation Based on Stacked Hourglass Neural Network for Aircraft Helmet Aiming

Helmet-mounted display (HMD) systems allow aircraft pilots to aim at targets by using head postures. However, the direct use of helmet orientation to indicate the aiming direction ignores human eye movements, which are more flexible and efficient for interaction. Since the opaqueness of the helmet goggle blocks the sight of external cameras to capture facial or eye images of the pilots, and traditional eye feature extraction methods may fail when encountering conditions such as poor lighting, occlusion, and shaking, which are common on fighter aircrafts. In this work, an eye gaze based aiming solution that adapts to pilots wearing HMDs is proposed, and a deep learning-based method is proposed to extract eye features robustly. The prototype experiments demonstrate the ability to pick and aim at targets in real-time (60FPS) and are capable to accurately locate the target markers on a screen with an average error of fewer than 2 degrees. Conclusively, the proposed method performs the tasks of eye feature extraction on real-person imagery and the estimation of the 3D aiming direction for users with helmets, displaying competitive results with similar research.

Holographic optics in planar optical systems for next generation small form factor mixed reality headsets

Helmet Mounted Displays (HMDs), such as in Virtual Reality (VR), Augmented Reality (AR), Mixed reality (MR), and Smart Glasses have the potential to revolutionize the way we live our private and professional lives, as in communicating, working, teaching and learning, shopping and getting entertained. Such HMD devices have to satisfy draconian requirements in weight, size, form factor, power, compute, wireless communication and of course display, imaging and sensing performances. We review in this paper the various optical technologies and architectures that have been developed in the past 10 years to provide adequate solutions for the drastic requirements of consumer HMDs, a market that has yet to become mature in the next years, unlike the existing enterprise and defense markets that have already adopted VR and AR headsets as practical tools to improve greatly effectiveness and productivity. We focus specifically our attention on the optical combiner element, a crucial element in Optical See-Through (OST) HMDs that combines the see-through scene with a world locked digital image. As for the technological platform, we chose optical waveguide combiners, although there is also a considerable effort today dedicated to free-space combiners. Flat and thin optics as in micro-optics, holographics, diffractives, metasurfaces and other nanostructured optical elements are key building blocks to achieve the target form factor.

Human Readiness Levels Explained

The Human Readiness Level scale complements and supplements the existing technology readiness level scale to support comprehensive and systematic evaluation of human system aspects throughout a system’s life cycle. The objective is to ensure humans can use a fielded technology or system as intended to support mission operations safely and effectively. This article defines the nine human readiness levels in the scale, explains their meaning, and illustrates their application using a helmet-mounted display example.

Increasing helicopter flight safety in maritime operations with a head-mounted display

To increase flight safety and operational availability for helicopters, the potential benefits of helmet mounted displays (HMD) are investigated, with a focus on maritime flight operations. Helicopters have long downtimes, due to harsh weather conditions or other visual impairments, especially in maritime scenarios. Flying in these poor conditions can drastically reduce flight safety. It is often difficult to recognize the horizon due to sea fog, and the absence of reference objects can complicate the maritime flight. These conditions and especially the downtimes cost money or, at worst, life's. Therefore, DLR integrated the augmented reality glasses Microsoft HoloLens into DLR's simulator AVES to use it as HMD for pilots. Subsequently, displays and symbolism were developed and evaluated. To carry out a piloted simulator study, a maritime scenario was created to measure changes in the pilots' performance with the HMD, like workload or situational awareness. The paper focuses (a) on the integration of the HoloLens into the simulator with its challenges, solutions and findings, (b) on the symbolism and (c) on the piloted simulator study. Both the quality of the HoloLens as HMD and the study results are very positive. The pilots rated high usability, reduced workload, increased situational awareness and increased safety.

Head Posture Recognition Method Based on POSIT Algorithm

Helmet Mounted Sight (HMS) is an optical aiming and display device installed on the fighter, which plays an important role in modern air combat, but it cost a lot to install a real HMS on ground fighter simulator, so it is a wise choice to design a low-cost simulation HMS instead. In this process, the head posture recognition method is the key point in simulation HMS design. In this paper, a head posture recognition method is proposed for Flight Training Simulation. Firstly, the development process and working principle of Helmet Mounted Sight (HMS) are deeply studied, then the principle of POSIT algorithm is introduced, after that the head posture recognition method based on POSIT is proposed. Finally, the hardware design and software workflow of head posture recognition system are given.

Mixed Reality System for Virtual Chemistry Lab

This paper presents a new method for improving teaching aids with the design and development of a chemistry education platform - the Mixed Reality (MR) Chemistry Lab. Our system provides a new educational experience in which students can simulate a chemistry experiment in a virtual lab and interact with objects using Oculus Helmet-Mounted Displays (HMD) and hand controller devices. The proposed system aims to familiarize students with experimental procedures and safety knowledge before conducting actual experiments. Moreover, students will be able to observe microscopic atomic structure models in three dimensions. Our research also includes a five-part, quantitative evaluation system user tests perform using a quantitative questionnaire consisting of five items, including hardware equipment, immersion, education, interaction sense of control, and degree of difficulty. The evaluation results confirm that this system will be helpful to the educational experience of conducting chemistry experiments with scientific evaluation methods, and the proposed system is also expected to have a broad range of applications in many other subjects.

Integration and use of an augmented reality display in a maritime helicopter simulator

Operating a helicopter in offshore wind parks with degraded visual environments from clouds or fog can endanger crew and material due to the presence of unseen obstacles. Utilizing on-board sensors such as LIDAR or radar, one can sense and record obstacles that could be potentially dangerous. One major challenge is to display the resulting raw sensor data in a way that the crew, especially the pilot, can make use of it without distracting them from their actual task. Augmented reality and mixed reality applications play an important role here. By displaying the data in a see-through helmet-mounted display (HMD), the pilot can be made aware of obstacles that are currently obscured by degraded visual conditions or even parts of the helicopter. This can be accomplished in one HMD. No attention sharing between the outside view and a head-down instrument is necessary. The German Aerospace Center (DLR) is continuously aiming at testing and evaluating both flight-proof and consumer grade HMDs. One particular widely known system is the Microsoft HoloLens. DLR will integrate this low-cost HMD into their experimental helicopter. For this, as a first step, a Microsoft HoloLens was integrated into DLR’s Air Vehicle Simulator (AVES). The integration process is detailed. The simulation capabilities are described, especially for conformal, open-loop LIDAR sensor data. Furthermore, first concepts of the display format are shown, and strengths and drawbacks of the HoloLens in a cockpit environment are discussed.

Editorial issue 31.2

This issue contains three papers. In the first paper, M. Hassaballah, Abdelraheem M. Aly, and Ahmad Abdelnaim, from South Valley University Qena, Egypt, propose for the Interactive simulations of fluids flow to obtain the stable pressure from modifications in the source term of pressure Poisson equation (PPE). The particles disorders are solved using a shifting technique with the current treatment of source term in PPE. In addition, the dummy boundary particles are used for the rigid boundary treatment. Performance of the stabilized ISPH model is tested on various numerical simulations with largely distorted free surface including liquid sloshing problems, fluid–fluid, fluid–structure interactions and dam-break over horizontal and inclined plane. To extend the applicability of the stabilized ISPH model, the postprocess including visual realism with the highly rendering scheme is coupled. In the second paper, Xianxuan Lin, Wang Xun, Jinyu Li, Bailin Yang, Kaili Zhang, Tianxiang Wei, from Zhejiang Gongshang University in Hangzhou, China, and Frederick W. B. Li, from Durham University, UK, propose a color migration framework for home scene images. It picks the coloring from a template image and transforms such coloring to a home scene image through a simple interaction. Their framework comprises three main parts. First, they carry out an interactive segmentation to divide an image into local regions and extract their corresponding colors. Second, they generate a matching color table by sampling the template image according to the color structure of the original home scene image. Finally, they transform colors from the matching color table to the target home scene image with boundary transition maintained. Experiment results demonstrated their framework can produce aesthetically attractive home scene coloring. In the last paper, Xiaoyu Cui, Ruifan Cai, Xiangjun Tang, Xiaogang Jin, from Zhejiang University, in Hangzhou, China, and Zhigang Deng, from University of Houston, Texas, United States, propose a novel shape-constrained fireworks simulation method with rich textures in a helmet-mounted display virtual environment using sketched feature lines as input. Their approach first retrieves an object from a three-dimensional (3D) model database using a sketch-based 3D shape retrieval algorithm. Then, in order to approximate models with complex structures, they introduce a novel point sampling algorithm based on Gaussian curvatures, which stores not only the positions of the selected vertices but also the texture (UV) coordinates information for texture display. In addition, they introduce a multilevel explosion process so that the fireworks can dynamically form specific, visually pleasing shapes. Through their experiments, they demonstrate that their approach can produce better results than state-of-the-art approaches.

Sketch‐based shape‐constrained fireworks simulation in head‐mounted virtual reality

AbstractIn this paper we present a novel shape‐constrained fireworks simulation method with rich textures in an HMD (Helmet Mounted Display) virtual environment using sketched feature lines as input. Our approach first retrieves an object from a three‐dimensional (3D) model database using a sketch‐based 3D shape retrieval algorithm. Then, in order to approximate models with complex structures, we introduce a novel point sampling algorithm based on Gaussian curvatures, which stores not only the positions of the selected vertices but also the texture (UV) coordinates information for texture display. In addition, we introduce a multilevel explosion process so that the fireworks can dynamically form specific, visually pleasing shapes. Through our experiments, we demonstrate that our approach can produce better results than state‐of‐the‐art approaches.

With flying colours: Pilot performance with colour-coded head-up flight symbology

The manipulation of colour in display symbology design has long been recognised as a method to improve operator experience and performance. Recent developments in colour head-up display (HUD) and helmet-mounted display (HMD) technology underline the necessity to understand the human factors considerations of symbology colour coding against conventional monochrome symbology formats. In this low-fidelity desktop human-in-the-loop experiment, the colour of flight symbology on an overlaid symbology set was coded as a redundant cue to indicate the accuracy of professional and non-professional pilots’ flight profile across a range of simulated flight manoeuvres. The main finding of this study was that colour coding flight symbology supported the manual flying performance of both professional and non-professional pilots. Notably, colour-coding of the bank indicator and airspeed tape minimised performance error during turning and altitude change manoeuvres, respectively. The usability of colour coded symbology was also rated higher than the monochrome symbology. We conclude that colour coded HUD/HMD symbology is preferred by the user and may improve performance during low workload manual flying tasks. A fuller understanding of performance and workload effects will require future studies to employ higher workload flying tasks and examine the utility of colour coding within higher fidelity environments.

Joint Helmet-Mounted Cueing System and Neck Muscle Activity During Air Combat Maneuvering.

BACKGROUND: The aim of the study was to determine the characteristics of cervical muscle activity in different head postures when using helmet-mounted display in one fighter vs. two aircraft air combat within visual range (WVR).METHODS: Cervical EMG was measured with eight F/A-18 pilots using the Joint Helmet Mounted Cueing System (JHMCS) during air combat maneuvering. In-flight Gz acceleration and continuous head position were recorded. EMG activity is divided and presented in a matrix with three-class rotation and five-class flexion-extension postures.RESULTS: The mean muscle activity in sternocleidomastoids and cervical extensors was 28.9% of maximal voluntary contraction (MVC) and 44.8% MVC, respectively. Cervical flexor and extensor muscles are subjected to loading over MVC during high Gz sorties. Cervical rotation combined with extension exceeded muscle force-producing capacity during high Gz, resulting in a decline in muscle activity.DISCUSSION: Awkward postures, especially rotational ones, are more prone to increase loading over muscles' capacity. Overloading of muscles increases the risk of muscular and ligamentous injury. In addition, the lack of muscular support potentially leads to the Gz loading being transferred to spinal structures via intervertebral discs and the vertebral column. The JHMCS helmet seems to change the pattern of most loading muscles toward the extensor (posterior) neck muscles.Sovelius R, Mäntylä M, Heini H, Oksa J, Valtonen R, Tiitola L, Leino T. Joint helmet-mounted cueing system and neck muscle activity during air combat maneuvering. Aerosp Med Hum Perform. 2019; 90(10):834-840.

A psychophysical and questionnaire investigation on the spatial disorientation triggered by cockpit layout and design

This study assessed Taiwanese air force combat jet pilots' ability to cope with spatial disorientation (SD) triggered by cockpit layout and design. The measurement items of psychophysical questionnaire were ranked by 325 air force combat jet pilots. This study prioritized the causes of SD perceived by combat jet pilots resulted from the unfamiliarity with instruments on aircraft undergoing flight training, the aircraft's cockpit canopy and the layout of instruments in the cockpit. A significant difference was found between with-SD experience and without-SD experience subjects' responses from adapting to the size of instruments in the cockpit, and from the helmet-mounted display (HMD). Considerable association was discernible between the position of instruments in the cockpit and combat jet models accrued; as well as between the helmet-mounted display (HMD) and combat jet models accrued. The study identify main cockpit layout and design causing SD of Taiwanese pilots based on combat jet models to eliminate the risk of SD, intensify SD training on preventing SD and provide pilots with skills to anticipate and assess the risk of SD during mission planning, and provide references to the military for future procurement of new aircraft models. Relevance to industryThese results provide important information to the defense manufacturers for designing the cockpit layout of combat jets to prevent SD of specific pilots. Finally, the results show the contribution of ergonomics to support military operational commanders in the design of training and prevention of SD.

Hybrid Helmet Attitude Tracking System using Adaptive Complementary Filter

This paper describes the design and performance analysis of HTS (Helmet attitude Tracking System), providing coincident LOS (Line of Sight) between aircraft and HMD (Helmet Mounted Display) which displays flight and mission information on Pilot helmet. The functionality and performance of the HMD system depends on the performance of HTS (Helmet Tracking System). The goal of HTS system design is to meet real-time performance by synchronizing the latency of non-periodic sensory systems and high accurate tracking performance. To prove the performance of the proposed approach using ACF (Adaptive Complementary Filter), a robust hybrid attitude tracking scheme with a fusion of optical sensor and inertial sensor, it took an experiment through an implemented test-bed. The experimental results show accurate and reliable tracking performance despite external errors.