Pilot Task Research Articles

Holographic visual cues for mission task elements

AbstractMission task elements (MTEs) are commonly used to evaluate the handling qualities (HQs) of rotorcraft. However, the traditional approach of physical ground courses faces challenges due to evolving rotorcraft designs and flight profiles. Infrastructure limitations and lack of flexibility for research pose significant obstacles, particularly for high-speed tasks. The emergence of new air vehicles, such as eVTOLs in the civil sector and future vertical lift configurations in the military domain, requires an adapted visual cueing process for previously unaddressed mission profiles. This paper proposes an innovative solution: an augmented reality (AR) system utilizing a head-mounted display to create virtual MTE courses called holographic visual cues (HVCs). A piloted simulation campaign performed at DLR’s AVES simulator compares the effectiveness of HVCs with dome projection-based visual cues, evaluating the performance of the AR system. The study investigated the impact of holographic visual cues on pilot handling qualities, workload ratings, and task performance, finding that although these cues typically do not significantly alter pilot ratings, individual responses varied, highlighting both the technology’s potential and the need for further refinement.

The usefulness of subjective task load assessment methods for predicting pilot task load in general aviation organizations

The usefulness of subjective task load assessment methods for predicting pilot task load in general aviation organizations

Dark light enhancement for dark scene urban object recognition

AbstractThis paper presents a low‐light image enhancement method to improve the performance of autonomous piloting tasks based on deep learning methods. In the low light environment, camera sensors cannot capture enough effective photon signals causing poor performance in vision‐based autonomous piloting. Moreover, the lack of training data makes single‐frame low‐light enhancement and denoising algorithms hard to generalize in real‐world scenarios. By analyzing the noise patterns of real‐world cameras under low‐light environments, a noise generation method is proposed to mimic real‐world dark‐light noise and generate noisy‐clean data pairs for training. A method that can calibrate the camera shot noise parameters is then designed to learn low‐light enhancement from the synthesized noisy data pairs. The neural network trained on a synthesized dataset can effectively enhance the dark light image quality. Consequently, the network improves the downstream auto‐piloting applications such as object detection and semantic segmentation. Qualitative and quantitative experiments have been conducted to demonstrate that the method outperforms previous methods in low‐light enhancement.

Efficiency of the Brain Network Is Associated with the Mental Workload with Developed Mental Schema.

The study of mental workload has attracted much interest in neuroergonomics, a frontier field of research. However, there appears no consensus on how to measure mental workload effectively because the mental workload is not only regulated by task difficulty but also affected by individual skill level reflected as mental schema. In this study, we investigated the alterations in the functional brain network induced by a 10-day simulated piloting task with different difficulty levels. Topological features quantifying global and local information communication and network organization were analyzed. It was found that during different tests, the global efficiency did not change, but the gravity center of the local efficiency of the network moved from the frontal to the posterior area; the small-worldness of the functional brain network became stronger. These results demonstrate the reconfiguration of the brain network during the development of mental schema. Furthermore, for the first two tests, the global and local efficiency did not have a consistent change trend under different difficulty levels, but after forming the developed mental schema, both of them decreased with the increase in task difficulty, showing sensitivity to the increase in mental workload. Our results demonstrate brain network reconfiguration during the motor learning process and reveal the importance of the developed mental schema for the accurate assessment of mental workload. We concluded that the efficiency of the brain network was associated with mental workload with developed mental schema.

Tracking shock movement on the surface of an oscillating, straked delta wing

Limit cycle oscillations (LCO), a mechanical nonlinearity similar to flutter, have affected high performance aircraft, like the F-16 and F-18, for years. The phenomena causes a lateral motion of the cockpit that makes performance of typical pilot tasks difficult. To better understand the nature of LCO and why high performance aircraft are typically afflicted, a computational model was developed to compare to wind tunnel tests and investigate the flow field around a straked, semispan delta wing and monitor the changes as the wing was pitched in an oscillatory fashion. The oscillations were intended to mimic LCO. By understanding the flow field around an oscillating wing, the fluid force that acts as the source of the motion could be discerned. The computational investigation focused on tracking shock movement along the top surface of the wing to confirm the presence of shock-induced trailing edge separation, one possible driver of LCO. The surface pressure was shown to be sensitive to the starting trim angle, oscillation frequency and oscillation amplitude. A separation bubble aft of the shock was observed as the source of sharp, localized negative surface pressure coefficient. The large change in surface pressure caused by the separation bubble could be an important phenomena contributing to the onset of LCO.

Design of automatic aircraft parking system module visually guidance display using ESP32 Microcontroller as practical media

A visual guidance display is a module that was created to assist the pilot task when parking his plane or helping to control aircraft parking at the airport. Up to now, the pilot task is still assisted by a parking attendant (Marshaller) whose aim is to guide the movement of the aircraft to the parking lot. So, a practicum module was made as a medium of learning and knowledge along with the use of alternative technology. This module uses a Visual Guidance Display (automatic aircraft parking aid). This module design uses 3 32x8 pixel dot matrix display boards with dimensions of 13x3.5 cm which are combined into one to form an appropriate combination to produce a parking direction display. The display used is a super right led dot matrix. The controller is an ESP32 microcontroller and a series of shift registers to display the right, left, straight, and stop motion views as well as the distance display of the aircraft to the parking stop point on the apron.

Deep clustering variational network for helicopter regime recognition in HUMS

The helicopter health and usage monitoring system is the core system to ensure its operational safety. Flight regime recognition is a key pilot task therein that affects subsequent decision-making. However, the current research on this topic has not aroused wide attention. Taking advantage of deep learning, a powerful pattern recognition tool, we proposed a deep clustering variational network to serve the helicopter regime recognition task. Through explicit feature distribution constraints and clustering loss function, we have made a clearer decision boundary and more significant category differences, thus achieving accurate recognition results. Two case studies show that deep clustering variational network can effectively recognize the regimes by utilizing vibration signals in time between overhaul experiments or online flight parameters.

Neuroadaptive Training via fNIRS in Flight Simulators.

Training to master a new skill often takes a lot of time, effort, and financial resources, particularly when the desired skill is complex, time sensitive, or high pressure where lives may be at risk. Professions such as aircraft pilots, surgeons, and other mission-critical operators that fall under this umbrella require extensive domain-specific dedicated training to enable learners to meet real-world demands. In this study, we describe a novel neuroadaptive training protocol to enhance learning speed and efficiency using a neuroimaging-based cognitive workload measurement system in a flight simulator. We used functional near-infrared spectroscopy (fNIRS), which is a wearable, mobile, non-invasive neuroimaging modality that can capture localized hemodynamic response and has been used extensively to monitor the anterior prefrontal cortex to estimate cognitive workload. The training protocol included four sessions over 2 weeks and utilized realistic piloting tasks with up to nine levels of difficulty. Learners started at the lowest level and their progress adapted based on either behavioral performance and fNIRS measures combined (neuroadaptive) or performance measures alone (control). Participants in the neuroadaptive group were found to have significantly more efficient training, reaching higher levels of difficulty or significantly improved performance depending on the task, and showing consistent patterns of hemodynamic-derived workload in the dorsolateral prefrontal cortex. The results of this study suggest that a neuroadaptive personalized training protocol using non-invasive neuroimaging is able to enhance learning of new tasks. Finally, we outline here potential avenues for further optimization of this fNIRS based neuroadaptive training approach. As fNIRS mobile neuroimaging is becoming more practical and accessible, the approaches developed here can be applied in the real world in scale.

Measured effects of workload and auditory feedback on remote pilot task performance

Absent or reduced sensory cueing can deprive pilots operating remotely piloted aircraft beyond visual line of sight (BVLOS) of vital information necessary for safe flight. The present study tested the effects of real-time auditory feedback on remote pilot perception and decision-making task performance in an automated BVLOS flight, under three levels of workload (Low, Moderate and High). Results from 36 participants revealed workload and auditory feedback influenced perception task performance in terms of error type count, with misses more frequent than wrong identifications. In terms of performance in the decision-making task, under low and moderate levels of workload, auditory feedback was found to improve performance. Conversely, under high workloads, an inflexion or tipping point occurred whereby auditory feedback became detrimental to task performance. These results correspond with the expected behavioural responses to external stressors as predicted by the Arousal and Maximal Adaptability theory, and build upon previous findings related to workload, auditory feedback and remote pilot task performance. Practitioner summary: This study tested the effect of real-time auditory feedback and dynamic workloads on remote pilots’ task performance. Auditory feedback and workload each influenced the perception tasks in terms of error types committed. Auditory feedback improved decision-making task performance under low and moderate workloads, and reduced performance under high workloads. These results may benefit practitioners by considering the nuanced effects of auditory feedback on human task performance within sensory deprived working environments, including those utilising teleoperated systems.

Experimental evaluation of tasking and teaming design patterns for human delegation of unmanned vehicles

This work discusses different approaches for the cooperation between humans as a supervisor and multiple unmanned vehicles (UVs). We evaluated the most promising approach experimentally with expert pilots of the German Air Force. The co-agency of humans and highly automated unmanned systems (i.e., human autonomy teaming, HAT) is described by the use of a design and description language for HAT design patterns. This design language is used to differentiate control modes for tasking, teaming, and swarming of UVs. The different control modes are then combined in a planner agent (PA) design pattern that further enables the UV guidance on scalable delegation levels from a single individual up to a team. The desired system behavior and interaction concept of the PA for these scalable delegation levels is then transferred to the domain of manned-unmanned teaming in fighter aircraft missions. To demonstrate the applicability of the system, we implemented the concept into our fast-jet simulator of the Institute of Flight Systems (IFS) and conducted an experimental campaign with expert pilots. The results of the experiment showed (1) task delegation with the PA design pattern is faster and reduces the error potential; (2) scalable delegation levels enable a pilot and situation-specific task delegation; (3) the delegation of teams is faster and reduces the error potential; however, in some situations, deeper access through the scalable delegation levels is needed; (4) the concept is intuitive and the transparency and trust in UVs and swarms were very high; and (5) the pilots could imagine operating such systems in the future. Overall speaking the presented PA design pattern is suited for the guidance of UVs and the scalable delegation levels are beneficial.

Autonomous flight cycles and extreme landings of airliners beyond the current limits and capabilities using artificial neural networks

We describe the Intelligent Autopilot System (IAS), a fully autonomous autopilot capable of piloting large jets such as airliners by learning from experienced human pilots using Artificial Neural Networks. The IAS is capable of autonomously executing the required piloting tasks and handling the different flight phases to fly an aircraft from one airport to another including takeoff, climb, cruise, navigate, descent, approach, and land in simulation. In addition, the IAS is capable of autonomously landing large jets in the presence of extreme weather conditions including severe crosswind, gust, wind shear, and turbulence. The IAS is a potential solution to the limitations and robustness problems of modern autopilots such as the inability to execute complete flights, the inability to handle extreme weather conditions especially during approach and landing where the aircraft’s speed is relatively low, and the uncertainty factor is high, and the pilots shortage problem compared to the increasing aircraft demand. In this paper, we present the work done by collaborating with the aviation industry to provide training data for the IAS to learn from. The training data is used by Artificial Neural Networks to generate control models automatically. The control models imitate the skills of the human pilot when executing all the piloting tasks required to pilot an aircraft between two airports. In addition, we introduce new ANNs trained to control the aircraft’s elevators, elevators’ trim, throttle, flaps, and new ailerons and rudder ANNs to counter the effects of extreme weather conditions and land safely. Experiments show that small datasets containing single demonstrations are sufficient to train the IAS and achieve excellent performance by using clearly separable and traceable neural network modules which eliminate the black-box problem of large Artificial Intelligence methods such as Deep Learning. In addition, experiments show that the IAS can handle landing in extreme weather conditions beyond the capabilities of modern autopilots and even experienced human pilots. The proposed IAS is a novel approach towards achieving full control autonomy of large jets using ANN models that match the skills and abilities of experienced human pilots and beyond.

Editorial: Special Issue on the Eighth Dialog System Technology Challenge

The 11 papers in this special section that were part of the Dialog System Technology Challenge. Research competitions have been a long-standing and valuable tradition in the speech and language community. They accelerate the development of new technologies by establishing a shared testbed, cross-validating many potential approaches, and attracting new researchers to the field. DSTC, the Dialog System Technology Challenge, has been a premier research competition for Dialog Systems since its inception in 2013. The earlier Dialog State Tracking Challenges focused on developing a single component for dialog state tracking on goaloriented human-machine conversations. Then, DSTC4 (2015) and DSTC5 (2016) introduced human-human conversations and started to offer multiple tasks not only for dialog state tracking, but also for other components in dialog systems as the pilot tasks. From the sixth challenge (2017), the DSTC has rebranded itself as “Dialog System Technology Challenge” and organized multiple main tracks in parallel to address a wider variety of end-to-end dialog related problems. In line with recent editions of this challenge, DSTC8 focuses on applying end-to-end technologies to Dialog Systems in a pragmatic way. We opened the call for track proposals to the dialog research community and finally held the four parallel main tracks: (1) Multi-domain Task Completion, (2) NOESIS II: Predicting Responses, Identifying Success, and Managing Complexity in Task-Oriented Dialogue, (3) Audio Visual SceneAware Dialog Track, and (4) Scalable Schema-Guided Dialogue State Tracking. This special issue presents more in-depth details of the challenge outcomes.

The estimation of aircraft control system stability boundaries by the describing function method

In this paper, the conditions for periodic modes formation in the closed-loop pilot-vehicle system in the compensatory control mode are established. The case of high gain pilot task is considered under the conditions of a sudden disturbance, as well as under the influence of the system nonlinearities, such as actuator position and rate limit. Sinusoidal input describing function method and parametric resonance equation were used to determine the onset conditions for self-oscillations and forced oscillations. The case when this methods lead to erroneous results is established. The numerical limits of permissible pilot gain, time delay and reference signal at which unstable periodic modes do not arise are calculated.

Methodology for assessing dependencies between factors influencing airline pilot performance reliability: A case of taxiing tasks

Pilot performance reliability is critical to civil air transportation safety. Internal and external factors influencing pilot performance reliability are important elements to estimate human error probability. However, dependencies between these factors, including state dependencies and effect dependencies, are always ignored or only partially considered in most human reliability analysis (HRA) method. This paper attempts to develop a methodology incorporating both state and effect dependencies between contributory factors into HRA for airline pilot tasks based on fuzzy logic and the Cognitive Reliability and Error Analysis Method (CREAM). Anchor points were determined for each common performance condition (CPC) state levels through airline pilot elicitation process and then corresponding state and effect fuzzy sets were constructed for each CPC. The CPC dependency fuzzy inference rules were constructed and transformed with reference to the CPC dependency assessing rule in CREAM. And a cognitive failure probability (CFP) calculation method based on the final CPC effect fuzzy sets and the extended CREAM was proposed. To illustrate the use of the proposed methodology, a case study about commercial airline pilot taxiing task was given. The results demonstrate the validity of the proposed methodology and its capability of representing the ambiguity and fuzziness of expert judgement.

Development and validation of a pilot activity load index (PALI) based on NASA-TLX template

Introduction: Workload can be defined as the hypothetical construct that represents the cost incurred by a human operator to achieve a particular level of performance. Several methods have been used to assess the workload. One of the most common methods of assessing mental workload is NASA-TLX method. The present study was conducted to develop and validate of a pilot activity load index (PALI) based on NASA-TLX template. Materials and Methods: This study was a cross-sectional and descriptive-analytical that was carried out among airline pilots of Civil Aviation Organization. At first, “Pilot Activity Load Index” or PALI subscales were developed based on NASA-TLX template. Validity of the subscales was evaluated by Content Validity Ratio (CVR) and Content Validity Index (CVI).Using Lawshe’s table. Finally, to determine the internal consistency of the subscales, 30 pilots completed the final version of the PALI in typical flight phases. The data obtained were then analyzed by SPSS using Cronbach's alpha to measure the reliability of the subscales and considering an acceptance level of 0.7. Results: The first results led to the emergence of an initial list with 17 subscales. Eleven subscales were irrelevant, redundant and impractical to use in operational environment, due to CVR less than 0.75 and were omitted from the subscales list. Finally, six PALI subscales were approved related to pilot tasks (CVI=0.79). The Verified subscales had a Cronbach's alpha value of 0.89 and were therefore considered a reliable tool. Conclusion: The results of this study showed that PALI questionnaire which validity and reliability has been approved can be used for assessment of workload in pilots.

Optimizing CNN Hyperparameters for Mental Fatigue Assessment in Demanding Maritime Operations

Human-related issues play an important role in accidents and causalities in demanding maritime operations. The industry lacks an approach capable of preventively assessing maritime operators’ mental fatigue and awareness levels before accidents happen. Aiming to reduce intrusiveness, we focused on improving the mental fatigue assessment capabilities of a combination of electroencephalogram and electrocardiogram sensors by investigating the optimization of convolutional neural networks by Bayesian optimization with Gaussian process. We proposed a mapping function to optimize the network structure without the need for a tree-like structure to define the domain of variables for the optimization process. We applied the proposed approach in a simulated vessel piloting task. Even though the mental fatigue assessment for the cross-subject case is a complex classification task, the trained convolutional neural network could achieve good generalization performance (97.6% test accuracy). Finally, we also proposed a method to improve the depiction of the mental fatigue build up process. The framework presented in this work can contribute for reducing accident risk in maritime operations by improving the accuracy and assessment quality of neural network-based mental fatigue assessment tools.

Transparency in Autonomous Teammates

Human–Machine teaming is a very near term standard for many occupational settings and still requires considerations for the design of autonomous teammates (ATs). Transparency of system processes is important for human–machine interaction and reliance but standards for its implementation are still being explored. Embedding social cues is a potential design approach, which may capture the social benefits of a team environment, yet vary with task setting. The current study examined the manipulation of transparency of benevolent intent from an AT within a piloting task requiring suppression of enemy defenses. Specifically, the benevolent AT maintained task communication as in a neutral condition, but included messages of support and awareness of errors. Benevolent communication reduced reported workload and increased reported team collaboration, indicating that this team intent was beneficial. In addition, trust and acceptance of the AT were rated higher by individuals tasked with depending on the system to protect them from missile threats. The need for information from ATs is beneficial, however may vary depending on team type.

Advancements in Predictions of Flying Qualities, Pilot-Induced Oscillation Tendencies, and Flight Safety

This work discusses the various groups of criteria for the prediction of the flying qualities (FQs) and pilot-induced oscillations developed at the Pilot-Vehicle Laboratory of the Moscow Aviation Institute over the last 25 years. The first criteria group includes requirements for the parameters of highly augmented aircraft. Four of these criteria for FQ and pilot-induced oscillation predictions were considered, and the same modification principles were proposed for all the criteria. The second criteria group is based on the requirements for the pilot–aircraft system, and some of the criteria developed at the Pilot-Vehicle Laboratory are discussed. The third criteria group developed at the Moscow Aviation Institute is used to calculate the pilot’s ratings group. Some of these criteria for single-loop, multichannel, and multimodal piloting tasks were considered. The relationship between the requirements for flight safety and the FQs demonstrates the necessity to modify these requirements for several aircraft classes.

A text classification framework for simple and effective early depression detection over social media streams

With the rise of the Internet, there is a growing need to build intelligent systems that are capable of efficiently dealing with early risk detection (ERD) problems on social media, such as early depression detection, early rumor detection or identification of sexual predators. These systems, nowadays mostly based on machine learning techniques, must be able to deal with data streams since users provide their data over time. In addition, these systems must be able to decide when the processed data is sufficient to actually classify users. Moreover, since ERD tasks involve risky decisions by which people’s lives could be affected, such systems must also be able to justify their decisions. However, most standard and state-of-the-art supervised machine learning models (such as SVM, MNB, Neural Networks, etc.) are not well suited to deal with this scenario. This is due to the fact that they either act as black boxes or do not support incremental classification/learning. In this paper we introduce SS3, a novel supervised learning model for text classification that naturally supports these aspects. SS3 was designed to be used as a general framework to deal with ERD problems. We evaluated our model on the CLEF’s eRisk2017 pilot task on early depression detection. Most of the 30 contributions submitted to this competition used state-of-the-art methods. Experimental results show that our classifier was able to outperform these models and standard classifiers, despite being less computationally expensive and having the ability to explain its rationale.

Influence of human-machine interactions and task demand on automation selection and use

A seminal work by Sheridan and Verplank depicted 10 levels of automation, ranging from no automation to an automation that acts completely autonomously without human support. These levels of automation were later complemented with a four-stage model of human information processing. Next, human-machine cooperation centred models and associated cooperation modes were introduced. The objective of the experiment was to test which human-machine theorie describe automation use better. The participants were asked to choose repeatedly between four automation types (i.e. no automation, warning, co-action, function delegation) to complete three multi-attribute task battery tasks. The results showed that the participants favour the selection of automation types offering the best human-machine interactions quality rather that the most effective automation type. Contrary to human-machine cooperation models, technology centred models could not predict accurately automation selection. The most advanced automation was not the most selected.Practitioner Summary: The experiment dealt with how people select different automation types to complete the multi-attribute task battery that emulates recreational aircraft pilot tasks. Automation performance was not the main criteria that explain automation use, as people tend to select an automation type based on the quality of the human-machine cooperation.