Automated Performance Measurement Research Articles

Algorithmic Performance Management in Higher Education: Viva! 365 Ways of Surveillance

This paper maps the emergence and consequences of automated Algorithmic Performance Management (APM) in the context of higher education. After reviewing the evolution of productivity management in academia, it argues that surveillance via APM shifts expectations not just about effectiveness at work but also about how work, and the good worker, come to be defined. In our paradigmatic case study of Office 365, we specify how the automated surveillance of workforce practices are deployed to redefine productivity in higher education: productive workers become good data subjects as well as producers of papers, grants, and other traditional outputs of success. Our analysis suggests performing well at work is managed in and by the platform via logics of the surveillance of wellness, time-regulation, and social connectivity to influence, manage, and control workers. We critique these automated performance measures in terms of platform capitalism, noting Office 365’s Viva Insights function as a telematic device of surveillance. The final section of the paper places these trends in Australia’s socio-legal context by showing how Viva is insufficient for considering performance given the range of practices that constitute “academic work,” including but not limited to the need for unmonitored activity. Yet, we observe that currently little can be done about Office 365’s surveillant presence given a regulatory regime that by and large excludes productivity surveillance from the scope of regulated surveillance activities.

The Virtual Kitchen Challenge: preliminary data from a novel virtual reality test of mild difficulties in everyday functioning

ABSTRACTBackground: Efficient, objective measures of mild functional difficulties are lacking. Preliminary data from a novel, non-immersive virtual reality, performance-based task (Virtual Kitchen Challenge; VKC) were obtained to address this gap.Methods: 14 older and 21 younger adults completed cognitive tests and two everyday tasks (breakfast, lunch) in the VKC with virtual objects and a touch-screen and in the Real Kitchen with real objects (order counterbalanced). Automated performance measures were obtained from the VKC program and human coders scored VKC and Real Kitchen videos for errors.Results: Older adults made more errors than younger adults on the VKC and Real Kitchen, with similar error patterns across measures. VKC automated measures were significantly related to measures from human coders, performance on the Real Kitchen, and cognitive test scores.Conclusion: The VKC is a valid and highly efficient performance-based measure of subtle functional difficulties with great potential for future clinical and research applications.

Automated performance measurement for 3D building modeling decisions

Abstract Building information modeling (BIM) is instrumental in documenting design, enhancing customer experience, and improving product functionality in capital projects. However, high-quality building models do not happen by accident, but rather because of a managed process that involves several participants from different disciplines and backgrounds. Throughout this process, the different priorities of design modelers often result in conflicts that can negatively impact project outcomes. To prevent such unwanted outcomes from occurring, the modeling process needs to be effectively managed. This effective management requires an ability to closely monitor the modeling process and correctly measure the modelers' performance. Nevertheless, existing methods of performance monitoring in building design practices lack an objective measurement system to quantify modeling progress. The widespread utilization of BIM tools presents a unique opportunity to retrieve granular design process data and conduct accurate performance measurements. This research improves upon previous efforts by presenting a novel application programming interface (API)-enabled approach to (a) automatically collect detailed model development data directly from BIM software packages in real-time, and (b) efficiently calculate several modeling performance measures during schematic and design development phases of building projects. These indicators can be used to properly arrange modeling teams in the quest for high-quality building models. The specific objectives of this study to examine the feasibility of a proposed automated design performance measurement framework, and to identify optimal modeling team configurations using empirical performance information. A passive data recording approach allows for the real-time capture of comprehensive user interface (UI) interaction and model element modification events. The proposed framework is implemented as an Autodesk Revit plugin. Next, an experiment is conducted to capture data using the developed Revit plugin. Experiment participants' individual production rates are estimated to establish the validity of the proposed approach to identify the optimal design team configuration. The presented approach uses the earliest due date (EDD) sequencing rule in combination with the critical path method (CPM) to calculate the maximum lateness for different design team arrangements.

Application-Driven OAM Framework for Heterogeneous IoT Environments

We present a novel approach for providing a comprehensive operational picture of heterogeneous networks by collecting system information from physical, data-link, network and application layers using extended methods and mechanisms for OAM, which take into account particularities of persistent access heterogeneity and IoT. A heterogeneous OAM (H-OAM) framework is proposed with a toolset for streamlined failure detection and isolation, and automated performance measurement and monitoring. The framework combines extended standardized OAM toolsets for physical and data-link layers with multiple well-defined and recognized IP network layer toolsets, which introduces possibilities for a tactical view of the monitored system and quick root cause analysis with unified interpretation and cross-correlation of horizontal and vertical levels. A practical deployment of an H-OAM system is presented and its use is demonstrated in a live mobile IoT testbed environment where performance is a function of a wide variety of physical layer parameters that need to be tuned and monitored by the operator. Results of two concrete usage scenarios performed in cooperation with two largest Slovenian mobile operators demonstrate how continuity check and connectivity verification, and performance diagnostics are conducted for Web-based IoT applications and network services of a live operational environment.

A scoping review to explore the suitability of interactive voice response to conduct automated performance measurement of the patient's experience in primary care.

Practice-based performance measurement is fundamental for improvement and accountability in primary care. Traditional performance measurement of the patient's experience is often too costly and cumbersome for most practices. This scoping review explores the literature on the use of interactive voice response (IVR) telephone surveys to identify lessons for its use for collecting data on patient-reported outcome measures at the primary care practice level. The literature suggests IVR could potentially increase the capacity to reach more representative patient samples and those traditionally most difficult to engage. There is potential for long-term cost effectiveness and significant decrease of the burden on practices involved in collecting patient survey data. Challenges such as low response rates, mode effects, high initial set-up costs and maintenance fees, are also reported and require careful attention. This review suggests IVR may be a feasible alternative to traditional patient data collection methods, which should be further explored.

Supplementing Signalized Intersection Infrastructure to Provide Automated Performance Measures with Existing Video Detection Equipment

The objective of this work was to retrofit a signalized intersection with the goal of developing event-based performance measures while leveraging existing infrastructure to its fullest extent. A TS1 intersection managed by the Arizona Department of Transportation in Flagstaff was chosen as the test location. In addition to video from the existing cameras, video detector interface cards were installed in the cabinet to provide contact closure outputs of vehicle flow rates. These outputs were then conveyed to an onboard event-based data logger within the traffic controller via TS2 communications. Performance measures were generated [volume-to-capacity (v/c) ratio and cumulative counts] from the video-generated data and compared with manually counted data over a 24-h analysis period. The v/c values generated by the video data were shown to be statistically different from those calculated with the manual count data; however, on all but one phase, the difference was not operationally significant. Analysis of cumulative count data showed operationally significant differences. A cost analysis determined that the price of equipping a similar intersection for this type of vehicle count capability is about $16,000 if the equipment specified for this project is used and if it is installed during the construction of the intersection. It is recommended that jurisdictions interested in developing this type of flow information consider equipping future intersections in a similar manner and that future work be completed in the evaluation of the use of video detection units for flow rate collection at signalized intersections.

Enhancing Team Training and Performance With Automated Performance Assessment Tools

Assessing team performance is a difficult task for instructors because of factors such as high workload. Automated performance measurement systems may provide assistance to instructors and improve the quality of performance assessment. However, little research is available to guide such interventions or to document their effects. This research compared the ability of flight instructors to give diagnostic ratings of team performance with or without computer assistance using the Enhancing Performance With Improved Coordination (EPIC) tool. It was found that the instructor group operating with EPIC (n = 10) had more differentiated and accurate ratings of team performance compared to the control group (n = 10). These results suggest that approaches such as EPIC can serve to complement and augment instructor capabilities, providing more powerful and diagnostic team performance assessment systems.

Making the Case for Critical Realism

This article seeks to address the dearth of practical examples of research in the area by proposing that critical realism be adopted as the underlying research philosophy for enterprise systems evaluation. We address some of the implications of adopting such an approach by discussing the evaluation and implementation of a number of automated performance measurement systems (APMS). Such systems are a recent evolution within the context of enterprise information systems. They collect operational data from integrated systems to generate values for key performance indicators, which are delivered directly to senior management. The creation and delivery of these data are fully automated, precluding manual intervention by middle or line management. Whilst these systems appear to be a logical progression in the exploitation of the available rich, real-time data, the statistics for APMS projects are disappointing. An understanding of the reasons is elusive and little researched. We describe how critical realism can provide a useful “underlabourer” for such research, by “clearing the ground a little ... removing some of the rubbish that lies in the way of knowledge” (Locke, 1894, p. 14). The implications of such an underlabouring role are investigated. Whilst the research is still underway, the article indicates how a critical realist foundation is assisting the research process.

Full-Cycle Adaptive Instructorless Training

For simulators to provide effective mission training, there must be capabilities for scenario selection, briefing, debriefing, and performance evaluation; typically provided by instructors. This paper describes an approach for automated scenario selection, guided brief and AAR, and automated performance measurement. Underlying these intelligent functions is a tight coupling of training objectives, performance measures, and scenario events derived through extensive cognitive task analysis. Training objectives drive scenario selection and instantiation with events that provide opportunities to demonstrate related competencies; automated performance measures then monitor and evaluate user actions in response to these events; and are in turn stored and aggregated for use in a guided after-action review (AAR). A cognitive agent conducts the AAR in much the same way as a real instructor would, tailoring it to the user's performance, selecting the most important performance measures to discuss, highlighting aspects of importance to the mission, and summarizing the user's performance. Since each performance measure is related to a training objective, the user's training profile can be updated for each training objective after the scenario, and subsequent scenarios can be selected to focus on training objectives that have not been achieved, beginning the cycle anew.

Macroergonomic Principles of Automated Performance Measurement

Retail associate productivity has increased dramatically over the last 15 years with the use of ever more sophisticated scanners and point of sale (POS) terminals. Data warehouses enable businesses to store remarkable detail about each transaction. Analyses of these data tend to focus on the items sold and money collected. However, if these systems also collect timestamp data of important events during transactions retailers would then have the basis for measuring associate activity. This paper discusses the macroergonomic principles that must guide the design of cashier performance measurement systems in the retail setting. Following these guidelines leads not only to increased manager effectiveness, but also to more accurate measurement of activity. It can also improve associates morale and the likelihood of incenting desirable and productive behaviors. These principles are generalizable to other domains in which automated performance measurement takes place.

A Method for Exploring Teamwork Modeling and Assessment within a Multiplayer Game

Massively multi-player games (MMPGs) have the potential to enable training at a level of participation, intensity, and fidelity previously unrealized. As a first step towards the implementation of automated performance measurement technology for MMPGs, we explored an approach to developing training and assessment of team performance during urban infantry operations simulated within the Asymmetric Warfare Team Training Technology (AW-VTT) under development by Forterra Systems and RDECOM-STTC. Our method entailed (1) mapping a theoretical framework to established Army doctrine, (2) mapping doctrinal descriptions to specific activities within a simple yet operationally-valid scenario, and (3) developing rule-based descriptions of these activities and formally representing them within finite state networks to validate their consistency and completeness. We then enacted the scenario within AW-VTT to demonstrate the environment's capability for simulating the behaviors required for assessment of teamwork in this scenario. We describe this approach and discuss lessons learned.

How well do automated performance measures assess guideline implementation for new-onset depression in the Veterans Health Administration?

Because most guidelines focus on patients with new episodes of depression, algorithms to identify such samples must be accurate. This study examined whether the Veterans Health Administration's (VHA's) electronic medical record database could identify valid cases of new-onset depression. Of 109 individuals receiving outpatient care at one of three VHA medical centers who were identified with newly diagnosed depressive disorder, 39 (35.8%) actually had documentation of depression diagnosis and antidepressant prescription or other treatment within the previous six months. Good to excellent agreement was found between indicators of guideline-concordant care using automated and manual chart review methods. Electronic medical records can validly identify many cases of new-onset depression, although with a higher-than-anticipated rate of false-positives. Half of depressed veterans received care consistent with clinical guidelines for psychopharmacological intervention, regardless of data source. Clinical managers, administrators, and policy advocates must weigh the cost-benefit of administrative versus medical record reviews to assess quality.

I'Ve Got Synthers, Who Could Ask for Anything More?

Synthetic agents can enrich the environment for practice of complex skills such as airborne command and control. Even in a robust simulation, however, synthetic teammates and opponents are not sufficient to train students. We identify several additional requirements for training: a well-focused curriculum, automated performance measures that drive feedback, strategies for managing real-time feedback and off-line instruction, and learning aids that provide instructional scaffolding for novice users. We describe methods of fulfilling these requirements and recommend future paths of research in the new domain of Intelligent Team Tutoring Systems.

The Effects of the Ah-64A Pilot's Night Vision System on the Performance of Seven Simulated Maneuver Tasks

The Pilot's Night Vision System (PNVS) allows the pilot of the AH-64A Apache helicopter to fly and navigate effectively at night. The PNVS restricts field of view (FOV) and degrades visual acuity, two factors that should adversely affect pilot performance during night operations, especially for flight maneuvers that rely on peripheral cues. Participants, all AH-64A instructor pilots, flew a mission profile in a high-fidelity simulation of the AH-64A. The profile consisted of seven routine maneuvers that were flown first under daytime (baseline) conditions (using a rear projection display with a 174° FOV) then under night conditions (with a simulated PNVS display with a 40° FOV). Automated performance measures were the main dependent variables. It was expected that pilot performance degradation would be significant on hovering turns, takeoffs, and landings, and negligible on stationary hovering and hover taxiing. Significant performance degradations were found for all maneuver tasks. Applications of the findings to training and pilot workload standards are discussed.

Development of a Prototype Performance Measurement System for Strike Warfare Training Assessment

This paper describes methods for developing automated performance measurement systems used with training ranges and simulators. A prototype automated measurement system designed to assess aircrew performance during strike warfare training is presented as an application of this methodology. Methods are also presented for displaying information which is useful in assessing student progress and for diagnosing training results.

Pilot Oriented Performance Measurement

Flight simulators provide a complete quantitative record of a pilot's flying performance. Evaluating this record is complicated by the volume of data and by its fine detail, dozens of flight parameters, sampled many times per second. Automated performance measurement systems (APMS) reduce the volume of data to an amount which is manageable and understandable. The usual APMS is aircraft state oriented. The APMS keys on aircraft state (e.g., X-Y position, bank angle) to define intervals over which performance data are integrated. This APMS is relatively insensitive to pilots' intentions and so may average performances which had differing objectives, based only on their having occurred at the same point during the task sequence. An alternative APMS has been developed which is pilot oriented. This APMS defines measurement intervals based on control inputs. Control inputs are identified by discrete changes in flight path. These intervals are psychologically relevant in that they begin with a goal-directed control input and end with a countervailing input. By relating performance in the pilot defined intervals to state defined intervals, it is possible to quantify performance on given flight segments (e.g., a level turn), and to specify factors which lead to a given level of performance.

New Technology for Training: An Evaluation of the Air Controller Exerciser (ACE)

This paper describes the evaluation of an experimental prototype training system called the Air Controller Exerciser (ACE). The system was developed to demonstrate the use of new technologies for training including computer speech recognition and generation, videodisc, automated instruction, automated performance measurement, syllabus control, and speech-interactive simulation. Empirical studies were conducted to determine the accuracy of the speech recognition system, validate the performance measurement system, and compare ACE with the traditional training program with a transfer of training test. ACE reasonably showed the potential for the use of new technologies in training. However, a number of changes and improvements would be necessary before ACE would be acceptable as an operational training system.

Subjective Performance Measures in the Test and Evaluation of Simulator Training Scenarios

Scenarios were developed to provide A-10 student pilots training in low level navigation (LLN) and elementary basic attack maneuvers (BAM). The primary objective of this effort was to evaluate the effectiveness of these scenarios using the A-10 Advanced Simulator for Pilot Training (ASPT) as the training medium. Training effectiveness was assessed using simulator automated performance measures and instructor pilot (IP) evaluations of student aircraft performances in a transfer of training paradigm. Forty-two subjects participated in the BAM phase of the experiment: thirty-six subjects were used in the LLN phase. Simulator measures showed enhanced performance as a function of training for three of the five BAM tasks trained: LLN simulator performance data were not analyzed due to difficulty with data collection apparatus. Power values, reliability tests and correlation results for aircraft performance measures made questionable the validity of using these measures for test purposes as was done in this experiment. Discussion focusses on the need for more objective performance metrics in future evaluation programs and reviews alternative solutions to fulfill this requirement.

Operational Performance Measures for Carrier Landing: Development and Application

Fifteen years of carrier landing performance criteria development and applications are described. An offshoot of the research resulted in the emergence of an automated performance measurement and appraisal system designed to enhance the effectiveness of night carrier landing.

Operational Performance Measures for Air Combat: Development and Application

The content of this paper summarizes four years of research designed to develop valid and reliable performance criteria for the Navy's Tactical Aircrew Combat Training System (TACTS). Performance measurement methods for assessing missile envelope recognition and air combat engagements have been developed and applied in an operational setting. TACTS measures used in performance assessment were selected on the basis of their operational importance and their demonstrated statistical relationship to successful completion of such air combat tasks as missile launch success and engagement outcomes. A measurement framework has evolved and may be appropriately applied to estimate overall air combat effectiveness, and to provide diagnostic performance analysis of critical air combat tasks. The resulting measurement framework has been applied operationally to evaluate U.S. Navy competitive air combat exercises and to provide diagnostic performance feedback to aircrews undergoing TACTS training. More recently, these measures and assessment methods have been incorporated in a computer-based TACTS debrief system called the Performance Assessment and Appraisal System (PAAS). The PAAS is representative of an emerging technology which uses automated performance measurement methods for enhancing the training process.