Systems In Real-world Settings Research Articles

Deep learning from \u201cpassive feeding\u201d to \u201cselective eating\u201d of real-world data

Artificial intelligence (AI) based on deep learning has shown excellent diagnostic performance in detecting various diseases with good-quality clinical images. Recently, AI diagnostic systems developed from ultra-widefield fundus (UWF) images have become popular standard-of-care tools in screening for ocular fundus diseases. However, in real-world settings, these systems must base their diagnoses on images with uncontrolled quality (“passive feeding”), leading to uncertainty about their performance. Here, using 40,562 UWF images, we develop a deep learning–based image filtering system (DLIFS) for detecting and filtering out poor-quality images in an automated fashion such that only good-quality images are transferred to the subsequent AI diagnostic system (“selective eating”). In three independent datasets from different clinical institutions, the DLIFS performed well with sensitivities of 96.9%, 95.6% and 96.6%, and specificities of 96.6%, 97.9% and 98.8%, respectively. Furthermore, we show that the application of our DLIFS significantly improves the performance of established AI diagnostic systems in real-world settings. Our work demonstrates that “selective eating” of real-world data is necessary and needs to be considered in the development of image-based AI systems.

Action-based solar education in architecture

This paper presents a classroom-field interactive teaching and research tool, named Interactive Satellite Solar Lab (ISSL) that transcends the physical boundary of conventional classrooms and expands the venue of teaching and learning to out-of-doors. Tethered wirelessly to the classroom, the satellite lab empowers students to examine real-time performance of solar systems in real world settings. While experiments are conducted in remote settings by a team of students, their experiments and collected data are communicated instantly to the classroom instructor and students. The visual images and collected data from field experiments are simulcast in the classroom where the instructor and other students are viewing them. The data gathered from the field are transmitted to the instructor’s and students’ PCs or cellular phones. Activities of onsite experiments can be recorded and projected in the classroom. The field team and the classroom students are able to assess field conditions, make decisions, and perform experiments together. The ISSL was developed by a multidisciplinary team of faculty and students in an effort to transform the environmental dimension of architectural education, and to explore and promote a next generation pedagogic paradigm for science and engineering.

Bridging the "last mile" gap between AI implementation and operation: "data awareness" that matters.

Interest in the application of machine learning (ML) techniques to medicine is growing fast and wide because of their ability to endow decision support systems with so-called artificial intelligence, particularly in those medical disciplines that extensively rely on digital imaging. Nonetheless, achieving a pragmatic and ecological validation of medical AI systems in real-world settings is difficult, even when these systems exhibit very high accuracy in laboratory settings. This difficulty has been called the “last mile of implementation.” In this review of the concept, we claim that this metaphorical mile presents two chasms: the hiatus of human trust and the hiatus of machine experience. The former hiatus encompasses all that can hinder the concrete use of AI at the point of care, including availability and usability issues, but also the contradictory phenomena of cognitive ergonomics, such as automation bias (overreliance on technology) and prejudice against the machine (clearly the opposite). The latter hiatus, on the other hand, relates to the production and availability of a sufficient amount of reliable and accurate clinical data that is suitable to be the “experience” with which a machine can be trained. In briefly reviewing the existing literature, we focus on this latter hiatus of the last mile, as it has been largely neglected by both ML developers and doctors. In doing so, we argue that efforts to cross this chasm require data governance practices and a focus on data work, including the practices of data awareness and data hygiene. To address the challenge of bridging the chasms in the last mile of medical AI implementation, we discuss the six main socio-technical challenges that must be overcome in order to build robust bridges and deploy potentially effective AI in real-world clinical settings.

Do-It-Yourself Artificial Pancreas Systems in Type 1 Diabetes: Perspectives of Two Adult Users, a Caregiver and Three Physicians.

Advances in continuous glucose monitoring and insulin pumps have allowed people with type 1 diabetes (T1D) and caregivers to accurately and continuously measure their glucose levels and make adjustments to insulin infusion. In recent years, algorithms for subcutaneous insulin dosing have been developed that can respond to changes in glucose in an automated fashion and “close the loop”. At present, a first-generation ‘hybrid closed-loop’, ‘artificial pancreas’ or ‘automated insulin dosing’ system, Medtronic 670G, is available commercially. Further systems are in clinical trials. Frustrated by the slow pace of innovation, people affected by diabetes have united online under the hashtag ‘#WeAreNotWaiting,’ to disseminate open-source diabetes technologies. One dimension of #WeAreNotWaiting is “looping” with a do-it-yourself artificial pancreas. Here we provide the perspectives of two adults with T1D, the parent of a child with T1D and three physicians who detail their experience with these systems. These personal and clinical perspectives highlight very clear metabolic and psychological benefits of these systems in real-world settings.

Remote Assessment of Cognitive Impairment Level Based on Serious Mobile Game Performance: An Initial Proof of Concept

Individuals with cognitive impairments are evaluated using clinically validated cognitive assessment tools, which need to be administered by trained therapists. This serves as a major barrier for frequent and longitudinal monitoring of patients' cognitive impairment level. We introduce Neuro-World, a set of six mobile games designed to challenge visuospatial short-term memory and selective attention, which allows one to self-administer the assessment of his/her cognitive impairment level. Game performance is analyzed to estimate a widely accepted clinical measure, the mini mental state examination (MMSE), which highlights the translational impact of the system in real-world settings. We collected game-specific performance data from 12 post-stroke patients at baseline and a three-month follow-up, which were used to train supervised machine learning models to estimate the corresponding MMSE scores. The results presented herein show that the proposed approach can estimate the MMSE scores with a normalized root mean square error of 5.75%. We also validate the system's responsiveness to longitudinal changes in cognitive impairment level and demonstrate the system's positive usability in cognitively impaired individuals and their willingness to adhere to the longitudinal use. This study demonstrates that Neuro-world has great potential to be used to evaluate the cognitive impairment level and monitor its long-term change. This study enables new clinical and research opportunities for accurate, longitudinal assessment of cognitive function via mobile games.

Multiple Myeloma Staging in Real World Clinical Practice Is Suboptimal: Absence of Beta-2-Microglobulin and Serum Lactase Dehydrogenase Testing Are Limiting Factors

Background: Staging systems, which help define prognosis and influence treatment decisions, for myeloma have evolved. The Durie-Salmon system (DS) was based largely on perceived myeloma cell tumor burden. In 2005, the International Staging System (ISS) was introduced and incorporated beta-2-microglobulin levels (a marker of tumor burden) and serum albumin (a marker of inflammatory cytokines). In 2015 the ISS was revised (R-ISS) to include high risk cytogenetics [17p-, t(4;14), t(14;16)} and serum lactase dehydrogenase (LDH) (correlating with tumor aggressiveness, extramedullary disease and proliferation rate). Although the ISS and R-ISS are relatively simple, they require additional laboratory studies outside of the diagnostic biopsy material and routine chemistry panel. The adequacy of staging using these newer systems in real world settings has not been reported, although only 69% of patients (pts) in the R-IPSS foundational study (Palumbo JCO 2015) had all of the elements available for complete staging.Methods: Cases were identified using the COTA Inc. database, which abstracts and organizes relevant demographic, diagnostic, treatment, and quality data from the electronic health records at all the clinical sites under business associate agreements. Trained COTA Inc. abstractors reviewed and confirmed the medical record-derived data. For the purposes of the present study, any laboratory or pathology report mentioning testing was counted as “tested,” regardless of method, vendor, or test completeness. The data were then merged with the study population using blinded identifiers for subsequent analysis. A limitation of this review is failure to document test results (or mention of test performance) in the electronic chart or physician notes, thus tests performed but not documented may not have been counted.Results: 857 pts diagnosed with MM between Jan 1, 2013 and Dec 31, 2016 were identified from 12 northeast USA centers (1 academic and 11 community based) treated by 81 hematologist-oncologists. At the time of MM presentation 831 pts (97%) had EHR documentation of immunoglobulin levels, hemoglobin, calcium concentration, renal function and number of bone lesions to permit Durie-Salmon staging. The ISS requires B2M and albumin; at the time of initial presentation 696 pts (81%) could be classified by the ISS but 161 pts (19%) had no documentation of B2M levels and thus could not be staged. The academic center was more likely to obtain B2M to permit complete ISS (84% academic obtained vs 72% community obtained; p<0.001). The R-ISS incorporates cytogenetic abnormalities and LDH. At the time of initial presentation, in addition to the pts without B2M, 353 pts (41%) had no documentation of LDH testing, 122 pts (14%) had no documentation of t(14;16) testing, 112 pts (13%) were without documentation of t(4;14) testing, and 91 pts (11%) were without documentation of del17 testing. The academic and community centers both failed to collect LDH equally (53% academic obtained vs 56% community; p=0.37; however notable that 18% of the academic patients were referred for transplant after initial staging in the community and may not have source documentation at the Cota center). Thus, only 350 pts (41%) could be classified by R-ISS. There was no difference in adequacy of obtaining R-ISS elements before (40%) and after (44%) publication of the R-ISS in late 2015 (p=0.22). The majority of the pts in the database were DS III at presentation. However DS did not correlate with ISS or R-ISS, thus demonstrating the need to perform the complete diagnostic/staging evaluation [Table1].Conclusions: In real world (non-research) settings staging of multiple myeloma using the ISS and R-ISS that require laboratories beyond diagnostic marrow based material (including cytogenetics) remains suboptimal and may reflect delayed incorporation of R-ISS (2015) into clinical use. Lack of obtaining B2M and LDH levels at initial presentation is the major deficiency.Table 1 Reclassification of DS staged pts by ISS and R-ISS DisclosuresBiran:Amgen: Consultancy, Speakers Bureau; Takeda: Consultancy, Speakers Bureau; Celgene: Consultancy, Honoraria, Speakers Bureau; BMS: Research Funding; Merck: Research Funding. Norden:Cota Inc: Employment. Siegel:Celgene: Consultancy, Honoraria, Research Funding, Speakers Bureau; Amgen: Consultancy, Honoraria, Speakers Bureau; Karyopharm: Consultancy, Honoraria; Novartis: Honoraria, Speakers Bureau; Takeda: Consultancy, Honoraria, Speakers Bureau; Janssen: Consultancy, Honoraria, Speakers Bureau; Merck: Consultancy, Honoraria, Speakers Bureau; BMS: Consultancy, Honoraria, Speakers Bureau. Goldberg:COTA Inc.: Employment, Equity Ownership.

Direct Collocation for Dynamic Behaviors With Nonprehensile Contacts: Application to Flipping Burgers

To realize robotic systems in real-world settings, e.g., in restaurants, it will be necessary to achieve dynamic manipulation of nontrivial objects. In this context, this letter discusses methodologies used to realize trajectories in a robotic arm platform, specifically, applied to flipping burgers as an example of nonprehensile object manipulation. Flipping of burgers involves a series of tasks—going to the burger location, scooping, picking up, and flipping. Since the goal is to obtain these trajectories in a reasonably fast manner, we employ direct collocation based multisegmented trajectory optimization. We will first describe the problem setup, and then describe the constraints, decision variables employed, and then, to conclude, we will demonstrate these behaviors in a 6-DOF robot experimentally.

Stable Contactless Sensing of Vital Signs Using RGB-Thermal Image Fusion System with Facial Tracking for Infection Screening.

Infrared thermography (IRT) has been used to screen febrile passengers in international airports for over a decade. However, fever-based infection screening using IRT suffered from low sensitivity because measurements can be affected by ambient temperature, humidity, etc. In our previous study, we proposed an RGB-thermal image fusion system to measure vital signs i.e., the RGB camera detects tiny changes in color from facial skin, associated with blood flow, to estimate heart rate, and IRT senses temperature changes around the nasal area, caused by respiration, to measure respiratory rate). The inclusion of heart and respiratory rates lead to increased screening accuracy. In the present study, to promote the widespread use of our system in real-world settings, a face detection and tracking method was developed and implemented into the system, thereby enabling the accurate and stable measurement of vital signs. We assessed heart and respiratory rate estimation via an RGB-thermal image fusion system using Bland-Altman plots and statistical analysis.

Plexiglass: Multiplexing Passive and Active Tasks for More Efficient Crowdsourcing

Efficiently scaling continuous real-time crowdsourcing tasks — which engage crowd workers over long periods of time to complete tasks, such as monitoring video for critical events — is challenging largely because of the cost of keeping people consistently engaged. Worse, for many continuous tasks on which progress cannot be immediately made, which we term passive tasks, this engagement effort is wasted until something becomes true about the environment (e.g., the annotation of a critical event cannot happen until the event is seen). In this paper, we present the idea of passive-active task multiplexing, in which continuous tasks that involve waiting for a change in state are completed concurrently with traditional (offline/non-real time) tasks. We then implement this idea in Plexiglass, a system that answers visual queries made by blind and low-vision users more efficiently by concurrently presenting a (passive) real-time sensing task and an (active) offline visual question answering tasks to workers. We explore different approaches to accomplishing this, and validate that task multiplexing can lead to an improvement in efficiency in these settings of more than 40% in terms of overall worker time taken and 85% decrease in cost of running continuous real-time tasks. This work has implications on how real-time crowdsourcing tasks are presented to workers, and it increases the feasibility of deploying continuous real-time crowdsourcing systems in real-world settings.

The relative importance of real-time in-cab and external feedback in managing fatigue in real-world commercial transport operations

ABSTRACTObjective: Real-time driver monitoring systems represent a solution to address key behavioral risks as they occur, particularly distraction and fatigue. The efficacy of these systems in real-world settings is largely unknown. This article has three objectives: (1) to document the incidence and duration of fatigue in real-world commercial truck-driving operations, (2) to determine the reduction, if any, in the incidence of fatigue episodes associated with providing feedback, and (3) to tease apart the relative contribution of in-cab warnings from 24/7 monitoring and feedback to employers.Methods: Data collected from a commercially available in-vehicle camera-based driver monitoring system installed in a commercial truck fleet operating in Australia were analyzed. The real-time driver monitoring system makes continuous assessments of driver drowsiness based on eyelid position and other factors. Data were collected in a baseline period where no feedback was provided to drivers. Real-time feedback to drivers then occurred via in-cab auditory and haptic warnings, which were further enhanced by direct feedback by company management when fatigue events were detected by external 24/7 monitors. Fatigue incidence rates and their timing of occurrence across the three time periods were compared.Results: Relative to no feedback being provided to drivers when fatigue events were detected, in-cab warnings resulted in a 66% reduction in fatigue events, with a 95% reduction achieved by the real-time provision of direct feedback in addition to in-cab warnings (p < 0.01). With feedback, fatigue events were shorter in duration a d occurred later in the trip, and fewer drivers had more than one verified fatigue event per trip.Conclusions: That the provision of feedback to the company on driver fatigue events in real time provides greater benefit than feedback to the driver alone has implications for companies seeking to mitigate risks associated with fatigue. Having fewer fatigue events is likely a reflection of the device itself and the accompanying safety culture of the company in terms of how the information is used. Data were analysed on a per-truck trip basis, and the findings are indicative of fatigue events in a large-scale commercial transport fleet. Future research ought to account for individual driver performance, which was not possible with the available data in this retrospective analysis. Evidence that real-time driver monitoring feedback is effective in reducing fatigue events is invaluable in the development of fleet safety policies, and of future national policy and vehicle safety regulations. Implications for automotive driver monitoring are discussed.

Energy Autonomous Wireless Sensing System Enabled by Energy Generated during Human Walking

Recently, there has been a huge amount of work devoted to wearable energy harvesting (WEH) in a bid to establish energy autonomous wireless sensing systems for a range of health monitoring applications. However, limited work has been performed to implement and test such systems in real-world settings. This paper reports the development and real-world characterisation of a magnetically plucked wearable knee-joint energy harvester (Mag-WKEH) powered wireless sensing system, which integrates our latest research progresses in WEH, power conditioning and wireless sensing to achieve high energy efficiency. Experimental results demonstrate that with walking speeds of 3∼7 km/h, the Mag-WKEH generates average power of 1.9∼4.5 mW with unnoticeable impact on the wearer and is able to power the wireless sensor node (WSN) with three sensors to work at duty cycles of 6.6%∼13%. In each active period of 2 s, the WSN is able to measure and transmit 482 readings to the base station.

EXPANDING JOINT VULNERABILITY ASSESSMENT BRANCH

David Rohret, the founder of the Joint Vulnerability Assessment Branch (JVAB) pondered this difficult question. Since 2003, he had been involved in building a team that was uniquely positioned to identify a wide range of vulnerabilities in military and commercial communications and web-based systems. He could cite numerous examples of past situations where the early use of JVAB’s services led to, or could have led to, tens of millions of dollars in savings—or possibly more, had the issues they detected been left unattended. The value that JVAB offered was gradually being recognized and, as a result, demand for their services was building. The problem was that it was nearly impossible to hire people with the skills necessary to meet the growing need. There were a number of aspects of JVAB’s approach that made it unique. First and foremost, it had been early to recognize that formerly distinct elements of communications systems were rapidly converging. Historically, communications using radio frequency (RF) signals had been the domain of electrical engineers, while network communications were handled by computer scientists. As network traffic was increasingly being handled using cellular and wifi signals, however, RF intrusions became a serious threat. By the same token, RF communications—such as those handled using high end hand-held devices and cell phones—often relied on the same IP protocols used by the Internet—making them a potential pathway to servers. Another key aspect of JVAB was its adversarial mindset. It prided itself on using the same tools and techniques as the black hat hackers that threatened systems in real world settings. Not only was this an attitude that was generally not cultivated in educational institutions, it also ran counter to the experience of individuals that has spent all their professional life dealing with security in a defensive posture. In the past, Rohret had hired high potential individuals, usually with military experience, and had helped them develop their skills over many years. The end result was the formation of a team with an extraordinary track record of success. But if JVAB were to meet the continuing demand for its services, it needed to figure out new ways to expand. That could be a real challenge in an organization where the most valuable assets all wore shoes.

Towards a product design assessment of visual analytics in decision support applications: a systematic review

There is currently an increasing effort to develop visual analytics (VA) tools that can support human analytical reasoning and decision making. In the last decade, advances in this field has allowed the application of various kinds of VA systems in real-world settings. While this represents a promising start from a product design perspective, part of the challenge to the research community is that current VA tools have evolved without due consideration of standardized design criteria and processes. Accordingly, some questions remain to be addressed on what are the useful, underlying attributes of effective VA tools and how their impact can be measured in human-product interaction. These considerations indicate a need to identify a specific range of VA tools and assess their capabilities through state-of-the-art empirical analysis. To address these issues, we conducted a systematic review of 470 VA papers published between 2006 and 2012. We report on the bibliometric techniques, the evaluation attributes and the metrics that were used to sample and analyze the body of literature. The analysis focused mainly on 26 papers that implemented visual analytics decision support tools. The results are presented in the form of six inductively derived design recommendations that, when taken together, uniquely contribute to the fields of product design and visual analytics.

Uptake and impact of a clinical diagnostic decision support tool at an academic medical center.

Use of differential diagnosis (DDX) generators may reduce the incidence of misdiagnosis-related harm, but there is a paucity of research examining the use and impact of such systems in real-world settings. In September 2012, the DDX generator VisualDx was made available across our entire academic healthcare system. We examined the use of VisualDx by month for the 18 months following its introduction. In addition, we compared the number of inpatient dermatology consults requested per month at the flagship hospital of our healthcare system for the 12 months before versus 18 months after VisualDx introduction. Across our entire academic healthcare system, there were a median of 474 (interquartile range 390-544) unique VisualDx sessions per month. VisualDx was accessed most frequently through mobile devices (35%) and the inpatient electronic health record (34%). Prior to VisualDx introduction, there was a non-significant increase in the number of inpatient dermatology consultations requested per month at the flagship hospital of our healthcare system (1.0 per month, 95% CI -2.5-4.6, p=0.54), which remained 1.0 consults per month (95% CI -0.9-2.9, p=0.27) following its introduction (p=0.99 comparing post- versus pre-introduction rates). The DDX generator VisualDx was regularly used, primarily on mobile devices and inpatient workstations, and was not associated with a change in inpatient dermatology consultation requests. Given the interest in DDX generators, it will be important to evaluate further the impact of such tools on the quality and value of care delivered.

Comparing Fixed-Route and Demand-Responsive Feeder Transit Systems in Real-World Settings

This research presents a method for comparing fixed-route transportation systems and demand-responsive feeder transit systems by using passenger survey data, published transit schedules, and optimal routing techniques. Demand-responsive transportation can be used to improve transit service levels in low-demand areas. Because demand in cities can vary significantly across the region and by time of day, an effective means of determining when demand-responsive services can outperform fixed-route services, and vice versa, is needed. This research builds on existing comparison techniques that focus on gridded street systems and expands the techniques to include all types of street networks, transit schedules, and passenger demand levels. The generic techniques are presented and a case study is given for Atlanta, Georgia, to determine where demand-responsive feeder systems could be implemented to improve customer satisfaction and reduce operating costs.

Model testing, prediction and experimental protocols in neuroscience: A case study

In their theoretical and experimental reflections on the capacities and behaviours of living systems, neuroscientists often formulate generalizations about the behaviour of neural circuits. These generalizations are highly idealized, as they omit reference to the myriads of conditions that could perturb the behaviour of the modelled system in real-world settings. This article analyses an experimental investigation of the behaviour of place cells in the rat hippocampus, in which highly idealized generalizations were tested by comparing predictions flowing from them with real-world experimental results. The aim of the article is to identify (1) under what conditions even single prediction failures regarding the behaviour of single cells sufficed to reject highly idealized generalizations, and (2) under what conditions prima facie counter-examples were deemed to be irrelevant to the testing of highly idealized generalizations. The results of this analysis may contribute to understanding how idealized models are tested experimentally in neuroscience and used to make reliable predictions concerning living systems in real-world settings.

Improving accuracy of medication identification in an older population using a medication bottle color symbol label system

BackgroundThe purpose of this pilot study was to evaluate and refine an adjuvant system of color-specific symbols that are added to medication bottles and to assess whether this system would increase the ability of patients 65 years of age or older in matching their medication to the indication for which it was prescribed.MethodsThis study was conducted in two phases, consisting of three focus groups of patients from a family medicine clinic (n = 25) and a pre-post medication identification test in a second group of patient participants (n = 100). Results of focus group discussions were used to refine the medication label symbols according to themes and messages identified through qualitative triangulation mechanisms and data analysis techniques. A pre-post medication identification test was conducted in the second phase of the study to assess differences between standard labeling alone and the addition of the refined color-specific symbols. The pre-post test examined the impact of the added labels on participants' ability to accurately match their medication to the indication for which it was prescribed when placed in front of participants and then at a distance of two feet.ResultsParticipants appreciated the addition of a visual aid on existing medication labels because it would not be necessary to learn a completely new system of labeling, and generally found the colors and symbols used in the proposed labeling system easy to understand and relevant. Concerns were raised about space constraints on medication bottles, having too much information on the bottle, and having to remember what the colors meant. Symbols and colors were modified if they were found unclear or inappropriate by focus group participants. Pre-post medication identification test results in a second set of participants demonstrated that the addition of the symbol label significantly improved the ability of participants to match their medication to the appropriate medical indication at a distance of two feet (p < 0.001) and approached significant improvement when placed directly in front of participants (p = 0.07).ConclusionsThe proposed medication symbol label system provides a promising adjunct to national efforts in addressing the issue of medication misuse in the home through the improvement of medication labeling. Further research is necessary to determine the effectiveness of the labeling system in real-world settings.

Measurement issues in knowledge engineering

There are five sources (or determinants) of knowledge base quality: domain experts, knowledge engineers, knowledge representation schemes, knowledge elicitation methods, and problem domains. The knowledge base for many expert systems is developed for a problem domain using one domain expert, one knowledge engineer, one knowledge representation scheme, and one elicitation method. Since there is minimal research demonstrating that the possible variation in each of these sources does not significantly affect the quality of the knowledge base, the generalizability (or validity) of such systems in real-world settings is questionable. Consequently, research is needed to assess the extent to which system validity is affected by these sources of variability. Toward this end, the results of reanalyzing the data from an experiment varying domain experts, knowledge engineers, and elicitation methods when developing a multiattributed representation scheme for combat readiness are presented. No significant effects were obtained for elicitation method or knowledge engineer.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>