Smart Glass System Research Articles

Captivates: A Smart Eyeglass Platform for Across-Context Physiological Measurements

We present Captivates, an open-source smartglasses system designed for long-term, in-the-wild psychophysiological monitoring at scale. Captivates integrate many underutilized physiological sensors in a streamlined package, including temple and nose temperature measurement, blink detection, head motion tracking, activity classification, 3D localization, and head pose estimation. Captivates was designed with an emphasis on (1) manufacturing and scalability, so we can easily support large-scale user studies for ourselves and offer the platform as a generalized tool for ambulatory psychophysiology research; (2) robustness and battery life, so long-term studies result in trustworthy data across an individual's entire day in natural environments without supervision or recharge; and (3) aesthetics and comfort, so people can wear them in their normal daily contexts without self-consciousness or changes in behavior.

Preliminary data on a novel smart glasses system for measuring the angle of deviation in strabismus.

To propose a novel smart glasses device for recording eye movement and compare its results to the prism alternate cover test (PACT). This method comparison study enrolled patients with strabismic conditions, who first underwent conventional strabismus evaluations (PACT in the primary position), followed by the smart glasses NeuroSpeed system (NSS) recording protocols. The video recordings were analysed using specialized software, to calculate the horizontal deviation from the primary position. The results were compared with those of the PACT using Passing‒Bablok regression and Bland‒Altman analysis. This study included 70 individuals aged 4 to 80 years, of which 38 were men and 32 were women. The overall analysis of horizontal deviations using the Passing‒Bablok regression revealed a correlation coefficient (r) of 0.969, with a systemic bias of 0.00, a proportional bias of 0.809, and a perpendicular residual standard deviation of 4.134. The predictive values of eye movement examinations recorded by the NSS were comparable to those of the PACT. Thus, this new system can provide additional information for ophthalmologists to aid in the diagnosis and measurement of strabismus.

15. Utilization of Augmented Reality Smart Glasses for the Provision of Remote Surgical Training in the Resource-Constrained Setting

Background: Surgical mission trips constitute an important strategy by which the global surgery campaign addresses inequities in surgical disease worldwide. Travel restrictions related to the COVID-19 pandemic, however, impose a challenging limit on the reach and impact of surgical mission work. In this study, we demonstrate the utility of a remote surgical mentorship program equipped with cutting-edge augmented reality (AR) technology for sustaining global surgery capacity building efforts within the constraints of the COVID-19 pandemic. Methods: Our remote surgical training program connects mentor and mentee surgeons via an AR-enabled digital headset and integrated smartphone application. The Vuzix® Smart Glasses system allows mentee surgeons to transmit a point-of-view livestream of activity in the operating room while receiving direct audio and visual feedback from their remote mentor in realtime. To facilitate their engagement, mentor and mentee surgeons interact using an integrated smartphone application that enables case scheduling, clinical documentation, and performance evaluation. Performance is assessed on the basis of three validated assessment tools that are well-established in the surgical education literature: the Objective Structured Assessment of Technical Skills, the Self-Confidence Scale, and the Student Evaluation of Educational Quality Survey. This survey is administered via the mobile application to both mentor and mentee surgeons at the time of their onboarding as well as after every subsequent three-month training quarter for the duration of the year-long program. Results: To date, 83 mentor and mentee surgeons across 33 countries and 6 continents are enrolled and actively participating in our remote training program. Baseline survey data suggest a mentee surgeon cohort that stands to benefit greatly from inclusion in our training program. A majority of mentee surgeons report incomplete knowledge and anxiety in association with their specialty prior to initiating training. Conclusion: The COVID-19 pandemic has created an urgent need for sustainable remote surgical capacity building strategies. Our remote surgical training program powered by AR-enabled telecommunication and smartphone technology constitutes a promising solution towards this end.

Feasibility of smart glasses in supporting spinal surgical procedures in low- and middle-income countries: experiences from East Africa.

Telemedicine technology has been developed to allow surgeons in countries with limited resources to access expert technical guidance during surgical procedures. The authors report their initial experience using state-of-the-art wearable smart glasses with wireless capability to transmit intraoperative video content during spine surgery from sub-Saharan Africa to experts in the US. A novel smart glasses system with integrated camera and microphone was worn by a spine surgeon in Dar es Salaam, Tanzania, during 3 scoliosis correction surgeries. The images were transmitted wirelessly through a compatible software system to a computer viewed by a group of fellowship-trained spine surgeons in New York City. Visual clarity was determined using a modified Snellen chart, and a percentage score was determined on the smallest line that could be read from the 8-line chart on white and black backgrounds. A 1- to 5-point scale (from 1 = unrecognizable to 5 = optimal clarity) was used to score other visual metrics assessed using a color test card including hue, contrast, and brightness. The same scoring system was used by the group to reach a consensus on visual quality of 3 intraoperative points including instruments, radiographs (ability to see pedicle screws relative to bony anatomy), and intraoperative surgical field (ability to identify bony landmarks such as transverse processes, pedicle screw starting point, laminar edge). All surgeries accomplished the defined goals safely with no intraoperative complications. The average download and upload connection speeds achieved in Dar es Salaam were 45.21 and 58.89 Mbps, respectively. Visual clarity with the modified white and black Snellen chart was 70.8% and 62.5%, respectively. The average scores for hue, contrast, and brightness were 2.67, 3.33, and 2.67, respectively. Visualization quality of instruments, radiographs, and intraoperative surgical field were 3.67, 1, and 1, respectively. Application of smart glasses for telemedicine offers a promising tool for surgical education and remote training, especially in low- and middle-income countries. However, this study highlights some limitations of this technology, including optical resolution, intraoperative lighting, and internet connection challenges. With continued collaboration between clinicians and industry, future iterations of smart glasses technology will need to address these issues to stimulate robust clinical utilization.

Automatic Fire Detection and Notification System Based on Improved YOLOv4 for the Blind and Visually Impaired.

The growing aging population suffers from high levels of vision and cognitive impairment, often resulting in a loss of independence. Such individuals must perform crucial everyday tasks such as cooking and heating with systems and devices designed for visually unimpaired individuals, which do not take into account the needs of persons with visual and cognitive impairment. Thus, the visually impaired persons using them run risks related to smoke and fire. In this paper, we propose a vision-based fire detection and notification system using smart glasses and deep learning models for blind and visually impaired (BVI) people. The system enables early detection of fires in indoor environments. To perform real-time fire detection and notification, the proposed system uses image brightness and a new convolutional neural network employing an improved YOLOv4 model with a convolutional block attention module. The h-swish activation function is used to reduce the running time and increase the robustness of YOLOv4. We adapt our previously developed smart glasses system to capture images and inform BVI people about fires and other surrounding objects through auditory messages. We create a large fire image dataset with indoor fire scenes to accurately detect fires. Furthermore, we develop an object mapping approach to provide BVI people with complete information about surrounding objects and to differentiate between hazardous and nonhazardous fires. The proposed system shows an improvement over other well-known approaches in all fire detection metrics such as precision, recall, and average precision.

Preliminary study of prehospital use of smart glasses.

A smart glasses system is a computerized communicator with a transparent screen and video camera that can be worn as a pair of glasses. There have been no reports on the use of smart glasses in the prehospital setting. To conduct a preliminary investigation on whether smart glasses were smoothly used bidirectionally between the prehospital setting and the receiving hospital. From March 1, 2022, to March 31, 2022, one smart glasses unit was installed in an ambulance at one branch of the fire department near our hospital. The physician on the computer, who received video and voice transmission from the smart glasses when the ambulance was dispatched, evaluated the transmission status with regard to video reception, voice reception, and voice transmission. In addition, the activity time was compared between cases in which the smart glasses system was used (patient) and not used (control). During the investigating period, 12 cases were analyzed as patients. The rate of good video reception was 75%, the rate of good voice reception was 50%, and the rate of good voice transmission was 25%. There was no significant change in the activity time between the patient and control groups. We performed a preliminary investigation on the usability of smart glasses in the prehospital setting. Using smart glasses, patient information was shared by video and voice before arrival at the hospital, and did not affect the activity time. However, the instability of the communication status should be recognized.

Smart Glass System Using Deep Learning for the Blind and Visually Impaired

Individuals suffering from visual impairments and blindness encounter difficulties in moving independently and overcoming various problems in their routine lives. As a solution, artificial intelligence and computer vision approaches facilitate blind and visually impaired (BVI) people in fulfilling their primary activities without much dependency on other people. Smart glasses are a potential assistive technology for BVI people to aid in individual travel and provide social comfort and safety. However, practically, the BVI are unable move alone, particularly in dark scenes and at night. In this study we propose a smart glass system for BVI people, employing computer vision techniques and deep learning models, audio feedback, and tactile graphics to facilitate independent movement in a night-time environment. The system is divided into four models: a low-light image enhancement model, an object recognition and audio feedback model, a salient object detection model, and a text-to-speech and tactile graphics generation model. Thus, this system was developed to assist in the following manner: (1) enhancing the contrast of images under low-light conditions employing a two-branch exposure-fusion network; (2) guiding users with audio feedback using a transformer encoder–decoder object detection model that can recognize 133 categories of sound, such as people, animals, cars, etc., and (3) accessing visual information using salient object extraction, text recognition, and refreshable tactile display. We evaluated the performance of the system and achieved competitive performance on the challenging Low-Light and ExDark datasets.

Captivates

We present Captivates, an open-source smartglasses system designed for long-term, in-the-wild psychophysiological monitoring at scale. Captivates integrate many underutilized physiological sensors in a streamlined package, including temple and nose temperature measurement, blink detection, head motion tracking, activity classification, 3D localization, and head pose estimation. Captivates were designed with an emphasis on: (1) manufacturing and scalability, so we can easily support large scale user studies for ourselves and offer the platform as a generalized tool for ambulatory psychophysiology research; (2) robustness and battery life, so long-term studies result in trustworthy data individual's entire day in natural environments without supervision or recharge; and (3) aesthetics and comfort, so people can wear them in their normal daily contexts without self-consciousness or changes in behavior. Captivates are intended to enable large scale data collection without altering user behavior. We validate that our sensors capture useful data robustly for a small set of beta testers. We also show that our additional effort on aesthetics was imperative to meet our goals; namely, earlier versions of our prototype make people uncomfortable to interact naturally in public, and our additional design and miniaturization effort has made a significant impact in preserving natural behavior. There is tremendous promise in translating psychophysiological laboratory techniques into real-world insight. Captivates serve as an open-source bridge to this end. Paired with an accurate underlying model, Captivates will be able to quantify the long-term psychological impact of our design decisions and provide real-time feedback for technologists interested in actuating a cognitively adaptive, user-aligned future.

6.7 AUTISM SPECTRUM DISORDER–FOCUSED AUGMENTED REALITY SMARTGLASSES SYSTEM FOR ENHANCING ENGAGEMENT AND PARENT-CHILD INTERACTION

Autism spectrum disorder (ASD) is a prevalent childhood-onset neurodevelopmental disorder. The current ASD interventions are primarily based on in-person behavioral therapies that are both costly and difficult to access, and are targeting ASD deficits in social communication, and the presence of stereotypies and other ASD-related behaviors. This secondary analysis evaluates the impact of ASD-focused augmented-reality Brain Power Smartglasses (BPS) system on engagement and parent-child interaction of children with higher-functioning ASD.

Social Communication Coaching Smartglasses: Well Tolerated in a Diverse Sample of Children and Adults With Autism.

BackgroundAugmented reality (AR) smartglasses are an emerging technology that is under investigation as a social communication aid for children and adults with autism spectrum disorder (ASD) and as a research tool to aid with digital phenotyping. Tolerability of this wearable technology in people with ASD is an important area for research, especially as these individuals may experience sensory, cognitive, and attentional challenges.ObjectiveThe aim of this study was to assess the tolerability and usability of a novel smartglasses system that has been designed as a social communication aid for children and adults with autism (the Brain Power Autism System [BPAS]). BPAS runs on Google Glass Explorer Edition and other smartglasses, uses both AR and affective artificial intelligence, and helps users learn key social and emotional skills.MethodsA total of 21 children and adults with ASD across a spectrum of severity used BPAS for a coaching session. The user’s tolerability to the smartglasses, user being able to wear the smartglasses for 1 minute (initial tolerability threshold), and user being able to wear the smartglasses for the entire duration of the coaching session (whole session tolerability threshold) were determined through caregiver report.ResultsOf 21 users, 19 (91%) demonstrated tolerability on all 3 measures. Caregivers reported 21 out of 21 users (100%) as tolerating the experience, while study staff found only 19 out of 21 users managed to demonstrate initial tolerability (91%). Of the 19 users who demonstrated initial tolerability, all 19 (100%) were able to use the smartglasses for the entire session (whole session tolerability threshold). Caregivers reported that 19 out of 21 users (91%) successfully used BPAS, and users surpassed caregiver expectations in 15 of 21 cases (71%). Users who could communicate reported BPAS as being comfortable (94%).ConclusionsThis preliminary report suggests that BPAS is well tolerated and usable to a diverse age- and severity-range of people with ASD. This is encouraging as these devices are being developed as assistive technologies for people with ASD. Further research should focus on improving smartglasses design and exploring their efficacy in helping with social communication in children and adults with ASD.

Feasibility of an Autism-Focused Augmented Reality Smartglasses System for Social Communication and Behavioral Coaching.

Autism spectrum disorder (ASD) is a childhood-onset neurodevelopmental disorder with a rapidly rising prevalence, currently affecting 1 in 68 children, and over 3.5 million people in the United States. Current ASD interventions are primarily based on in-person behavioral therapies that are both costly and difficult to access. These interventions aim to address some of the fundamental deficits that clinically characterize ASD, including deficits in social communication, and the presence of stereotypies, and other autism-related behaviors. Current diagnostic and therapeutic approaches seldom rely on quantitative data measures of symptomatology, severity, or condition trajectory. Given the current situation, we report on the Brain Power System (BPS), a digital behavioral aid with quantitative data gathering and reporting features. The BPS includes customized smartglasses, providing targeted personalized coaching experiences through a family of gamified augmented-reality applications utilizing artificial intelligence. These applications provide children and adults with coaching for emotion recognition, face directed gaze, eye contact, and behavioral self-regulation. This preliminary case report, part of a larger set of upcoming research reports, explores the feasibility of the BPS to provide coaching in two boys with clinically diagnosed ASD, aged 8 and 9 years. The coaching intervention was found to be well tolerated and rated as being both engaging and fun. Both males could easily use the system, and no technical problems were noted. During the intervention, caregivers reported improved non-verbal communication, eye contact, and social engagement during the intervention. Both boys demonstrated decreased symptoms of ASD, as measured by the aberrant behavior checklist at 24-h post-intervention. Specifically, both cases demonstrated improvements in irritability, lethargy, stereotypy, hyperactivity/non-compliance, and inappropriate speech. Smartglasses using augmented reality may have an important future role in helping address the therapeutic needs of children with ASD. Quantitative data gathering from such sensor-rich systems may allow for digital phenotyping and the refinement of social communication constructs of the research domain criteria. This report provides evidence for the feasibility, usability, and tolerability of one such specialized smartglasses system.

Can Machine Learning Decode Depression?

Can Machine Learning Decode Depression?

Smart Glass and Its Potential in Energy Savings

Smart glass is such that its properties may be changed by application of a potential across it. The change in properties may be engineered to alter the amount of heat energy that can penetrate the glass which provides heating and cooling design options. Therein lies its potential in energy savings. Smart glass may be classified into three types: electrochromic, suspended particle, and polymer dispersed liquid crystal (PDLC). Each of these types has their own mechanisms, advantages, and disadvantages. Electrochromic smart glass is the most popular, currently it utilizes an electrochromic film with an ion storage layer and ion conductor placed between two transparent plates. The electrochromic film is usually made of tungsten oxide, owing to the electrochromic nature of transition metals. An electric potential initiates a redox reaction of the electrochromic film transitioning the color and the transparency of the smart glass. Suspended particle smart glass has needle shaped particles suspended within an organic gel placed between two electrodes. In its off state, the particles are randomly dispersed and have a low light transmittance. Once a voltage is applied, the needle particles will orient themselves to allow for light to pass through. PDLC smart glass works similarly to the suspended particle variety. However, in PDLC smart glass, the central layer is a liquid crystal placed within a polymer matrix between electrodes. Similar in behavior to the suspended particles, in the off position the liquid crystals are randomly dispersed and have low transmittance. With the application of a voltage, the liquid crystals orient themselves, thereby allowing for the transmittance of light. These different smart glasses have many different applications, but with one hindrance. The requirement of a voltage source is a major disadvantage which greatly complicates the overall installation and manufacturing processes. However, the integration of photovoltaic (PV) devices into smart glass technology has provided one solution. Photovoltaic films attached in the smart glass will provide the necessary voltage source. The photovoltaic film may even be designed to produce more voltage than needed. The use a photovoltaic smart glass system provides significant cost savings in regards to heating, cooling, lighting, and overall energy bills. Smart glass represents a technology with a great deal of potential to reduce energy demand. Action steps have been identified to propagate the popular use of smart glass.