Eyewear Devices Research Articles

Silent Speech Eyewear Interface: Silent Speech Recognition Method Using Eyewear and an Ear-Mounted Microphone with Infrared Distance Sensors.

As eyewear devices such as smart glasses become more common, it is important to provide input methods that can be used at all times for such situations and people. Silent speech interaction (SSI) has the potential to be useful as a hands-free input method for various situations and people, including those who have difficulty with voiced speech. However, previous methods have involved sensor devices that are difficult to use anytime and anywhere. We propose a method for SSI that involves using an eyewear device equipped with infrared distance sensors. The proposed method measures facial skin movements associated with speech from the infrared distance sensor mounted on an eyewear device and recognizes silent speech commands by applying machine learning to time series sensor data. The proposed method was applied to a prototype system including a sensor device consisting of eyewear and ear-mounted microphones to measure the movements of the cheek, jaw joint, and jaw. Evaluations 1 and 2 showed that five speech commands could be recognized with an F value of 0.90 and ten longer speech commands with an F value of 0.83. Evaluation 3 showed how the recognition accuracy changes with the combination of sensor points. Evaluation 4 examined whether the proposed method can be used for a larger number of speech commands with 21 commands by using deep learning LSTM and a combination of DTW and kNN. Evaluation 5 examined the recognition accuracy in some situations affecting recognition accuracy such as re-attaching devices and walking. These results show the feasibility of the proposed method for a simple hands-free input interface, such as with media players and voice assistants. Our study provides the first wearable sensing method that can easily apply SSI functions to eyewear devices.

Unveiling Causal Attention in Dogs' Eyes with Smart Eyewear

Our goals are to better understand dog cognition, and to support others who share this interest. Existing investigation methods predominantly rely on human-manipulated experiments to examine dogs' behavioral responses to visual stimuli such as human gestures. As a result, existing experimental paradigms are usually constrained to in-lab environments and may not reveal the dog's responses to real-world visual scenes. Moreover, visual signals pertaining to dog behavioral responses are empirically derived from observational evidence, which can be prone to subjective bias and may lead to controversies. We aim to overcome or reduce the existing limitations of dog cognition studies by investigating a challenging issue: identifying the visual signal(s) from dog eye motion that can be utilized to infer causal explanations of its behaviors, namely estimating causal attention. To this end, we design a deep learning framework named Causal AtteNtIon NEtwork (CANINE) to unveil the dogs' causal attention mechanism, inspired by the recent advance in causality analysis with deep learning. Equipped with CANINE, we developed the first eyewear device to enable inference on the vision-related behavioral causality of canine wearers. We demonstrate the technical feasibility of the proposed CANINE glasses through their application in multiple representative experimental scenarios of dog cognitive study. Various in-field trials are also performed to demonstrate the generality of the CANINE eyewear in real-world scenarios. With the proposed CANINE glasses, we collect the first large-scale dataset, named DogsView, which consists of automatically generated annotations on the canine wearer's causal attention across a wide range of representative scenarios. The DogsView dataset is available online to facilitate research.

Exploring Gaze Movement Gesture Recognition Method for Eye-Based Interaction Using Eyewear with Infrared Distance Sensor Array

With the spread of eyewear devices, people are increasingly using information devices in various everyday situations. In these situations, it is important for eyewear devices to have eye-based interaction functions for simple hands-free input at a low cost. This paper proposes a gaze movement recognition method for simple hands-free interaction that uses eyewear equipped with an infrared distance sensor. The proposed method measures eyelid skin movement using an infrared distance sensor inside the eyewear and applies machine learning to the time-series sensor data to recognize gaze movements (e.g., up, down, left, and right). We implemented a prototype system and conducted evaluations with gaze movements including factors such as movement directions at 45-degree intervals and the movement distance difference in the same direction. The results showed the feasibility of the proposed method. The proposed method recognized 5 to 20 types of gaze movements with an F-value of 0.96 to 1.0. In addition, the proposed method was available with a limited number of sensors, such as two or three, and robust against disturbance in some usage conditions (e.g., body vibration, facial expression change). This paper provides helpful findings for the design of gaze movement recognition methods for simple hands-free interaction using eyewear devices at a low cost.

Do Smart Glasses Dream of Sentimental Visions?

Emotion recognition in smart eyewear devices is valuable but challenging. One key limitation of previous works is that the expression-related information like facial or eye images is considered as the only evidence of emotion. However, emotional status is not isolated; it is tightly associated with people's visual perceptions, especially those with emotional implications. However, little work has examined such associations to better illustrate the causes of emotions. In this paper, we study the emotionship analysis problem in eyewear systems, an ambitious task that requires classifying the user's emotions and semantically understanding their potential causes. To this end, we describe EMOShip, a deep-learning-based eyewear system that can automatically detect the wearer's emotional status and simultaneously analyze its associations with semantic-level visual perception. Experimental studies with 20 participants demonstrate that, thanks to its awareness of emotionship, EMOShip achieves superior emotion recognition accuracy compared to existing methods (80.2% vs. 69.4%) and provides a valuable understanding of the causes of emotions. Further pilot studies with 20 additional participants further motivate the potential use of EMOShip to empower emotion-aware applications, such as emotionship self-reflection and emotionship life-logging.

FlexiSee: flexible configuration, customization, and control of mediated and augmented vision for users of smart eyewear devices

Smart eyewear and Augmented Reality technology have been examined closely in the scientific community to provide vision rehabilitation to people with visual impairments as well as augmented vision to people with and without visual impairments for various application scenarios and contexts of use. However, current systems lack flexibility in the configuration and customization of the features and functionalities they present to their users, as we show in this paper by means of a thorough literature review and categorization of prior work on augmented and mediated vision for smart eyewear devices. To address the flexibility aspect that has been missing in prior work, we introduce FlexiSee, an application for smart eyewear devices, such as see-through Augmented Reality glasses and Head-Mounted Mixed Reality Displays, specifically designed to enable flexible configuration, customization, and control of both augmented and mediated vision. FlexiSee achieves this desiderata by implementing visual filters (e.g., color correction, edge highlighting, contrast adjustment, and others) that are coupled with a web-based interface, readily accessible from smartphones, tablets, smartwatches, and other devices with web browsers, where authorized users can specify and apply custom parameters for the visual filters implemented by FlexiSee. We also introduce FlexiSee-DS, a three-dimensional design space for FlexiSee-like applications, that includes mediation & augmentation, user categories, and control design dimensions to specify a variety of FlexiSee-like systems. We show how the dimensions of FlexiSee-DS were applied to inform the design of our FlexiSee system, and we highlight and focus on the distinction between primary users and vision monitors and assistants, where the latter two categories represent new types of users for augmented and mediated vision that have various degrees of control, from their remote locations, over the visual reality delivered to and perceived by the primary users of smart eyewear devices. We conduct a user study to understand the perception of vision monitors and assistants regarding our new FlexiSee concept and system, and we report empirical results about usability aspects (e.g., we found an average SUS score of 75.3 and high ratings for the perceived usefulness of FlexiSee) as well as user feedback and suggestions to inform further developments of FlexiSee-like systems and applications.

SAR analysis of tri-band antennas for a 5G eyewear device

AbstractThe goal of this study is to analyze the specific absorption rate (SAR) distribution of the projected 5G frequencies below 6 GHz and at Wi-Fi frequency (2.45 GHz) on a human head, for eyewear device applications. Two separate tri-band printed dipole antennas for this purpose are designed and fabricated at operating frequencies of 2.45/3.8/6 GHz for prototype-1 and at operating frequencies of 2.45/3.6/4.56 GHz for prototype-2. In order to obtain the desired frequencies: first, the prototypes of the proposed antennas are fine-tuned via Computer Simulation Technology Microwave Studio (CST) and then fabricated on the FR4 layer. The reflection coefficient (S11) is tested and the simulation results are confirmed. In order to analyze the effect of wearing a pair of glasses' frame including a tri-band 5G antenna, a frame is designed and produced via 3D printer with polylactic acid material which has high dielectric constant (ɛr = 8.1). The SAR results of the proposed antennas have been examined for the cases where the antenna is embedded in the frame and is used alone. Both cases were analyzed by using the homogeneous specific anthropomorphic mannequin and the heterogeneous visible human head phantoms and the results have been evaluated in terms of SAR10 g values.

Automatic Pupillary Light Reflex Detection in Eyewear Computing

There are many benefits to facilitating “always-on” pupillary light reflex (PLR)-aware pupil size measurement in eyewear, including improving the reliability of pupil-based cognitive and affective load monitoring and enabling PLR-based diagnosis of cognitive and eye-related diseases which have neurological symptoms manifested in the form of aberrant PLR responses. However, the detection of PLR responses for application in eyewear devices for everyday usage, beyond PLR measurement in confined clinical sessions, has not been investigated. To this end, a means of characterizing PLRs in less controlled environmental settings is investigated and subsequently a method of PLR detection is developed and evaluated. A low-cost head-mounted Web camera was used to record near-field eye video sequences which were processed with the self-tuning threshold algorithm for pupil diameter estimation and blink detection. PLR was induced by luminance change of a monitor and brightness change of the displayed image on a monitor. A transient model-based PLR detection algorithm which utilizes the general correlation of PLR amplitude and velocity was developed and evaluated on the data sets in terms of false alarm and false rejection rates. The findings from this research suggest that the PLR can be detected reliably using low-cost wearable pupil-measurement systems without using a separate sensor for detecting the luminance conditions. The correlation between pupil diameter amplitude and maximum velocity of PLR was shown to be sufficiently consistent for PLR detection.

Dual-Antenna Concept With Complementary Radiation Patterns for Eyewear Applications

A dual-antenna concept to be integrated in a smart eyewear device is presented. The two antennas are printed on FR4 substrate and placed inside an Acrylonitrile butadiene styrene dielectric eyewear frame manufactured using 3-D printing technology. One of the two antennas is a monopole situated at the back of the eyewear with a specifically shaped ground plane underneath. The second antenna is a balanced-fed loop antenna placed in front of the eyewear with an integrated balun circuit. Both antennas are designed to operate in the 2.4 GHz band of the 802.11b/g/n standard. When compared to a reference single monopole antenna usually used in commercial eyewear, higher gain and complementary radiation patterns are obtained in several useful directions using our dual-antenna solution: 1) in front of the eyewear to communicate with a laptop; 2) in the side direction of the eyewear to communicate with a fixed hotspot access point; and 3) in downward direction of the eyewear to communicate with a peripheral smartphone in the pocket of the user. S-parameters and radiation pattern measurements are given with the eyewear placed over a specific anthropomorphic mannequin head phantom to validate the advantages of this novel dual-antenna concept. Specific absorption rate simulation results are presented showing the proposed antennas stay within the authorized limits.

A Framework for Learning Analytics Using Commodity Wearable Devices.

We advocate for and introduce LEARNSense, a framework for learning analytics using commodity wearable devices to capture learner’s physical actions and accordingly infer learner context (e.g., student activities and engagement status in class). Our work is motivated by the observations that: (a) the fine-grained individual-specific learner actions are crucial to understand learners and their context information; (b) sensor data available on the latest wearable devices (e.g., wrist-worn and eye wear devices) can effectively recognize learner actions and help to infer learner context information; (c) the commodity wearable devices that are widely available on the market can provide a hassle-free and non-intrusive solution. Following the above observations and under the proposed framework, we design and implement a sensor-based learner context collector running on the wearable devices. The latest data mining and sensor data processing techniques are employed to detect different types of learner actions and context information. Furthermore, we detail all of the above efforts by offering a novel and exemplary use case: it successfully provides the accurate detection of student actions and infers the student engagement states in class. The specifically designed learner context collector has been implemented on the commodity wrist-worn device. Based on the collected and inferred learner information, the novel intervention and incentivizing feedback are introduced into the system service. Finally, a comprehensive evaluation with the real-world experiments, surveys and interviews demonstrates the effectiveness and impact of the proposed framework and this use case. The F1 score for the student action classification tasks achieve 0.9, and the system can effectively differentiate the defined three learner states. Finally, the survey results show that the learners are satisfied with the use of our system (mean score of 3.7 with a standard deviation of 0.55).

Dual-Band 4G Eyewear Antenna and SAR Implications

This communication presents low-cost and easy-to-manufacture dual-band antenna solution to achieve cellular 4G (LTE-Advanced) coverage in smart eyewear devices. Coupling element type antenna has been evaluated with appropriate matching networks to cover the target bands of 700–960 MHz and 1.7–2.7GHz. To emulate a realistic device, an ABS plastic dielectric frame has been designed and manufactured using 3-D printing technology. Simulations for the antenna are carried out in three typical use-case scenarios which are “with user’s head,” “with head and hand,” and “free space.” The simulations are validated through S-parameters, efficiency and radiation pattern measurements using fabricated frame and antenna prototype in the presence of head and hand phantoms. The specific absorption rate (SAR) behavior of the antenna designs is also investigated through simulations and measurements. It is demonstrated that SAR values are found to be above the limitations which might be problematic in practical use if the transmit power of the eyewear is not reduced.

Ultrabroadband Antenna With Robustness to Body Detuning for 4G Eyewear Devices

An ultrabroadband antenna design for 4G cellular coverage in eyewear devices is presented in this letter. The antenna has been designed considering a realistic eyewear printed circuit board and its dielectric frame. In order to overcome any negative effects from the antenna's surroundings (frequency detuning due to the head and the hand of the user), the antenna was designed and optimized to cover continuously the 700–2700-MHz band with a reflection coefficient below –6 dB instead of a dual-band behavior usually required for 4G operation: 700–960 and 1700–2700 MHz. The antenna was of capacitive coupling element type with its appropriate matching network to cover the mentioned frequency interval. To emulate a realistic device, an ABS plastic dielectric frame was manufactured using 3-D printing technology. The antenna was simulated and tested in three different use cases as “default case with user's head,” “with head and hand,” and “free space” cases. State-of-the art performance is demonstrated. The specific absorption rate behavior was also investigated through simulations and measurements to check the compliance with regulations.

Ball Grid Array-Module With Integrated Shaped Lens for WiGig Applications in Eyewear Devices

A ball grid array-module (BGA-module) incorporating a low-cost shaped dielectric lens is proposed for wireless communications in the 60-GHz WiGig band between a smart eyewear, where it is integrated and facing a laptop or TV. The module, which is codesigned with a 60-GHz transceiver, consists of two separate identical antennas for transmitting (Tx) and receiving (Rx). The in-plane separation of these elements is 6.9 mm both being offset from the lens focus. This poses a challenge to the lens design to ensure coincident beam pointing directions for Rx and Tx. The shaped lens is further required to narrow the angular coverage in the elevation plane and broaden it in the horizontal plane. A 3-D-printed eyewear frame with an integrated lens and a recess for proper BGA-module integration is fabricated in ABS-plastic material. Measurements show a reflection coefficient below $-12\; \text{dB}$ in the 57–66 GHz band. A maximum gain of 11 dBi is obtained at 60 GHz, with 24° and 96° beamwidth at 5-dBi gain, respectively, in the vertical and horizontal planes. The radiation exposure is evaluated for a homogeneous SAM head phantom and a heterogeneous visible human head. The simulated power density values for both models are found to be lower than the existing standards.

4G antennas for wireless eyewear devices and related SAR

In this paper, we first present a feasibility study to design 4G antennas (700–960 MHz and 1.7–2.7 GHz) for eyewear devices. Those eyewear devices should be connected to the last generation cellular networks, Wireless Local Area Networks or wireless hotspots. Three coupling element type antennas with their matching networks are evaluated in terms of reflection coefficient and total radiation efficiency when the eyewear is placed on the user's head. We also present Specific Absorption Rate (SAR) simulations when the eyewear is positioned over a homogeneous SAM phantom and over a heterogeneous VH (Visible Human) phantom: the SAR levels are compared to international limit values. In a second step, we present experimental results obtained with 3D printed eyewear and coupling elements etched on a classical PCB substrate where the matching circuits are optimized close to the feeding point of the coupling element. Simulated and measured values are in very good agreement: 7 to 16% and 9 to 35% total efficiency are respectively obtained for the low- and high-frequency bands. However, simulated SAR values are somewhat higher than authorized levels with preoccupant high electromagnetic field distribution close to the eye of the user. Dans cet article, nous présentons tout d'abord une étude de faisabilité d'antennes 4G (700–960 MHz et 1,7–2,7 GHz) intégrables dans une paire de lunettes connectées. Cet objet est destiné à être connecté aux réseaux de communication cellulaires de dernière génération ainsi qu'aux réseaux personnels de type WLAN. Le concept antennaire d'élément de couplage associé à un circuit d'adaptation est ainsi évalué en termes de coefficient de réflexion et d'efficacité rayonnée lorsque les lunettes sont positionnées sur la tête de l'utilisateur. Nous présentons également l'analyse dosimétrique des prototypes optimisés pour le fantôme homogène SAM et le fantôme évolué hétérogène Visible Human (VH) : les niveaux de DAS dans ces deux fantômes sont simulés et comparés aux normes internationales. Dans un second temps, nous présentons les premiers résultats expérimentaux obtenus à partir de prototypes fabriqués par une imprimante 3D pour la monture des lunettes et des procédés de réalisation classique pour les PCB qui accueillent les éléments de couplage antennaire et leur circuit d'adaptation. Les valeurs théoriques et expérimentales concernant les coefficients de réflexion et les efficacités totales des antennes sont en très bon accord, attestant ainsi une efficacité rayonnée variant de 10 à 12% en bande basse et de 20 à 35% en bande haute. Cependant, les simulations dosimétriques révèlent des niveaux de DAS qui peuvent dépasser les normes du fait des fortes efficacités rayonnées obtenues avec les antennes conçues, notamment avec des pics préoccupants observés au niveau de l'œil.