Wearable Camera Research Articles

Experimental evaluation of wearable LED strip and side-emitting fiber for optical camera communications systems.

This paper presents an experimental evaluation of two types of light-emitting diode (LED)-based distributed transmitters, namely an LED strip and an LED-coupled side-emitting optical fiber, in both laboratory and wearable optical camera communication (OCC) systems. We study the system performance in terms of success of reception (SoR) with regard to the transmission distance. The best value of SoR is achieved when the camera is facing directly to the transmitter (Tx) from a close distance of 1 m. Additionally, we compare the power consumption, the signal-to-noise ratio performance (SNR) and all the obtained values under optimal conditions are better than the forward error correction (FEC) limit in OCC systems.

Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis.

Conventional approaches to dietary assessment are primarily grounded in self-reporting methods or structured interviews conducted under the supervision of dietitians. These methods, however, are often subjective, potentially inaccurate, and time-intensive. Although artificial intelligence (AI)-based solutions have been devised to automate the dietary assessment process, prior AI methodologies tackle dietary assessment in a fragmented landscape (e.g., merely recognizing food types or estimating portion size), and encounter challenges in their ability to generalize across a diverse range of food categories, dietary behaviors, and cultural contexts. Recently, the emergence of multimodal foundation models, such as GPT-4V, has exhibited transformative potential across a wide range of tasks (e.g., scene understanding and image captioning) in various research domains. These models have demonstrated remarkable generalist intelligence and accuracy, owing to their large-scale pre-training on broad datasets and substantially scaled model size. In this study, we explore the application of GPT-4V powering multimodal ChatGPT for dietary assessment, along with prompt engineering and passive monitoring techniques. We evaluated the proposed pipeline using a self-collected, semi free-living dietary intake dataset comprising 16 real-life eating episodes, captured through wearable cameras. Our findings reveal that GPT-4V excels in food detection under challenging conditions without any fine-tuning or adaptation using food-specific datasets. By guiding the model with specific language prompts (e.g., African cuisine), it shifts from recognizing common staples like rice and bread to accurately identifying regional dishes like banku and ugali. Another GPT-4V's standout feature is its contextual awareness. GPT-4V can leverage surrounding objects as scale references to deduce the portion sizes of food items, further facilitating the process of dietary assessment.

Assessing personal screen exposure with ever-changing contexts using wearable cameras and computer vision

Electronic screens are ubiquitous in daily life and support everyday activities in different contexts. Moderate screen use is crucial to individuals' physical and mental health in modern society, while empirical research has verified the potential of altering psychology and behavioral choices by improving micro-environmental qualities. With the hypothesis that the ever-changing contexts may influence people's screen behavior, this article explores how design interventions in these micro-environments can promote healthier and more balanced screen use. To achieve this, the fundamental step is understanding how people utilize screens in diverse contexts, including indoor and outdoor spaces in free-living scenarios. Despite its importance, current literature offers limited methodologies for precisely examining screen behaviors and their contexts simultaneously. To address this gap, this study proposes an automated method with wearable cameras and Computer Vision technologies to quantify screen exposure and related daily contexts extracted from the collected life-logging images (N = 30,186). “Indoor” and “desk” are found to be positively associated with the occurrence of screen behavior. Conversely, “greenery,” “crowd,” “travel,” and “food” exhibit robust and negative relationships with screen exposure. This study offers a new approach to objectively and automatically measure screen exposure, enhancing efficiency and reliability over conventional methods. Moreover, it establishes a replicable framework for future research on broader datasets and informs the fields of architectural and urban design on molding healthier and more balanced screen use among individuals.

Survey on Blind Assist System using ML and Image Processing

This paper presents a Blind Assist System (BAS) leveraging Machine Learning (ML) and Image Processing (IP) techniques to enhance the autonomy and safety of visually impaired individuals. The system utilizes a convolutional neural network (CNN) to process real-time image inputs from a wearable camera device. Through ML classification, it identifies objects, obstacles, and environmental cues. The IP module further refines data, providing depth perception and spatial awareness. Leveraging ML's adaptability, the system continuously learns and improves its recognition accuracy. Integration with auditory feedback facilitates intuitive interaction, conveying vital information to users. In evaluations, the BAS demonstrates promising results in aiding navigation and increasing users' independence. The fusion of ML and IP offers a robust solution for empowering the visually impaired in navigating complex environments

CollaborativeBEV: Collaborative bird eye view for reconstructing crowded environment

Constructing a virtual world for the Metaverse based on real-world data is crucial, yet creating virtual environments for crowded scenes poses challenges in accurately tracking individuals using egocentric wearable cameras due to occlusions caused by crowded pedestrians. To address this, we propose a collaborative perception strategy that leverages multiple agents equipped with multi-view cameras to construct an occupancy map for a crowded environment. To fuse the multi-view perceptions of multiple agents, we propose a Collaborative Bird Eye View fusion network, called CollaborativeBEV (C-BEV), in which, we leverage a depth-based BEV network as a feature extractor, and propose a feature enhancement module to improve perception fusion in overlapping area. A designed loss function is introduced to address data imbalance during training, and a BEV enhancement strategy is proposed to augment the sample pool for training the BEV decoder. Experiment on the Sean2.0 dataset demonstrates that our C-BEV method performs better than the baseline method in terms of a 5.3% IoU increase. Our code will be released on github. https://github.com/RYaNzzZ1/CollaborativeBEV.

Strategic Analysis of Feature Selection Methods for Enhanced Dental Therapy Recognition in Machine Learning Applications

The popularity of smart clinics has increased significantly as a result of technical developments in fields like computer vision. At the heart of such systems is the ability to recognize objects and activities as well as perceive the environment as a whole. This is crucial for both eco-independent systems and human-machine interaction, especially in settings with constrained workspaces, like dental care. Our study delves into an extensive analysis of multiple machine learning models designed to robustly predict dental treatments. These models encompass Lazy Instance-based Learning, Sequential Minimal Optimization, Hoeffding Tree, and Random Tree. Leveraging object-oriented input sourced from gaze-guided wearable cameras, we scrutinize intricate attributes such as material properties, patients' dental conditions, and the array of instruments in use. Notably, we exploit the insight that identifying visual cues during an activity holds the potential to address the specific therapy identification challenge. Utilizing a dental data set that we gathered in the real world, we conducted our experiments and discovered that combining multiple criteria enhances accuracy in comparison to using each one alone. We did see, nevertheless, that in certain circumstances employing the symptoms alone produced superior outcomes. Additionally, symptoms demonstrated to have lesser error than combination in terms of RMS error convergence. Finally, we observed that the machine learning models' build and test durations increased as a result of the combined method. This demonstrates that adding additional parameters does not necessarily result in better outcomes in machine learning applications generally and in medical/dental applications in particular. Instead, it relies on the machine learning tool used, the settings taken into account and the input data provided. The versatility of our approach extends beyond dental contexts. It has been systematically validated across diverse domains, including the recognition of kitchen activities within smart home environments. This methodology holds relevance for various outdoor scenarios where the focal point of attention guides ongoing activities.

Learning to Detect Attended Objects in Cultural Sites with Gaze Signals and Weak Object Supervision

Cultural sites such as museums and monuments are popular tourist destinations worldwide. Visitors come to these places to learn about the cultures, histories, and arts of a particular region or country. However, for many cultural sites, traditional visiting approaches are limited and may fail to engage visitors. To enhance visitors’ experiences, previous works have explored how wearable devices can be exploited in this context. Among the many functions that these devices can offer, understanding which artwork or detail the user is attending to is fundamental to provide additional information on the observed artworks, understand the visitor’s tastes, and provide recommendations. This motivates the development of algorithms for understanding visitor attention from egocentric images. We considered the attended object detection task, which involves detecting and recognizing the object observed by the camera wearer, from an input RGB image and gaze signals. To study the problem, we collect a dataset of egocentric images collected by subjects visiting a museum. Since collecting and labeling data in cultural sites for real applications is a time-consuming problem, we present a study comparing unsupervised, weakly supervised, and fully supervised approaches for attended object detection. We evaluate the considered approaches on the collected dataset, assessing also the impact of training models on external datasets such as COCO and EGO-CH. The experiments show that weakly supervised approaches requiring only a 2D point label related to the gaze can be an effective alternative to fully supervised approaches for attended object detection. To encourage research on the topic, we publicly release the code and the dataset at the following url: https://iplab.dmi.unict.it/EGO-CH-Gaze/ .

A Multi-Modal Egocentric Activity Recognition Approach towards Video Domain Generalization.

Egocentric activity recognition is a prominent computer vision task that is based on the use of wearable cameras. Since egocentric videos are captured through the perspective of the person wearing the camera, her/his body motions severely complicate the video content, imposing several challenges. In this work we propose a novel approach for domain-generalized egocentric human activity recognition. Typical approaches use a large amount of training data, aiming to cover all possible variants of each action. Moreover, several recent approaches have attempted to handle discrepancies between domains with a variety of costly and mostly unsupervised domain adaptation methods. In our approach we show that through simple manipulation of available source domain data and with minor involvement from the target domain, we are able to produce robust models, able to adequately predict human activity in egocentric video sequences. To this end, we introduce a novel three-stream deep neural network architecture combining elements of vision transformers and residual neural networks which are trained using multi-modal data. We evaluate the proposed approach using a challenging, egocentric video dataset and demonstrate its superiority over recent, state-of-the-art research works.

HUMAN ACTIVITY DETECTION USING MMWAVE SENSOR

Human Activity Recognition (HAR) is becoming increasingly useful for applications such as well-being monitoring and personalizing smart spaces. Traditional methods for HAR often require wearable devices or cameras. The former is not feasible for every environment and the latter has strong privacy concerns. The mmWave radar has been shown to be a promising alternative as it does not imply the same privacy concerns and does not require the users to have wearable devices. In this paper we have used a low- cost mmWave radar to generate micro-Doppler spectrograms to ultimately classify different activities. For this, multiple classifiers and methods of spectrogram filtering have been examined. Finally, a Time-Distributed Convolutional Neural Network in conjunction with a Bi-Directional Long Short-Term Memory has attained an average accuracy of 99.62% on a dataset of 5 activities, involving 2 participants.

Inferring in-air gestures in complex indoor environment with less supervision

People have high demands for comfort and technology in indoor environments. Gestures, as a natural and friendly human computer interaction (HCI) method, have received widespread attention and have been the subject of many research studies. Traditional approaches are based on wearable devices and cameras, which can be cumbersome to operate and infringe upon users’ privacy. Millimeter-wave (mmWave) radar avoids these problems by detecting gestures in a noninvasive manner. However, it encounters practical challenges in complex indoor environments, such as dynamic disturbance from surroundings, variable usage conditions and diverse gesture patterns, which conventionally require considerable manual effort to address. In this paper, we attempt to minimize human supervision and propose a noninvasive gesture recognition method named RaGe that involves a commercial mmWave indoor radar. First, a parameter optimization framework considering gesture prior constraints is proposed for radar configuration, which functions to weaken the disturbance from surroundings. Second, we alleviate data shortages in variable usage conditions and achieve low-cost data augmentation by applying affine transformations. Third, we combine deformable convolution operations with an unsupervised attention mechanism, thus exploring the intrinsic features involved in diverse gesture patterns. Experimental results show that RaGe is able to recognize 7 gestures with 99.3% accuracy and less human supervision, surpassing the state-of-the-art methods in comparative experiments.

Through each other's eyes: initial results and protocol for the co-design of an observational measure of adolescent-parent interaction using first-person perspective

BackgroundCurrent observational methods to understand adolescent-parent interaction are limited in terms of ecological and content validity. We outline initial results and a protocol for future work from a programme of work to: (1) establish a new method for data capture of adolescent-parent interaction at home using wearable cameras and; (2) develop a new relevant and comprehensive observational micro-coding scheme. In Part 1, we report our completed preliminary work, comprised of an initial scoping review, and public engagement work. In Part 2, we present a protocol for the development of the new measure.MethodsPart 1—We searched Pubmed for existing observational measures of adolescent-parent interaction for the scoping review. We also undertook public engagement work utilising a mobile research van, taken to multiple locations around Bristol, UK to engage with a variety of populations through interactive methods. Part 2—Our protocol describes plans for: (1) A systematic review of the psychometric properties of observational measures of adolescent-parent interaction; (2) Focussed public engagement workshops; (3) Harmonisation of information from existing coding schemes and literature with information from public engagement with adolescents and parents; (4) A pilot study to assess the acceptability and feasibility of the method; (5) Development of a coding scheme in consultation with expert and lay panels, and through real-life application to recorded videos from a pilot sample.ResultsScoping review: we identified 21 adolescent-parent observational schemes, of which eight used micro-coding and 13 used globalcoding schemes. The majority of micro-coding schemes were not developed specifically for adolescents. Most studies used conflict or problem-solving tasks, which may not adequately capture positive adolescent-parent interactions. The mobile van event received views from 234 young people and/or parents. Families were positive about taking part in research using headcams. “Trust” and “understanding” were most frequently reported as important adolescent-parent relationship constructs.ConclusionsThis work represents the first attempt to truly co-design a method to assess parenting in adolescence. We hope to develop an observational measure using novel technological methods that can be used across a range of research and therapeutic settings.

The association between diet quality and chrononutritional patterns in young adults

PurposeYoung adults eat erratically and later in the day which may impact weight and cardiometabolic health. This cross-sectional study examined relationships between chrononutritional patterns and diet quality in two young adult populations: a university and community sample.MethodsThree days of dietary data were collected including food images captured using wearable cameras. Chrononutritional variables were extracted: time of first and last eating occasions, caloric midpoint (time at which 50% of daily energy was consumed), number of eating occasions per day, eating window, day-to-day variability of the above metrics, and evening eating (≥20:00h). The Healthy Eating Index for Australian Adults scored diet quality. Statistical analyses controlled for gender, body mass index, and socio-economic status.ResultsNo significant associations between chrononutritional patterns and diet quality were found for all participants (n = 95). However, differences in diet quality were found between university (n = 54) and community (n = 41) samples with average diet quality scores of 59.1 (SD 9.7) and 47.3 (SD 14.4), respectively. Of those who extended eating ≥20:00 h, university participants had better diet quality (62.9±SE 2.5 vs. 44.3±SE 2.3, p < 0.001) and discretionary scores (7.9±SE 0.9 vs. 1.6±SE 0.6, p < 0.001) than community participants. University participants consumed predominately healthful dinners and fruit ≥20:00h whereas community participants consumed predominately discretionary foods.ConclusionFor the general young adult population, meal timing needs to be considered. Food choices made by this cohort may be poorer during evenings when the desire for energy-dense nutrient-poor foods is stronger. However, meal timing may be less relevant for young adults who already engage in healthy dietary patterns.

Daily Activity Lifelogs of People With Heart Failure: Observational Study.

Globally, heart failure (HF) affects more than 64 million people, and attempts to reduce its social and economic burden are a public health priority. Interventions to support people with HF to self-manage have been shown to reduce hospitalizations, improve quality of life, and reduce mortality rates. Understanding how people self-manage is imperative to improve future interventions; however, most approaches to date, have used self-report methods to achieve this. Wearable cameras provide a unique tool to understand the lived experiences of people with HF and the daily activities they undertake, which could lead to more effective interventions. However, their potential for understanding chronic conditions such as HF is unclear. This study aimed to determine the potential utility of wearable cameras to better understand the activities of daily living in people living with HF. The "Seeing is Believing (SIB)" study involved 30 patients with HF who wore wearable cameras for a maximum of 30 days. We used the E-Myscéal web-based lifelog retrieval system to process and analyze the wearable camera image data set. Search terms for 7 daily activities (physical activity, gardening, shopping, screen time, drinking, eating, and medication intake) were developed and used for image retrieval. Sensitivity analysis was conducted to compare the number of images retrieved using different search terms. Temporal patterns in daily activities were examined, and differences before and after hospitalization were assessed. E-Myscéal exhibited sensitivity to specific search terms, leading to significant variations in the number of images retrieved for each activity. The highest number of images returned were related to eating and drinking, with fewer images for physical activity, screen time, and taking medication. The majority of captured activities occurred before midday. Notably, temporal differences in daily activity patterns were observed for participants hospitalized during this study. The number of medication images increased after hospital discharge, while screen time images decreased. Wearable cameras offer valuable insights into daily activities and self-management in people living with HF. E-Myscéal efficiently retrieves relevant images, but search term sensitivity underscores the need for careful selection.

Hand Grasp Classification in Egocentric Video After Cervical Spinal Cord Injury.

The hand function of individuals with spinal cord injury (SCI) plays a crucial role in their independence and quality of life. Wearable cameras provide an opportunity to analyze hand function in non-clinical environments. Summarizing the video data and documenting dominant hand grasps and their usage frequency would allow clinicians to quickly and precisely analyze hand function. We introduce a new hierarchical model to summarize the grasping strategies of individuals with SCI at home. The first level classifies hand-object interaction using hand-object contact estimation. We developed a new deep model in the second level by incorporating hand postures and hand-object contact points using contextual information. In the first hierarchical level, a mean of 86% ±1.0% was achieved among 17 participants. At the grasp classification level, the mean average accuracy was 66.2 ±12.9%. The grasp classifier's performance was highly dependent on the participants, with accuracy varying from 41% to 78%. The highest grasp classification accuracy was obtained for the model with smoothed grasp classification, using a ResNet50 backbone architecture for the contextual head and a temporal pose head. We introduce a novel algorithm that, for the first time, enables clinicians to analyze the quantity and type of hand movements in individuals with spinal cord injury at home. The algorithm can find applications in other research fields, including robotics, and most neurological diseases that affect hand function, notably, stroke and Parkinson's.

Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring.

Camera-based passive dietary intake monitoring is able to continuously capture the eating episodes of a subject, recording rich visual information, such as the type and volume of food being consumed, as well as the eating behaviors of the subject. However, there currently is no method that is able to incorporate these visual clues and provide a comprehensive context of dietary intake from passive recording (e.g., is the subject sharing food with others, what food the subject is eating, and how much food is left in the bowl). On the other hand, privacy is a major concern while egocentric wearable cameras are used for capturing. In this article, we propose a privacy-preserved secure solution (i.e., egocentric image captioning) for dietary assessment with passive monitoring, which unifies food recognition, volume estimation, and scene understanding. By converting images into rich text descriptions, nutritionists can assess individual dietary intake based on the captions instead of the original images, reducing the risk of privacy leakage from images. To this end, an egocentric dietary image captioning dataset has been built, which consists of in-the-wild images captured by head-worn and chest-worn cameras in field studies in Ghana. A novel transformer-based architecture is designed to caption egocentric dietary images. Comprehensive experiments have been conducted to evaluate the effectiveness and to justify the design of the proposed architecture for egocentric dietary image captioning. To the best of our knowledge, this is the first work that applies image captioning for dietary intake assessment in real-life settings.

Objective assessment of shared plate eating using a wearable camera in urban and rural households in Ghana.

Shared plate eating (SPE), defined as two or more individuals eating directly from the same plate or bowl, is a common household food consumption practice in many Low- and Middle-Income Countries (LMICs). Examination of household engagement in SPE remains largely unexplored, highlighting a gap in research when interpreting dietary information obtained from these settings. The dearth of research into SPE can be attributed to the inherent limitations of traditional dietary assessment methods which constrain their usability in settings where SPE is common. In this expository narrative, we describe what SPE is when it is practiced in an LMIC such as Ghana; and also compare the frequency of SPE versus individual plate eating (IPE) by different household members in rural and urban households using a wearable camera (Automatic Ingestion Monitor version 2: AIM-2). Purposive convenience sampling was employed to recruit and enroll 30 households each from an urban and a rural community (n = 60 households) in Ghana. The AIM-2 was worn on eyeglass frames for 3 days by selected household members. The AIM-2, when worn, automatically collects images to capture food consumption in participants' environments, thus enabling passive capture of household SPE dynamics. A higher percentage of SPE occasions was observed for rural (96.7%) compared to urban (36.7%) households (p < 0.001). Common SPE dynamics included only adults sharing, adults and children sharing, only children sharing, and non-household member participation in SPE. The wearable camera captured eating dynamics within households that would have likely been missed or altered by traditional dietary assessment methods. Obtaining reliable and accurate data is crucial for assessing dietary intake in settings where SPE is a norm.

Using a wearable micro-camera for dynamic glare evaluation: Corrections and verification assessment

Micro-cameras such as the ‘vision-in-package’ (VIP) can be used as a wearable for user-centric glare estimations. Yet, there is a lack of methodologies to use such head-mounted device and their outputs to obtain reliable glare evaluations. This paper focuses proposes a methodology for dynamic glare evaluation based on wearable micro-cameras. The methodology involves a novel image rotation algorithm and pixel overflow correction based the solar disk luminance measured using an HDR camera. We tested the methodology during an experiment where 21 participants were exposed to direct sunlight in an office-like set-up in Lausanne, Switzerland. For our dataset, the pixel overflow correction avoids an underestimation of the DGP metric in 94.8% of the cases. These corrections appear crucial for the correct estimation of glare using wearable VIP cameras.

Make some Noise: Acoustic Classification of Manual Work Steps Towards Adaptive Assistance Systems

With 32 million people working in the European manufacturing sector, human work still plays a crucial role in industry. However, due to lot size one manufacturing and increased quality requirements, the complexity of products and processes is growing. Therefore, numerous approaches introduce adaptive assistance systems in assembly to support workers during complex work tasks and adapt their level of assistance to the current situation. To enable adaptivity, human action recognition must be incorporated into the consideration of context. Until now, research has focused on providing context to the machine through wearable sensors or cameras. However, wearable sensors hinder worker’s movements and cameras have difficulties distinguishing between work steps of high visual similarity. To mitigate these challenges, we present a new method to classify manual work steps only by their typical acoustic characteristics. The proposed method uses log-Mel spectrograms of work sounds fed into a convolutional neural network (CNN), thus learning their characteristic structure. Moreover, we present a new public dataset for the acoustic classification of manual work steps. The dataset includes typical sources of sounds in manufacturing, such as working with a bench grinder, cordless screwdriver, fling, or grabbing screws. Before feeding the data to the CNN, we apply various pre-processing and data augmentation techniques to increase generalisation capabilities. Our method can detect work steps with reliable accuracy while requiring less parameters than other techniques, proving that detecting work context through acoustics is possible and feasible.

Feasibility and usability of remote monitoring in Alzheimer's disease.

Remote monitoring technologies (RMTs) can measure cognitive and functional decline objectively at-home, and offer opportunities to measure passively and continuously, possibly improving sensitivity and reducing participant burden in clinical trials. However, there is skepticism that age and cognitive or functional impairment may render participants unable or unwilling to comply with complex RMT protocols. We therefore assessed the feasibility and usability of a complex RMT protocol in all syndromic stages of Alzheimer's disease and in healthy control participants. For 8 weeks, participants (N = 229) used two activity trackers, two interactive apps with either daily or weekly cognitive tasks, and optionally a wearable camera. A subset of participants participated in a 4-week sub-study (N = 45) using fixed at-home sensors, a wearable EEG sleep headband and a driving performance device. Feasibility was assessed by evaluating compliance and drop-out rates. Usability was assessed by problem rates (e.g., understanding instructions, discomfort, forgetting to use the RMT or technical problems) as discussed during bi-weekly semi-structured interviews. Most problems were found for the active apps and EEG sleep headband. Problem rates increased and compliance rates decreased with disease severity, but the study remained feasible. This study shows that a highly complex RMT protocol is feasible, even in a mild-to-moderate AD population, encouraging other researchers to use RMTs in their study designs. We recommend evaluating the design of individual devices carefully before finalizing study protocols, considering RMTs which allow for real-time compliance monitoring, and engaging the partners of study participants in the research.

Automated Hand Prehension Assessment From Egocentric Video After Spinal Cord Injury.

Hand function assessments in a clinical setting are critical for upper limb rehabilitation after spinal cord injury (SCI) but may not accurately reflect performance in an individual's home environment. When paired with computer vision models, egocentric videos from wearable cameras provide an opportunity for remote hand function assessment during real activities of daily living (ADLs). This study demonstrates the use of computer vision models to predict clinical hand function assessment scores from egocentric video. SlowFast, MViT, and MaskFeat models were trained and validated on a custom SCI dataset, which contained a variety of ADLs carried out in a simulated home environment. The dataset was annotated with clinical hand function assessment scores using an adapted scale applicable to a wide range of object interactions. An accuracy of 0.551±0.139, mean absolute error (MAE) of 0.517±0.184, and F1 score of 0.547±0.151 was achieved on the 5-class classification task. An accuracy of 0.724±0.135, MAE of 0.290±0.140, and F1 score of 0.733±0.144 was achieved on a consolidated 3-class classification task. This novel approach, for the first time, demonstrates the prediction of hand function assessment scores from egocentric video after SCI.