Personal Lifelog Research Articles

Lifelog Retrieval From Daily Digital Data: Narrative Review.

BackgroundOver the past decade, the wide availability and small size of different types of sensors, together with the decrease in pricing, have allowed the acquisition of a substantial amount of data about a person’s life in real time. These sensors can be incorporated into personal electronic devices available at a reasonable cost, such as smartphones and small wearable devices. They allow the acquisition of images, audio, location, physical activity, and physiological signals among other data. With these data, usually denoted as lifelog data, we can then analyze and understand personal experiences and behaviors. This process is called lifelogging.ObjectiveThe objective of this paper was to present a narrative review of the existing literature about lifelogging over the past decade. To achieve this goal, we analyzed lifelogging applications used to retrieve relevant information from daily digital data, some of them with the purpose of monitoring and assisting people with memory issues and others designed for memory augmentation. We aimed for this review to be used by researchers to obtain a broad idea of the type of data used, methodologies, and applications available in this research field.MethodsWe followed a narrative review methodology to conduct a comprehensive search for relevant publications in Google Scholar and Scopus databases using lifelog topic–related keywords. A total of 411 publications were retrieved and screened. Of these 411 publications, 114 (27.7%) publications were fully reviewed. In addition, 30 publications were manually included based on our bibliographical knowledge of this research field.ResultsFrom the 144 reviewed publications, a total of 113 (78.5%) were selected and included in this narrative review based on content analysis. The findings of this narrative review suggest that lifelogs are prone to become powerful tools to retrieve memories or increase knowledge about an individual’s experiences or behaviors. Several computational tools are already available for a considerable range of applications. These tools use multimodal data of different natures, with visual lifelogs being one of the most used and rich sources of information. Different approaches and algorithms to process these data are currently in use, as this review will unravel. Moreover, we identified several open questions and possible lines of investigation in lifelogging.ConclusionsThe use of personal lifelogs can be beneficial to improve the quality of our life, as they can serve as tools for memory augmentation or for providing support to people with memory issues. Through the acquisition and analysis of lifelog data, lifelogging systems can create digital memories that can be potentially used as surrogate memory. Through this narrative review, we understand that contextual information can be extracted from lifelogs, which provides an understanding of the daily life of a person based on events, experiences, and behaviors.

Personal Trajectory with Ring Structure Network: Algorithms and Experiments

Network theory has provided a new analytical tool for the study of human trajectory and has also achieved rapid development in the complex network field. Conventional network model or complex network model ignores some details and cannot display the most remarkable features for a GPS based personal trajectory. It is necessary to set up a new personal trajectory model. For the purpose of researching the characteristics of trajectory for one person in a long time, we collected a GPS based personal LifeLog dataset named Liu Lifelog in the past 9 years. This paper analyzed the Liu Lifelog and proposed a ring structure personal trajectory (RSPT) model based on the basic complex network model. We discussed the definition, source, characteristic and attribute of the RSPT model and tested the model with the dataset which was provided by the Geolife project and verified that the model described the characteristic of trajectory for a person well. The result shows that this model is feasible and it can predict the human behavior characteristics more accurately and effectively.

Recommendation of indoor luminous environment for occupants using big data analysis based on machine learning

To provide an optimal luminous environment for occupants, a personalized luminous environment can be recommended by analyzing personal lifelog data. A basic platform for collecting lifelog data was constructed based on a previous study, and the collected data were classified into three types: task, fatigue, and emotion. Twelve tasks were defined, and appropriate ranges of illuminance and correlated color temperature (CCT) were recommended for each task. In addition, fatigue was divided into four levels, and the most appropriate values of illuminance and CCT were specified within the ranges recommended for each task. In addition, the lighting colors that can alter or improve emotions were designated by selecting the first and second priorities among the five emotions. Totally, 31,680 luminous environment data were processed based on the collected lifelog data, which were divided into training and test datasets. A machine learning (ML)-based luminous environment recommendation system was constructed by applying four ML algorithms (K-nearest neighbor, decision tree, random forest, and support vector machine). The system was designed to recommend an occupant-customized luminous environment based on the task type, fatigue level, and emotion class, and showed an accuracy of approximately 92% or higher.

Toward storytelling from personal informative lifelogging

The authors began collecting personal informative lifelogging data in 2011. The data collected in this article is a discontinuous data set, each of which includes one or more images, a GPS message, a description of a location, and a description of the text, we call this informative lifelogging. However, for the purposes automatically building a story from a huge collection of unstructured egocentric data presents major challenges. This paper first introduces the structure and characteristics of the collected data and uses the DB-scan algorithm to classify the data. Then a model for generating a story is proposed, and a model of story generation based on a story template is proposed in the model. The author implemented a complete software system through code, described a story generation model, and gave the key algorithm to generate stories. Through this system, 418 stories were generated automatically, of which 62% of the stories were particularly accurate. The experimental results verify that it is feasible to automatically generate stories based on personal Informative lifelogging data.

Invited papers] Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge (LSC2018)

The Lifelog Search Challenge (LSC) is an international content retrieval competition that evaluates search for personal lifelog data. At the LSC, content-based search is performed over a multi-modal dataset, continuously recorded by a lifelogger over 27 days, consisting of multimedia content, biometric data, human activity data, and information activities data. In this work, we report on the first LSC that took place in Yokohama, Japan in 2018 as a special workshop at ACM International Conference on Multimedia Retrieval 2018 (ICMR 2018). We describe the general idea of this challenge, summarise the participating search systems as well as the evaluation procedure, and analyse the search performance of the teams in various aspects. We try to identify reasons why some systems performed better than others and provide an outlook as well as open issues for upcoming iterations of the challenge.

Requirement Analysis and Implementation of Smart Emergency Medical Services

Emergency medical service (EMS) occurs in a high-pressure and error-prone environment, where paramedics must provide prompt decisions in care while recording information with limited time, incomplete data, restricted resources, and competing priorities. The EMS requires cooperative workflows between patients or caregivers, paramedics and medical centers in the community. In a conventional EMS, they have difficulties in obtaining causes of emergencies and personal medical histories, which are important for a rapid and proper response. We analyzed the requirement of a smart EMS (SEMS) system and derived the key components in connected care environments leveraging information and communication technology. A survey of paramedics (n=113) revealed that a SEMS system using IoT technology should integrate personal lifelogs, electronic medical records, and patient monitoring in ambulances into pre-hospital care recording systems. It also addressed context-awareness in the EMS accelerates first responder’s activities, while supporting personalized care not only at the scene of the emergency but also during the entire hospital stay. Based on requirement analysis, we designed and implemented SEMS using health information standards to provide interoperability between devices and systems. As an application of SEMS, an example service is introduced: lifelog-connected EMS for stroke patients with a real-time location service for managing timeline of treatment.

Personal search system based on android using lifelog and machine learning

Lifelog is the foundation on which lifelong services and healthcare services are implemented in a smart home system. It also plays a major role in the sub-processes of the system because it acquires information about the home's residents for home automation and entertainment. Providing personalized services to individuals by acquiring and managing this personal lifelog information has great advantages in terms of service satisfaction and effectiveness. In this paper, we implemented a personal search system based on android that collected and stored an individual's lifelog based on nine smart phone sensors and used it to derive new meaningful information about the user. The activity recognition module for classifying the user's behavior, the naive Bayesian method, showed an accuracy of 88.23% and the area under the ROC curve value of 0.941. We designed and implemented density-based spatial clustering method in the module for extracting the point of interest and the participants filled out a satisfaction questionnaire to evaluate the search system. The proposed system efficiently uses a large amount of lifelog data and automates the process of extracting meaningful information, associating it according to the user's intention.

Multiple density maps information fusion for effectively assessing intensity pattern of lifelogging physical activity

Physical activity (PA) measurement is a crucial task in healthcare technology aimed at monitoring the progression and treatment of many chronic diseases. Traditional lifelogging PA measures require relatively high cost and can only be conducted in controlled or semi-controlled environments, though they exhibit remarkable precision of PA monitoring outcomes. Recent advancement of commercial wearable devices and smartphones for recording one’s lifelogging PA has popularized data capture in uncontrolled environments. However, due to diverse life patterns and heterogeneity of connected devices as well as the PA recognition accuracy, lifelogging PA data measured by wearable devices and mobile phones contains much uncertainty thereby limiting their adoption for healthcare studies. To improve the feasibility of PA tracking datasets from commercial wearable/mobile devices, this paper proposes a lifelogging PA intensity pattern decision making approach for lifelong PA measures. The method is to firstly remove some irregular uncertainties (IU) via an Ellipse fitting model, and then construct a series of monthly based hour-day density map images for representing PA intensity patterns with regular uncertainties (RU) on each month. Finally it explores Dempster-Shafer theory of evidence fusing information from these density map images for generating a decision making model of a final personal lifelogging PA intensity pattern. The approach has significantly reduced the uncertainties and incompleteness of datasets from third party devices. Two case studies on a mobile personalized healthcare platform MHA [1] connecting the mobile app Moves are carried out. The results indicate that the proposed approach can improve effectiveness of PA tracking devices or apps for various types of people who frequently use them as a healthcare indicator.

Information practices meet lifelogging technologies: Towards a successful multimethod research framework

ABSTRACTInformation Practice research emphasises on contextual factors such as the situation, social roles, collaborators, or communications of information behaviour. This wider scope allows us to gain enriched, deeper, and complex understanding of how people interact with information in various contexts. However, this also poses researchers many challenges in data collection and analyses, since conventional methods such as observations, interviews, surveys might not either possible or complete. Meanwhile, a range of wearable devices have become commodity and collecting a large amount of personal lifelog data has become much easier than before. On the other hand, the researchers who are familiar with qualitative methods and computational methods do not often communicate, although they are likely to have common research agenda.This proposed panel provides a unique opportunity to invite panellists from the two research communities, and discusses how lifelogging technologies may (or may not!) contribute and transform information practice research. The panel will also involve the audience in sharing their personal or scientific experience with lifelogging technologies, or their expectations or concerns regarding the use of lifelogging technologies in their research. Outcomes from the panel discussion should yield some clear pointers that can facilitate multimethod research in Information Practices.

Particle Swarm Optimization Using Adaptive Boundary Correction for Human Activity Recognition

As a kind of personal lifelog data, activity data have been considered as one of the most compelling information to understand the users habits and to calibrate diagnoses. In this paper, we proposed a robust algorithm to sampling rates for human activity recognition, which identifies a user’s activity using accelerations from a triaxial accelerometer in a smartphone. Although a high sampling rate is required for high accuracy, it is not desirable for actual smartphone usage, battery consumption, or storage occupancy. Activity recognitions with well-known algorithms, including MLP, C4.5, or SVM, suffer from a loss of accuracy when a sampling rate of accelerometers decreases. Thus, we start from particle swarm optimization (PSO), which has relatively better tolerance to declines in sampling rates, and we propose PSO with an adaptive boundary correction (ABC) approach. PSO with ABC is tolerant of various sampling rate in that it identifies all data by adjusting the classification boundaries of each activity. The experimental results show that PSO with ABC has better tolerance to changes of sampling rates of an accelerometer than PSO without ABC and other methods. In particular, PSO with ABC is 6%, 25%, and 35% better than PSO without ABC for sitting, standing, and walking, respectively, at a sampling period of 32 seconds. PSO with ABC is the only algo-rithm that guarantees at least 80% accuracy for every activity at a sampling period of smaller than or equal to 8 seconds.

Health Avatar: An Informatics Platform for Personal and Private Big Data

Health Avatar: An Informatics Platform for Personal and Private Big Data

Hierarchical structured data logging system for effective lifelog management in ubiquitous environment

The researches for collecting personal daily behaviors and providing lifelog services with them have been recently increasing. Recent advances in mobile devices and sensor technologies have motivated to collect a huge amount of personal lifelog data in real time. With the rapid growth of the need for the research, there is a coming need for the effective lifelog management system which collects time-series big lifelog data sent from sensing devices and extracts major activities through processing them. For the effective lifelog management, the lifelog data can be processed in separated computing resources depending on the size and level of data. In this paper, we propose hierarchical structured data logging to support lifelog based personal services and to reduce the processing complexity and storage cost. First, we present the architecture of personal lifelog management system. With the system we present hierarchical lifelog data logging to optimally utilize computing and storage resources. Then we describe cost analysis and performance comparison for demonstrating the efficacy of our proposed system. Finally, as an initial step for experiments in our research, we describe experimental results of recognizing physical activities and extracting lifelog data which indicate major activities from them.

Personal Lifelog Upstreaming System for Secure and Reliable Lifelog Transmission

Personal Lifelog Upstreaming System for Secure and Reliable Lifelog Transmission

A single tri-axial accelerometer-based real-time personal life log system capable of human activity recognition and exercise information generation

Recording a personal life log (PLL) of daily activities in a ubiquitous environment is an emerging application of information technology. In this work, we present a single tri-axial accelerometer-based PLL system capable of human activity recognition and exercise information generation. Our PLL system exhibits two main functions: activity recognition and exercise information generation. For activity recognition, the system first recognizes a state of daily activities based on the statistical and spectral features of the accelerometer signals. An activity within the recognized state is then recognized using a set of augmented features, including autoregressive coefficients, signal magnitude area, and tilt angle, via linear discriminant analysis and hierarchical artificial neural networks. Upon the recognition of each activity, the system further estimates exercise information that includes energy expenditure based on metabolic equivalents, stride length, step count, walking distance, and walking speed. Our PLL system operates in real-time, and the life log information it generates is archived in a daily log database. We have validated our PLL system for six daily activities (i.e., lying, standing, walking, going-upstairs, going-downstairs, and driving) via subject-independent and subject-dependent recognition on a total of twenty subjects, achieving an average recognition accuracy of 94.43 and 96.61%, respectively. Our results demonstrate the feasibility of a portable real-time PLL system that could be used for u-lifecare and u-healthcare services in the near future.

다양한 기기로부터의 데이터 단일 표현을 통한 개인 미멕스 시스템

한 개인이 일상 생활을 하면서 겪는 사건을 자동으로 기록하고 탐색하는 시스템은 비교적 최근에 활발히 연구되고 있는 분야이다. 개인 미멕스(personal memex) 또는 라이프로그(life log)라 불리는 이러한 시스템은 MyLifeBits 프로젝트의 센스캠(SenseCam)처럼 보통 이를 위한 전용 디지털 기기를 수반한다. 본 연구에서는 개인의 일상 생활을 담는 매체로서 이러한 현대인이 일상적으로 사용하는 휴대폰, 신용카드, 디지털 카메라 등의 매체를 주목하였다. 이러한 매체에 직접 기록되는 일상 생활 또는 이러한 매체를 통해 상용 서비스를 제공하는 업체의 사이트 (예를 들면 휴대폰 회사 통화 기록)에 기록되는 일상 생활을 전송받아 체계적으로 저장하고 추후 빠르게 참조할 수 있도록 하였다. MyMemex라 불리는 본 시스템의 데이터 수집 에이전트는 이러한 싸이트에서 제공하는 웹 서비스를 이용하여 개인의 라이프로그 '웹 데이터'를 수집하여 서버에 저장한다. 또한 디지털 기기에 저장되는 '파일 데이터'를 로드하여 서버에 저장한다. 본 연구에서는 이러한 웹 데이터 또는 파일 데이터를 4W1H로 기술되는 하나의 미멕스 사건으로 보아 각 서비스마다 다른 형태를 가지는 데이터를 4 W1H 미멕스 사건 데이터로 변환한다. 이러한 변환에는 미멕스 사건 온톨로지를 사용한다. 웹 기반으로 제공되는 본 시스템에서 사용자는 자신의 일상 기록을 시간 순으로 볼 수 있고 특정 키워드를 이용하여 검색할 수 있다. 또한 미멕스 사건들이 문장으로 변환되어 일기 또는 이야기 형식으로 전개될 수 있다. 관련된 일련의 미멕스 사건들은 '에피소드'로 자동으로 군집화되어 보여진다. 저자 중 한명의 실제 라이프로그 데이터를 사용하여 에피소드 군집화를 실험한 결과 높은 정확도를 보였다. The researches on the system that automatically records and retrieves one's everyday life is relatively actively worked recently. These systems, called personal memex or life log, usually entail dedicated devices such as SenseCam in MyLifeBits project. This research paid attention to the digital devices such as mobile phones, credit cards, and digital camera that people use everyday. The system enables a person to store everyday life systematically that are saved in the devices or the deviced-related web pages (e.g., phone records in the cellular phone company) and to refer this quickly later. The data collection agent in the proposed system, called MyMemex, collects the personal life log "web data" using the web services that the web sites provide and stores the web data into the server. The "file data" stored in the off-line digital devices are also loaded into the server. Each of the file data or web data is viewed as a memex event that can be described by 4W1H form. The different types of data in different services are transformed into the memex event data in 4W1H form. The memex event ontology is used in this transform. Users can sign in to the web server of this service to view their life logs in the chronological manner. Users can also search the life logs using keywords. Moreover, the life logs can be viewed as a diary or story style by converting the memex events to sentences. The related memex events are grouped to be displayed as an "episode" by a heuristic identification method. A result with high accuracy has been obtained by the experiment for the episode identification using the real life log data of one of the authors.