Behavior In Natural Settings Research Articles

The visual experience dataset: Over 200 recorded hours of integrated eye movement, odometry, and egocentric video.

We introduce the Visual Experience Dataset (VEDB), a compilation of more than 240hours of egocentric video combined with gaze- and head-tracking data that offer an unprecedented view of the visual world as experienced by human observers. The dataset consists of 717 sessions, recorded by 56 observers ranging from 7 to 46 years of age. This article outlines the data collection, processing, and labeling protocols undertaken to ensure a representative sample and discusses the potential sources of error or bias within the dataset. The VEDB's potential applications are vast, including improving gaze-tracking methodologies, assessing spatiotemporal image statistics, and refining deep neural networks for scene and activity recognition. The VEDB is accessible through established open science platforms and is intended to be a living dataset with plans for expansion and community contributions. It is released with an emphasis on ethical considerations, such as participant privacy and the mitigation of potential biases. By providing a dataset grounded in real-world experiences and accompanied by extensive metadata and supporting code, the authors invite the research community to use and contribute to the VEDB, facilitating a richer understanding of visual perception and behavior in naturalistic settings.

Neural Mechanisms Determining the Duration of Task-free, Self-paced Visual Perception.

Humans spend hours each day spontaneously engaging with visual content, free from specific tasks and at their own pace. Currently, the brain mechanisms determining the duration of self-paced perceptual behavior remain largely unknown. Here, participants viewed naturalistic images under task-free settings and self-paced each image's viewing duration while undergoing EEG and pupillometry recordings. Across two independent data sets, we observed large inter- and intra-individual variability in viewing duration. However, beyond an image's presentation order and category, specific image content had no consistent effects on spontaneous viewing duration across participants. Overall, longer viewing durations were associated with sustained enhanced posterior positivity and anterior negativity in the ERPs. Individual-specific variations in the spontaneous viewing duration were consistently correlated with evoked EEG activity amplitudes and pupil size changes. By contrast, presentation order was selectively correlated with baseline alpha power and baseline pupil size. Critically, spontaneous viewing duration was strongly predicted by the temporal stability in neural activity patterns starting as early as 350msec after image onset, suggesting that early neural stability is a key predictor for sustained perceptual engagement. Interestingly, neither bottom-up nor top-down predictions about image category influenced spontaneous viewing duration. Overall, these results suggest that individual-specific factors can influence perceptual processing at a surprisingly early time point and influence the multifaceted ebb and flow of spontaneous human perceptual behavior in naturalistic settings.

Picturing self-harm: Investigating flash-forward mental imagery as a proximal and modifiable driver of non-suicidal self-injury.

Non-suicidal self-injury (NSSI) is theorized to be reinforced by its emotional consequences. Mental images of NSSI are commonly reported as occurring prior to NSSI. Based on the known functional properties of anticipatory mental imagery as an emotional and motivational amplifier, this study investigated whether NSSI mental imagery constitutes a proximal and dynamic mechanism underpinning NSSI risk. An intensive ecological momentary assessment (EMA) study was conducted to track the occurrence and characteristics of NSSI mental imagery alongside NSSI urge and behavior in naturalistic settings. A sample of N = 43 individuals aged 17 to 24 with a history of repetitive NSSI completed EMA surveys seven times a day for 14 days. Mental preoccupation in the form of NSSI mental imagery-based flash-forwards to the actions, bodily sensations, and emotional benefits of NSSI was found to occur when NSSI urge was high but not when urge was low. Critically, objective cross-panel analyses showed that higher frequencies of NSSI imagery occurrence predicted greater future NSSI urge and increased likelihood of acting on urge, over and above current urge. Mental imagery of NSSI is not simply an epiphenomenal by-product of NSSI urge and may constitute a dynamic and proximal novel intervention target.

Intonation Units in Spontaneous Speech Evoke a Neural Response.

Spontaneous speech is produced in chunks called intonation units (IUs). IUs are defined by a set of prosodic cues and presumably occur in all human languages. Recent work has shown that across different grammatical and sociocultural conditions IUs form rhythms of ∼1 unit per second. Linguistic theory suggests that IUs pace the flow of information in the discourse. As a result, IUs provide a promising and hitherto unexplored theoretical framework for studying the neural mechanisms of communication. In this article, we identify a neural response unique to the boundary defined by the IU. We measured the EEG of human participants (of either sex), who listened to different speakers recounting an emotional life event. We analyzed the speech stimuli linguistically and modeled the EEG response at word offset using a GLM approach. We find that the EEG response to IU-final words differs from the response to IU-nonfinal words even when equating acoustic boundary strength. Finally, we relate our findings to the body of research on rhythmic brain mechanisms in speech processing. We study the unique contribution of IUs and acoustic boundary strength in predicting delta-band EEG. This analysis suggests that IU-related neural activity, which is tightly linked to the classic Closure Positive Shift (CPS), could be a time-locked component that captures the previously characterized delta-band neural speech tracking.SIGNIFICANCE STATEMENT Linguistic communication is central to human experience, and its neural underpinnings are a topic of much research in recent years. Neuroscientific research has benefited from studying human behavior in naturalistic settings, an endeavor that requires explicit models of complex behavior. Usage-based linguistic theory suggests that spoken language is prosodically structured in intonation units. We reveal that the neural system is attuned to intonation units by explicitly modeling their impact on the EEG response beyond mere acoustics. To our understanding, this is the first time this is demonstrated in spontaneous speech under naturalistic conditions and under a theoretical framework that connects the prosodic chunking of speech, on the one hand, with the flow of information during communication, on the other.

Animals and diplomacy: on the prospect for interspecies diplomacy

If diplomacy is considered an alternative to war, can the ongoing human ‘war against animals’ be replaced with diplomacy between humans and other animals? While many scholars and practitioners of diplomacy can be expected to dismiss such an idea out of hand, this essay encourages us to think more seriously and thoroughly about what it might imply to engage diplomatically with nonhuman animals. Doing so requires a somewhat unconventional conception of diplomacy, and some scholars have already done much to rethink diplomacy in suitable ways (despite the persistent anthropocentrism). Combining such work with political science scholarship on human-animal relations, indigenous peoples’ relations with animals, various notions of animal ambassadorship and the study of animal behaviour in natural settings, the essay argues that interspecies diplomacy is possible and urges scholars to further explore this and how the possibility in question can be translated into reality.

Wild social behavior differs following experimental loss of vision in social hermit crabs

Even for animals with multiple senses at their disposal, there may be a strong reliance on a single sense, like vision, for social behavior. Experimentally blocking or eliminating vision offers a powerful means of testing impacts on social behavior, though few studies have followed experimentally blinded individuals in the wild to test potential changes in social behavior in natural settings. Here we conducted experiments with social hermit crabs (Coenobita compressus), applying opaque material overtop their eyes to temporarily blind individuals. We then released these experimentally blinded individuals and non-blinded control individuals into the wild as well as into captive social settings. Compared to control individuals, experimentally blinded individuals initiated significantly fewer social contacts with conspecifics in the wild. These experimentally blinded individuals were not, however, differentially targeted by conspecifics. Interestingly, unlike the wild experiments, the captive experiments showed no differences in social behavior between experimentally blinded and non-blinded control individuals, suggesting that experiments in natural settings in the wild may be essential to fully unraveling impacts of blindness on social behavior. Broadly, for social animals that are highly reliant on the visual modality, social behavior may change dramatically if they lose their vision.

Behaviors of pedestrians on streets with roadside strips

Pedestrian behaviors in natural settings have been investigated mainly on roads with sidewalks or the corridors outside or inside buildings. The present study examined pedestrian behaviors on roads with roadside strips, which are popular in Japanese local areas. The purpose of the study was to investigate pedestrian behaviors and the factors affecting them. Observations were made at two single-lane streets in Tokyo with 0.9 m and 0.5 m roadside strips. A total of 2,476 pedestrians were observed for 2 days (1 weekday, 1 weekend day) in each location. Walking with traffic was more popular (61%) than walking against traffic. People seemed to walk either side of the street depending on their turn direction at the forward intersection. Illegal behavior of walking into the roadway was also observed in 46% of pedestrians. This behavior was more likely to appear on the streets with narrower roadside strips and when people walked in a group. As expected, males, young and mature adults, and single pedestrians walked faster than females, children and senior adults, and pedestrians in a group. When a female and a male walked abreast, males tended to reduce their walking speed considerably and walk on the roadway side (62%), taking care of females. Similar positional consideration for a vulnerable partner was also found in parent–child pairs. It was concluded that people are likely to violate traffic law if the violation serves a purpose, and they tend to behave altruistically towards others, even if there is no relevant rule.

PainRhythms: Machine learning prediction of chronic pain from circadian dysregulation using actigraph data — a preliminary study

Chronic pain is currently diagnosed using verbal self-reports, which present a challenge for patients with cognitive or physiological disorders. Prior work has explored machine learning prediction of pain from clinical data, which requires active user involvement and does not capture their behavior in natural settings. Passive objective assessment is desirable. Circadian Rhythms, including sleep-wake cycles, are biological processes that reoccur every 24 h and can be derived from physiological data such as heart rate, activity, and sleep, gathered using widely-owned smart wearables. This study investigated the feasibility of using machine learning and rest-activity circadian rhythm features to predict patients’ pain, including pain intensity, its interference with the patient’s life (dysregulation), and their difficulty in performing physical functions using passively gathered actigraphy data. To predict pain on day N , actigraphy data collected over that day were analyzed. Three sets of feature were extracted: (1) Activity (total sedentary bouts/time/breaks, % in sedentary/light/ moderate activity), (2) Sleep (sleep efficiency/latency, wake after sleep onset), and (3) Rest Activity Rhythm (mesor, acrophase, Intradaily Variability (IV)). These features were then classified using various machine learning algorithms. Our proposed PainRhythms approach achieved an average AUC-ROC of 0.97 with a stacking machine learning classifier for predicting pain, 0.67 and 0.62 with logistic regression for pain intensity and interference, and 0.56 with gradient boosting for physical function. We found that chronic pain predictions were more accurate using rest-activity rhythm features than sleep or activity features. Of all the rhythmic features, Intradaily Variability (IV) was the most predictive feature, with elevated values in pain associated with disturbed sleep. PainRhythms provides preliminary evidence that rest-activity rhythms can effectively detect subjects with chronic pain. In future work, we aim to gather more data and confirm our preliminary findings on a large, class-balanced and diverse dataset. • PainRhythms provides preliminary evidence that rest-activity rhythms can effectively detect subjects with chronic pain. • Chronic pain predictions were more accurate using rest-activity rhythms than sleep or activity. • Intradaily Variability (IV) was the most predictive feature, with elevated values in pain associated with disturbed sleep. • Sleep disturbance was more associated with pain and pain intensity. • Besides sleep disturbance, features encompassing the subject’s physical activity and sleep quality also influenced pain interference and physical function.

Ten Lessons About Infants’ Everyday Experiences

Audio recorders, accelerometers, and cameras that infants wear throughout their everyday lives capture the experiences that are available to shape development. Using sensors to capture behaviors in natural settings can reveal patterns within the everyday hubbub that are unknowable using methods that capture shorter, more isolated, or more planned slices of behavior. Here, we review 10 lessons learned from recent endeavors in which researchers neither designed nor participated in infants’ experiences and instead quantified patterns that arose within infants’ own spontaneously arising everyday experiences. The striking heterogeneity of experiences—the fact that there is no meaningfully “representative” hour of a day, instance of a category, interaction context, or infant—inspires next steps in theory and practice that embrace the complex, dynamic, and multiple pathways of human development.

Determination of Chewing Count from Video Recordings Using Discrete Wavelet Decomposition and Low Pass Filtration.

Several studies have shown the importance of proper chewing and the effect of chewing speed on the human health in terms of caloric intake and even cognitive functions. This study aims at designing algorithms for determining the chew count from video recordings of subjects consuming food items. A novel algorithm based on image and signal processing techniques has been developed to continuously capture the area of interest from the video clips, determine facial landmarks, generate the chewing signal, and process the signal with two methods: low pass filter, and discrete wavelet decomposition. Peak detection was used to determine the chew count from the output of the processed chewing signal. The system was tested using recordings from 100 subjects at three different chewing speeds (i.e., slow, normal, and fast) without any constraints on gender, skin color, facial hair, or ambience. The low pass filter algorithm achieved the best mean absolute percentage error of 6.48%, 7.76%, and 8.38% for the slow, normal, and fast chewing speeds, respectively. The performance was also evaluated using the Bland-Altman plot, which showed that most of the points lie within the lines of agreement. However, the algorithm needs improvement for faster chewing, but it surpasses the performance of the relevant literature. This research provides a reliable and accurate method for determining the chew count. The proposed methods facilitate the study of the chewing behavior in natural settings without any cumbersome hardware that may affect the results. This work can facilitate research into chewing behavior while using smart devices.

Is repairability enough? big data insights into smartphone obsolescence and consumer interest in repair

A dominant narrative surrounding smartphone lifespans suggests that their objective functional capabilities deteriorate rapidly and that if only devices were more repairable consumers would use them longer thereby reducing demand for new production and e-waste generation. Here we use a big-data approach to help unpack this narrative and examine two related yet distinct aspects: smartphone performance and obsolescence, and consumers interest in repair. Examining over 3.5 million iPhone benchmarking test scores, we reveal that the objective performance of devices remains very stable over time and does not rapidly deteriorate as common wisdom might suggest. In contrast, testing frequency varies substantially. This discrepancy suggests that factors other than objective performance meaningfully influence consumers' perceptions of smartphone functionality and obsolescence. Relatedly, our analysis of 22 million visits to a website offering free repair manuals revels that interest in repair declines exponentially over time and that repairability does not necessarily prolong consumer's interest in repair. Taken together, our findings indicate that non-technical aspects, such as mental depreciation and perceived obsolescence play a critical role in determining smartphone lifespans, and suggest that focus on the technical aspects of repairability as currently discussed by policy makers is unlikely to yield the desired extension in smartphone lifespan. We propose that sustainability advocates try to avoid narratives of planned obsolescence which might have counterproductive impacts on perceived obsolescence and consumer's’ interest in repair, and instead highlight how well devices perform over time. More broadly, this work demonstrates the potential of using novel datasets to directly observe consumer behavior in natural settings, and improve our general understanding of issues such as planned obsolescence and repair.

Values, Beliefs, Norms, and Conservation-Oriented Behaviors toward Native Fish Biodiversity in Rivers: Evidence from Four European Countries

Riverine biodiversity in Europe is under threat from a range of anthropogenic factors. Key to effective biodiversity conservation is the public's willingness to support restoration efforts. Based on value-belief-norm (VBN) theory and using a longitudinal survey design with n = 1,000 respondents per each of four countries (France, Germany, Norway, Sweden) we measured individual conservation-oriented behaviors in natural settings over time (e.g., signing a petition, donating money) that benefit native river fish biodiversity. We also examined sociopsychological determinants of these behaviors. In addition to behavioral intentions and self-reported behaviors, we measured actual behavior (monetary donations). We found broad support for the VBN theory but also relevant cultural diversity. In France, Norway, and Sweden fish value orientations affected conservation-oriented behaviors, whereas in Germany general ecological worldviews had more explanatory power. Conservation-oriented outreach and information campaigns will be most effective when taking between-country differences in the relationship between beliefs and behaviors into account.

Automated measurement of long-term bower behaviors in Lake Malawi cichlids using depth sensing and action recognition

In the wild, behaviors are often expressed over long time periods in complex and dynamic environments, and many behaviors include direct interaction with the environment itself. However, measuring behavior in naturalistic settings is difficult, and this has limited progress in understanding the mechanisms underlying many naturally evolved behaviors that are critical for survival and reproduction. Here we describe an automated system for measuring long-term bower construction behaviors in Lake Malawi cichlid fishes, in which males use their mouths to sculpt sand into large species-specific structures for courtship and mating. We integrate two orthogonal methods, depth sensing and action recognition, to simultaneously track the developing bower structure and the thousands of individual sand manipulation behaviors performed throughout construction. By registering these two data streams, we show that behaviors can be topographically mapped onto a dynamic 3D sand surface through time. The system runs reliably in multiple species, across many aquariums simultaneously, and for up to weeks at a time. Using this system, we show strong differences in construction behavior and bower form that reflect species differences in nature, and we gain new insights into spatial, temporal, social dimensions of bower construction, feeding, and quivering behaviors. Taken together, our work highlights how low-cost tools can automatically quantify behavior in naturalistic and social environments over long timescales in the lab.

Mental Shopping Calculations: A Transcranial Magnetic Stimulation Study.

One of the most critical skills behind consumer’s behavior is the ability to assess whether a price after a discount is a real bargain. Yet, the neural underpinnings and cognitive mechanisms associated with such a skill are largely unknown. While there is general agreement that the posterior parietal cortex (PPC) on the left is critical for mental calculations, and there is also recent repetitive transcranial magnetic stimulation (rTMS) evidence pointing to the supramarginal gyrus (SMG) of the right PPC as crucial for consumer-like arithmetic (e.g., multi-digit mental addition or subtraction), it is still unknown whether SMG is involved in calculations of sale prices. Here, we show that the neural mechanisms underlying discount arithmetic characteristic for shopping are different from complex addition or subtraction, with discount calculations engaging left SMG more. We obtained these outcomes by remodeling our laboratory to resemble a shop and asking participants to calculate prices after discounts (e.g., $8.80–25 or $4.80–75%), while stimulating left and right SMG with neuronavigated rTMS. Our results indicate that such complex shopping calculations as establishing the price after a discount involve SMG asymmetrically, whereas simpler calculations such as price addition do not. These findings have some consequences for neural models of mathematical cognition and shed some preliminary light on potential consumer’s behavior in natural settings.

Constant Sub-second Cycling between Representations of Possible Futures in the Hippocampus

Cognitive faculties such as imagination, planning, and decision-making entail the ability to represent hypothetical experience. Crucially, animal behavior in natural settings implies that the brain can represent hypothetical future experience not only quickly but also constantly over time, as external events continually unfold. To determine how this is possible, we recorded neural activity in the hippocampus of rats navigating a maze with multiple spatial paths. We found neural activity encoding two possible future scenarios (two upcoming maze paths) in constant alternation at 8Hz: one scenario per ∼125-ms cycle. Further, we found that the underlying dynamics of cycling (both inter- and intra-cycle dynamics) generalized across qualitatively different representational correlates (location and direction). Notably, cycling occurred across moving behaviors, including during running. These findings identify a general dynamic process capable of quickly and continually representing hypothetical experience, including that of multiple possible futures.

ClassEx — an online tool for lab-in-the-field experiments with smartphones

classEx is an online tool for running experiments with smartphones. It can be employed for lab-in-the-field experiments, allowing the investigation of behavior in natural settings and with large groups, even if their size is unknown and varying. Standard experiments with multiple treatments, monetary incentives, roles, groups, rounds and stages are available in a ready-made format. An easy-to-use modular backend system allows users to conveniently implement their own experiments. Replication is facilitated by the possibility of sharing experiments and results with other researchers. We provide some applications, describe the functionality and show how classEx tackles the challenges of lab-in-the-field research.

Outdoor Fitness Equipment Usage Behaviors in Natural Settings.

Outdoor fitness equipment (OFE) areas have become a popular form of built environment infrastructure in public open spaces as a means to improve public health through increased physical activity. However, the benefits of using OFE are not consistent, and several OFE accidents have been reported. In this study, we videotaped how OFE users operate OFE in parks and selected four types of popular OFE (the waist twister, air walker, ski machine, and waist/back massager) for video content analysis. Furthermore, we established coding schemes and compared results with the instructions provided by OFE manufacturers. The results revealed various usage behaviors for the same OFE types. In addition, we observed that a significant portion of user behaviors did not follow manufacturers’ instructions, which might pose potential risks or actually cause injuries. Children are especially prone to act improperly. This study provides empirical evidence indicating the existence of potential safety risks due to inappropriate usage behaviors that might lead to accidents and injuries while using OFE. This study provides crucial information that can be used to evaluate the effectiveness of OFE and to develop future park or open space initiatives.

Extraction without tooling around — The first comprehensive description of the foraging- and socio-ecology of wild Goffin’s cockatoos (Cacatua goffiniana)

Abstract When tested under laboratory conditions, Goffin’s cockatoos (Cacatua goffiniana) demonstrate numerous sophisticated cognitive skills. Most importantly, this species has shown the ability to manufacture and use tools. However, little is known about the ecology of these cockatoos, endemic to the Tanimbar Islands in Indonesia. Here we provide first insights into the feeding- and socio-ecology of the wild Goffin’s cockatoos and propose potential links between their behaviour in natural settings and their advanced problem-solving capacities shown in captivity. Observational data suggests that Goffin’s cockatoos rely on a large variety of partially seasonal resources. Furthermore, several food types require different extraction techniques. These ecological and behavioural characteristics fall in line with current hypotheses regarding the evolution of complex cognition and innovativeness. We discuss how the efficiency of (extractive) foraging may preclude tool use in wild Goffin’s cockatoos, even though the corresponding cognitive and ecological prerequisites seem to be present.

Fuelling on the wing: sensory ecology of hawkmoth foraging

Hawkmoths (Lepidoptera, Sphingidae) comprise around 1500 species, most of which forage on nectar from flowers in their adult stage, usually while hovering in front of the flower. The majority of species have a nocturnal lifestyle and are important nocturnal pollinators, but some species have turned to a diurnal lifestyle. Hawkmoths use visual and olfactory cues including CO2 and humidity to detect and recognise rewarding flowers; they find the nectary in the flowers by means of mechanoreceptors on the proboscis and vision, evaluate it with gustatory receptors on the proboscis, and control their hovering flight position using antennal mechanoreception and vision. Here, we review what is presently known about the sensory organs and sensory-guided behaviour that control feeding behaviour of this fascinating pollinator taxon. We also suggest that more experiments on hawkmoth behaviour in natural settings are needed to fully appreciate their sensory capabilities.

Analyzing the effect of fog weather conditions on driver lane-keeping performance using the SHRP2 naturalistic driving study data

Introduction: Driving in foggy weather conditions has been recognized as a major safety concern for many years. Driver behavior and performance can be negatively affected by foggy weather conditions due to the low visibility in fog. A number of previous studies focused on driver performance and behavior in simulated environments. However, very few studies have examined the impact of foggy weather conditions on specific driver behavior in naturalistic settings. Method: This study utilized the second Strategic Highway Research Program (SHRP2) Naturalistic Driving Study (NDS) dataset to evaluate driver lane-keeping behavior in clear and foggy weather conditions. Preliminary descriptive analysis was conducted and a lane-keeping model was developed using the ordered logistic regression approach to achieve the study goals. Results: This study found that individual variables such as visibility, traffic conditions, lane change, driver marital status, and geometric characteristics, as well as some interaction terms (i.e., weather and gender, surface condition and driving experience, speed limit and mileage last year) significantly affect lane-keeping ability. An important finding of this study illustrated that affected visibility caused by foggy weather conditions decreases lane-keeping ability significantly. More specifically, drivers in affected visibility conditions showed 1.37 times higher Standard Deviation of Lane Position (SDLP) in comparison with drivers who were driving in unaffected visibility conditions. Conclusions: These results provide a better understanding of driver lane-keeping behavior and driver perception of foggy weather conditions. Moreover, the results might be used to improve Lane Departure Warning (LDW) systems algorithm by allowing them to account for the effects of fog on visibility. Practical Applications: These results provide a better understanding of driver lane-keeping behavior and driver perception of foggy weather conditions. Moreover, the results might be used to improve Lane Departure Warning (LDW) systems algorithm by allowing them to account for the effects of fog on visibility.