Human-centric Tasks Research Articles

Data augmentation in human-centric vision

AbstractThis survey presents a comprehensive analysis of data augmentation techniques in human-centric vision tasks, a first of its kind in the field. It delves into a wide range of research areas including person ReID, human parsing, human pose estimation, and pedestrian detection, addressing the significant challenges posed by overfitting and limited training data in these domains. Our work categorizes data augmentation methods into two main types: data generation and data perturbation. Data generation covers techniques like graphic engine-based generation, generative model-based generation, and data recombination, while data perturbation is divided into image-level and human-level perturbations. Each method is tailored to the unique requirements of human-centric tasks, with some applicable across multiple areas. Our contributions include an extensive literature review, providing deep insights into the influence of these augmentation techniques in human-centric vision and highlighting the nuances of each method. We also discuss open issues and future directions, such as the integration of advanced generative models like Latent Diffusion Models, for creating more realistic and diverse training data. This survey not only encapsulates the current state of data augmentation in human-centric vision but also charts a course for future research, aiming to develop more robust, accurate, and efficient human-centric vision systems.

Algorithmic management and human-centered task design: a conceptual synthesis from the perspective of action regulation and sociomaterial systems theory.

This paper aims to explain potential psychological effects of algorithmic management (AM) on human-centered task design and with that also workers' mental well-being. For this, we link research on algorithmic management (AM) with Sociomaterial System Theory and Action Regulation Theory (ART). Our main assumption is that psychological effects of sociomaterial systems, such as AM, can be explained by their impact on human action. From the synthesis of the theories, mixed effects on human-centered task design can be derived: It can be expected that AM contributes to fewer action regulation opportunities (i.e., job resources like job autonomy, transparency, predictability), and to lower intellectual demands (i.e., challenge demands like task complexity, problem solving). Moreover, it can be concluded that AM is related with more regulation problems (i.e., hindrance demands like overtaxing regulations) but also fewer regulation problems (like regulation obstacles, uncertainty). Based on these considerations and in line with the majority of current research, it can be assumed that the use of AM is indirectly associated with higher risks to workers' mental well-being. However, we also identify potential positive effects of AM as some stressful and demotivating obstacles at work are often mitigated. Based on these considerations, the main question of future research is not whether AM is good or bad for workers, but rather how work under AM can be designed to be humane. Our proposed model can guide and support researchers and practitioners in improving the understanding of the next generation of AM systems.

The first step is the hardest: pitfalls of representing and tokenizing temporal data for large language models.

Large language models (LLMs) have demonstrated remarkable generalization and across diverse tasks, leading individuals to increasingly use them as personal assistants due to their emerging reasoning capabilities. Nevertheless, a notable obstacle emerges when including numerical/temporal data into these prompts, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. This article discusses the challenges of representing and tokenizing temporal data. It argues that naively passing timeseries to LLMs can be ineffective due to the modality gap between numbers and text. We conduct a case study by tokenizing a sample mobile sensing dataset using the OpenAI tokenizer. We also review recent works that feed timeseries data into LLMs for human-centric tasks, outlining common experimental setups like zero-shot prompting and few-shot learning. The case study shows that popular LLMs split timestamps and sensor values into multiple nonmeaningful tokens, indicating they struggle with temporal data. We find that preliminary works rely heavily on prompt engineering and timeseries aggregation to "ground" LLMs, hinting that the "modality gap" hampers progress. The literature was critically analyzed through the lens of models optimizing for expressiveness versus parameter efficiency. On one end of the spectrum, training large domain-specific models from scratch is expressive but not parameter-efficient. On the other end, zero-shot prompting of LLMs is parameter-efficient but lacks expressiveness for temporal data. We argue tokenizers are not optimized for numerical data, while the scarcity of timeseries examples in training corpora exacerbates difficulties. We advocate balancing model expressiveness and computational efficiency when integrating temporal data. Prompt tuning, model grafting, and improved tokenizers are highlighted as promising directions. We underscore that despite promising capabilities, LLMs cannot meaningfully process temporal data unless the input representation is addressed. We argue that this paradigm shift in how we leverage pretrained models will particularly affect the area of biomedical signals, given the lack of modality-specific foundation models.

Collaborative human-centered design of manufacturing tasks: a multi-user immersive VR experience

Work-related musculoskeletal disorders are among the top factors impacting the individual’s performance and the overall company’s productivity. Virtual Reality (VR) has enabled proactive consideration of human factors. However, there are still barriers impeding the broader adoption of human-in-the-loop approaches, including the number and roles of people involved in simulation stages and the complexity of modelling lengthy task sequences. This paper presents a multi-user immersive VR simulation framework for collaborative human-centered task evaluation involving engineers and operators. The software architecture enabling multi-user VR sessions is discussed, together with a hierarchical approach for modelling complex manufacturing tasks in VR applications and collecting structured data from the simulations. Metrics such as heatmap and Rapid Upper Limb Assessment are integrated for informed decision-making about task redesign. The proposed approach was tested in a case study inspired by large parts manufacturing, where operators need to work in teams. The server-client network enabled the engineer to comprehensively assess the design of the task having two operators executing virtual actions in parallel as they would in the actual workstation. The collection of data from the team of operators and their intuitive visualization enhanced the effectiveness in identifying aspects introducing discomfort and assessing the overall suitability of the design for human operators.

Human-Centered Task Allocation: A Simulation-Based Case Study

This study investigates human-centered task allocation, focusing on factors such as cognitive load, physical demand, and ergonomics. A discrete event simulator was developed to validate the task allocation results of the Q-learning optimization. The simulation evaluates the process and resulting task allocation plan based on predefined human-centered objectives. Through a case study on remanufacturing, we demonstrate how to optimize the coordination of a robotic arm and two human operators to reduce overall workforce requirements. The simulation allows for the analysis of operators’ cognitive and physical workloads over time, enabling exploration of not only typical time balances but also cognitive and physical burdens.

Multi-Objective Multi-Resource Task Allocation For Collaborative Robots Systems

The shift from Industry 4.0 to Industry 5.0 has led to a greater focus on workers’ needs in the workplace. Collaborative robots have been introduced to promote a fair division of tasks and reduce physical and mental strain on workers. However, there is a lack of research on how to implement human-centered task allocation. This study proposes a model for multi-objective task allocation, including minimizing makespan, energy expenditure, and mental workload. The study also suggests a method for evaluating mental workload. Results show that the strictness of task sequence affects makespan and energy expenditure, and a new constraint related to idle times is proposed. The optimal level of worker saturation is one that minimizes makespan while minimizing increases in energy expenditure and mental workload.

Robot-Assisted Disassembly Sequence Planning With Real-Time Human Motion Prediction

This article presents a disassembly task planning algorithm considering human–robot collaboration (HRC) and human behavior prediction (HBP). Unlike assembly procedures, the disassembly of end-of-life (EOL) products has been a labor-intensive process with uncertainties difficult to cope with. Meanwhile, it is usually challenging to obtain an optimal sequence efficiently without excessive computational cost. Also, the conventional human-centered task planning, in which the robot has to halt frequently due to unsafe interruptions by human motions, may decrease the efficiency of the disassembly process. In this article, a sequence planner is proposed to assign tasks in real time between a human operator and a robot to overcome the aforementioned challenges. The cost function includes the effort of the human and the robot in terms of both movement distance and time spent on the tasks. The constraints include the disassembly rules and the safety of the human operation. The optimal sequence is generated by solving an optimization problem in a receding-horizon way. In particular, at each step, the proposed disassembly sequence planner locates the workers (a human operator and a robot) and the to-be-disassembled components, predicts human movement for the next several steps, and obtains the optimal disassembly sequence for the next several steps following disassembly rules and safety constraints. Experiments have been extensively conducted on the disassembly of a wooden toybox and a used hard disk drive (HDD) to validate the proposed disassembly sequence planner. The planner has successfully generated the disassembly sequence in an HRC setting explicitly considering real-time human motion prediction and assigned the human operator and the robot to collaboratively complete disassembly tasks without violating disassembly rules and safety constraints.

From Handcrafted to Deep Features for Pedestrian Detection: A Survey.

Pedestrian detection is an important but challenging problem in computer vision, especially in human-centric tasks. Over the past decade, significant improvement has been witnessed with the help of handcrafted features and deep features. Here we present a comprehensive survey on recent advances in pedestrian detection. First, we provide a detailed review of single-spectral pedestrian detection that includes handcrafted features based methods and deep features based approaches. For handcrafted features based methods, we present an extensive review of approaches and find that handcrafted features with large freedom degrees in shape and space have better performance. In the case of deep features based approaches, we split them into pure CNN based methods and those employing both handcrafted and CNN based features. We give the statistical analysis and tendency of these methods, where feature enhanced, part-aware, and post-processing methods have attracted main attention. In addition to single-spectral pedestrian detection, we also review multi-spectral pedestrian detection, which provides more robust features for illumination variance. Furthermore, we introduce some related datasets and evaluation metrics, and a deep experimental analysis. We conclude this survey by emphasizing open problems that need to be addressed and highlighting various future directions. Researchers can track an up-to-date list at https://github.com/JialeCao001/PedSurvey.

Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis.

The human language can be expressed through multiple sources of information known as modalities, including tones of voice, facial gestures, and spoken language. Recent multimodal learning with strong performances on human-centric tasks such as sentiment analysis and emotion recognition are often black-box, with very limited interpretability. In this paper we propose Multimodal Routing, which dynamically adjusts weights between input modalities and output representations differently for each input sample. Multimodal routing can identify relative importance of both individual modalities and cross-modality features. Moreover, the weight assignment by routing allows us to interpret modality-prediction relationships not only globally (i.e. general trends over the whole dataset), but also locally for each single input sample, mean-while keeping competitive performance compared to state-of-the-art methods.

Generating Sketch-Based Synthetic Seismic Images With Generative Adversarial Networks

The characterization of the subsurface is paramount for the exploration and production life cycle and, more specifically, for the identification of potential hydrocarbon accumulations. In recent years, the oil and gas industry has increased its interest in applying machine learning to accelerate the seismic interpretation process, which is regarded as a time-consuming and human-centered task. Although machine learning has been successfully used in many applications ranging from stratigraphic segmentation to salt dome detection, a usual bottleneck is the need for a large amount of high-quality annotated data. To overcome this, data augmentation approaches are commonly used, and one of the most powerful is synthetic data generation. In addition, sketch-based synthetic seismic images can be used to support image retrieval applications to help oil companies leverage petabytes of seismic data sets. In this context, this letter investigates the generation of synthetic seismic images based on sketches using generative adversarial networks (GANs). To the best of our knowledge, this is the first work to propose such an approach. Experiments with five different sketch types in a public seismic data set indicate that realistic seismic images can be synthesized using rather simple sketches.

A neural signature of pattern separation in the monkey hippocampus

The CA3 and dentate gyrus (DG) regions of the hippocampus are considered key for disambiguating sensory inputs from similar experiences in memory, a process termed pattern separation. The neural mechanisms underlying pattern separation, however, have been difficult to compare across species: rodents offer robust recording methods with less human-centric tasks, while humans provide complex behavior with less recording potential. To overcome these limitations, we trained monkeys to perform a visual pattern separation task similar to those used in humans while recording activity from single CA3/DG neurons. We find that, when animals discriminate recently seen novel images from similar (lure) images, behavior indicative of pattern separation, CA3/DG neurons respond to lure images more like novel than repeat images. Using a population of these neurons, we are able to classify novel, lure, and repeat images from each other using this pattern of firing rates. Notably, one subpopulation of these neurons is more responsible for distinguishing lures and repeats-the key discrimination indicative of pattern separation.

Neck kinematics and muscle activity during mobile device operations

Neck pain is a significant health problem due to its high incident rates and economic costs. Increased use of touch screen mobile devices is becoming pervasive in the modern society and may further influence this already prevalent health problem. However, our current understanding of the cervical spine biomechanics during the operation of touch screen mobile devices is very limited. This study evaluated the neck extensor muscle activities and the kinematics of the cervical spine during the operation of a touch screen tablet and a smart phone. Three-variables, DEVICE, LOCATION and TASK were treated as the independent variables. NASA TLX revealed that “Gaming” was the least difficult task and “Typing” was the most difficult task. Participants of this study maintained significantly deeper neck flexion when operating a smart phone (44.7°), with the mobile devices set on a table (46.4°), and while performing a “Typing” task (45.6°). Lower levels of neck muscle activities were observed while performing a “Reading” task and holding mobile devices with hand. Lower levels of neck muscle activities were also observed when using a smart phone vs. a tablet, however such change was statistically insignificant. Relevance to industryThe current study demonstrated that users maintain deep neck flexion when using touchscreen mobile devices. In the recent years, there is an increasing popularity of mobile smart devices in various occupational environments. The findings of this study may be useful in implementing human-centered task designs to reduce neck injury risks among mobile device users.

From Abstract Task Knowledge to Executable Robot Programs

Robots that are capable of learning new tasks from humans need the ability to transform gathered abstract task knowledge into their own representation and dimensionality. New task knowledge that has been collected e.g. with Programming by Demonstration approaches by observing a human does not a-priori contain any robot-specific knowledge and actions, and is defined in the workspace of the human demonstrator. This article presents a new approach for mapping abstract human-centered task knowledge to a robot execution system based on the target system properties. Therefore the required background knowledge about the target system is examined and defined explicitly.

Model-based human-centered task automation: A case study in ACC system design

Engineers, business managers, and governments are increasingly aware of the importance and difficulty of integrating technology and humans. The presence of technology can enhance human comfort, efficiency, and safety, but the absence of human-factors analysis can lead to uncomfortable, inefficient, and unsafe systems. Systematic human-centered design requires a basic understanding of how humans generate and manage tasks. A very useful model of human behavior generation can be obtained by recognizing the task-specific role of mental models in not only guiding execution of skills but also managing initiation and termination of these skills. By identifying the human operator's mental models and using them as templates for automating different tasks, we experimentally support the hypothesis that natural and safe interaction between human operator and automation is facilitated by this model-based human-centered approach. The design of adaptive cruise control (ACC) systems is used as a case study in the design of model-based task automation systems. Such designs include identifying ecologically appropriate perceptual states, identifying perceptual triggering events for managing transitions between skilled behaviors, and coordinating the actions of automation and operator.