Vision-based Motion Research Articles

In-process part tracking and shape measurement using vision-based motion capture for automated English wheeling

An English wheel is an exceedingly adaptable instrument in traditional metalworking. It is a manual manufacturing technique, enabling skilled craftsmen and blacksmiths to shape complex compound curves in sheet metal panels. Accurate measurements and precise adjustments are essential when operating an English wheel to ensure that the metal is shaped with the desired curvature. An automated method to form English wheeled panels through robot forming has recently been proposed. For such a method to be successful, accurate tracking of sheet information including positions, orientations, and deformation is important for error compensation and the design of the subsequent tool paths. In this study, a Vicon motion capture system is employed to monitor the position and shape of the sheet metal during the English wheeling process. The initial experimental results demonstrate the potential of such an in-process metrology system, along with possible avenues for future work.

Binocular Vision-Based Motion Planning of an AUV: A Deep Reinforcement Learning Approach

Binocular Vision-Based Motion Planning of an AUV: A Deep Reinforcement Learning Approach

Audio Matters Too! Enhancing Markerless Motion Capture with Audio Signals for String Performance Capture

In this paper, we touch on the problem of markerless multi-modal human motion capture especially for string performance capture which involves inherently subtle hand-string contacts and intricate movements. To fulfill this goal, we first collect a dataset, named String Performance Dataset (SPD), featuring cello and violin performances. The dataset includes videos captured from up to 23 different views, audio signals, and detailed 3D motion annotations of the body, hands, instrument, and bow. Moreover, to acquire the detailed motion annotations, we propose an audio-guided multi-modal motion capture framework that explicitly incorporates hand-string contacts detected from the audio signals for solving detailed hand poses. This framework serves as a baseline for string performance capture in a completely markerless manner without imposing any external devices on performers, eliminating the potential of introducing distortion in such delicate movements. We argue that the movements of performers, particularly the sound-producing gestures, contain subtle information often elusive to visual methods but can be inferred and retrieved from audio cues. Consequently, we refine the vision-based motion capture results through our innovative audio-guided approach, simultaneously clarifying the contact relationship between the performer and the instrument, as deduced from the audio. We validate the proposed framework and conduct ablation studies to demonstrate its efficacy. Our results outperform current state-of-the-art vision-based algorithms, underscoring the feasibility of augmenting visual motion capture with audio modality. To the best of our knowledge, SPD is the first dataset for musical instrument performance, covering fine-grained hand motion details in a multi-modal, large-scale collection. It holds significant implications and guidance for string instrument pedagogy, animation, and virtual concerts, as well as for both musical performance analysis and generation. Our code and SPD dataset are available at https://github.com/Yitongishere/string_performance.

Motion Capture Technology in Sports Scenarios: A Survey.

Motion capture technology plays a crucial role in optimizing athletes' skills, techniques, and strategies by providing detailed feedback on motion data. This article presents a comprehensive survey aimed at guiding researchers in selecting the most suitable motion capture technology for sports science investigations. By comparing and analyzing the characters and applications of different motion capture technologies in sports scenarios, it is observed that cinematography motion capture technology remains the gold standard in biomechanical analysis and continues to dominate sports research applications. Wearable sensor-based motion capture technology has gained significant traction in specialized areas such as winter sports, owing to its reliable system performance. Computer vision-based motion capture technology has made significant advancements in recognition accuracy and system reliability, enabling its application in various sports scenarios, from single-person technique analysis to multi-person tactical analysis. Moreover, the emerging field of multimodal motion capture technology, which harmonizes data from various sources with the integration of artificial intelligence, has proven to be a robust research method for complex scenarios. A comprehensive review of the literature from the past 10 years underscores the increasing significance of motion capture technology in sports, with a notable shift from laboratory research to practical training applications on sports fields. Future developments in this field should prioritize research and technological advancements that cater to practical sports scenarios, addressing challenges such as occlusion, outdoor capture, and real-time feedback.

Experimental Evaluation of Pedestrian-Induced Multiaxial Gait Loads on Footbridges: Effects of the Structure-to-Human Interaction by Lateral Vibrating Platforms.

The introduction of resistant and lightweight materials in the construction industry has led to civil structures being vulnerable to excessive vibrations, particularly in footbridges exposed to human-induced gait loads. This interaction, known as Human-Structure Interaction (HSI), involves a complex interplay between structural vibrations and gait loads. Despite extensive research on HSI, the simultaneous effects of lateral structural vibrations with fundamental frequencies close to human gait frequency (around 1.0 Hz) and wide amplitudes (over 30.0 mm) remain inadequately understood, posing a contemporary structural challenge highlighted by incidents in iconic bridges like the Millennium Bridge in London, Solferino Bridge in Paris, and Premier Bridge in Cali, Colombia. This paper focuses on the experimental exploration of Structure-to-Human Interaction (S2HI) effects using the Human-Structure Interaction Multi-Axial Test Framework (HSI-MTF). The framework enables the simultaneous measurement of vertical and lateral loads induced by human gait on surfaces with diverse frequency ranges and wide-amplitude lateral harmonic motions. The study involved seven test subjects, evaluating gait loads on rigid and harmonic lateral surfaces with displacements ranging from 5.0 to 50.0 mm and frequency content from 0.70 to 1.30 Hz. A low-cost vision-based motion capture system with smartphones analyzed the support (Tsu) and swing (Tsw) periods of human gait. Results indicated substantial differences in Tsu and Tsw on lateral harmonic protocols, reaching up to 96.53% and 58.15%, respectively, compared to rigid surfaces. Normalized lateral loads (LL) relative to the subject's weight (W0) exhibited a linear growth proportional to lateral excitation frequency, with increased proportionality constants linked to higher vibration amplitudes. Linear regressions yielded an average R2 of 0.815. Regarding normalized vertical load (LV) with respect to W0, a consistent behavior was observed for amplitudes up to 30.0 mm, beyond which a linear increase, directly proportional to frequency, resulted in a 28.3% increment compared to rigid surfaces. Correlation analyses using Pearson linear coefficients determined relationships between structural surface vibration and pedestrian lateral motion, providing valuable insights into Structure-to-Human Interaction dynamics.

Vision-based tactile intelligence with soft robotic metamaterial

Robotic metamaterials represent an innovative approach to creating synthetic structures that combine desired material characteristics with embodied intelligence, blurring the boundaries between materials and machinery. Inspired by the functional qualities of biological skin, integrating tactile intelligence into these materials has gained significant interest for research and practical applications. This study introduces a Soft Robotic Metamaterial (SRM) design featuring omnidirectional adaptability and superior tactile sensing, combining vision-based motion tracking and machine learning. The study compares two sensory integration methods to a state-of-the-art motion tracking system and force/torque sensor baseline: an internal-vision design with high frame rates and an external-vision design offering cost-effectiveness. The results demonstrate the internal-vision SRM design achieving an impressive tactile accuracy of 98.96%, enabling soft and adaptive tactile interactions, especially beneficial for dexterous robotic grasping. The external-vision design offers similar performance at a reduced cost and can be adapted for portability, enhancing material science education and robotic learning. This research significantly advances tactile sensing using vision-based motion tracking in soft robotic metamaterials, and the open-source availability on GitHub fosters collaboration and further exploration of this innovative technology (https://github.com/bionicdl-sustech/SoftRoboticTongs).

Robust real-time upper limb motion transfer from human to humanoid for manipulation skill teaching

This paper presents a framework for real-time high-fidelity upper limb motion transfer from a human to a humanoid robot, intending to provide robust support for robots to learn manipulation skills. Robot imitation of human motion is modeled as a simultaneous wrist and elbow tracking problem, where the coordination between multiple imitation objectives and physical constraints receives special attention. First, for the vision-based motion capture system, an efficient preprocessing method is introduced to map the raw human skeletal points analogously or reflectively to the robot’s workspace. Then, the generation of velocity-level imitation motion commands is formalized as a constrained hierarchical quadratic programing (HQP) problem, in which multiple objectives for achieving imitation are organized into a hierarchy. The wrist tracking task is set as the highest priority, followed by the adjustment of the elbow and the minimization of the joint velocity. Physical limits of the robot joints, as well as strict kinematic constraints regarding self-motion, are considered as constraints for the optimization problem. Finally, a hierarchical recurrent neural network (RNN) is designed to solve this HQP problem through its unique neural dynamics. The performance of this network is theoretically guaranteed with high computational efficiency. Quantitative results show that the proposed motion imitation control scheme offers excellent integrated tracking accuracy of [Formula: see text] m and is robust to different arm length ratios. Using this method, the humanoid robot can imitate the upper body motions of a human as accurately as possible under physical constraints, without significant delays.

Identifying activity level related movement features of children with ASD based on ADOS videos

Autism spectrum disorder (ASD) is a neurodevelopmental disorder that affects about 2% of children. Due to the shortage of clinicians, there is an urgent demand for a convenient and effective tool based on regular videos to assess the symptom. Computer-aided technologies have become widely used in clinical diagnosis, simplifying the diagnosis process while saving time and standardizing the procedure. In this study, we proposed a computer vision-based motion trajectory detection approach assisted with machine learning techniques, facilitating an objective and effective way to extract participants’ movement features (MFs) to identify and evaluate children’s activity levels that correspond to clinicians’ professional ratings. The designed technique includes two key parts: (1) Extracting MFs of participants’ different body key points in various activities segmented from autism diagnostic observation schedule (ADOS) videos, and (2) Identifying the most relevant MFs through established correlations with existing data sets of participants’ activity level scores evaluated by clinicians. The research investigated two types of MFs, i.e., pixel distance (PD) and instantaneous pixel velocity (IPV), three participants’ body key points, i.e., neck, right wrist, and middle hip, and five activities, including Table-play, Birthday-party, Joint-attention, Balloon-play, and Bubble-play segmented from ADOS videos. Among different combinations, the high correlations with the activity level scores evaluated by the clinicians (greater than 0.6 with p < 0.001) were found in Table-play activity for both the PD-based MFs of all three studied key points and the IPV-based MFs of the right wrist key point. These MFs were identified as the most relevant ones that could be utilized as an auxiliary means for automating the evaluation of activity levels in the ASD assessment.

Ultra-Precise High-Speed Untethered Manipulation of Magnetic Scanning Microprobe in Aqueous Solutions

This article describes the design and development of an integrated system that renders ultra-precise and high-speed, untethered manipulation of a single magnetic scanning microprobe in aqueous solutions under a microscope. The system uses a six-input-three-output hexapole electromagnetic actuator to control the magnetic gradient force exerted on the probe and a three-dimensional vision-based motion tracking system to enable feedback control. The control system has three core functions that enables superior manipulation of the magnetic microprobe. The six-input-six-output closed-loop magnetic flux control significantly suppresses the effect of magnetic hysteresis and greatly increases the bandwidth of magnetic flux generation. The optimal flux allocation solves four issues in multipole magnetic force generation: redundancy; coupling; nonlinearity; and position-dependency. The position-dependent discrete-time motion control law determines the required force for stabilization and tracking. These three core functions have been implemented using a high-speed field programmable gate array (FPGA) system. Experiments have verified that the use of digital control technology, high-speed electronics, mathematical modeling, and real-time computation has achieved superior manipulation of a single magnetic microprobe in aqueous solutions.

Design and Development of an Automatic Microinjection System for High-Throughput Injection of Zebrafish Larvae

The microinjection of zebrafish larvae is a complicated work due to their small size and shape variations. In this article, for the first time, we design an automatic microinjection system dedicated to high-throughput injection of zebrafish larvae. The proposed custom-built microarray Petri dish combined with machine vision-based motion control enables high-efficiency automatic identification and injection of a batch of zebrafish larvae. The feasibility of the developed system is demonstrated by experimental testing on live zebrafish larvae. Results show that the developed system works fast with more stable injection success rate and higher survival rate than the skilled experimenter. The proposed system offers a good working efficiency and maintains a consistent success rate and stability in the process of high-throughput injection. It can be applied in various biomedical experiments based on zebrafish microinjection. Note to Practitioners—Zebrafish is widely used in biomedical experiments due to its obvious advantages. At present, the batch injection task of larvae is mainly conducted by experimenters manually. As the number of injections increases, experimenters will get tired, which makes it difficult for manual injection to maintain a stable success rate. This article reports the first automatic microinjection system for high-throughput batch injection of zebrafish larvae. A new microarray Petri dish has been proposed to fix the larvae, facilitating the microinjection process. Experimental results show that the system can complete a large number of injection tasks quickly while ensuring a stable success rate and survival rate of the larvae. It can free experimenters from repeated and tedious injection work. It also enables efficient zebrafish larvae microinjection for inexperienced experimenters.

MiShape

We propose MiShape, a millimeter-wave (mmWave) wireless signal based imaging system that generates high-resolution human silhouettes and predicts 3D locations of body joints. The system can capture human motions in real-time under low light and low-visibility conditions. Unlike existing vision-based motion capture systems, MiShape is privacy non-invasive and can generalize to a wide range of motion tracking applications at-home. To overcome the challenges with low-resolution, specularity, and aliasing in images from Commercial-Off-The-Shelf (COTS) mmWave systems, MiShape designs deep learning models based on conditional Generative Adversarial Networks and incorporates the rules of human biomechanics. We have customized MiShape for gait monitoring, but the model is well adaptive to any tracking applications with limited fine-tuning samples. We experimentally evaluate MiShape with real data collected from a COTS mmWave system for 10 volunteers, with diverse ages, gender, height, and somatotype, performing different poses. Our experimental results demonstrate that MiShape delivers high-resolution silhouettes and accurate body poses on par with an existing vision-based system, and unlocks the potential of mmWave systems, such as 5G home wireless routers, for privacy-noninvasive healthcare applications.

Assessing Team Situational Awareness in the Operating Room via Computer Vision.

Situational awareness (SA) at both individual and team levels, plays a critical role in the operating room (OR). During the pre-incision time-out, the entire OR team comes together to deploy the surgical safety checklist (SSC). Worldwide, the implementation of the SSC has been shown to reduce intraoperative complications and mortality among surgical patients. In this study, we investigated the feasibility of applying computer vision analysis on surgical videos to extract team motion metrics that could differentiate teams with good SA from those with poor SA during the pre-incision time-out. We used a validated observation-based tool to assess SA, and a computer vision software to measure body position and motion patterns in the OR. Our findings showed that it is feasible to extract surgical team motion metrics captured via off-the-shelf OR cameras. Entropy as a measure of the level of team organization was able to distinguish surgical teams with good and poor SA. These findings corroborate existing studies showing that computer vision-based motion metrics have the potential to integrate traditional observation-based performance assessments in the OR.

A Vision-Based Motion Control Framework for Water Quality Monitoring Using an Unmanned Aerial Vehicle

In this paper, we present a vision-aided motion planning and control framework for the efficient monitoring and surveillance of water surfaces using an Unmanned Aerial Vehicle (UAV). The ultimate goal of the proposed strategy is to equip the UAV with the necessary autonomy and decision-making capabilities to support First Responders during emergency water contamination incidents. Toward this direction, we propose an end-to-end solution, based on which the First Responder indicates visiting and landing waypoints, while the envisioned strategy is responsible for the safe and autonomous navigation of the UAV, the refinement of the way-point locations that maximize the visible water surface area from the onboard camera, as well as the on-site refinement of the appropriate landing region in harsh environments. More specifically, we develop an efficient waypoint-tracking motion-planning scheme with guaranteed collision avoidance, a local autonomous exploration algorithm for refining the way-point location with respect to the areas visible to the drone’s camera, water, a vision-based algorithm for the on-site area selection for feasible landing and finally, a model predictive motion controller for the landing procedure. The efficacy of the proposed framework is demonstrated via a set of simulated and experimental scenarios using an octorotor UAV.

Using the Improved SSD Algorithm to Motion Target Detection and Tracking.

Computer vision-based motion target detection and tracking, which is widely used in video surveillance, human-computer interaction, range interpretation, and other fields, is one of the current research hotspots in the field of computer vision. In engineering scenarios, the two are inseparable and need to work together to accomplish specific tasks. The related research is progressing rapidly, but there is still room for improving its timeliness, accuracy, and automation. In this paper, we summarize and classify some classical target detection methods, analyze the basic principles of convolutional neural networks, and analyze the classical detection algorithms based on region suggestion and deep regression networks. After that, we improve the SSD algorithm for the shortage of low-level feature convolution layers, which has insufficient feature extraction and leads to poor detection of small targets. For the motion target tracking problem, this paper studies the motion target tracking method based on support vector machine and proposes the tracking method of support vector regression and the corresponding online support vector regression solution method based on the analysis of support vector tracking method and structural support vector tracking method. In this paper, we propose a tracking method that fuses structural support vector machines and correlation filtering. The structure is based on the idea of Inception, which adds and replaces some feature convolution layers of the original network while maintaining the original lightweight backbone. The final experiments on the VOC data set demonstrate that the improved algorithm improves the average detection accuracy by 2.6% compared to the original algorithm and basically maintains the real-time speed as well. Experimental simulations on a subset of VOC data (human set) show a significant improvement in AP values and more effective and reliable detection tracking of moving targets. The stability and accuracy of motion target detection and tracking are improved by setting parameters, such as confidence level; the effectiveness and continuity of detection and tracking are judged by setting the interframe centroid distance.

Pure Vision-Based Motion Tracking for Data-Driven Design – A Simple, Flexible, and Cost-Effective Approach for Capturing Static and Dynamic Interactions

AbstractThis paper presents an exploratory case study where video-based pose estimation is used to analyse human motion to support data-driven design. It provides two example use cases related to design. Results are compared to ground truth measurements showing high correlation for the estimated pose, with an RMSE of 65.5 mm. The paper exemplifies how design projects can benefit from a simple, flexible, and cost-effective approach to capture human-object interactions. This also entails the possibility of implementing interaction and body capturing in the earliest stages of design, at minimal effort.

Convolutional Neural Network and Computer Vision-Based Action Correction Method for Long-Distance Running Technology

For long-distance running tactics, a computer vision-based motion correction solution is presented. The depth information is merged into the KCF algorithm to enhance it, which overcomes the classic KCF method’s incapacity to tackle the tracking drift issue caused by occlusion and extracts the technical features of long-distance running motions. Based on computer vision, the posture area of long-distance runners is detected, and the technical movements of long-distance runners are recognized. Calculate the centroid coordinates of the wrong technical movement correction area in long-distance running, and generate a tracking image of the wrong technical movement in long-distance running. The foreground and background information of the image is separated by the optical flow feature of machine vision, and the motion trajectory of the long-distance running error technique is extracted, and the motion correction of the long-distance running technique based on computer vision is realized. The experimental results show that the method in this paper has better accuracy in extracting long-distance running motion features and can accurately identify and correct the technical movements of the long-distance running, the technical movements of the head, and the technical movements of the body balance, and the correction efficiency is high.

Characteristics and Applications of Technology-Aided Hand Functional Assessment: A Systematic Review.

Technology-aided hand functional assessment has received considerable attention in recent years. Its applications are required to obtain objective, reliable, and sensitive methods for clinical decision making. This systematic review aims to investigate and discuss characteristics of technology-aided hand functional assessment and their applications, in terms of the adopted sensing technology, evaluation methods and purposes. Based on the shortcomings of current applications, and opportunities offered by emerging systems, this review aims to support the design and the translation to clinical practice of technology-aided hand functional assessment. To this end, a systematic literature search was led, according to recommended PRISMA guidelines, in PubMed and IEEE Xplore databases. The search yielded 208 records, resulting into 23 articles included in the study. Glove-based systems, instrumented objects and body-networked sensor systems appeared from the search, together with vision-based motion capture systems, end-effector, and exoskeleton systems. Inertial measurement unit (IMU) and force sensing resistor (FSR) resulted the sensing technologies most used for kinematic and kinetic analysis. A lack of standardization in system metrics and assessment methods emerged. Future studies that pertinently discuss the pathophysiological content and clinimetrics properties of new systems are required for leading technologies to clinical acceptance.

Image Motion Extraction of Structures Using Computer Vision Techniques: A Comparative Study.

Vibrational measurements play an important role for structural health monitoring, e.g., modal extraction and damage diagnosis. Moreover, conditions of civil structures can be mostly assessed by displacement responses. However, installing displacement transducers between the ground and floors in real-world buildings is unrealistic due to lack of reference points and structural scales and complexity. Alternatively, structural displacements can be acquired using computer vision-based motion extraction techniques. These extracted motions not only provide vibrational responses but are also useful for identifying the modal properties. In this study, three methods, including the optical flow with the Lucas–Kanade method, the digital image correlation (DIC) with bilinear interpolation, and the in-plane phase-based motion magnification using the Riesz pyramid, are introduced and experimentally verified using a four-story steel-frame building with a commercially available camera. First, the three displacement acquiring methods are introduced in detail. Next, the displacements are experimentally obtained from these methods and compared to those sensed from linear variable displacement transducers. Moreover, these displacement responses are converted into modal properties by system identification. As seen in the experimental results, the DIC method has the lowest average root mean squared error (RMSE) of 1.2371 mm among these three methods. Although the phase-based motion magnification method has a larger RMSE of 1.4132 mm due to variations in edge detection, this method is capable of providing full-field mode shapes over the building.

American society of biomechanics early career achievement award 2020: Toward portable and modular biomechanics labs: How video and IMU fusion will change gait analysis

The field of biomechanics is at a turning point, with marker-based motion capture set to be replaced by portable and inexpensive hardware, rapidly improving markerless tracking algorithms, and open datasets that will turn these new technologies into field-wide team projects. Despite progress, several challenges inhibit both inertial and vision-based motion tracking from reaching the high accuracies that many biomechanics applications require. Their complementary strengths, however, could be harnessed toward better solutions than those offered by either modality alone. The drift from inertial measurement units (IMUs) could be corrected by video data, while occlusions in videos could be corrected by inertial data. To expedite progress in this direction, we have collected the CMU Panoptic Dataset 2.0, which contains 86 subjects captured with 140 VGA cameras, 31 HD cameras, and 15 IMUs, performing on average 6.5 min of activities, including range of motion activities and tasks of daily living. To estimate ground-truth kinematics, we imposed simultaneous consistency with the video and IMU data. Three-dimensional joint centers were first computed by geometrically triangulating proposals from a convolutional neural network applied to each video independently. A statistical meshed model parametrized in terms of body shape and pose was then fit through a top-down optimization approach that enforced consistency with both the video-based joint centers and IMU data. As proof of concept, we used this dataset to benchmark pose estimation from a sparse set of sensors, showing that incorporation of complementary modalities is a promising frontier that can be further strengthened through physics-informed frameworks.

Wireless Sensor Acquisition of Human Motion Parameters Based on Blockchain

With the rapid development of intelligent technology, people’s lives have gradually entered the era of information and intelligence. Traditional vision‐based and sensor‐based human motion detection methods have shortcomings such as high illumination requirements, difficulty in preserving privacy, and poor portability. The wireless sensor detection method based on a wireless network can overcome the shortcomings of these methods, so it has become a hot research direction. Human motion parameters are very important data. If leaked, it may cause unimaginable harm to individuals. Blockchain technology is an emerging technology in the field of information technology. Its decentralized nature, distributed storage, and data is not easy. The modification provides a feasible solution for us to realize the data upload. Combining the advantages of wireless sensing and blockchain technology, this paper proposes a method of wireless sensing collection of human motion parameters based on blockchain, and designs a human motion recognition system. The experiment found that under the experiments given in this article, the traditional accuracy rate of sensor data collection is 71.82%, and the accuracy rate of the wireless sensing collection method of human motion parameters based on the blockchain in this article has reached 95.31%, which verifies the effectiveness and superiority of the system designed in this article.