Visual Guidance Research Articles

Measuring User Response to Attention Guidance Using the Integrated Cognitive User Assistance System

Humans are limited in their attentional capacity and miss salient cues when multitasking. We built a platform, ICUa (Integrated Cognitive User assistance) that simulates parts of MATBII, a computer-based multitasking testbed, adding gaze-based attention guidance in an attempt to improve multitasking performance. ICUa uses software agents for monitoring the interface, human behavior and eye gaze in order to guide attention (displaying highlights and arrows). We varied the difficulty of our testbed and found effects on human performance (based on error times) and characterized the types of errors that occurred with higher difficulty and how agent response changed. There was no clear effect of ICUa guidance on gaze patterns or human performance. The visual guidance we deployed may not be sufficient to measurably influence behavior. Future work will develop the detection eye gaze patterns and effective cues to redirect attention using this human-agent interface for experiments on attention guidance.

Design of a Leaf-Bottom Pest Control Robot with Adaptive Chassis and Adjustable Selective Nozzle

Pest control is an important guarantee for agricultural production. Pests are mostly light-avoiding and often gather on the bottom of crop leaves. However, spraying agricultural machinery mostly adopts top-down spraying, which suffers from low pesticide utilization and poor insect removal effect. Therefore, the upward spraying mode and intelligent nozzle have gradually become the research hotspot of precision agriculture. This paper designs a leaf-bottom pest control robot with adaptive chassis and adjustable selective nozzle. Firstly, the adaptive chassis is designed based on the MacPherson suspension, which uses shock absorption to drive the track to swing within a 30° angle. Secondly, a new type of cone angle adjustable selective nozzle was developed, which achieves adaptive selective precision spraying under visual guidance. Then, based on a convolutional block attention module (CBAM), the multi-CBAM-YOLOv5s network model was improved to achieve a 70% recognition rate of leaf-bottom spotted bad point in video streams. Finally, functional tests of the adaptive chassis and the adjustable selective spraying system were conducted. The data indicate that the adaptive chassis can adapt to diverse single-ridge requirements of soybeans and corn while protecting the ridge slopes. The selective spraying system achieves 70% precision in pesticide application, greatly reducing the use of pesticides. The scheme explores a ridge-friendly leaf-bottom pest control plan, providing a technical reference for improving spraying effect, reducing pesticide usage, and mitigating environmental pollution.

Conventional ultrasonography enabled with augmented reality needle guidance for percutaneous kidney access: An innovative methodologies randomized controlled trial.

Successful needle puncture of the renal collecting system is a critical but difficult procedure in percutaneous nephrolithotomy (PCNL). Although fluoroscopy and ultrasound are the standard imaging techniques to guide puncture during PCNL, both have known limitations. To assess the feasibility and safety of a new navigation system for needle puncture in ultrasound-guided PCNL. This study employed a single-center randomized controlled trial (RCT) design to assess the feasibility and safety of a new navigation system for needle puncture in ultrasound-guided PCNL. Conducted between May 2021 and November 2021, the trial utilized computer-generated random numbers for participant allocation to control for selection bias. The trial was executed at the *********, which serves as an academic medical center. All patients who met the inclusion criteria were randomly divided into two groups, with 29 patients in each group. One group underwent PCNL procedures using the new navigation system, while the control group underwent standard ultrasound-guided PCNL procedures. Included patients had renal pelvis or caliceal calculi larger than 2.0cm in diameter or had multiple or staghorn stones. The puncture procedure was performed with the support of real-time ultrasound imaging and visual guidance displayed on the screen. The primary outcome was system feasibility and puncture success rate. Secondary outcomes included puncture time, total surgical time, number of attempts, post-procedure complications, and one-year and three-year stone recurrence rates. Stone clearance was defined by postoperative CT. Descriptive statistics summarized patient demographics, stone size, and location. Independent samples t-tests analyzed puncture time and total surgical time. Chi-square or Fisher's exact tests compared stone clearance, complications, socioeconomic status, renal hydronephrosis, stone location, race, and medical history. Linear regression examined the correlation between BMI and puncture time. Significance was set at P<0.05. For all 58 patients undergoing PCNL, needle punctures of the renal collecting system were completed with a success rate of 100%. The average time from planning the puncture protocol to successful puncture was significantly shorter in the AcuSee guidance system group (3.12min, range 0.2-6.88min) compared to the standard ultrasound-guided group (7.58min, range 5.41-10.68min), representing a reduction of approximately 59%. The total surgical time was also shorter in the AcuSee group for patients with no and mild hydronephrosis (P<0.05). Complication rates were lower in the AcuSee group, with no major complications observed. However, 3 patients in the standard ultrasound-guided group have adverse effects after the PCNL procedure. The one-year stone recurrence rate was significantly lower in the AcuSee group (3.4%) compared to the standard group (24.1%), and the three-year recurrence rate was also lower (6.9% vs. 41.4%). Patient-specific factors such as BMI, renal morphology, and prior surgical history did not significantly affect the performance of the AcuSee system. We report the first clinical application of a new navigation system for needle puncture in ultrasound-guided PCNL. It has been demonstrated that it is feasible and safe compared to the standard ultrasound-guided group in percutaneous renal puncture. This technology provides intuitive and easy-to-use visual guidance, which may facilitate safe, accurate and fast needle puncture of the kidney.

Polyfunctional Arylamine Based Nanofiltration Membranes with Enhanced Aggressive Organic Solvents Resistance.

Organic solvent nanofiltration (OSN) membranes with high separation performance and excellent stability in aggressive organic solvents are urgently desired for chemical separation. Herein, we utilized a polyfunctional arylamine tetra-(4-aminophenyl) ethylene (TAPE) to prepare a highly cross-linked polyamide membrane with a low molecular weight cut-off (MWCO) of 312 Da. Owing to its propeller-like conformation, TAPE formed micropores within the polyamide membrane and provided fast solvent transport channels. Importantly, the rigid conjugated skeleton and high connectivity between micropores effectively prevented the expansion of the polyamide matrix in aggressive organic solvents. The membrane maintained high separation performance even immersed in N,N-dimethylformamide for 90 days. Based on the aggregation-induced emission (AIE) effect of TAPE, the formation of polyamide membrane can be visually monitored by fluorescence imaging technology, which achieved visual guidance for membrane fabrication. This work provides a vital foundation for utilizing polyfunctional monomers in the interfacial polymerization reaction to prepare high-performance OSN membranes.

A Visual Environment for Data Driven Protein Modeling and Validation.

In structural biology, validation and verification of new atomic models are crucial and necessary steps which limit the production of reliable molecular models for publications and databases. An atomic model is the result of meticulous modeling and matching and is evaluated using a variety of metrics that provide clues to improve and refine the model so it fits our understanding of molecules and physical constraints. In cryo electron microscopy (cryo-EM) the validation is also part of an iterative modeling process in which there is a need to judge the quality of the model during the creation phase. A shortcoming is that the process and results of the validation are rarely communicated using visual metaphors. This work presents a visual framework for molecular validation. The framework was developed in close collaboration with domain experts in a participatory design process. Its core is a novel visual representation based on 2D heatmaps that shows all available validation metrics in a linear fashion, presenting a global overview of the atomic model and provide domain experts with interactive analysis tools. Additional information stemming from the underlying data, such as a variety of local quality measures, is used to guide the user's attention toward regions of higher relevance. Linked with the heatmap is a three-dimensional molecular visualization providing the spatial context of the structures and chosen metrics. Additional views of statistical properties of the structure are included in the visual framework. We demonstrate the utility of the framework and its visual guidance with examples from cryo-EM.

Multimodal heterogeneous graph entity-level fusion for named entity recognition with multi-granularity visual guidance

Multimodal heterogeneous graph entity-level fusion for named entity recognition with multi-granularity visual guidance

Precision-Targeted Injection Laryngoplasty and Longitudinal Biomaterial Effects Evaluation Using High-resolution Ultrasonography in a Rat Model.

Current laryngeal injection models rely on the transoral route and are suboptimal due to limited view, narrowed working space, and the need to sacrifice animals for investigation of the injectables. In the present study, a novel surgical model for laryngeal intervention therapy utilizing an ultra-high frequency ultrasound imaging system was proposed. Based on this system, we developed a systemic evaluation approach, from guidance of the injection process, documentation of the injection site of the material, to in vivo longitudinal follow-up on the augmentation and medialization effect by analyzing the ultrasonography data. In vivo animal study. Academic institution. Injection laryngoplasty with hyaluronic acid under ultrasonography guidance was performed on Sprague-Dawley rats one week after induced unilateral vocal paralysis. Ultrasonography was performed at preinjection, immediately postinjection, on Day 2, Day 7 and then weekly for 4 weeks to obtain measurements, including the glottic area, angle between bilateral folds, and vocal fold width ratio. Laryngoscopic and histologic studies were also performed. Unilateral injections to the paralyzed fold were successfully performed as demonstrated by ultrasonographic, laryngoscopic, and histologic studies. The width ratio was significantly increased after injection for 4 weeks, while the glottic airway area was unchanged. Here, a novel surgical model for laryngeal injection utilizing ultrasonography in rats was established. In addition to providing visual guidance for precise localization of the injection, robust documentation of the treatment effect was also demonstrated. This methodology could be beneficial for screening therapeutic agents for treatment of glottic insufficiency.

Real-World Evidence From a Digital Health Treatment Program for Female Urinary Incontinence: Observational Study of Outcomes Following User-Centered Product Design.

Urinary incontinence (UI) affects millions of women with substantial health and quality-of-life impacts. Supervised pelvic floor muscle training (PFMT) is the recommended first-line treatment. However, multiple individual and institutional barriers impede women's access to skilled care. Evidence suggests that digital health solutions are acceptable and may be effective in delivering first-line incontinence treatment, although these technologies have not yet been leveraged at scale. The primary objective is to describe the effectiveness and safety of a prescribed digital health treatment program to guide PFMT for UI treatment among real-world users. The secondary objectives are to evaluate patient engagement following an updated user platform and identify the factors predictive of success. This retrospective cohort study of women who initiated device use between January 1, 2022, and June 30, 2023, included users aged ≥18 years old with a diagnosis of stress, urgency, or mixed incontinence or a score of >33.3 points on the Urogenital Distress Inventory Short Form (UDI-6). Users are prescribed a 2.5-minute, twice-daily, training program guided by an intravaginal, motion-based device that pairs with a smartphone app. Data collected by the device or app include patient-reported demographics and outcomes, adherence to the twice-daily regimen, and pelvic floor muscle performance parameters, including angle change and hold time. Symptom improvement was assessed by the UDI-6 score change from baseline to the most recent score using paired 2-tailed t tests. Factors associated with meeting the UDI-6 minimum clinically important difference were evaluated by regression analysis. Of 1419 users, 947 met inclusion criteria and provided data for analysis. The mean baseline UDI-6 score was 46.8 (SD 19.3), and the mean UDI-6 score change was 11.3 (SD 19.9; P<.001). Improvement was reported by 74% (697/947) and was similar across age, BMI, and incontinence subtype. Mean adherence was 89% (mean 12.5, SD 2.1 of 14 possible weekly uses) over 12 weeks. Those who used the device ≥10 times per week were more likely to achieve symptom improvement. In multivariate logistic regression analysis, baseline incontinence symptom severity and maximum angle change during pelvic floor muscle contraction were significantly associated with meeting the UDI-6 minimum clinically important difference. Age, BMI, and UI subtype were not associated. This study provides real-world evidence to support the effectiveness and safety of a prescribed digital health treatment program for female UI. A digital PFMT program completed with visual guidance from a motion-based device yields significant results when executed ≥10 times per week over a period of 12 weeks. The program demonstrates high user engagement, with 92.9% (880/947) of users adhering to the prescribed training regimen. First-line incontinence treatment, when implemented using this digital program, leads to statistically and clinically substantial symptom improvements across age and BMI categories and incontinence subtypes.

A Construction Optimization for Laser SLAM Based on Odometer Constraint Fusion

The traditional laser SLAM (Simultaneous Localization and Mapping) algorithm uses the global relative poses and local ones to form residual blocks. Its constructed map is not smooth enough and the constraint construction is too simplex under some special scenarios. Thus, this paper proposes an odometer constraint fusion method called FOSLAM (Fusion Odometer SLAM) to construct residual blocks between constrains and solve the nonlinear least squares by Ceres. The effectiveness and accuracy of this method have been verified through comparative experiments. Experimental results showed that without increasing the time and space complexity, by involving the odometer constraint into the SLAM optimization process, the convergence of scan matching scores can be improved and the constructed grid map edges are smoother and the jagged phenomenon can be reduced. Under sophisticated scene, FOSLAM is able to acquire more accurate maps and laser odometer trajectory than Cartographer method. Therefore, it is suitable to be used on indoor robot for cleaning and inspection and can be further deployed on autonomous unmanned vehicles involving spatial visualization and neuro-heuristic guidance.

Embedded Zero-Shot Image Classification Based on Bidirectional Feature Mapping

The zero-shot image classification technique aims to explore the semantic information shared between seen and unseen classes through visual features and auxiliary information and, based on this semantic information, to complete the knowledge migration from seen to unseen classes in order to complete the classification of unseen class images. Previous zero-shot work has either not extracted enough features to express the relationship between the sample classes or has only used a single feature mapping method, which cannot fully explore the information contained in the features and the connection between the visual–semantic features. To address the above problems, this paper proposes an embedded zero-shot image classification model based on bidirectional feature mapping (BFM). It mainly contains a feature space mapping module, which is dominated by a bidirectional feature mapping network and supplemented with a mapping network from visual to category label semantic feature space. Attention mechanisms based on attribute guidance and visual guidance are further introduced to weight the features to reduce the difference between visual and semantic features to alleviate the modal difference problem, and then the category calibration loss is utilized to assign a larger weight to the unseen class to alleviate the seen class bias problem. The BFM model proposed in this paper has been experimented on three public datasets CUB, SUN, and AWA2, and has achieved 71.9%, 62.8%, and 69.3% and 61.6%, 33.2%, and 66.6% accuracies under traditional and generalized zero-sample image classification settings, respectively. The experimental results verify the superiority of the BFM model in the field of zero-shot image classification.

Research on Unmanned Aerial Vehicle (UAV) Visual Landing Guidance and Positioning Algorithms

Considering the weak resistance to interference and generalization ability of traditional UAV visual landing navigation algorithms, this paper proposes a deep-learning-based approach for airport runway line detection and fusion of visual information with IMU for localization. Firstly, a coarse positioning algorithm based on YOLOX is designed for airport runway localization. To meet the requirements of model accuracy and inference speed for the landing guidance system, regression loss functions, probability prediction loss functions, activation functions, and feature extraction networks are designed. Secondly, a deep-learning-based runway line detection algorithm including feature extraction, classification prediction and segmentation networks is designed. To create an effective detection network, we propose efficient loss function and network evaluation methods Finally, a visual/inertial navigation system is established based on constant deformation for visual localization. The relative positioning results are fused and optimized with Kalman filter algorithms. Simulation and flight experiments demonstrate that the proposed algorithm exhibits significant advantages in terms of localization accuracy, real-time performance, and generalization ability, and can provide accurate positioning information during UAV landing processes.

Antarstick: Extracting Snow Height From Time‐Lapse Photography

AbstractThe evolution and accumulation of snow cover are among the most important characteristics influencing Antarctica's climate and biotopes. The changes in Antarctica are also substantially impacting global climate change. Therefore, detailed monitoring of snow evolution is key to understanding such changes. One way to conduct this monitoring is by installing trail cameras in a particular region and then processing the captured information. This option is affordable, but has some drawbacks, such as the fully automatic solution for the extraction of snow height from these images is not feasible. Therefore, it still requires human intervention, manually correcting the inaccurately extracted information. In this paper, we present Antarstick, a tool for visual guidance of the user to potentially wrong values extracted from poor‐quality images and support for their interactive correction. This tool allows for much quicker and semi‐automated processing of snow height from time‐lapse photography.

Use of a pressure sensor array for multifunctional patient monitoring in radiotherapy: A feasibility study.

Modern radiotherapeutic techniques, such as intensity-modulated radiation therapy or stereotactic body radiotherapy, require high-dose delivery precision. However, the precise localization of tumors during patient respiration remains a challenge. Therefore, it is essential to investigate effective methods for monitoring respiration to minimize potential complications. Despite several systems currently in clinical use, there are drawbacks, including the complexity of the setup, the discomfort to the patient, and the high cost. This study investigated the feasibility of using a novel pressure sensor array (PSA) as a tool to monitor respiration during radiotherapy treatments. The PSA was positioned between the treatment couch and the back of the patient lying on it and was intended to overcome some limitations of current methods. The main objectives included assessing the PSA's capability in monitoring respiratory behavior and to investigate prospective applications that extend beyond respiratory monitoring. A PSA with 31 pressure-sensing elements was used in 12 volunteers. The participants were instructed to breathe naturally while lying on a couch without any audio or visual guidance. The performance of the PSA was compared to that of a camera-based respiratory monitoring system (RPM, Varian, USA), which served as a reference. Several metrics, including pressure distribution, weight sensitivity, and correlations between PSA and RPM signals, were analyzed. The PSA's capacity to provide information on potential applications related to patient stability was also investigated. The linear relationship between the weight applied to the PSA and its output was demonstrated in this study, confirming its sensitivity to pressure changes. A comparison of PSA and RPM curves revealed a high correlation coefficient of 0.9391 on average, indicating consistent respiratory cycles. The PSA also effectively measured the weight distribution at the volunteer's back in real-time, which allows for monitoring the patient's movements during the radiotherapy. PSA is a promising candidate for effective respiratory monitoring during radiotherapy treatments. Its performance is comparable to the established RPM system, and its additional capabilities suggest its multifaceted utility. This paper shows the potential use of PSA for patient monitoring in radiotherapy and suggests possibilities for further research, including performance comparisons with other existing systems and real-patient applications with respiratory training.

Research on Six-Degree-of-Freedom Refueling Robotic Arm Positioning and Docking Based on RGB-D Visual Guidance

The main contribution of this paper is the proposal of a six-degree-of-freedom (6-DoF) refueling robotic arm positioning and docking technology guided by RGB-D camera visual guidance, as well as conducting in-depth research and experimental validation on the technology. We have integrated the YOLOv8 algorithm with the Perspective-n-Point (PnP) algorithm to achieve precise detection and pose estimation of the target refueling interface. The focus is on resolving the recognition and positioning challenges of a specialized refueling interface by the 6-DoF robotic arm during the automated refueling process. To capture the unique characteristics of the refueling interface, we developed a dedicated dataset for the specialized refueling connectors, ensuring the YOLO algorithm’s accurate identification of the target interfaces. Subsequently, the detected interface information is converted into precise 6-DoF pose data using the PnP algorithm. These data are used to determine the desired end-effector pose of the robotic arm. The robotic arm’s movements are controlled through a trajectory planning algorithm to complete the refueling gun docking process. An experimental setup was established in the laboratory to validate the accuracy of the visual recognition and the applicability of the robotic arm’s docking posture. The experimental results demonstrate that under general lighting conditions, the recognition accuracy of this docking interface method meets the docking requirements. Compared to traditional vision-guided methods based on OpenCV, this visual guidance algorithm exhibits better adaptability and effectively provides pose information for the robotic arm.

Synergy and medial effects of multimodal cueing with auditory and electrostatic force stimuli on visual field guidance in 360° VR

This study investigates the effects of multimodal cues on visual field guidance in 360° virtual reality (VR). Although this technology provides highly immersive visual experiences through spontaneous viewing, this capability can disrupt the quality of experience and cause users to miss important objects or scenes. Multimodal cueing using non-visual stimuli to guide the users’ heading, or their visual field, has the potential to preserve the spontaneous viewing experience without interfering with the original content. In this study, we present a visual field guidance method that imparts auditory and haptic stimulations using an artificial electrostatic force that can induce a subtle “fluffy” sensation on the skin. We conducted a visual search experiment in VR, wherein the participants attempted to find visual target stimuli both with and without multimodal cues, to investigate the behavioral characteristics produced by the guidance method. The results showed that the cues aided the participants in locating the target stimuli. However, the performance with simultaneous auditory and electrostatic cues was situated between those obtained when each cue was presented individually (medial effect), and no improvement was observed even when multiple cue stimuli pointed to the same target. In addition, a simulation analysis showed that this intermediate performance can be explained by the integrated perception model; that is, it is caused by an imbalanced perceptual uncertainty in each sensory cue for orienting to the correct view direction. The simulation analysis also showed that an improved performance (synergy effect) can be observed depending on the balance of the uncertainty, suggesting that a relative amount of uncertainty for each cue determines the performance. These results suggest that electrostatic force can be used to guide 360° viewing in VR, and that the performance of visual field guidance can be improved by introducing multimodal cues, the uncertainty of which is modulated to be less than or comparable to that of other cues. Our findings on the conditions that modulate multimodal cueing effects contribute to maximizing the quality of spontaneous 360° viewing experiences with multimodal guidance.

The role of prediction and visual tracking strategies during manual interception: An exploration of individual differences.

The interception (or avoidance) of moving objects is a common component of various daily living tasks; however, it remains unclear whether precise alignment of foveal vision with a target is important for motor performance. Furthermore, there has also been little examination of individual differences in visual tracking strategy and the use of anticipatory gaze adjustments. We examined the importance of in-flight tracking and predictive visual behaviors using a virtual reality environment that required participants (n = 41) to intercept tennis balls projected from one of two possible locations. Here, we explored whether different tracking strategies spontaneously arose during the task, and which were most effective. Although indices of closer in-flight tracking (pursuit gain, tracking coherence, tracking lag, and saccades) were predictive of better interception performance, these relationships were rather weak. Anticipatory gaze shifts toward the correct release location of the ball provided no benefit for subsequent interception. Nonetheless, two interceptive strategies were evident: 1) early anticipation of the ball's onset location followed by attempts to closely track the ball in flight (i.e., predictive strategy); or 2) positioning gaze between possible onset locations and then using peripheral vision to locate the moving ball (i.e., a visual pivot strategy). Despite showing much poorer in-flight foveal tracking of the ball, participants adopting a visual pivot strategy performed slightly better in the task. Overall, these results indicate that precise alignment of the fovea with the target may not be critical for interception tasks, but that observers can adopt quite varied visual guidance approaches.

Research and Implementation of Robotic Arm Positioning and Grasping Based on Visual Guidance

Abstract This paper proposes a robotic arm vision guidance system based on the ROS platform, which combines the Hikrobot Robotics 2D vision sensor with the Xinsong GCR5 robotic arm to successfully realise vision-guided robotic arm positioning and grasping. Camera calibration, hand-eye calibration, and deep learning techniques are used to establish an accurate transformation relationship between the base coordinate system of the robotic arm and the camera coordinate system to achieve efficient target goal recognition and localisation. Adopting the MoveIt framework and RRT-connect algorithm, the robotic arm can flexibly plan the movement path and successfully complete the target grasping, which provides the theoretical basis and practical experience for robotic arm intelligent grasping.

Preparatory movement state enhances premovement EEG representations for brain–computer interfaces

Objective. Motor-related brain–computer interface (BCI) have a broad range of applications, with the detection of premovement intentions being a prominent use case. However, the electroencephalography (EEG) features during the premovement phase are not distinctly evident and are susceptible to attentional influences. These limitations impede the enhancement of performance in motor-based BCI. The objective of this study is to establish a premovement BCI encoding paradigm that integrates the preparatory movement state and validates its feasibility in improving the detection of movement intentions. Methods. Two button tasks were designed to induce subjects into a preparation state for two movement intentions (left and right) based on visual guidance, in contrast to spontaneous premovement. The low frequency movement-related cortical potentials (MRCPs) and high frequency event-related desynchronization (ERD) EEG data of 14 subjects were recorded. Extracted features were fused and classified using task related common spatial patterns (CSP) and CSP algorithms. Differences between prepared premovement and spontaneous premovement were compared in terms of time domain, frequency domain, and classification accuracy. Results. In the time domain, MRCPs features reveal that prepared premovement induce lower amplitude and earlier latency on both contralateral and ipsilateral motor cortex compared to spontaneous premovement, with susceptibility to the dominant hand’s influence. Frequency domain ERD features indicate that prepared premovement induce lower ERD values bilaterally, and the ERD recovery speed after button press is the fastest. By using the fusion approach, the classification accuracy increased from 78.92% for spontaneous premovement to 83.59% for prepared premovement (p < 0.05). Along with the 4.67% improvement in classification accuracy, the standard deviation decreased by 0.95. Significance. The research findings confirm that incorporating a preparatory state into premovement enhances neural representations related to movement. This encoding enhancement paradigm effectively improves the performance of motor-based BCI. Additionally, this concept has the potential to broaden the range of decodable movement intentions and related information in motor-related BCI.

Visual and Haptic Guidance for Enhancing Target Search Performance in Dual-Task Settings

In complex environments, users frequently need to manage multiple tasks simultaneously, which poses significant challenges for user interface design. For instance, when driving, users must maintain continuous visual attention on the road ahead while also monitoring rearview mirrors and performing shoulder checks. These multitasking scenarios present substantial design challenges in effectively guiding users. To address these challenges, we focus on investigating and designing visual and haptic guidance systems to augment users’ performance. We initially propose the use of visual guidance, specifically employing a dynamic arrow as a guidance technique. Our evaluation shows that dynamic arrows significantly expedite both reaction and selection times. We further introduce and evaluate haptic feedback, which users perceive as more salient than visual guidance, leading to quicker responses when switching from primary to secondary tasks. This allows users to maintain visual attention on the primary task while simultaneously responding effectively to haptic cues. Our findings suggest that multimodal guidance, especially haptic guidance, can enhance both reaction time and user experience in dual-task environments, offering promising practical implications and guidelines for designing more user-friendly interfaces and systems.

A digital twin-driven human-machine interactive assembly method based on lightweight multi-target detection and assembly feature generation

ABSTRACT In the field of industrial assembly, human-machine interactive assembly methods are frequently used. Lack of virtual and physical mapping, a convoluted guiding system, and low effect precision in the interactive process, a digital twin-driven human-machine interactive assembly method system is proposed as a solution to the mentioned issues. The YOLOv7-tiny lightweight model is used to perform accurate detection of parts. By incorporating attention modules into the backbone network, the feature extraction capability of the model in complicated assembly environments is enhanced. The assembly method proposed is validated using the assembly process of the reducer as an instance. The OpenCV method is employed to produce geometric reference features for parts. The experimental results show that the proposed assembly method can provide visual guidance for the assembly process, improve the traditional list-type assembly component retrieval method, solve the drawbacks of the pre-set assembly guidance in the guidance system that may not be able to adapt to the changes of the assembly results in the actual operation, and can accurately instruct novices how to assemble, which is characterised by easy implementation, low cost and high accuracy, and is of great significance for improving the success rate and assembly efficiency of human-machine interactive assembly.