Augmented Reality Scene Research Articles

AR/VR assisted integrated framework of autonomous disassembly system for industrial products

An immersive Augmented Reality (AR) technology enhances productivity and flexibility in manufacturing by providing real-time instructions within workers’ field of view, aiding in complex tasks. According to cognitive load and attention-based theories, AR guidance can reduce cognitive strain, enhancing task performance. The affordance of AR in industrial processes like assembly or disassembly for repair/maintenance has been exploited broadly. However, the prevailing studies on automated user interfaces for AR instruction development, specifically virtual scene creation and target generation for object mapping rendering, still need to be explored. Notably, production industries currently rely on outsourcing skilled professionals to create AR overlaying instructions using manufacturer input. However, the exactness of the disassembly repository is substantial to avoid defects in AR guidance particulars. This underscores the significance of generalized automated AR instruction development and emphasizes the demands for further research. In this work, the autonomous AR-integrated framework is developed for the disassembly context, empowering manufacturers to acquire desired outcomes across various modules. These modules include the validation of the manual user disassembly sequence repository during the prefatory phase of AR instruction creation and the provision of Disassembly Sequence Planning (DSP) input. Additionally, the framework supports the simulation/rendering of disassembly efforts in a virtual platform and the generation of rendering virtual display scenes for appropriate AR-guided object mapping. Furthermore, it enables the generation of essential model targets for registering the AR scenes. Subsequently, the extracted repositories (virtual scenes and model targets) are input into the Unity AR tool to create AR mapping guidance for disassembly operations. The performance of the autonomous disassembly module is evaluated through three case studies with different AR visualizations. Furthermore, the potential of the proposed framework is exploited by extracting feedback from experts regarding flexibility, compatibility, completion time, and cognitive level.

Interactive technology of high voltage tower acceptance UAV Based on AR and YOLOv7

To address many shortcomings of the current unmanned aerial vehicle (UAV) inspection mode of the transmission line high-voltage towers and improve the inspection efficiency, the UAV tower foreign matter detection technology based on Augmented Reality (AR) and You Only Look Once: Unified, Real-Time Object Detection (YOLOv7) has been developed. Firstly, the system architecture, including UAV, ground station, and AR glasses is established. Then, the AR scene is built by using the Unity engine, and the interactive control mode between AR glasses and UAV is designed. Subsequently, the mapping mode of head action and UAV command is devised to synchronize the UAV action and head action. Finally, the tower is detected for foreign objects by using YOLOv7 vision detection technology. The experimental results validate the efficacy of the built AR scene, demonstrating seamless functionality in completing the video streaming and attitude data transmission functions. The use of AR glasses can achieve interactive control of the UAV attitude. In addition, the YOLOv7 visual detection algorithm can accurately detect the attachment of foreign objects to the high-voltage tower.

Do we need to squeeze large tele-AR scenes to fit them into a small room?

This study aims to check the impact of the limited room space and the presence of furniture on the assessment of the distance of virtual objects displayed in augmented reality using low-cost equipment, namely a smartphone in cardboard goggles. The stimuli presented to participants were partially displayed outside the physical room in which the experiment was conducted. Empirical results confirmed a linear relationship between the perceived and actual distances, although the former is strongly underestimated, even though there is an obstacle in the form of a wall at some distance from the user. Additionally, we found that the presence of real-world objects, namely furniture, influenced participants’ estimation of the object’s distance. The results suggest that limiting the size of the room and the proximity of its walls or the presence of the furniture do not affect the perception of distances in augmented reality scenes exceeding the room size.

For higher accuracy: 3D eye movement gaze depth estimation method based on Mix-TCN

Currently, the biggest obstacle in applying eye tracking technology in Virtual Reality (VR) and Augmented Reality (AR) scenes is the particular difficulty in choosing distance and object distance in 3D scenes. In previous research, geometric calculation methods using vestibular ocular reflex (VOR) and research on binocular visual angle have been studied to some extent, but unfortunately, their effects have not reached a practical level. The paper proposes a new research idea to estimate the depth of binocular gaze using a method of time series eye movement data analysis based on depth learning, and proposes a Mix-Temporal Convolutional Network (TCN) optical time series network. By combining VOR and deep learning theory, the paper has realized the current state-of-the-art technology for estimating the depth of gaze from the movement of the gaze.

Visual coherence in augmented reality training systems considering aerospace specific features

In May 2022, Saudi Arabian Military Industries, a Saudi government agency, acquired an augmented reality training platform for pilots. In September, the Boeing Corporation began the development of an augmented reality pilot simulator. In November, a similar project was launched by BAE Systems, a leading British developer of aeronautical engineering. These facts allow us to confidently speak about the beginning of a new era of aviation simulators – simulators using the augmented reality technology. One of the promising advantages of this technology is the ability to safely simulate dangerous situations in the real world. A necessary condition for using this advantage is to ensure the visual coherence of augmented reality scenes: virtual objects must be indistinguishable from real ones. All the global IT leaders consider augmented reality as the subsequent surge of radical changes in digital electronics, so visual coherence is becoming a key issue for the future of IT, and in aerospace applications, visual coherence has already acquired practical significance. The Russian Federation lags far behind in studying the problems of visual coherence in general and for augmented reality flight simulators in particular: at the time of publication the authors managed to find only two papers on the subject in the Russian research space, while abroad their number is already approximately a thousand. The purpose of this review article is to create conditions for solving the problem. Visual coherence depends on many factors: lighting, color tone, shadows from virtual objects on real ones, mutual reflections, textures of virtual surfaces, optical aberrations, convergence and accommodation, etc. The article reviews the publications devoted to methods for assessing the conditions of illumination and color tone of a real scene and transferring them to virtual objects using various probes and by individual images, as well as by rendering virtual objects in augmented reality scenes, using neural networks.

Self-training based augmented reality for robust 3D object registration and task assistance

Typical augmented reality (AR) technologies superimpose virtual information onto the real environment for task guidance and assistance by tracking fiducial markers. However, these have narrow recognition ranges due to the markers, and it is difficult to track markers when occlusion occurs. Therefore, it is challenging to seamlessly register real objects in AR scenes captured by the camera without requiring fiducial markers. Deep learning–based approaches to 3D object registration and task assistance have recently indicated promising directions for superimposing 3D virtual objects onto corresponding real objects or their surroundings seamlessly. However, since supervised learning-based object registration requires many pose annotations on images for the locations and orientations of real objects, this process is a time-consuming and labor-intensive task. Thus, this paper proposes a new self-training based augmented reality method for seamless real object registration and task assistance without time-consuming pose annotations. As the industrial components or objects have their 3D geometric models, the proposed approach utilizes them as their synthetic data. The automatically generated synthetic dataset is used for training the deep learning model, which outputs a colored coordinate map of objects to be detected. 2D–3D key point matching between the 2D colored coordinate map and 3D colored coordinate model is then performed for 3D object registration in an AR environment. In particular, self-training is performed for fine-tuning the deep learning model by generating pseudo-labels with the highest probability for mask intersection over union (IoU) in the unlabeled real object dataset. We compared the proposed approach with previous studies to confirm its effectiveness and advantages. Also, we performed ablation studies by training a custom dataset based on various industrial 3D virtual models. Finally, we present AR-based task guidance and assistance, such as industrial robot manipulation.

Feasibility Study of Using Augmented Mirrors for Alignment Task during Orthopaedic Procedures in Mixed Reality.

Accurate depth estimation poses a significant challenge in egocentric Augmented Reality (AR), particularly for precision-dependent tasks in the medical field, such as needle or tool insertions during percutaneous procedures. Augmented Mirrors (AMs) provide a unique solution to this problem by offering additional non-egocentric viewpoints that enhance spatial understanding of an AR scene. Despite the perceptual advantages of using AMs, their practical utility has yet to be thoroughly tested. In this work, we present results from a pilot study involving five participants tasked with simulating epidural injection procedures in an AR environment, both with and without the aid of an AM. Our findings indicate that using AM contributes to reducing mental effort while improving alignment accuracy. These results highlight the potential of AM as a powerful tool for AR-enabled medical procedures, setting the stage for future exploration involving medical professionals.

A Radiotherapy Positioning Method for Both Coarse Guidance and Precise Verification Based on Integration of AR and Optical Surface Imaging

Traditional methods of radiotherapy positioning have shortcomings such as fragile skin-markers, additional doses and lack of information integration. Emerging technologies may provide alternatives for the relevant clinical practice. We proposed a noninvasive radiotherapy positioning method integrating augmented reality (AR) and optical surface, and evaluated its feasibility in clinical workflow. AR and structured light-based surface were integrated to implement the coarse-to-precise positioning through two coherent steps, i) the AR-based coarse guidance. To implement quality assurance, recognition of face and pattern was used for patient authentication, case association and accessory validation in AR scenes. The holographic images reconstructed from simulation computed tomography (CT) images, guided the initial posture correction by virtual-real alignment. ii) optical surface-based precise verification. The point clouds were fused, with the calibration and pose estimation of structured light cameras, and segmented according to the preset regions of interest (ROIs). The global-to-local registration for cross-source point clouds was achieved to calculate couch shifts in 6 degrees-of-freedom (DoF), which were ultimately transmitted to AR scenes. The evaluation based on phantom and human-body (4 volunteers) included, i) quality assurance workflow, ii) errors of both steps and correlation analysis, and iii) receiver operating characteristic (ROC). The maximum errors in phantom evaluation were 3.4±2.5 mm in Vrt and 1.4±1.0° in Pitch for the coarse guidance step, while 1.6±0.9 mm in Vrt and 0.6±0.4° in Pitch for the precise verification step. The Pearson correlation coefficients between precise verification and cone beam CT (CBCT) results were distributed in the interval [0.81, 0.85]. In ROC analysis, the areas under the curve (AUC) were 0.87 and 0.89 for translation and rotation respectively. In human body-based evaluation, the errors of thorax and abdomen (T&A) were significantly greater than those of head and neck (H&N) in Vrt (2.6±1.3 vs. 1.7±1.1, p<0.01), Lng (2.4±1.3 vs. 1.4±0.1, p<0.01) and Rtn (0.8±0.5 vs. 0.6±0.4, p = 0.03) while relatively similar in Lat (1.7±1.0 vs. 1.9±1.1, p = 0.13). The combination of AR and optical surface has utility and feasibility for patient positioning, in terms of both safety and accuracy.

Visual Coherence for Augmented Reality

Introduction. The 2020s were marked by the emergence of a new generation of computer simulators using augmented reality. One of the promising advantages of augmented reality technology is the ability to safely simulate hazardous situations real-world. A prerequisite for realizing this advantage is to provide the visual coherence of augmented reality scenes: virtual objects must be indistinguishable from real ones. All IT leaders consider augmented reality as a next “big wave”; thus, the visual coherence is becoming a key issue for IT in general. However, it is in aerospace applications that the visual coherence has already acquired practical significance. An example is Boeing's development of an augmented reality flight simulator, which began in 2022. Visual coherence is a complex problem, one of the aspects of which is to provide the correct overall coloration of virtual objects in an augmented reality scene. The objective of the research was to develop a new method of such tinting.Materials and Methods. The developed method (called spectral transplantation) uses two-dimensional spectral image transformations.Results. A spectral transplantation technology is proposed that provides direct transfer of color, brightness, and contrast characteristics from the real background to virtual objects. An algorithm for automatic selection of the optimal type of spectral transformation has been developed.Discussion and Conclusion. Being a fully automatic process without recording lighting conditions, spectral transplantation solves a number of complex problems of visual coherence. Spectral transplantation can be a valuable addition to other methods of providing visual coherence.

Real-Time Target Tracking and Localization in Augmented Reality Scenes with Virtual Reality-Based Computer Vision Technology

Real-Time Target Tracking and Localization in Augmented Reality Scenes with Virtual Reality-Based Computer Vision Technology

Innovative integration of augmented reality and optical surface imaging: A coarse-to-precise system for radiotherapy positioning.

Traditional methods of radiotherapy positioning have shortcomings such as fragile skin-markers, additional doses, and lack of information integration. Emerging technologies may provide alternatives for the relevant clinical practice. To propose a noninvasive radiotherapy positioning system integrating augmented reality (AR) and optical surface, and to evaluate its feasibility in clinical workflow. AR and structured light-based surface were integrated to implement the coarse-to-precise positioning through two coherent steps, the AR-based coarse guidance and the optical surface-based precise verification. To implement quality assurance, recognition of face and pattern was used for patient authentication, case association, and accessory validation in AR scenes. The holographic images reconstructed from simulation computed tomography (CT) images, guided the initial posture correction by virtual-real alignment. The point clouds of body surface were fused, with the calibration and pose estimation of structured light cameras, and segmented according to the preset regions of interest (ROIs). The global-to-local registration for cross-source point clouds was achieved to calculate couch shifts in six degrees-of-freedom (DoF), which were ultimately transmitted to AR scenes. The evaluation based on phantom and human-body (4 volunteers) included, (i) quality assurance workflow, (ii) errors of both steps and correlation analysis, (iii) receiver operating characteristic (ROC), (iv) distance characteristics of accuracy, and (v) clinical positioning efficiency. The maximum errors in phantom evaluation were 3.4±2.5mm in Vrt and 1.4±1.0° in Pitch for the coarse guidance step, while 1.6±0.9mm in Vrt and 0.6±0.4° in Pitch for the precise verification step. The Pearson correlation coefficients between precise verification and cone beam CT (CBCT) results were distributed in the interval [0.81, 0.85]. In ROC analysis, the areas under the curve (AUC) were 0.87 and 0.89 for translation and rotation, respectively. In human body-based evaluation, the errors of thorax and abdomen (T&A) were significantly greater than those of head and neck (H&N) in Vrt (2.6±1.1vs. 1.7±0.8, p<0.01), Lng (2.3±1.1vs. 1.4±0.9, p<0.01), and Rtn (0.8±0.4vs. 0.6±0.3, p=0.01) while relatively similar in Lat (1.8±0.9vs. 1.7±0.8, p=0.07). The translation displacement range, after coarse guidance step, required for high accuracy of the optical surface component of the integrated system was 0-42mm, and the average positioning duration of the integrated system was significantly less than that of conventional workflow (355.7±21.7 vs. 387.7±26.6 s, p<0.01). The combination of AR and optical surface has utility and feasibility for patient positioning, in terms of both safety and accuracy.

Competencies of Industrial Engineers for Implementing Augmented Reality Metadata Systems

The paper focuses on the use of augmented reality (AR) by industrial engineers, especially for determining the necessary competencies required for its use. Industrial engineers are not inherently programmers. Nevertheless, augmented reality is a modern trend in their field, especially in relation to the concept of Industry 4.0 and industry in general, where it has a higher potential than virtual reality. In the first part of this paper, we placed augmented reality and the competencies required for its use in the context of industrial engineering. Subsequently, we described our own methods of implementing an augmented reality industrial metadata visualization system, namely Help Lightning Fieldbit and Unity 3D, using the Vuforia extension. We chose the metadata used in the methods with regard to their environmental potential. In this part of the paper, we also described the chosen and applied testing methodology using a questionnaire survey. Subsequently, we described the results from the questionnaire surveys of both these approaches of implementing augmented reality methods. Finally, we evaluated the results and compared them with each other and with results from other authors. As the results show, the most important competencies for creating the described AR environments are analytical competencies. We draw conclusions from the collected data regarding the necessary competencies for the creation of AR scenes using these methods and their deployment in industry, including an outline for further research.

Підвищення ефективності формування сцен доповненої реальності за допомогою нейронних мереж

On the way to the fourth wave of industrial technological progress, visualization and virtualization tools have received a wide range of applications and integration into the multi-industry space. The technology of creating additional visual images is currently used in the medical field, the field of education, the industrial and industrial field, the field of advertising and trade, in the field of modeling and design, in the scientific field, the cultural and entertainment field, etc. The potential of using visualization tools is inexhaustible, because the integration of additional information in the form of graphic objects helps to increase the perception of the data flow of reality and develops analytical capabilities for users of augmented reality technology. Modern means of creating scenes of augmented reality and additional visual images have increased requirements for the consumption of computing power, as they require dynamic adaptive interaction with streams of real data, which actually leads to the formation of extremely complex algorithms and corresponding hardware-analog and software-digital solutions. Optimizing and improving the efficiency of the augmented reality scene creation technology is a scientific problem that needs to be solved, including within the scope of the current research. According to the bibliographic search and analysis of modern trends and profile developments, the potential possibility of using neural network tools to create additional visual objects in augmented reality scenes has been established. Neural networks have a high adaptive capacity for learning and an adequate reaction to external conditions of functioning. Therefore, neural network tools are surprisingly suitable for integration into technological solutions for the functioning of augmented reality technology. Known topological solutions for arranging and organizing the functioning of neural networks, which can be applied to solve a certain scientific problem of optimizing the consumption of computing power and increasing the efficiency of creating augmented reality scenes, have a number of limitations to their application, which prompts the further search for adaptive solutions. A promising solution is the formation of combined-hybrid technologies for constructing the topology of neural networks. Thus, the relevance of the research is outlined, the scientific issues are formed, and the vector of scientific research to solve the identified issues is proposed.

A 3D Scene Information Enhancement Method Applied in Augmented Reality

Aiming at the problem that the detection of small planes with unobvious texture is easy to be missed in augmented reality scene, a 3D scene information enhancement method to grab the planes for augmented reality scene is proposed based on a series of images of a real scene taken by a monocular camera. Firstly, we extract the feature points from the images. Secondly, we match the feature points from different images, and build the three-dimensional sparse point cloud data of the scene based on the feature points and the camera internal parameters. Thirdly, we estimate the position and size of the planes based on the sparse point cloud. The planes can be used to provide extra structural information for augmented reality. In this paper, an optimized feature points extraction and matching algorithm based on Scale Invariant Feature Transform (SIFT) is proposed, and a fast spatial planes recognition method based on a RANdom SAmple Consensus (RANSAC) is established. Experiments show that the method can achieve higher accuracy compared to the Oriented Fast and Rotated Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK) and Super Point. The proposed method can effectively solve the problem of missing detection of faces in ARCore, and improve the integration effect between virtual objects and real scenes.

SURG-37. NEURO-ONCOLOGICAL AUGMENTED REALITY: A MORE INTUITIVE APPROACH TO RESECTION PLANNING

Abstract BACKGROUND When preparing for the resection of an intracranial lesion, its borders and optimal approach are often determined using neuronavigation with a tracked pointer. This can sometimes prove challenging, especially for deep-seated lesions. Augmented reality (AR) can simplify and improve this step by directly displaying the lesion on the patient's skin. METHODS A proprietary inside-out infrared tracking solution was developed, allowing for heads-up displaying AR scenes on the Microsoft HoloLens II without the need for an external tracking camera or computer. We included twenty patients with an intracerebral lesion planned for resection. After semi-automatic hologram-to-patient registration, different participants marked the lesion outlines on the patient’s skin, consecutively aided by the Brainlab neuronavigation system and the HoloLens. Each registration on both systems provided a registration transform that was compared for accuracy and consistency. Participant performance was quantified in terms of duration and accuracy for both patient registration and lesion delineation, and compared to expert performance. RESULTS When using AR both registration and delineation were significantly faster than with conventional neuronavigation (p = 0.02 and p &amp;lt; 0.001, respectively, and p &amp;lt; 0.001 for the total duration), taking 79.23 ± 17.48 and 39.58 ± 39.10 seconds while neuronavigation required 96.61 ± 24.54 and 90.80 ± 44.09 seconds. AR had a registration offset of 3.3mm and 3.4°, and was more consistent compared to neuronavigation. AR facilitated more accurate and detailed lesion delineation, while neuronavigation often overestimated lesion size. CONCLUSION Augmented reality provides a faster and more accurate alternative for resection planning. Lesion delineation was more intuitive while retaining high accuracy. Future research should focus on further intraoperative implementations.

P07.02.B Neuro-oncological augmented reality planning (NOA-p)

Abstract Background When preparing for the resection of an intracranial lesion, neuronavigation with a tracked pointer is most often used to determine lesion borders and the optimal approach. This can sometimes prove challenging, especially for deep-seated lesions. Augmented Reality (AR), directly displaying the lesion on the patient’s skin, can simplify and improve this step. Material and Methods We developed a system for inside-out infrared tracking that does not require an external tracking camera or external computer and allows for heads-up displaying an AR scene on the Microsoft HoloLens II. Twenty patients planned for the resection of an intracerebral lesion were included in our study. After patient registration, the lesion outlines were marked on the patient’s skin by different participants, consecutively using the Brainlab neuronavigation system and the HoloLens. Each registration on both systems provided a registration transform that was compared for accuracy and consistency. The performance of the participants was measured in terms of duration and accuracy and compared to expert registration and delineation. Results Both registration and delineation were significantly faster with AR (p=0.02 and p&amp;lt;0.001, respectively, and p&amp;lt;0.001 for the total duration), taking 79.23±17.48 and 39.58±39.10 seconds while neuronavigation required 96.61±24.54 and 90.80±44.09 seconds. AR had a registration offset of 3.3mm and 3.4°, and was more consistent compared to neuronavigation. AR facilitated more accurate and detailed lesion delineation, while neuronavigation often overestimated lesion size. Conclusion Augmented reality provides a faster and more accurate alternative for resection planning. Lesion delineation is more intuitive while remaining high in accuracy. Future research should focus on further intraoperative implementations.

A Quick Development Toolkit for Augmented Reality Visualization (QDARV) of a Factory

Augmented reality (AR) technology has been applied to the industrial field; however, at present, most industrial AR applications are developed for specific application scenarios, which are difficult to develop and have a long development cycle, and lack universality. To this end, this paper proposed a quick development toolkit for augmented reality visualization (QDARV) of a factory by using the script configuration and parsing approach. With QDARV, designers can quickly develop industrial AR applications, achieve AR registration based on quick response (QR) code and simultaneous localization and mapping (SLAM), and superpose information such as disassembly animations, industrial instruments, pictures, and texts on real scenes. In QDARV, an AR registration method based on SLAM map reloading is proposed. Through saving and reloading the map and the configuration and analysis of the SLAM-AR display content script, the AR scene configured by the designer is displayed. Meanwhile, the objects detected using YOLOv5 are used as a landmark to assist the SLAM system in positioning. The QDARV can be applied to realize AR visualization of factories with a large area.

Reconstructing real object appearance with virtual materials using mobile augmented reality

In augmented reality (AR) applications, detecting and tracking real-world objects remains a challenge. Another challenge is making the loaded virtual objects truly reflect the lighting and shadow conditions of the actual environment in which they are located. In this paper, we proposed an improved method for real-time light estimation of environmental lighting in AR applications, tracking the movement of physical objects, and making the corresponding virtual objects reflect the lighting and shadows of the actual environment. First, we proposed an effective method for estimating light in an AR scene by training a deep neural network once on different scenes from the input of RGB-D images to determine the direction of the brightest light. Second, we experimented with advanced methods of detecting and tracking different types of real objects (flat surfaces, simple geometries, and complex geometries) for visualizing virtual materials on the top of these surfaces, where virtual materials will acquire the conditions of the real place in terms of lighting, reflection, and shadows depending on our proposed method for light estimation. Our method of light estimation was tested via experiments on the three types of real objects mentioned above. Results indicate that our method achieves higher resolution and lower error rate compared with other work. Our methods have achieved very good results in terms of detection accuracy, tracking accuracy, realistic visualization of materials, matching real shade, and lighting conditions.

Augmented reality calibration using feature triangulation iteration-based registration for surgical navigation

BackgroundMarker-based augmented reality (AR) calibration methods for surgical navigation often require a second computed tomography scan of the patient, and their clinical application is limited due to high manufacturing costs and low accuracy. MethodsThis work introduces a novel type of AR calibration framework that combines a Microsoft HoloLens device with a single camera registration module for surgical navigation. A camera is used to gather multi-view images of a patient for reconstruction in this framework. A shape feature matching-based search method is proposed to adjust the size of the reconstructed model. The double clustering-based 3D point cloud segmentation method and 3D line segment detection method are also proposed to extract the corner points of the image marker. The corner points are the registration data of the image marker. A feature triangulation iteration-based registration method is proposed to quickly and accurately calibrate the pose relationship between the image marker and the patient in the virtual and real space. The patient model after registration is wirelessly transmitted to the HoloLens device to display the AR scene. ResultsThe proposed approach was used to conduct accuracy verification experiments on the phantoms and volunteers, which were compared with six advanced AR calibration methods. The proposed method obtained average fusion errors of 0.70 ± 0.16 and 0.91 ± 0.13 mm in phantom and volunteer experiments, respectively. The fusion accuracy of the proposed method is the highest among all comparison methods. A volunteer liver puncture clinical simulation experiment was also conducted to show the clinical feasibility. ConclusionsOur experiments proved the effectiveness of the proposed AR calibration method, and revealed a considerable potential for improving surgical performance.

Data-Driven Image Interaction-Based Software Infrastructure for Graphic Design Research and Implementation

This article constructs and studies a graphic design software model based on data-driven image interactions. The semantic information of image interaction is integrated into the framework of the augmented reality label placement algorithm. A new feature map, Guidance Map, is proposed to combine image saliency information with semantic information in the task scenario, accurately describing the importance of different regions in the user’s field of view to derive a more reasonable label placement strategy. This database is different from other label placement algorithms that only use data-driven saliency detection. This paper proposes a data-driven augmented reality labeling method, which first presents the design requirements for augmented reality scene labeling placement, then designs the energy function for augmented reality scene characteristics, transforms the labeling placement problem into an optimization problem, and solves it. This paper proposes a feature representation form oriented to pixels, elements, interelement relationships, planes, and applications and quantifies the geometric features, perceptual features, and stylistic features in them. The model can fit the probability density distribution of the parts to predict the optimal element placement and element color under the target conditions.