Appearance Model Research Articles

A Hierarchical Architecture for Neural Materials

AbstractNeural reflectance models are capable of reproducing the spatially‐varying appearance of many real‐world materials at different scales. Unfortunately, existing techniques such as NeuMIP have difficulties handling materials with strong shadowing effects or detailed specular highlights. In this paper, we introduce a neural appearance model that offers a new level of accuracy. Central to our model is an inception‐based core network structure that captures material appearances at multiple scales using parallel‐operating kernels and ensures multi‐stage features through specialized convolution layers. Furthermore, we encode the inputs into frequency space, introduce a gradient‐based loss, and employ it adaptive to the progress of the learning phase. We demonstrate the effectiveness of our method using a variety of synthetic and real examples.

Correlation Filters for UAV Online Tracking Based on Complementary Appearance Model and Reversibility Reasoning

Correlation Filters for UAV Online Tracking Based on Complementary Appearance Model and Reversibility Reasoning

Self-supervised learning for RGB-D object tracking

Recently, there has been a growing interest in RGB-D object tracking thanks to its promising performance achieved by combining visual information with auxiliary depth cues. However, the limited volume of annotated RGB-D tracking data for offline training has hindered the development of a dedicated end-to-end RGB-D tracker design. Consequently, the current state-of-the-art RGB-D trackers mainly rely on the visual branch to support the appearance modelling, with the depth map utilised for elementary information fusion or failure reasoning of online tracking. Despite the achieved progress, the current paradigms for RGB-D tracking have not fully harnessed the inherent potential of depth information, nor fully exploited the synergy of vision-depth information. Considering the availability of ample unlabelled RGB-D data and the advancement in self-supervised learning, we address the problem of self-supervised learning for RGB-D object tracking. Specifically, an RGB-D backbone network is trained on unlabelled RGB-D datasets using masked image modelling. To train the network, the masking mechanism creates a selective occlusion of the input visible image to force the corresponding aligned depth map to help with discerning and learning vision-depth cues for the reconstruction of the masked visible image. As a result, the pre-trained backbone network is capable of cooperating with crucial visual and depth features of the diverse objects and background in the RGB-D image. The intermediate RGB-D features output by the pre-trained network can effectively be used for object tracking. We thus embed the pre-trained RGB-D network into a transformer-based tracking framework for stable tracking. Comprehensive experiments and the analysis of the results obtained on several RGB-D tracking datasets demonstrate the effectiveness and superiority of the proposed RGB-D self-supervised learning framework and the following tracking approach.

Predicting Tibia-Fibula Geometry and Density From Anatomical Landmarks Via Statistical Appearance Model: Influence of Errors on Finite Element-Calculated Bone Strain.

State-of-the-art participant-specific finite element models require advanced medical imaging to quantify bone geometry and density distribution; access to and cost of imaging is prohibitive to the use of this approach. Statistical appearance models may enable estimation of participants' geometry and density in the absence of medical imaging. The purpose of this study was to: (1) quantify errors associated with predicting tibia-fibula geometry and density distribution from skin-mounted landmarks using a statistical appearance model and (2) quantify how those errors propagate to finite element-calculated bone strain. Participant-informed models of the tibia and fibula were generated for thirty participants from height and sex and from twelve skin-mounted landmarks using a statistical appearance model. Participant-specific running loads, calculated using gait data and a musculoskeletal model, were applied to participant-informed and CT-based models to predict bone strain using the finite element method. Participant-informed meshes illustrated median geometry and density distribution errors of 4.39-5.17 mm and 0.116-0.142 g/cm3, respectively, resulting in large errors in strain distribution (median RMSE = 476-492 με), peak strain (limits of agreement =±27-34%), and strained volume (limits of agreement =±104-202%). These findings indicate that neither skin-mounted landmark nor height and sex-based predictions could adequately approximate CT-derived participant-specific geometry, density distribution, or finite element-predicted bone strain and therefore should not be used for analyses comparing between groups or individuals.

Numerical study of osteophyte effects on preoperative knee functionality in patients undergoing total knee arthroplasty.

Osteophytes are routinely removed during total knee arthroplasty, yet the preoperative planning currently relies on preoperative computed tomography (CT) scans of the patient's osteoarthritic knee, typically including osteophytic features. This complicates the surgeon's ability to anticipate the exact biomechanical effects of osteophytes and the consequences of their removal before the operation. The aim of this study was to investigate the effect of osteophytes on ligament strains and kinematics, and ascertain whether the osteophyte volume and location determine the extent of this effect. We segmented preoperative CT scans of 21 patients, featuring different osteophyte severity, using image-based active appearance models trained to identify the osteophytic and preosteophytic bone geometries and estimate the cartilage thickness in the segmented surfaces. The patients' morphologies were used to scale a template musculoskeletal knee model. Osteophytes induced clinically relevant changes to the knee's functional behavior, but these were variable and patient-specific. Generally, severe osteophytic knees significantly strained the oblique popliteal ligament (OPL) and posterior capsule (PC) relative to the preosteophytic state. Furthermore, there was a marked effect on the lateral collateral ligament and anterolateral ligament (ALL) strains compared to mild and moderate osteophytic knees, and concurrent alterations in the tibial lateral-medial translation and external-internal rotation. We found a strong correlation between the OPL, PC, and ALL strains and posterolateral condylar and tibial osteophytes, respectively. Our findings may have implications for the preoperative planning in total knee arthroplasty, toward reproducing the physiological knee biomechanics as close as feasibly possible.

Xception transfer learning with early stopping for facial age estimation

The rapid development of deep learning attracts more attention to the analysis of person's face images. Deep learning methodsof facial age estimation are more effective compared to methods based on anthropometric models, models of active appearance, texture models, subspace of aging patterns. However, deep learning networks require more computing power to process images. Pre-trained models do not need a large training set and their training time is less. However, the parameters obtained as a result of transfer learning of the pre-training network significantly affect its efficiency. It is also necessary to take into account the properties of the processed images, in particular, the conditions under which they were obtained.Recently, the facial age estimation is implemented in applications in devices with limited resources of computing, for example, in smartphones. The memory size and power consumption of such applications are limited by the computing power of mobile devices. In addition, when photographing a person's face with a smartphone camera, it is very difficult to ensure the uniform lighting. The aim of the research is reducing the error of facial age estimation from uneven illuminated images by applying an early stopping of transfer learning of the Xception network. The proposed technique of transfer learning includes an early stopping of training, if the improvement of the results is not observed within a certain number of epochs. Then the network weights from the epoch with the lowest validation loss are saved. As a result of the proposed technique applying, the average absolute error of age estimation was about five years from unevenly illuminated test images. A number of parameters of the used in this case Xceptionnetwork is less than that of other deep learning neural networks which solved the age estimation problem. Then applying of the Xception network reduces the resource consumption of devices with limited computing power. Prospects for further research are reducing the unevenness of facial image lighting to decrease the error of age estimation. Also, to reduce the computing resources, it is promising to use fast transforms in the Xception convolutional layers.

Application of Multimedia Man-Machine Interaction in College Physical Education Teaching

In the realm of physical education in college, the traditional classroom teaching approach primarily relies on paper textbooks and demonstrations by the teacher. However, in today's physical education classrooms, there is a need to change these traditional teaching methods. Utilizing multimedia technology and interactive interfaces, teachers can demonstrate the essential actions and intricate details through multimedia videos and other formats such as push-pull and shaking. This chapter introduces a multi-segment human body tracking algorithm that focuses on real-time tracking. The human body target is divided into multiple segments, and an online learning method based on Hough Forests is used to learn the overall appearance of the human body and the appearance model of each segment. The research findings indicate that when the merged area exceeds 82.36% of the pre-segmentation area using watershed analysis, smaller areas lead to faster segmentation speed, while larger areas result in slower segmentation speed. Compared to other methods, this approach yields better recognition results.

Explicit Visual Prompts for Visual Object Tracking

How to effectively exploit spatio-temporal information is crucial to capture target appearance changes in visual tracking. However, most deep learning-based trackers mainly focus on designing a complicated appearance model or template updating strategy, while lacking the exploitation of context between consecutive frames and thus entailing the when-and-how-to-update dilemma. To address these issues, we propose a novel explicit visual prompts framework for visual tracking, dubbed EVPTrack. Specifically, we utilize spatio-temporal tokens to propagate information between consecutive frames without focusing on updating templates. As a result, we cannot only alleviate the challenge of when-to-update, but also avoid the hyper-parameters associated with updating strategies. Then, we utilize the spatio-temporal tokens to generate explicit visual prompts that facilitate inference in the current frame. The prompts are fed into a transformer encoder together with the image tokens without additional processing. Consequently, the efficiency of our model is improved by avoiding how-to-update. In addition, we consider multi-scale information as explicit visual prompts, providing multiscale template features to enhance the EVPTrack's ability to handle target scale changes. Extensive experimental results on six benchmarks (i.e., LaSOT, LaSOText, GOT-10k, UAV123, TrackingNet, and TNL2K.) validate that our EVPTrack can achieve competitive performance at a real-time speed by effectively exploiting both spatio-temporal and multi-scale information. Code and models are available at https://github.com/GXNU-ZhongLab/EVPTrack.

Action-guided CycleGAN for Bi-directional Video Prediction

Most of the earlier proposed action prediction methods, such as PoseGAN, C2GAN, etc. are limited to unidirectional prediction and not intended to perform the desired class of action. In this paper, we propose an action-guided Cycle Generative Adversarial Network (AGCGAN), which is a bi-directional video prediction model to anticipate future frames from current visual frames and vice versa. The proposed CycleGAN architecture consists of two generators and two discriminators along with two additional keypoint detectors to make keypoint loss measurements. The adversarial and cycle losses perform the appearance modelling whereas the keypoint loss provides the necessary motion correction. We evaluated the effectiveness of our proposed method on the UT-Interaction dataset. Furthermore, we also tested our model on the standard Market-1501 dataset against the state-of-the-art schemes. Results show that our proposed method provides comparable visual quality with the state-of-the-art methods.

RGB road scene material segmentation

We introduce RGB road scene material segmentation, i.e., per-pixel segmentation of materials in real-world driving views with pure RGB images, as a novel computer vision task by building a benchmark dataset and by deriving a new method. Our dataset, KITTI-Materials, is based on the well-established KITTI dataset and consists of 1000 frames covering 24 different road scenes of urban/suburban landscapes, carefully annotated with one of 20 material categories for every pixel. It is the first dataset for RGB material segmentation in real driving scenes. Through careful analysis of KITTI-Materials, we identify the extraction and fusion of texture and image context as the key to accurate modeling of road scene material appearance. For this, we introduce Road scene Material Segmentation Network (RMSNet) as a baseline method for this challenging task. RMSNet encodes multi-scale hierarchical features with efficient Transformer layers. We construct the decoder of RMSNet based on a novel efficient self-attention model, which we refer to as SAMixer which adaptively fuses texture and context cues across multiple feature levels. Extensive experiments on KITTI-Materials validate the effectiveness of our RMSNet. We believe our work lays a solid foundation for further studies on RGB road scene material segmentation.

Probabilistic 3D motion model for object tracking in aerial applications

AbstractVisual object tracking, crucial in aerial applications such as surveillance, cinematography, and chasing, faces challenges despite AI advancements. Current solutions lack full reliability, leading to common tracking failures in the presence of fast motions or long‐term occlusions of the subject. To tackle this issue, a 3D motion model is proposed that employs camera/vehicle states to locate a subject in the inertial coordinates. Next, a probability distribution is generated over future trajectories and they are sampled using a Monte Carlo technique to provide search regions that are fed into an online appearance learning process. This 3D motion model incorporates machine‐learning approaches for direct range estimation from monocular images. The model adapts computationally by adjusting search areas based on tracking confidence. It is integrated into DiMP, an online and deep learning‐based appearance model. The resulting tracker is evaluated on the VIOT dataset with sequences of both images and camera states, achieving a 68.9% tracking precision compared to DiMP's 49.7%. This approach demonstrates increased tracking duration, improved recovery after occlusions, and faster motions. Additionally, this strategy outperforms random searches by about 3.0%.

CHEAT DETECTION IN ONLINE EXAMINATIONS USING ARTIFICIAL INTELLIGENCE

With increasing use of ICT and technical advancements in the education sector, distant and online education as well as examinations are being carried out frequently. However, online examination, as a method of assessment offers the risk of an unmonitored setting where students have full access to external resources. Online-proctored exams are the most efficient way for educational institutions to ensure academic honesty and ethics to counteract this. Typically, proctoring requires human assistance in the form of online proctors who remotely monitor students' performance. Yet, due to the rising demand for personnel and the intrusive nature of human proctoring, it is imperative to explore other areas. To tackle this pressing issue, this research work aims to devise a novel architecture that, through the development of a robust and automated Artificial Intelligence system, enables students to take exams remotely and reduces proctor involvement. The method overcomes the shortcomings of the previous automated proctoring system by combining important components of online exam cheating detection with cost-effective and efficient hardware. By proposing a Hybrid of FaceNet Model, Lucas Kanade Algorithm, and Active Appearance Model for Face Detection and Activity Monitoring of the student, the proposed system extracts semantic indicators to evaluate whether an applicant is cheating in an online examination. The proposed Cheat Detection system's experimental results measured via an F-score of 0.94 demonstrate its efficacy, and promising performance compared to the standard baseline techniques.

Online Learning Samples and Adaptive Recovery for Robust RGB-T Tracking

With the increasing diversity of visual tracking tasks, object tracking in RGB and thermal (RGB-T) modalities has received widespread interest. Most of the existing RGB-T tracking methods mainly improve tracking performance by integrating hierarchically complementary information from RGB and thermal modalities, however, they are insufficient in handling tracking failures due to the lack of re-detection capability. To address these issues, we propose a new RGB-T tracking method with online learning samples and adaptive object recovery. First, the features of RGB and thermal modalities are concatenated for robust appearance modeling. Second, a multimodal fusion strategy is designed to stably integrate reliable information of modalities and propose to use similarity to measure tracking confidence. Finally, a detector with online learning of positive and negative samples and adaptive recovery is developed to correct unreliable tracking results. Numerical results on five recent large-scale benchmark datasets demonstrate that the proposed tracker achieves competitive performance compared to other state-of-the-art methods.

Simple color appearance model (sCAM) based on simple uniform color space (sUCS).

A new color appearance model named sCAM has been developed, including a uniform color space, sUCS. The model has a simple structure but provides comprehensive functions for color related applications. It takes input from either XYZ D65 or signals from an RGB space. Their accuracy has been extensively tested. sUCS performed the best or second-best to the overall 28 datasets for space uniformity and the 6 datasets for hue linearity comparing the state of the art UCSs. sCAM also performed the best to fit all available one- and two-dimensional color appearance datasets. It is recommended to have field tests for all color related applications.

An AI-based novel system for predicting respiratory support in COVID-19 patients through CT imaging analysis

The proposed AI-based diagnostic system aims to predict the respiratory support required for COVID-19 patients by analyzing the correlation between COVID-19 lesions and the level of respiratory support provided to the patients. Computed tomography (CT) imaging will be used to analyze the three levels of respiratory support received by the patient: Level 0 (minimum support), Level 1 (non-invasive support such as soft oxygen), and Level 2 (invasive support such as mechanical ventilation). The system will begin by segmenting the COVID-19 lesions from the CT images and creating an appearance model for each lesion using a 2D, rotation-invariant, Markov–Gibbs random field (MGRF) model. Three MGRF-based models will be created, one for each level of respiratory support. This suggests that the system will be able to differentiate between different levels of severity in COVID-19 patients. The system will decide for each patient using a neural network-based fusion system, which combines the estimates of the Gibbs energy from the three MGRF-based models. The proposed system were assessed using 307 COVID-19-infected patients, achieving an accuracy of 97.72%pm 1.57, a sensitivity of 97.76%pm 4.08, and a specificity of 98.87%pm 2.09, indicating a high level of prediction accuracy.

Exploring the Path of Innovative College Dance Teaching Models with Motion Capture Technology

Abstract The continuous development of artificial intelligence has brought new exploration paths and ideas for the teaching field, however, the current application and research of motion capture technology in college dance teaching mode is less. In this study, the human body and the appearance model are modeled in three dimensions: the simulated annealing particle swarm algorithm is used to optimize the human body modeling posture. Then, the feature vector matching algorithm is used to capture the human body’s movement posture, construct the college dance teaching mode, and carry out practical application. According to the study, the capture effect of the five movements in the dance teaching classroom video clip is between 0.865 and 0.945, and each movement is recognized with an accuracy of above 95%. After using this model, it was concluded by analyzing the effect of dance teaching that the number of excellent grades in the experimental group was 2 more than that in the control group, and the number of good grades was 2 more than that in the control group. This study provides a new development path for dance teaching in colleges and universities.

One-Stream Vision-Language Memory Network for Object Tracking

Most existing tracking methods try to represent the target by exploiting visual information as much as possible based on the various deep networks. However, the appearance model hardly describes the attribute feature of the target well, which makes the trackers fail to adapt to the complex visual surrounding. In this paper, inspired by brain-like intelligence, we propose an One-stream Vision-Language Memory network (OVLM) for object tracking. Firstly, we use the combination of vision and language to build the target model and use the semantic information in the language to compensate for the instability of visual information, making the target model more stable in the face of complex appearance changes. Secondly, to build a more compact target model, we propose a memory token selection mechanism that utilizes linguistic information to eliminate tokens that do not contain target information. Furthermore, to provide better visual information for target modeling, we propose a language-based evaluation method to select high-quality target samples to be stored in the memory. Finally, OVLM achieves a 64.7% success rate on the large-scale tracking benchmark dataset TNL2K, outperforming the previous best result (VLT) by 11.6%. By exposing the possibility of the vision-language memory network, we aim to draw greater attention to it and open up new avenues for vision-language tracking.

The effect of femur positioning on dual-energy X-ray absorptiometry (DXA) measures and statistical shape and appearance modeling (SSAM) fracture risk assessments.

The diagnosis of osteoporosis using Dual-energy X-ray Absorptiometry (DXA) relies on accurate hip scans, whereby variability in measurements may be introduced by altered patient positioning, as could occur with repeated scans over time. The goal herein was to test how altered postures affect diagnostic metrics (i.e., standard clinical metrics and a newer image processing tool) for femur positioning. A device was built to support cadaveric femurs and adjust their orientation in 3° increments in flexion and internal/external rotation. Seven isolated femurs were scanned in six flexion postures (0° (neutral) to 15° of flexion) and eleven rotational postures (15° external to 15° internal rotation) while collecting standard clinical DXA-based measures for each scan. The fracture risk tool was applied to each scan to calculate fracture risk. Two separate one-way repeated measures ANOVAs (α = 0.05) were performed on the DXA-based measures and fracture risk prediction output. Flexion had a significant effect on T-score, Bone Mineral Density (BMD), and Bone Mineral Content (BMC), but not area, at angles greater than 12°. Internal and external rotation did not have a significant effect on any clinical metric. Fracture risk (as assessed by the image processing tool) was not affected by either rotation mode. Overall, this suggests clinicians can adjust patient posture to accommodate discomfort if deviations are less than 12 degrees, and the greatest care should be taken in flexion. Furthermore, the tool is relatively insensitive to postural adjustments, and as such may be a good option for tracking risk over repeated patient scans.

Pose-Aided Video-Based Person Re-Identification via Recurrent Graph Convolutional Network

Existing methods for video-based person re-identification (ReID) mainly learn the appearance feature of a given pedestrian via a feature extractor and a feature aggregator. However, the appearance models would fail to learn a large inter-class variance when different pedestrians have similar appearances. Considering that different pedestrians have different walking postures and body proportions, we propose to learn the discriminative pose feature beyond the appearance feature for video retrieval. Specifically, we implement a two-branch architecture to separately learn the appearance feature and pose feature, and then concatenate them together for inference. To learn the pose feature, we first detect the pedestrian pose in each frame through an off-the-shelf pose detector, and construct a temporal graph using the pose sequence. We then exploit a recurrent graph convolutional network (RGCN) to learn the node embeddings of the temporal pose graph, which devises a global information propagation mechanism to simultaneously achieve the neighborhood aggregation of intra-frame nodes and message passing among inter-frame graphs. Finally, we propose a dual-attention method (DAM) consisting of node-attention and time-attention to obtain the temporal graph representation from the node embeddings, where the self-attention mechanism is employed to learn the importance of each node and each frame. We verify the proposed method on three video-based ReID datasets, i.e., Mars, DukeMTMC and iLIDS-VID, whose experimental results demonstrate that the learned pose feature can effectively improve the performance of existing appearance models.

NerfCap: Human Performance Capture With Dynamic Neural Radiance Fields.

This paper addresses the challenge of human performance capture from sparse multi-view or monocular videos. Given a template mesh of the performer, previous methods capture the human motion by non-rigidly registering the template mesh to images with 2D silhouettes or dense photometric alignment. However, the detailed surface deformation cannot be recovered from the silhouettes, while the photometric alignment suffers from instability caused by appearance variation in the videos. To solve these problems, we propose NerfCap, a novel performance capture method based on the dynamic neural radiance field (NeRF) representation of the performer. Specifically, a canonical NeRF is initialized from the template geometry and registered to the video frames by optimizing the deformation field and the appearance model of the canonical NeRF. To capture both large body motion and detailed surface deformation, NerfCap combines linear blend skinning with embedded graph deformation. In contrast to the mesh-based methods that suffer from fixed topology and texture, NerfCap is able to flexibly capture complex geometry and appearance variation across the videos, and synthesize more photo-realistic images. In addition, NerfCap can be pre-trained end to end in a self-supervised manner by matching the synthesized videos with the input videos. Experimental results on various datasets show that NerfCap outperforms prior works in terms of both surface reconstruction accuracy and novel-view synthesis quality.