Thangka Image Research Articles

Thangka Image—Text Matching Based on Adaptive Pooling Layer and Improved Transformer

Image–text matching is a research hotspot in the multimodal task of integrating image and text processing. In order to solve the difficult problem of associating image and text data in the multimodal knowledge graph of Thangka, we propose an image and text matching method based on the Visual Semantic Embedding (VSE) model. The method introduces an adaptive pooling layer to improve the feature extraction capability of semantic associations between Thangka images and texts. We also improved the traditional Transformer architecture by combining bidirectional residual concatenation and mask attention mechanisms to improve the stability of the matching process and the ability to extract semantic information. In addition, we designed a multi-granularity tag alignment module that maps global and local features of images and text into a common coding space, leveraging inter- and intra-modal semantic associations to improve image and text accuracy. Comparative experiments on the Thangka dataset show that our method achieves significant improvements compared to the VSE baseline method. Specifically, our method improves the recall by 9.4% and 10.5% for image-matching text and text-matching images, respectively. Furthermore, without any large-scale corpus pre-training, our method outperforms all models without pre-training and outperforms two out of four pre-trained models on the Flickr30k public dataset. Also, the execution efficiency of our model is an order of magnitude higher than that of the pre-trained models, which highlights the superior performance and efficiency of our model in the image–text matching task.

Thangka Image Captioning Based on Semantic Concept Prompt and Multimodal Feature Optimization.

Thangka images exhibit a high level of diversity and richness, and the existing deep learning-based image captioning methods generate poor accuracy and richness of Chinese captions for Thangka images. To address this issue, this paper proposes a Semantic Concept Prompt and Multimodal Feature Optimization network (SCAMF-Net). The Semantic Concept Prompt (SCP) module is introduced in the text encoding stage to obtain more semantic information about the Thangka by introducing contextual prompts, thus enhancing the richness of the description content. The Multimodal Feature Optimization (MFO) module is proposed to optimize the correlation between Thangka images and text. This module enhances the correlation between the image features and text features of the Thangka through the Captioner and Filter to more accurately describe the visual concept features of the Thangka. The experimental results demonstrate that our proposed method outperforms baseline models on the Thangka dataset in terms of BLEU-4, METEOR, ROUGE, CIDEr, and SPICE by 8.7%, 7.9%, 8.2%, 76.6%, and 5.7%, respectively. Furthermore, this method also exhibits superior performance compared to the state-of-the-art methods on the public MSCOCO dataset.

Few Shot Object Detection for Tangka Seats Based on Deformable Convolution

According to deformable target sizes in the detection for Thangka Seats, an optimized few shot Thangka detection model is proposed. The deep residual network based on the deformable convolution is designed to extract the Thangka image features to decrease the missing detections rate. Experimental results show that the proposed method has better performance than the FSOD model on the Thangka dataset. The AP of 2-way 1-shot is 28.5 %, and the AP50 reaches 67.6 %, which increases by 4.7 % and 4.7 %, respectively.

Application of YOLOv5 Neural Network Based on Improved Attention Mechanism in Recognition of Thangka Image Defects

Application of YOLOv5 Neural Network Based on Improved Attention Mechanism in Recognition of Thangka Image Defects

Thangka Image Segmentation Method Based on Enhanced Receptive Field

The portrait thangka image is a kind of religious scroll painting that expresses figures’ identity and duties through portraits, sitting platforms, and backlighting. The segmentation of significant semantic objects in the image is one of the essential ways for scholars to study and understand the image’s content. To better understand this content, we elaborately collected a dataset of portrait-like thangkas, which consists of 4086 images covering four object categories. We provide rich annotation for this dataset. In addition, we propose an end-to-end deep learning method that effectively solves the problems of blurred target edges, segmentation errors, and missed segmentation in thangka image segmentation. First, regular convolution and atrous convolution of different sizes are concatenated after the high-level feature output. This method can effectively improve the receptive field of the model while obtaining more image feature information. Then, the attention module is introduced to fully utilize the spatial relationship between the image’s semantic content and enhance the discriminative ability of the feature representation on the thangka image. Finally, cross-layer feature fusion is added to reduce the loss of edge details and improve the accuracy of target edge segmentation. The results show that compared to the base model, the mPA and mIoU indices of the model proposed in this paper reach 90.75% and 85.66%, respectively, which effectively improved the accuracy of the Tangka image segmentation.

Application of YOLOv5 Based on Attention Mechanism and Receptive Field in Identifying Defects of Thangka Images

Aiming at the problems of target detection network in the defect detection field of thangka images with complex background colors, such as poor small target detection effect, insufficient feature information extraction, prone to error detection and leak detection, and low accuracy of defect detection, this paper proposed the YOLOv5 defect detection algorithm combining attention mechanism and receptive field. First of all, the Backbone network is used for feature extraction, integrating attention mechanism to represent different features, so that the network can fully extract the texture and semantic features of the defect area, and the extracted features are weighted and fused to reduce information loss. Secondly, a weighted fusion of features of different dimensions is transferred by the Neck network, and the combination of FPN and PAN is used to realize the fusion of semantic features and texture features of different layers and to locate the defect target more accurately. Finally, while replacing the GIoU loss function with CIoU, the receptive field is added to the network, so that the algorithm uses a four-channel detection mechanism to expand the detection range of receptive fields, and fuses semantic information between different network layers, so as to achieve fast location and more refined processing of small targets. The experimental results show that compared with the original YOLOv5 network, the detection accuracy of YOLOV5-scSE and YOLOV5-CA networks proposed in this paper has improved by 9.96 percentage points and 12.22 percentage points respectively, and the verification index has been significantly improved. It can quickly and more accurately identify and locate the location of the defect area and has a stronger ability to generalize the defect category, which greatly improves the accuracy of thangka image defect detection.

A Semantic Segmentation Model for Headdresses in Thangka Image Based on Line Drawing Augmentation and Spatial Prior Knowledge

Thangka is a religious scroll painting. As a window to Tibetan traditional culture and religious customs, the segmentation of important semantic objects in the images can help the public understand the image content and realize high-level semantic cognition of the visual content. This paper proposes a semantic segmentation network for the central figures’ headdresses in the portrait-type Thangka images based on the rich structure and standard composition of Thangka images. Meanwhile, it has also constructed the first pixel-level semantic annotation data set of the central figures’ headdresses in Thangka images. Firstly, an original Thangka image witnesses edge augmentation through the line drawing augmentation module. Then the Feature Extraction Network is used to extract the feature map. After that, it uses the Half-RPN network to get the region of interest (ROI). Finally, Mask R-CNN Head completes segmentation and class prediction. The experimental results show that the performance of the proposed model is 10% to 19% better than that of the state-of-the-art models such as Mask R-CNN and DeepLab V3.

Imbalanced Thangka Image Classification research Based on the ResNet Network

Aiming at the problem of performance degradation caused by ignoring the softmax score of the wrong class in the training process of the unbalanced data set of Thangka images and the problem of the loss of negative feature information in the propagation process of the ReLU activation function, a new loss calculation method is proposed. Firstly, the parameters of the pre-training model on the COCO data set are used as the initial parameters. Secondly, CE and CCE are used to calculate the loss during the calculation of loss in the back propagation. Finally, AReLU activation function is used and a weight assigned to CE and CCE is added as final loss to update the parameters. The experimental results show that this algorithm improves the convergence speed and accuracy of the model with respect to imbalanced data. Compared with other loss functions, ours method performance is state-of-the-art, such as complement cross entropy.

Few-shot Thangka image classification based on improved DenseNet

Thangka is an important part of Tibetan culture, and the classification of Thangka image is one of the basic works of Thangka research. DenseNet(Densely connected convolutional networks) has achieved a very good effect in the field of image classification. Considering that the DenseNet adopts ReLU function which loses the negative feature of the image in the feature propagation process, this paper proposed an improved DenseNet, called L-DenseNet that Leaky ReLU replaces ReLU function to increase the negative feature of propagation. In order to solve the problem of insufficient Thangka image sample, we adopted the method of based fine-tuned network. Experimental results show that L-DenseNet obtains an outstanding performance, which improves 1.1% performance compared with DenseNet. Compared with other CNNs, such as VGG16, ResNet50 and InceptionV3, L-DenseNet obtains state-of-the-art performance in the classification of Thangka image.

Bibliometric Analysis of the Research Status of Tangka Images at Home and Abroad

Carrying out research on Thangka images is of great significance for accelerating the construction of China’s cultural industry, especially the protection of Thangka’s cultural heritage. This paper will analyse the current research situation of Thangka images at home and abroad to provide reference for relevant personnel. This paper uses Chinese academic journals, doctoral and master’s degree thesis, international conferences, and English journals as information sources to perform quantitative analysis on the research results of the Thangka images published in them. The search process does not set a time limit, and uses CNKI and Excel analysis tools to relevant topics of the 55 Chinese documents and 21 foreign literatures are subject to annual trend analysis, document source analysis, author analysis, and distribution agency research. By this paper, the development trends, important information sources, important authors, and important research institutions of Thangka image research are obtained.

基于多特征的彩色唐卡修复图像无参考质量评价方法

基于多特征的彩色唐卡修复图像无参考质量评价方法

Research on a Thangka Image Classification Method Based on Support Vector Machine

As an art image, Thangka images have rich themes, various forms of expression, complex picture content and many layers of color representation. This paper mainly constructs a multi-core support vector machine (SVM) based on the information entropy feature-weighted radial basis kernel function. In this paper, the kernel function is optimized, and the feature reduction is performed by using the random forest feature selection algorithm with average accuracy degradation. Finally, the effective classification of the icon image and the mandala image in Thangka is realized. The research results provide support for the follow-up study of Thangka image annotation and retrieval.

A new method of Thangka image inpainting quality assessment

In order to solve the problem of Thangka image inpainting quality assessment (IIQA) and existing quality evaluation methods are not suitable for inpainting Thangka image, this paper proposes a new non-reference quality evaluation method which can effectively solve this problem. Firstly, due to lack of original Thangka image, the proposed method using symmetry of Thangka images to predicted an undamaged image. Secondly, we extract harries corner in Thangka inpainting images to show the structural feature of total graph. Thirdly, demonstrate subjective evaluation score of inpainting Thangka image by caparison difference of structural feature between inpainting and predicted original area. Finally, in order to compensate the lack of Thangka images in existing database, we add Generative Adversarial Nets (GANs) to generate large number of available image. Experiments show that our proposed method generates a state-of-the-art index for Thangka image inpainting quality which correlated with human vision.

A new quality assessment for Thangka image inpainting

SummaryThis paper presents an efficient metric for evaluation the effect of the inpainted Thangka images. In contrast to standard image quality metrics, the proposed one takes into account some constraints and characteristics related to the specific goals of inpainting techniques. The key is that we proposed a method to decompose the reference and distorted image and to compare the intensity of fuzzy edge for structure component and similarity for texture component. By comparative analyzing the experimental results with other metrics, the proposed image inpainting quality index delivers high consistency with subjective evaluation results, which will help the selecting algorithm in image restoration and could be also applied to most of inpainting image approaches.

Algorithm Optimization for the Edge Extraction of Thangka Images

A novel image edge detection algorithm developed to extract object region from a Thangka image by gradient magnitude and two-step gradient. The idea of our image edge detection algorithm is similar to the canny edge detection algorithm, but the implementation is rather different. By using this algorithm, we can realize the object region extraction by four steps. Firstly, the Gaussian algorithm is used to smooth the Thangka image. Then, the second order gradient amplitude and gradient angle for the Thangka image are calculated. Thirdly, non-maximum suppression is used to detect the edge of the Thangka image. At last, the double threshold is performed to achieve the outline of the Tang image extraction.

Damaged region filling by improved criminisi image inpainting algorithm for thangka

In order to solve the problems of the criminisi algorithm in inpainting thangka image, such as the mistake matching phenomenon, image structure information inconsistent and the inaccurate matching standards, the new image inpainting algorithm based on thangka image structure information is proposed in this paper. The following three steps are the keys of this method. (1) The correlation of repaired block and its neighborhood block is introduced and the priority of formula is improved; (2) The exemplar-based size selection is improved, the adaptive patch size is automatically adjusted according to the exemplar-based information changes; (3) In order to solve mistake matching problem, the structure information of thangka image and color of Euclidean distance are combined as the new matching criterion. The experimental results show that the mistake matching phenomenon by the proposed method for thangka image is significantly reduced, the structure of thangka image is more fluent and smooth than them in comparative literature.

Thangka Image Retrieval System Based on GLCM

Thangka Image Retrieval System is a challenging task in Thangka image classification. The Thangka images obtained from different ways are not all true Thangka images, which may lead to the misclassification of Thangka images. Therefore, a retrieval system is needed to distinguish true and false Thangka images. Thangka has rich texture characteristics, so this paper proposes a kind of feature extraction based on gray level co-occurrence matrix (GLCM) algorithm, to extract the statistical features of the matrix of thangka image retrieval, the distinction between has obtained the good effect.

Improved exemplar-based inpainting algorithm for broken Thangka images

The algorithm of exemplar-based image inpainting was introduced to the digital protection of Thangka image due to the advantage that can repair the broken structure and texture effectively at the same time.This algorithm is good at repairing the specific broken images of Thangka,but is not appropriate to other kinds of broken images.Hence,this paper proposed two aspects of improvement for the deficiencies of exemplar-based broken Thangka image inpainting:the improvement of computing method of confidence and the improvement of computing method of isophote intensity.The problem that optimum exemplar block is not exclusive has been solved.The experimental results show that the improved algorithm can not only get a satisfied inpainting result but also improve repairing efficiency.