ModelsKG:A Design and Research on Knowledge Graph of Multimodal Curriculum Based on PaddleOCR and DeepKE

Abstract

Multimodel deep learning system has attracted more and more attention. The traditional deep learning system mostly focuses on single modal processing and application. However, many applications require various modes to complete a certain task. For example, in the teaching scene, teaching materials are mostly displayed in text mode, video and PPT modes are also used to transfer the content of knowledge points. However, there is often a lack of connection between one mode and another mode, resulting in the fragmentation, complexity and redundancy of knowledge. Based on this consideration, the paper puts forward the design idea and frame of multimodal curriculum knowledge graph based on paddleOCR and DeepKE. Use DeepKE to extract the triple relationship between subject knowledge points and store it in the neo4j graph database, so as to build the knowledge graph of subject knowledge points, then use paddeOCR to identify the text content in the teaching video, generate the video frame description text, use NLP processing technologies such as text similarity to realize the understanding of video segments, and finally link the fine-grained video segments to the text knowledge graph, so as to build the multi-modal curriculum knowledge graph, In order to realize the purpose of intelligent search and intelligent construction of learning link.