Distillation Scheme Research Articles

Semantic segmentation via fusing 2D image and 3D point cloud data with shared multi-layer perceptron

ABSTRACT Three-dimensional (3D) semantic segmentation based on point clouds has become a critical technology in the field of intelligent spatial perception and scene understanding. However, most existing methods cannot fully exploit the spatial feature of 3D point cloud data and the orderliness of two-dimensional (2D) image information, resulting in an inability to effectively improve segmentation accuracy. This study proposes a novel 2D image and 3D point cloud fusion method named shared multi-layer perceptron fusion semantic segmentation (SMFusionSeg) for semantic segmentation. The rich features of 2D image segmentation and spatial information of 3D point clouds are integrated by designing a fusion architecture that facilitates the transfer of 2D semantics to 3D semantics. Specifically, the fusion architecture concatenates the features from both 2D and 3D domains, which are then further integrated through a shared multi-layer perceptron (MLP) and optimized using an attention mechanism to enhance feature effectiveness and distinctiveness. Moreover, a knowledge distillation strategy is introduced to improve the learning capability and segmentation accuracy in parsing complex scenes. Experimental results on the SemanticKITTI dataset demonstrate that the proposed SMFusionSeg achieves a mean Intersection over Union ( mIoU ) of 65.4%, significantly outperforming traditional single-modal methods and many existing segmentation techniques, thereby effectively enhancing the precision and robustness of geographic feature recognition.

Non‐Autoregressive Translation Algorithm Based on LLM Knowledge Distillation in English Corpus

ABSTRACTAlthough significant advancements have been made in the quality of machine translation by large‐scale language models, their high computational costs and resource consumption have hindered their widespread adoption in practical applications. So this research introduces an English corpus‐based machine translation algorithm that leverages knowledge distillation from large language model, with the goal of enhancing translation quality and reducing the computational demands of the model. Initially, we conducted a thorough analysis of the English corpus to identify prevalent language patterns and structures. Following this, we developed a knowledge distillation approach that transfers the translation expertise of a large teacher model to a smaller student model, thereby achieving increased translation accuracy and efficiency. We designed a dynamic temperature hyperparameter distillation strategy that effectively enhances the precision of translations. In the experimental phase, we utilized several standard English corpora to train and assess our algorithm. The findings indicate that, compared to current machine translation systems, our method significantly reduces the need for computational resources while preserving translation quality.

Multimodal Cross-Lingual Summarization for Videos: A Revisit in Knowledge Distillation Induced Triple-Stage Training Method.

Multimodal summarization (MS) for videos aims to generate summaries from multi-source information (e.g., video and text transcript), showing promising progress recently. However, existing works are limited to monolingual scenarios, neglecting non-native viewers' needs to understand videos in other languages. It stimulates us to introduce multimodal cross-lingual summarization for videos (MCLS), which aims to generate cross-lingual summaries from multimodal input of videos. Considering the challenge of high annotation cost and resource constraints in MCLS, we propose a knowledge distillation (KD) induced triple-stage training method to assist MCLS by transferring knowledge from abundant monolingual MS data to those data with insufficient volumes. In the triple-stage training method, a video-guided dual fusion network (VDF) is designed as the backbone network to integrate multimodal and cross-lingual information through diverse fusion strategies in the encoder and decoder; What's more, we propose two cross-lingual knowledge distillation strategies: adaptive pooling distillation and language-adaptive warping distillation (LAWD), designed for encoder-level and vocab-level distillation objects to facilitate effective knowledge transfer across cross-lingual sequences of varying lengths between MS and MCLS models. Specifically, to tackle lingual sequences of varying lengths between MS and MCLS models. Specifically, to tackle the challenge of unequal length of parallel cross-language sequences in KD, LAWD can directly conduct cross-language distillation while keeping the language feature shape unchanged to reduce potential information loss. We meticulously annotated the How2-MCLS dataset based on the How2 dataset to simulate MCLS scenarios. Experimental results show that the proposed method achieves competitive performance compared to strong baselines, and can bring substantial performance improvements to MCLS models by transferring knowledge from the MS model.

Overcoming the challenges of multi-modal medical image sharing: A novel data distillation strategy via contrastive learning

Multi-modal medical images serve as a critical basis for disease diagnosis. However, sharing multi-modal medical data between hospitals is a significant challenge due to privacy protection concerns as well as the substantial costs associated with data transfer and storage. These hurdles are further compounded by two primary issues: (1) the complexity involved in sharing data features, which arises from the intricate process of multi-modal image registration; (2) the difficulty and high cost of acquiring accurate data labels. To address these issues, we propose a novel multi-modal medical dataset distillation method based on contrastive learning. We consider each modality as a unique approach to data augmentation for the corresponding imaging region, offering a unique perspective on the same anatomical region, which is then fused to collectively contribute to a comprehensive global representation of the medical image. Validation results on two datasets indicate our method is capable of reducing the original dataset to just 5% of its original size while retaining classification performance comparable to that of the full dataset, and notably, this efficiency is achieved using only 30% of the total labels, which significantly lowers the cost and effort associated with dataset labeling.

Self-distillation with model averaging

Knowledge distillation (KD) and model averaging (MA) are prominent techniques for enhancing the efficiency and effectiveness of deep neural networks (DNNs). MA produces an average model by averaging multiple checkpoints along the trajectory of the stochastic gradient descent (SGD), which tends to converge toward the flatter side of the local minimum (Fig. 2). However, MA operates offline and does not influence the specific local minimum to which the network converges. By contrast, KD transfers knowledge from a teacher model to a student model, guiding the student toward a better local minimum. Despite this, KD can still result in a poor convergence location within the minimum if the teacher over/under-regularizes the student model's behavior. By combining KD and MA, we aim to leverage KD for a favorable local minimum and MA for a robust convergence location within the local minimum. However, our results revealed that these two techniques were incompatible. This study empirically analyzed the causes of this incompatibility and proposed an annealing knowledge distillation (AKD) scheme to address it. Building on MA and AKD, we introduced a general self-distillation framework referred to as self-distillation with model averaging (SDMA). To the best of our knowledge, SDMA is the first general distillation framework that aims to achieve a favorable local minimum and a robust convergence location. Extensive experiments demonstrated that SDMA significantly improved the generalization performance of DNNs.

AdaptFL: Adaptive Federated Learning Framework for Heterogeneous Devices

With the development of the Internet of Things (IoT), Federated Learning (FL) is extensively employed in smart cities and industrial IoT, involving numerous heterogeneous devices with varying computational and storage capabilities. Traditional FL assumes that clients have enough resources to train a unified global model from the beginning to the end of training. However, it ignores the problem of uneven and real-time changes in client resources. Additionally, there are aggregation difficulties between heterogeneous client models and global model. To address these challenges, we propose an Adaptive Federated Learning Framework for Heterogeneous Devices (AdaptFL). In AdaptFL, we employ a resource-aware neural architecture search method, which searches for models based on each client’s resource conditions. It enables AdaptFL to automatically assign customized models tailored to each client’s specific resource conditions in the current round. Additionally, we employ a staged knowledge distillation strategy to facilitate efficient distribution and aggregation between the heterogeneous global model and the client models. Experimental results demonstrate that, compared to state-of-the-art model-level heterogeneous ablation methods, AdaptFL improves global test accuracy by 4% to 15% on the SVHN dataset and enhances accuracy by 5% to 14% in scenarios with heterogeneous data. Additionally, AdaptFL effectively reduces communication overhead by over 50% across all datasets. Furthermore, it offers a degree of resilience against model poisoning attacks.

Multi-scale contextual learning for medical image segmentation via dual distillation.

Recently, many studies have explored fusing features extracted from Convolutional neural networks (CNNs) and transformers to integrate multi-scale representations for better performance in medical image segmentation tasks. Although these hybrid models have achieved better results than previous CNN-based and transformer-based methods, they suffer from high computation and spacecomplexities. The purpose of this research is to address the prohibitive computation and space complexities of hybrid models, which limit their application in clinical practice where computational resources are usuallyconstrained. We propose a novel model equipped with a dual distillation scheme to sufficiently harness the complementary advantages of CNNs and transformers without compromising model efficiency. We further propose a multi-scale prior-knowledge distillation (MPD) module to effectively distill multi-scale knowledge from features extracted from transformers. In addition, to cooperate with the knowledge distillation scheme, we also propose an efficient and robust Selective Fusion module in the studentnetwork. We extensively evaluate the proposed model against fourteen different network frameworks on two representative datasets: SipakMed and ISIC 2017. In the SipakMed dataset, 3037 Pap smear images are used for training and 1012 for testing. In the ISIC 2017 dataset, 2000 dermoscopic images are used for training, 150 for validation, and 600 for testing. Experimental results demonstrate that our method not only surpasses existing methods by a considerable margin with respect to the evaluation metrics of mean Intersection over Union, mean Dice coefficient, mean average symmetric surface distance, but also requires fewer computational resources in terms of model parameters and floating-point operations persecond. Comprehensive comparisons in terms of segmentation accuracy and computational complexity unequivocally confirm that our method effectively and efficiently integrates the advantages of both CNNs and transformers, showing its suitability and significance for clinicalapplications.

SCARF: Scalable Continual Learning Framework for Memory‐efficient Multiple Neural Radiance Fields

AbstractThis paper introduces a novel continual learning framework for synthesising novel views of multiple scenes, learning multiple 3D scenes incrementally, and updating the network parameters only with the training data of the upcoming new scene. We build on Neural Radiance Fields (NeRF), which uses multi‐layer perceptron to model the density and radiance field of a scene as the implicit function. While NeRF and its extensions have shown a powerful capability of rendering photo‐realistic novel views in a single 3D scene, managing these growing 3D NeRF assets efficiently is a new scientific problem. Very few works focus on the efficient representation or continuous learning capability of multiple scenes, which is crucial for the practical applications of NeRF. To achieve these goals, our key idea is to represent multiple scenes as the linear combination of a cross‐scene weight matrix and a set of scene‐specific weight matrices generated from a global parameter generator. Furthermore, we propose an uncertain surface knowledge distillation strategy to transfer the radiance field knowledge of previous scenes to the new model. Representing multiple 3D scenes with such weight matrices significantly reduces memory requirements. At the same time, the uncertain surface distillation strategy greatly overcomes the catastrophic forgetting problem and maintains the photo‐realistic rendering quality of previous scenes. Experiments show that the proposed approach achieves state‐of‐the‐art rendering quality of continual learning NeRF on NeRF‐Synthetic, LLFF, and TanksAndTemples datasets while preserving extra low storage cost.

Multi-Teachers Distillation Strategy for Target-Oriented Collision-Free Grasping in Clutter

Multi-Teachers Distillation Strategy for Target-Oriented Collision-Free Grasping in Clutter

Personalized blood pressure estimation using multiview fusion information of wearable physiological signals and transfer learning

Continuous blood pressure (BP) monitoring is crucial for individual health management, yet the significant inter-individual variations among patients pose challenges to achieving precision medicine. In response to this issue, we propose a parallel cross-hybrid architecture that integrates a convolutional neural network backbone and a Mix-Transformer backbone. This model, grounded in multi-view physiological signals and personalized fine-tuning strategies, aims to estimate BP, facilitating the capture of physiological information across diverse receptive fields and enhancing network expressive capabilities. Our proposed architecture exhibits superior performance in estimating systolic blood pressure and diastolic blood pressure, with average absolute errors of 3.94 mmHg and 2.24 mmHg, respectively. These results surpass existing baseline models and align with the standards set by the British Hypertension Society, the Association for the Advancement of Medical Instrumentation, and the Institute of Electrical and Electronics Engineers for BP measurement. Additionally, this study explores a personalized model fine-tuning strategy by adjusting specific layers and incorporating individual information, presenting an optimal solution. The model's generalization ability is validated through transfer learning across databases (public and self-made). To enhance the proposed architecture's usability in wearable devices, this study employs a knowledge distillation strategy for model lightweighting, with preliminary application in our designed real-time BP estimation system. This study provides an efficient and accurate solution for personalized BP estimation, exhibiting broad potential applications.

Effective Bi-decoding networks for rail-surface defect detection by knowledge distillation

No-service rail-surface defect detection is a crucial method for assessing the quality of railroad tracks. However, the low-contrast and dark-tone characteristics of track-surface textures pose challenges to current defect-monitoring techniques. Real-time and on-site online inspections are important to ensure safe railway operation; however, most complex models for no-service inspections are difficult to deploy on mobile devices. To address these challenges and overcome the detection difficulties associated with complex scenes, we designed a knowledge distillation-based double decoding-layer refinement network (EBDNet-KD). The first decoding process is guided by a bimodal high-level semantic feature map obtained by extending the attention-based graph convolution to incrementally enhance the dual-stream features and obtain an image restoration prior. A divide-and-conquer decoder is then designed to distinguish features using different decoding layers. The prior is then used in the second decoding layer, which enables the bimodal features to interact fully and obtain the final prediction map. We introduce a knowledge distillation strategy that enables a lightweight, compact student network to learn a complex teacher network’s feature extraction process. This facilitates pixel-consistent learning of the knowledge within the bi-decoder layer, as well as bidirectional learning of the focused contextual response knowledge to optimize the model. The EBDNet-KD significantly reduces computational costs while guaranteeing performance with a parameter count of only 28 M. EBDNet-KD demonstrated superior performance over 15 state-of-the-art methods in experiments conducted on NEU RSDDS-AUG, an industrial RGB-depth dataset. We assessed the generalizability of EBDNet-KD by evaluating its performance on three additional public datasets, yielding competitive results. The source code and results can be found at https://github.com/Wuyue15/EBDNet.

A Self-Distilled Learning to Rank Model for Ad Hoc Retrieval

Learning to rank models are broadly applied in ad hoc retrieval for scoring and sorting documents based on their relevance to textual queries. The generalizability of the trained model in the learning to rank approach, however, can have an impact on the retrieval performance, particularly when data includes noise and outliers, or is incorrectly collected or measured. In this paper, we introduce a Self-Distilled Learning to Rank (SDLR) framework for ad hoc retrieval, and analyze its performance over a range of retrieval datasets and also in the presence of features’ noise. SDLR assigns a confidence weight to each training sample, aiming at reducing the impact of noisy and outlier data in the training process. The confidence weight is approximated based on the feature’s distributions derived from the values observed for the features of the documents labeled for a query in a listwise training sample. SDLR includes a distillation process that facilitates passing on the underlying patterns in assigning confidence weights from the teacher model to the student one. We empirically illustrate that SDLR outperforms state-of-the-art learning to rank models in ad hoc retrieval. We thoroughly investigate the SDLR performance in different settings including when no distillation strategy is applied; when different portion of data are used for training the teacher and the student models, and when both teacher and student models are trained over identical data. We show that SDLR is more effective when training data are split between a teacher and a student model. We also show that SDLR’s performance is robust when data features are noisy.

UCC: A unified cascade compression framework for vision transformer models

In recent years, Vision Transformer (ViT) and its variants have dominated many computer vision tasks. However, the high computational consumption and training data requirements of ViT make it challenging to be deployed directly on resource-constrained devices and environments. Model compression is an effective approach to accelerate deep learning networks, but existing methods for compressing ViT models are limited in their scopes and struggle to strike a balance between performance and computational cost. In this paper, we propose a novel Unified Cascaded Compression Framework (UCC) to compress ViT in a more precise and efficient manner. Specifically, we first analyze the frequency information within tokens and prune them based on a joint score of their both spatial and spectral characteristics. Subsequently, we propose a similarity-based token aggregation scheme that combines the abundant contextual information contained in all pruned tokens with the host tokens according to their weights. Additionally, we introduce a novel cumulative cascaded pruning strategy that performs bottom-up cascaded pruning of tokens based on cumulative scores, avoiding information loss caused by individual idiosyncrasies of blocks. Finally, we design a novel two-level distillation strategy, incorporating imitation and exploration, to ensure the diversity of knowledge and better performance recovery. Extensive experiments demonstrate that our unified cascaded compression framework outperforms most existing state-of-the-art approaches, compresses the floating-point operations of ViT-Base as well as DeiT-Base models by 22 % and 54.1 %, and improves the recognition accuracy of the models by 3.74 % and 1.89 %, respectively, significantly reducing model computational consumption while enhancing performance, which enables efficient end-to-end training of compact ViT models.

MBQuant: A novel multi-branch topology method for arbitrary bit-width network quantization

Arbitrary bit-width network quantization has received significant attention due to its high adaptability to various bit-width requirements during runtime. However, in this paper, we investigate existing methods and observe a significant accumulation of quantization errors caused by switching weight and activations bit-widths, leading to limited performance. To address this issue, we propose MBQuant, a novel method that utilizes a multi-branch topology for arbitrary bit-width quantization. MBQuant duplicates the network body into multiple independent branches, where the weights of each branch are quantized to a fixed 2-bit and the activations remain in the input bit-width. For completing the computation of a desired bit-width, MBQuant selects multiple branches, ensuring that the computational costs match those of the desired bit-width, to carry out forward propagation. By fixing the weight bit-width, MBQuant substantially reduces quantization errors caused by switching weight bit-widths. Additionally, we observe that the first branch suffers from quantization errors caused by all bit-widths, leading to performance degradation. Thus, we introduce an amortization branch selection strategy that amortizes the errors. Specifically, the first branch is selected only for certain bit-widths, rather than universally, thereby the errors are distributed among the branches more evenly. Finally, we adopt an in-place distillation strategy that uses the largest bit-width to guide the other bit-widths to further enhance MBQuant’s performance. Extensive experiments demonstrate that MBQuant achieves significant performance gains compared to existing arbitrary bit-width quantization methods. Code is made publicly available at https://github.com/zysxmu/MBQuant.

Taming Overconfident Prediction on Unlabeled Data From Hindsight.

Minimizing prediction uncertainty on unlabeled data is a key factor to achieve good performance in semi-supervised learning (SSL). The prediction uncertainty is typically expressed as the entropy computed by the transformed probabilities in output space. Most existing works distill low-entropy prediction by either accepting the determining class (with the largest probability) as the true label or suppressing subtle predictions (with the smaller probabilities). Unarguably, these distillation strategies are usually heuristic and less informative for model training. From this discernment, this article proposes a dual mechanism, named adaptive sharpening (ADS), which first applies a soft-threshold to adaptively mask out determinate and negligible predictions, and then seamlessly sharpens the informed predictions, distilling certain predictions with the informed ones only. More importantly, we theoretically analyze the traits of ADS by comparing it with various distillation strategies. Numerous experiments verify that ADS significantly improves state-of-the-art SSL methods by making it a plug-in. Our proposed ADS forges a cornerstone for future distillation-based SSL research.

Video-based AI module with raw-scale and ROI-scale information for thyroid nodule diagnosis

ObjectivesUltrasound examination is a primary method for detecting thyroid lesions in clinical practice. Incorrect ultrasound diagnosis may lead to delayed treatment or unnecessary biopsy punctures. Therefore, our objective is to propose an artificial intelligence model to increase the precision of thyroid ultrasound diagnosis and reduce puncture rates. MethodsWe consecutively collected ultrasound recordings from 672 patients with 845 nodules across two Chinese hospitals. This dataset was divided into training, validation, and internal test sets in a ratio of 7:1:2. We constructed and tested six different model variants based on different video feature distillation strategies and whether additional information from ROI (Region of Interest) scales was used. The models' performances were evaluated using the internal test set and an additional external test set containing 126 nodules from a third hospital. ResultsThe dual-stream model, which contains both raw-scale and ROI-scale streams with the time dimensional convolution layer, achieved the best performance on both internal and external test sets. On the internal test set, it achieved an AUROC (Area Under Receiver Operating Characteristic Curve) of 0.969 (95 % confidence interval, CI: 0.944–0.993) and an accuracy of 92.6 %, outperforming other variants (AUROC: 0.936–0.955, accuracy: 80.2%–88.3 %) and experienced radiologists (accuracy: 91.9 %). The AUROC of the best model in the external test was 0.931 (95 % CI: 0.890–0.972). ConclusionIntegrating a dual-stream model with additional ROI scale information and the time dimensional convolution layer can improve performance in diagnosing thyroid ultrasound videos.

A new lightweight framework based on knowledge distillation for reducing the complexity of multi-modal solar irradiance prediction model

The inherent uncertainty of solar energy brings great difficulties to the grid connection and short-term energy planning and dispatching. Deep learning method makes it possible to predict the short-term solar energy with its powerful learning ability, but its complex model structure and huge trainable parameters bring great difficulties to the practical deployment. Therefore, this paper proposes a lightweight framework based on knowledge distillation strategy, which greatly reduces the complexity of multi-modal solar irradiance prediction model meanwhile ensuring an acceptable accuracy, facilitating the practical deployment. Firstly, a teacher model with multi-modal structure and good accuracy is built based on ResNet18-Informer. Then, the lightweight model is obtained by the proposed lightweight framework depending on the knowledge of teacher model. The comparisons of various models and the optimal settings of knowledge distillation are analyzed. Results show that the lightweight model can reduce the trainable parameters, inference time, and GPU memory by 97.7%, 52.5%, and 36.3%, respectively. The normalized root mean square error is reduced by 24.87% compared with the same structure model but without knowledge distillation, verifying the superiority of the proposed framework. The soft loss using the light loss with the ratio of 0.3 can obtain the best training results for the lightweight model. The structure with 3 residual blocks and 3 LSTM layers is proved to be the best for the lightweight model in the solar irradiance prediction task.

Research on Lightweight Method of Insulator Target Detection Based on Improved SSD.

Aiming at the problems of a large volume, slow processing speed, and difficult deployment in the edge terminal, this paper proposes a lightweight insulator detection algorithm based on an improved SSD. Firstly, the original feature extraction network VGG-16 is replaced by a lightweight Ghost Module network to initially achieve the lightweight model. A Feature Pyramid structure and Feature Pyramid Network (FPN+PAN) are integrated into the Neck part and a Simplified Spatial Pyramid Pooling Fast (SimSPPF) module is introduced to realize the integration of local features and global features. Secondly, multiple Spatial and Channel Squeeze-and-Excitation (scSE) attention mechanisms are introduced in the Neck part to make the model pay more attention to the channels containing important feature information. The original six detection heads are reduced to four to improve the inference speed of the network. In order to improve the recognition performance of occluded and overlapping targets, DIoU-NMS was used to replace the original non-maximum suppression (NMS). Furthermore, the channel pruning strategy is used to reduce the unimportant weight matrix of the model, and the knowledge distillation strategy is used to fine-adjust the network model after pruning, so as to ensure the detection accuracy. The experimental results show that the parameter number of the proposed model is reduced from 26.15 M to 0.61 M, the computational load is reduced from 118.95 G to 1.49 G, and the mAP is increased from 96.8% to 98%. Compared with other models, the proposed model not only guarantees the detection accuracy of the algorithm, but also greatly reduces the model volume, which provides support for the realization of visible light insulator target detection based on edge intelligence.

A Semi-Supervised Transformer-Based Deep Learning Framework for Automated Tooth Segmentation and Identification on Panoramic Radiographs.

Automated tooth segmentation and identification on dental radiographs are crucial steps in establishing digital dental workflows. While deep learning networks have been developed for these tasks, their performance has been inferior in partially edentulous individuals. This study proposes a novel semi-supervised Transformer-based framework (SemiTNet), specifically designed to improve tooth segmentation and identification performance on panoramic radiographs, particularly in partially edentulous cases, and establish an open-source dataset to serve as a unified benchmark. A total of 16,317 panoramic radiographs (1589 labeled and 14,728 unlabeled images) were collected from various datasets to create a large-scale dataset (TSI15k). The labeled images were divided into training and test sets at a 7:1 ratio, while the unlabeled images were used for semi-supervised learning. The SemiTNet was developed using a semi-supervised learning method with a label-guided teacher-student knowledge distillation strategy, incorporating a Transformer-based architecture. The performance of SemiTNet was evaluated on the test set using the intersection over union (IoU), Dice coefficient, precision, recall, and F1 score, and compared with five state-of-the-art networks. Paired t-tests were performed to compare the evaluation metrics between SemiTNet and the other networks. SemiTNet outperformed other networks, achieving the highest accuracy for tooth segmentation and identification, while requiring minimal model size. SemiTNet's performance was near-perfect for fully dentate individuals (all metrics over 99.69%) and excellent for partially edentulous individuals (all metrics over 93%). In edentulous cases, SemiTNet obtained statistically significantly higher tooth identification performance than all other networks. The proposed SemiTNet outperformed previous high-complexity, state-of-the-art networks, particularly in partially edentulous cases. The established open-source TSI15k dataset could serve as a unified benchmark for future studies.

StaRS: Learning a Stable Representation Space for Continual Relation Classification.

Relation classification (RC) aims to detect the semantic relation between two annotated entities in a piece of sentence, serving as an essential task in automatic knowledge graph construction. Due to the emergence of new relations, there is a recent trend to train RC models in continual settings. To overcome the catastrophic forgetting problem in continual learning, existing research is devoted in a two-stage training paradigm, fast adaptation to novel relations, and memory replay for all historical relations. These memory-replay-based methods explore different techniques to mitigate the forgetting problem of continual RC (CRC) models during the memory replay stage. However, we find that the representation space undergoes distortion due to the incoming of fresh relations in the fast adaptation phase. To address this issue, we propose using a knowledge distillation strategy and designing a margin loss, aiming to maintain the stability of the RC model during adaptation to new relations. In addition, in the second stage, with a limited number of typical memory instances available, we introduce a self-contrastive learning objective to facilitate learning a balanced decision boundary for RC. Through training in two stages, our objective is to acquire a stable representation space to encode instances for CRC. We experimentally demonstrate the superiority of our model over competing methods in various settings, and the results suggest that our tailored designs can achieve better performance in CRC.