The Influence Mechanism of Knowledge Network Allocation Mechanism on Knowledge Distillation of High-Tech Enterprises

Unsupervised semantic hashing has emerged as an indispensable technique for fast image search, which aims to convert images into binary hash codes without relying on labels. Recent advancements in the field demonstrate that employing large-scale backbones (e.g., ViT) in unsupervised semantic hashing models can yield substantial improvements. However, the inference delay has become increasingly difficult to overlook. Knowledge distillation provides a means for practical model compression to alleviate this delay. Nevertheless, the prevailing knowledge distillation approaches are not explicitly designed for semantic hashing. They ignore the unique search paradigm of semantic hashing, the inherent necessities of the distillation process, and the property of hash codes. In this paper, we propose an innovative Bit-mask Robust Contrastive knowledge Distillation (BRCD) method, specifically devised for the distillation of semantic hashing models. To ensure the effectiveness of two kinds of search paradigms in the context of semantic hashing, BRCD first aligns the semantic spaces between the teacher and student models through a contrastive knowledge distillation objective. Additionally, to eliminate noisy augmentations and ensure robust optimization, a cluster-based method within the knowledge distillation process is introduced. Furthermore, through a bit-level analysis, we uncover the presence of redundancy bits resulting from the bit independence property. To mitigate these effects, we introduce a bit mask mechanism in our knowledge distillation objective. Finally, extensive experiments not only showcase the noteworthy performance of our BRCD method in comparison to other knowledge distillation methods but also substantiate the generality of our methods across diverse semantic hashing models and backbones. The code for BRCD is available at https://github.com/hly1998/BRCD.

Efficient model deployment is a key focus in deep learning. This has led to the exploration of methods such as knowledge distillation and network pruning to compress models and increase their performance. In this study, we investigate the potential synergy between knowledge distillation and network pruning to achieve optimal model efficiency and improved generalization. We introduce an innovative framework for model compression that combines knowledge distillation, pruning, and fine-tuning to achieve enhanced compression while providing control over the degree of compactness. Our research is conducted on popular datasets, CIFAR-10 and CIFAR-100, employing diverse model architectures, including ResNet, DenseNet, and EfficientNet. We could calibrate the amount of compression achieved. This allows us to produce models with different degrees of compression while still being just as accurate, or even better. Notably, we demonstrate its efficacy by producing two compressed variants of ResNet 101: ResNet 50 and ResNet 18. Our results reveal intriguing findings. In most cases, the pruned and distilled student models exhibit comparable or superior accuracy to the distilled student models while utilizing significantly fewer parameters.

A popular method for shrinking over-parameterized networks nowadays is pruning, which can efficiently reduce the number of computational parameters and computational cost of the network and has almost the same high accuracy as the original network. The general weighted pruning algorithm can only reduce the number of parameters based on the original network structure, but cannot reduce the width and depth of the pruned network. While the knowledge distillation algorithm can solve the problem by compressing the network structure, it cannot make further modifications on the processed network. To further reduce the network structure, we propose a model compression algorithm, HKDP, a hybrid method combining knowledge distillation and network pruning that can significantly reduce the overall size of the network and maintain substantial accuracy. This approach obtains the advantages of knowledge distillation and pruning, which achieves 10 times higher compression rate and 2 percent higher accuracy than using either algorithm alone. Concretely, we apply a stage-wise knowledge distillation algorithm in the front that can quickly and efficiently reduce the original model structure; we also apply a Stochastic Gradient Descent (SGD) based pruning method and introduce the concept of global sparsity, which allows us to customize the compression rate of the model. Our experiments on CIFAR-10 and MNIST show that our hybrid optimization algorithm has higher model accuracy and model compression ratio compared to other competitors' network compression algorithms.

Knowledge innovation ability is the source of value realization of high-tech enterprises, and the acquisition of high-value knowledge is important. Taking knowledge as the intermediary variable, knowledge field activity and knowledge fermentation as mediating variables, and knowledge mobilization and knowledge network position transition as moderating variables, the conceptual model and theoretical analysis framework of the impact mechanism of knowledge innovation network fragmentation fault on technology practices is constructed and the moderated mediating effect model is derived. Taking high-tech enterprises as empirical samples, 538 valid questionnaires were obtained online and offline and the nonpercentile bootstrap method based on deviation correction was used to empirically investigate the influence mechanism and transmission path of knowledge innovation network fragmentation fault on high-tech enterprises' technological practices. The empirical results show that the main effect of knowledge innovation network fragmentation fault on high-tech enterprise technology practices is significant. Knowledge field activity and knowledge fermentation play a differential mediating role in knowledge innovation network split fault and Technology Convention. Knowledge field activity and knowledge fermentation play a partial mediating role in knowledge innovation network split fault and Technology Convention. Knowledge mobilization partially moderates the split fault of knowledge innovation network and technological practices. Knowledge mobilization positively moderates the positive effect of split fault of knowledge innovation network on technological practices and significantly positively moderates the mediating effect of knowledge field activity and knowledge fermentation, resulting in the moderated mediating effect. Knowledge network location transition plays a part of moderating role in knowledge innovation network split fault and Technology Convention. Knowledge network location transition positively moderates the positive impact of knowledge innovation network split fault on Technology Convention and significantly positively moderates the mediating role of knowledge field activity and knowledge fermentation, resulting in a moderated mediating effect. Knowledge innovation network split fault, knowledge field activity and knowledge fermentation, knowledge mobilization and knowledge network location transition, and the combination of technological practices can be the antecedents of promoting technological practices in high-tech enterprises. Through the research on the mechanism of knowledge innovation network split fault in the technological practices of high-tech enterprises, the connotation of knowledge innovation network split fault is enriched, the influencing factors of technological practices are clarified, and the value-added knowledge is promoted and has guiding and reference significance for the innovation knowledge acquisition and competitiveness improvement of high-tech enterprises.

At this stage, the popular deep neural network models often encounter problems of high latency, difficult deployment and high hardware requirements in practical applications. Knowledge distillation is a good approach to solve these problems. We adopted an innovative knowledge distillation approach and formulated data augmentation strategies for the tasks, and obtained a lightweight model with 6. 7x acceleration ratio and 13. 6x compression ratio compared to the baseline model BERT-base, and the average performance of the lightweight model reached 95% of BERT-base for each task. We continue to conduct in-depth research to investigate some of the issues that remain in the knowledge distillation phase. To address the problems in distillation model selection and model fine-tuning, we propose a teacher model and student model selection strategy and a two-stage model fine-tuning strategy before and after the knowledge distillation stage. These two strategies further improve the average performance of the models to 98% of BERT-base. Finally, we developed a lightweight model evaluation scheme based on different types of downstream tasks, which provides a reference for subsequent practical applications when encountering similar tasks.

Neural network compression techniques, such as knowledge distillation (KD) and network pruning, have received increasing attention. Recent work `Prune, then Distill' reveals that a pruned student-friendly teacher network can benefit the performance of KD. However, the conventional teacher-student pipeline, which entails cumbersome pre-training of the teacher and complicated compression steps, makes pruning with KD less efficient. In addition to compressing models, recent compression techniques also emphasize the aspect of efficiency. Early pruning demands significantly less computational cost in comparison to the conventional pruning methods as it does not require a large pre-trained model. Likewise, a special case of KD, known as self-distillation (SD), is more efficient since it requires no pre-training or student-teacher pair selection. This inspires us to collaborate early pruning with SD for efficient model compression. In this work, we propose the framework named Early Pruning with Self-Distillation (EPSD), which identifies and preserves distillable weights in early pruning for a given SD task. EPSD efficiently combines early pruning and self-distillation in a two-step process, maintaining the pruned network's trainability for compression. Instead of a simple combination of pruning and SD, EPSD enables the pruned network to favor SD by keeping more distillable weights before training to ensure better distillation of the pruned network. We demonstrated that EPSD improves the training of pruned networks, supported by visual and quantitative analyses. Our evaluation covered diverse benchmarks (CIFAR-10/100, Tiny-ImageNet, full ImageNet, CUB-200-2011, and Pascal VOC), with EPSD outperforming advanced pruning and SD techniques.

The economic stability of many countries is closely tied to agriculture, where crop quality and yield are heavily affected by weather conditions and disease control. Unpredictable climate patterns and plant diseases remain major challenges, emphasizing the need for more reliable disease detection methods. Traditionally, plant disease identification has relied on visual examination, a method that is often inaccurate. To address this, our study proposes a two-stage model for improved disease detection in mango leaves. In the first stage, we implement an innovative, block-based feature extraction technique using Local Directional Patterns (LDP) and Local Directional Pattern variance (LDPv) on a comprehensive dataset, MangoLeafBD, consisting of 4000 images, achieving satisfactory results in terms of detection accuracy, sensitivity, specificity, and false negative rate. In the second stage, we introduce a Knowledge Distillation (KD) process to further enhance model performance by transferring knowledge from a larger teacher model to a smaller student model. Our results demonstrate significant advancement, with the KD-enhanced model achieving an improvement in detection accuracy from 89.2% to 95.6%, sensitivity from 7.8% to 4.1%, and specificity from 71.2% to 88.9% for Anthracnose disease. Similar improvements were observed in detecting other diseases, making our approach a more robust and efficient solution for mango plant disease detection. • Novel two-stage model detects mango leaf disease with high accuracy and low cost. • Uses LDP and LDPv for fast, robust feature extraction with minimal resources. • Knowledge distillation boosts accuracy by training a compact student model. • Designed for resource-limited devices, avoiding heavy deep learning models. • Outperforms ML and DL baselines on MangoLeafBD across all performance metrics.

Research on semantic segmentation for remote sensing road scenes advanced significantly, driven by autonomous driving technology. However, motion blur from camera or subject movements hampers segmentation performance. To address this issue, we propose a knowledge distillation-based semantic segmentation network (KDS-Net) that is robust to motion blur, eliminating the need for image restoration networks. KDS-Net leverages innovative knowledge distillation techniques and edge-enhanced segmentation loss to refine edge regions and improve segmentation precision across various receptive fields. To enhance the interpretability of segmentation quality under motion blur, we incorporate fractal dimension estimation to quantify the geometric complexity of class-specific regions, allowing for a structural assessment of predictions generated by the proposed knowledge distillation framework for autonomous driving. Experiments on well-known motion-blurred remote sensing road scene datasets (CamVid and KITTI) demonstrate mean IoU scores of 72.42% and 59.29%, respectively, surpassing state-of-the-art methods. Additionally, the lightweight KDS-Net (21.44 M parameters) enables real-time edge computing, mitigating data privacy concerns and communication overheads in internet of vehicles scenarios.

Knowledge distillation in transformers with tripartite attention: Multiclass brain tumor detection in highly augmented MRIs

Distillation embedded absorbable pruning for fast object re-identification

Knowledge distillation which learns a lightweight student model by distilling knowledge from a cumbersome teacher model is an attractive approach for learning compact deep neural networks (DNNs). Recent works further improve student network performance by leveraging multiple teacher networks. However, most of the existing knowledge distillation-based multi-teacher methods use separately pretrained teachers. This limits the collaborative learning between teachers and the mutual learning between teachers and student. Network quantization is another attractive approach for learning compact DNNs. However, most existing network quantization methods are developed and evaluated without considering multi-teacher support to enhance the performance of quantized student model. In this paper, we propose a novel framework that leverages both multi-teacher knowledge distillation and network quantization for learning low bit-width DNNs. The proposed method encourages both collaborative learning between quantized teachers and mutual learning between quantized teachers and quantized student. During learning process, at corresponding layers, knowledge from teachers will form an importance-aware shared knowledge which will be used as input for teachers at subsequent layers and also be used to guide student. Our experimental results on CIFAR-100 and ImageNet datasets show that the compact quantized student models trained with our method achieve competitive results compared to other state-of-the-art methods, and in some cases, surpass the full precision models.

Lightweight service identification models are very important for resource-constrained distribution grid systems. To address the increasingly larger deep learning models, we provide a method for the lightweight identification of complex power services based on knowledge distillation and network pruning. Specifically, a pruning method based on Taylor expansion is first used to rank the importance of the parameters of the small-scale network and delete some of the parameters, compressing the model parameters and reducing the amount of operation and complexity. Then, knowledge distillation is used to migrate the knowledge from the large-scale network ResNet50 to the small-scale network so that the small-scale network can fit the soft-label information output from the large-scale neural network through the loss function to complete the knowledge migration of the large-scale neural network. Experimental results show that this method can compress the model size of the small network and improve the recognition accuracy. Compared with the original small network, the model accuracy is improved by 2.24 percentage points to 97.24%. The number of model parameters is compressed by 81.9% and the number of floating-point operations is compressed by 92.1%, making it more suitable for deployment in resource-constrained devices.

Aiming at the problem that the training effectiveness of deep learning models is closely related to labeled training data, a federated learning framework, FedPrune, is proposed with high training efficiency and accuracy; a knowledge distillation method is adopted and furthermore a network pruning scheme is used to generate a suitable local student model from a global teacher model, which realizes the lightweight intelligent algorithms to be executed efficiently in the resource-constrained devices; comparison experiments are conducted with the 2 local solutions and 2 global solutions on CIFAR10, CIFAR100 datasets for comparison experiments, FedPrune has higher accuracy and shorter training time.

Despite the remarkable performance on various Natural Language Processing (NLP) tasks, the parametric complexity of pretrained language models has remained a major obstacle due to limited computational resources in many practical applications. Techniques such as knowledge distillation, network pruning, and quantization have been developed for language model compression. However, it has remained challenging to achieve an optimal tradeoff between model size and inference accuracy. To address this issue, we propose a novel and efficient uncertainty-driven knowledge distillation compression method for transformer-based pretrained language models. Specifically, we design a method of parameter retention and feedforward network parameter distillation to compress N-stacked transformer modules into one module in the fine-tuning stage. A key innovation of our approach is to add the uncertainty estimation module (UEM) into the student network such that it can guide the student network's feature reconstruction in the latent space (similar to the teacher's). Across multiple datasets in the natural language inference tasks of GLUE, we have achieved more than 95% accuracy of the original BERT, while only using about 50% of the parameters.

Dual network embedding is an emerging way for small and medium-sized enterprises to break the resource dilemma. It is of great significance for small and medium-sized enterprises to explore the innovation capability improvement based on dual network embedding. In addition, knowledge inertia and dual innovation based on dual theory have become important research objects in the academic circle to improve innovation ability. This article adopts the method of literature research, the first in this paper, the dual network embedded characteristics of small and medium-sized enterprises, on this basis, further analysis of cooperation network and knowledge of inertia, the relationship between knowledge network and dual innovation, based on the dual network of complementary and synergies attempt to propose a can reflect the cooperation network, knowledge network, knowledge of inertia, dual innovation and the relationship between the small and medium-sized enterprise innovation ability promote model. In this paper reveals the dual embedded network for small and medium-sized enterprises improve innovation ability, deepen the knowledge of inertia and dual innovation of small and medium-sized enterprises behavior in the dual influences of network structure, based on dual network is discussed under the complementary synergy effects on innovation ability to ascend, and in the future related research is discussed.