Proposed Method Research Articles

Adaptive Constraint Penalty-Based Multiobjective Operation Optimization of an Industrial Dynamic System With Complex Multiconstraint.

Aiming at the operation optimization of the wastewater treatment process (WWTP) with nonstationary time-varying dynamics and complex multiconstraint, this article proposes a novel adaptive constraint penalty decomposed multiobjective evolutionary algorithm with synthetical distance (SD)-based cross-generation crossover. First, the concept of spatial SD is presented to comprehensively evaluate the similarity of individual solutions from two aspects of distance and angle, and the individual information between two adjacent generations is used to enhance the diversity of individuals and accelerate the convergence of the algorithm. Second, aiming at the complex multiconstraint during the operation optimization of WWTP, an adaptive penalty algorithm is further adopted to punish the individual solutions that violate the constraints, so as to improve the handling efficiency and success rate of constraints. Furthermore, in view of the time-varying dynamics of actual WWTP, a recursive bilinear subspace identification method based on sliding window is adopted to establish the optimization models as well as the constraint models with self-learning parameter, which provides accurate model guarantee for high-performance multiobjective operation optimization. Finally, the effectiveness, superiority, and practicability of the proposed method are verified through test function experiments as well as operation optimization control experiments of WWTP.

Sliding-Mode Control for Perturbed MIMO Systems With Time-Synchronized Convergence.

This article introduces a novel approach called terminal sliding-mode control for achieving time-synchronized convergence in multi-input-multi-output (MIMO) systems under disturbances. To enhance controller design, the systems are categorized into two groups: 1) input-dimension-dominant and 2) state-dimension-dominant, based on signal dimensions and their potential for achieving thorough time-synchronized convergence. We explore sufficient Lyapunov conditions using terminal sliding-mode designs and develop adaptive controllers for the input-dimension-dominant case. To handle perturbations, we design a multivariable disturbance observer with a super-twisting structure, which is integrated into the controller. By utilizing the sliding-mode technique and the disturbance observer, the proposed controller ensures simultaneous convergence of all output dimensions. In the state-dimension-dominant case, where a full-rank system matrix is absent, only specific output elements converge to equilibrium simultaneously. We conduct comparative simulations on a practical system to highlight the effectiveness of our proposed method for the input-dimension-dominant case. Statistical results reveal the benefits of shorter output trajectories and reduced energy consumption. For the state-dimension-dominant case, we present numerical examples to validate the semi-time-synchronized property.

Cu-Co Bimetallic Catalyst-based Electrochemical Sensing Platform for Determination of Bisoprolol in Clinical Samples.

Bisoprolol (BIS) is a selective beta-blocker. It has been successfully used to treat hypertension and angina pectoris. An overdose of BIS can lead to serious complications. An overdose is a medical emergency that requires immediate medical attention to overcome the adverse effects of the overdose. Hence, sensitive, reliable, and cost-effective methods are required for the determination of BIS. In this work, a new electrochemical sensing platform based on a bimetallic catalyst was developed for the determination of BIS. The Cu-Co nanocatalyst was easily synthesized by galvanic displacement onto a carbon paste electrode (CPE). Then, field emission scanning electron microscopy (FESEM), energy dispersive spectroscopy (EDS), and cyclic voltammetry (CV) were utilized for the characterization of the Cu-Co catalyst. The galvanic displacement of Cu metal significantly affected the electro-catalytic behavior of the Cu-Co catalyst and the Cu-Co/CPE electrode displayed a very sensitive and accurate response towards BIS. Under optimized conditions, the response was linear in the 3 to 120 μM concentration range, sensitivity of 631.1 μA mM-1 and a detection limit of as low as 0.4 μM using cyclic voltammetry. The simple proposed method was also successfully employed in the analysis of BIS in biological and pharmaceutical samples. The advantages of Cu-Co/CPE are its fast and simple manufacturing and the possibility of a repeated surface regeneration of the sensing platform, as well as its application for the detection of BIS in tablets and biological samples, making Cu-Co significant promise for use in clinical diagnostics. Besides, the synthesized catalysts showed excellent reusability and stability. The presence of Cu metal due to galvanic displacement increased the sensitivity. These findings suggest that the new nanocatalyst has potential applications in sensors and electronics.

Task-Related Saliency for Few-Shot Image Classification.

A weakness of the existing metric-based few-shot classification method is that task-unrelated objects or backgrounds may mislead the model since the small number of samples in the support set is insufficient to reveal the task-related targets. An essential cue of human wisdom in the few-shot classification task is that they can recognize the task-related targets by a glimpse of support images without being distracted by task-unrelated things. Thus, we propose to explicitly learn task-related saliency features and make use of them in the metric-based few-shot learning schema. We divide the tackling of the task into three phases, namely, the modeling, the analyzing, and the matching. In the modeling phase, we introduce a saliency sensitive module (SSM), which is an inexact supervision task jointly trained with a standard multiclass classification task. SSM not only enhances the fine-grained representation of feature embedding but also can locate the task-related saliency features. Meanwhile, we propose a self-training-based task-related saliency network (TRSN) which is a lightweight network to distill task-related salience produced by SSM. In the analyzing phase, we freeze TRSN and use it to handle novel tasks. TRSN extracts task-relevant features while suppressing the disturbing task-unrelated features. We, therefore, can discriminate samples accurately in the matching phase by strengthening the task-related features. We conduct extensive experiments on five-way 1-shot and 5-shot settings to evaluate the proposed method. Results show that our method achieves a consistent performance gain on benchmarks and achieves the state-of-the-art.

Ensemble Clustering via Co-Association Matrix Self-Enhancement.

Ensemble clustering integrates a set of base clustering results to generate a stronger one. Existing methods usually rely on a co-association (CA) matrix that measures how many times two samples are grouped into the same cluster according to the base clusterings to achieve ensemble clustering. However, when the constructed CA matrix is of low quality, the performance will degrade. In this article, we propose a simple, yet effective CA matrix self-enhancement framework that can improve the CA matrix to achieve better clustering performance. Specifically, we first extract the high-confidence (HC) information from the base clusterings to form a sparse HC matrix. By propagating the highly reliable information of the HC matrix to the CA matrix and complementing the HC matrix according to the CA matrix simultaneously, the proposed method generates an enhanced CA matrix for better clustering. Technically, the proposed model is formulated as a symmetric constrained convex optimization problem, which is efficiently solved by an alternating iterative algorithm with convergence and global optimum theoretically guaranteed. Extensive experimental comparisons with 12 state-of-the-art methods on ten benchmark datasets substantiate the effectiveness, flexibility, and efficiency of the proposed model in ensemble clustering. The codes and datasets can be downloaded at https://github.com/Siritao/EC-CMS.

HPLC method for the determination of thymoquinone in growth cell medium.

Preclinical drug testing requires in vitro and in vivo assessments that are vital for studying drug pharmacokinetics and toxicity. Distinct factors that play an important role in drug screening, such as hydrophobicity, solubility of the substance and serum protein binding can be challenging by inducing result inconsistencies. Hence, establishing accurate methods to quantify drug concentrations in cell cultures becomes pivotal for reliable and reproducible results important for in vivo dosing predictions. This research focuses on developing an optimized analytical approach via high-pressure liquid chromatography (HPLC) to determine thymoquinone (TQ) levels in monolayer cell cultures. The method's validation adheres to the International Council for Harmonisation (ICH) guideline M10, ensuring its acceptance and applicability. Using an HPLC system with a Diode Array Detector (DAD), the study fine-tuned various parameters to achieve an efficient separation of TQ. Validation covered specificity, sensitivity, matrix effects, linearity, precision, and accuracy, alongside assessing TQ stability in RPMI-1640 medium. The HPLC method exhibited remarkable TQ specificity, free from interfering peaks at the analyte retention. Sensitivity analysis at the lower limit of quantification (LLOQ) revealed 5.68% %CV and 98.37% % mean accuracy. Matrix effect evaluation showcased accuracy within 85-115%. Linearity spanned in the concentration range of 2-10 μM with a correlation coefficient (r2) of 0.9993. Precision and accuracy were aligned with acceptance criteria. The proposed method was found to be greener in terms of usage of persistent, bioaccumulative, and toxic chemicals and solvents, corrosive samples, and waste production. The developed HPLC-DAD method emerges as specific, accurate, sensitive, and reliable for TQ determination in cell cultures. It ensures robust TQ quantification, enhancing precise in vitro assessments and dependable dosing predictions for in vivo studies. Further research is advocated to investigate TQ's stability across diverse environmental conditions.

Learning-Assisted Fast Determination of Regularization Parameter in Constrained Image Reconstruction.

To leverage machine learning (ML) for fast selection of optimal regularization parameter in constrained image reconstruction. Constrained image reconstruction is often formulated as a regularization problem and selecting a good regularization parameter value is an essential step. We solved this problem using an ML-based approach by leveraging the finding that for a specific constrained reconstruction problem defined for a fixed class of image functions, the optimal regularization parameter value is weakly subject-dependent and the dependence can be captured using few experimental data. The proposed method has four key steps: a) solution of a given constrained reconstruction problem for a few (say, 3) pre-selected regularization parameter values, b) extraction of multiple approximated quality metrics from the initial reconstructions, c) predicting the true quality metrics values from the approximated values using pre-trained neural networks, and d) determination of the optimal regularization parameter by fusing the predicted quality metrics. The effectiveness of the proposed method was demonstrated in two constrained reconstruction problems. Compared with L-curve-based method, the proposed method determined the regularization parameters much faster and produced substantially improved reconstructions. Our method also outperformed state-of-the-art learning-based methods when trained with limited experimental data. This paper demonstrates the feasibility and improved reconstruction quality by using machine learning to determine the regularization parameter in constrained reconstruction. The proposed method substantially reduces the computational burden of the traditional methods (e.g., L-curve) or relaxes the requirement of large training data by modern learning-based methods, thus enhancing the practical utility of constrained reconstruction.

Segmentation Method of Concrete Small Cracks Based on UAV Images

Introduction:: Cracks are one of the major problems in modern concrete buildings, especially in locations that are difficult to map manually, such as bridges and high-rise buildings. Accurate analysis of unmanned aerial vehicle (UAV) images has become the key to determining whether a building needs maintenance. Methods:: Traditional image processing methods are easily interfered by high-frequency background. Neural network methods need fine datasets, which increase labor costs. Therefore, this paper proposes a segmentation algorithm based on UNet3+ network. After obtaining the UAV image, the rough location of the crack can be obtained by only rough labeling. And then, the sample balance can be carried out by clipping the target area. The UNet3+ network is used to train the processed datasets and extract the region of interest to ignore the non-target texture. Finally, the region of interest is further segmented by color clustering and edge detection methods. Results:: The proposed method can detect the cracks accurately. In all test images, the relative errors are less than 13%. Especially in test images whose width is less than 0.2mm, the maximum absolute error is only 0.0237mm, which is completely acceptable in actual production. The proposed method has higher practicability in the detection of concrete crack images taken by UAV. The results show that the proposed method outperforms the cutting-edge method published in the journal "Sensor", when the background is complex. Conclusion:: The proposed method can segment and detect cracks effectively, which can remove the high-frequency interference region from the images.

Robust Commonsense Reasoning Against Noisy Labels Using Adaptive Correction.

Commonsense reasoning based on knowledge graphs (KGs) is a challenging task that requires predicting complex questions over the described textual contexts and relevant knowledge about the world. However, current methods typically assume clean training scenarios with accurately labeled samples, which are often unrealistic. The training set can include mislabeled samples, and the robustness to label noises is essential for commonsense reasoning methods to be practical, but this problem remains largely unexplored. This work focuses on commonsense reasoning with mislabeled training samples and makes several technical contributions: 1) we first construct diverse augmentations from knowledge and model, and offer a simple yet effective multiple-choice alignment method to divide the training samples into clean, semi-clean, and unclean parts; 2) we design adaptive label correction methods for the semi-clean and unclean samples to exploit the supervised potential of noisy information; and 3) finally, we extensively test these methods on noisy versions of commonsense reasoning benchmarks (CommonsenseQA and OpenbookQA). Experimental results show that the proposed method can significantly enhance robustness and improve overall performance. Furthermore, the proposed method is generally applicable to multiple existing commonsense reasoning frameworks to boost their robustness. The code is available at https://github.com/xdxuyang/CR_Noisy_Labels.

Fast Filter Pruning via Coarse-to-Fine Neural Architecture Search and Contrastive Knowledge Transfer.

Filter pruning is the most representative technique for lightweighting convolutional neural networks (CNNs). In general, filter pruning consists of the pruning and fine-tuning phases, and both still require a considerable computational cost. So, to increase the usability of CNNs, filter pruning itself needs to be lightweighted. For this purpose, we propose a coarse-to-fine neural architecture search (NAS) algorithm and a fine-tuning structure based on contrastive knowledge transfer (CKT). First, candidates of subnetworks are coarsely searched by a filter importance scoring (FIS) technique, and then the best subnetwork is obtained by a fine search based on NAS-based pruning. The proposed pruning algorithm does not require a supernet and adopts a computationally efficient search process, so it can create a pruned network with higher performance at a lower cost than the existing NAS-based search algorithms. Next, a memory bank is configured to store the information of interim subnetworks, i.e., by-products of the above-mentioned subnetwork search phase. Finally, the fine-tuning phase delivers the information of the memory bank through a CKT algorithm. Thanks to the proposed fine-tuning algorithm, the pruned network accomplishes high performance and fast convergence speed because it can take clear guidance from the memory bank. Experiments on various datasets and models prove that the proposed method has a significant speed efficiency with reasonable performance leakage over the state-of-the-art (SOTA) models. For example, the proposed method pruned the ResNet-50 trained on Imagenet-2012 up to 40.01% with no accuracy loss. Also, since the computational cost amounts to only 210 GPU hours, the proposed method is computationally more efficient than SOTA techniques. The source code is publicly available at https://github.com/sseung0703/FFP.

Coupled Multimodal Emotional Feature Analysis Based on Broad-Deep Fusion Networks in Human-Robot Interaction.

A coupled multimodal emotional feature analysis (CMEFA) method based on broad-deep fusion networks, which divide multimodal emotion recognition into two layers, is proposed. First, facial emotional features and gesture emotional features are extracted using the broad and deep learning fusion network (BDFN). Considering that the bi-modal emotion is not completely independent of each other, canonical correlation analysis (CCA) is used to analyze and extract the correlation between the emotion features, and a coupling network is established for emotion recognition of the extracted bi-modal features. Both simulation and application experiments are completed. According to the simulation experiments completed on the bimodal face and body gesture database (FABO), the recognition rate of the proposed method has increased by 1.15% compared to that of the support vector machine recursive feature elimination (SVMRFE) (without considering the unbalanced contribution of features). Moreover, by using the proposed method, the multimodal recognition rate is 21.22%, 2.65%, 1.61%, 1.54%, and 0.20% higher than those of the fuzzy deep neural network with sparse autoencoder (FDNNSA), ResNet-101 + GFK, C3D + MCB + DBN, the hierarchical classification fusion strategy (HCFS), and cross-channel convolutional neural network (CCCNN), respectively. In addition, preliminary application experiments are carried out on our developed emotional social robot system, where emotional robot recognizes the emotions of eight volunteers based on their facial expressions and body gestures.

Spectral Super-Resolution via Model-Guided Cross-Fusion Network.

Spectral super-resolution, which reconstructs a hyperspectral image (HSI) from a single red-green-blue (RGB) image, has acquired more and more attention. Recently, convolution neural networks (CNNs) have achieved promising performance. However, they often fail to simultaneously exploit the imaging model of the spectral super-resolution and complex spatial and spectral characteristics of the HSI. To tackle the above problems, we build a novel cross fusion (CF)-based model-guided network (called SSRNet) for spectral super-resolution. In specific, based on the imaging model, we unfold the spectral super-resolution into the HSI prior learning (HPL) module and imaging model guiding (IMG) module. Instead of just modeling one kind of image prior, the HPL module is composed of two subnetworks with different structures, which can effectively learn the complex spatial and spectral priors of the HSI, respectively. Furthermore, a CF strategy is used to establish the connection between the two subnetworks, which further improves the learning performance of the CNN. The IMG module results in solving a strong convex optimization problem, which adaptively optimizes and merges the two features learned by the HPL module by exploiting the imaging model. The two modules are alternately connected to achieve optimal HSI reconstruction performance. Experiments on both the simulated and real data demonstrate that the proposed method can achieve superior spectral reconstruction results with relatively small model size. The code will be available at https://github.com/renweidian.

Learning Robust Shape Regularization for Generalizable Medical Image Segmentation.

Generalizable medical image segmentation enables models to generalize to unseen target domains under domain shift issues. Recent progress demonstrates that the shape of the segmentation objective, with its high consistency and robustness across domains, can serve as a reliable regularization to aid the model for better cross-domain performance, where existing methods typically seek a shared framework to render segmentation maps and shape prior concurrently. However, due to the inherent texture and style preference of modern deep neural networks, the edge or silhouette of the extracted shape will inevitably be undermined by those domain-specific texture and style interferences of medical images under domain shifts. To address this limitation, we devise a novel framework with a separation between the shape regularization and the segmentation map. Specifically, we first customize a novel whitening transform-based probabilistic shape regularization extractor namely WT-PSE to suppress undesirable domain-specific texture and style interferences, leading to more robust and high-quality shape representations. Second, we deliver a Wasserstein distance-guided knowledge distillation scheme to help the WT-PSE to achieve more flexible shape extraction during the inference phase. Finally, by incorporating domain knowledge of medical images, we propose a novel instance-domain whitening transform method to facilitate a more stable training process with improved performance. Experiments demonstrate the performance of our proposed method on both multi-domain and single-domain generalization.

Multimodal Medical Image Fusion based on the VGG19 Model in the NSCT Domain

Aim: To deal with the drawbacks of the traditional medical image fusion methods, such as the low preservation ability of the details, the loss of edge information, and the image distortion, as well as the huge need for the training data for deep learning, a new multi-modal medical image fusion method based on the VGG19 model and the non-subsampled contourlet transform (NSCT) is proposed, whose overall objective is to simultaneously make the full use of the advantages of the NSCT and the VGG19 model. Methodology: Firstly, the source images are decomposed into the high-pass and low-pass subbands by NSCT, respectively. Then, the weighted average fusion rule is implemented to produce the fused low-pass sub-band coefficients, while an extractor based on the pre-trained VGG19 model is constructed to obtain the fused high-pass subband coefficients. Result and Discussion: Finally, the fusion results are reconstructed by the inversion transform of the NSCT on the fused coefficients. To prove the effectiveness and the accuracy, experiments on three types of medical datasets are implemented. Conclusion: By comparing seven famous fusion methods, both of the subjective and objective evaluations demonstrate that the proposed method can effectively avoid the loss of detailed feature information, capture more medical information from the source images, and integrate them into the fused images.

An Identification Method for Anomaly Types of Active Distribution Network Based on Data Mining

With the increasing penetration of distributed generators (DGs) and the growing demand for reliable power sources, it has become imperative to promptly identify anomalies in active distribution networks (ADNs). Additionally, anomaly identification is also crucial to assisting in targeting maintenance to handle failures after the action of relay protections. This study presents a highly accurate and rapidly responsive approach to identifying anomalies in ADNs. The proposed method uses three layers of intricately interconnected modules. In the first module, an iterative clustering layer is devised to process the original data. In the second module, an association rule layer is designed to identify hidden patterns based on the types of anomalies and the post-fault voltage. In the third module, a regression training layer is employed to train accurate identification models. The case study utilizes data samples collected by an IEEE 33-bus system with distributed generations as well as post-fault data from a practical distribution network. The results of comparative tests demonstrate that the proposed data-mining structure achieves reliable performance with high identification accuracy and short reaction times, indicating its ability to be applied to the state awareness system of ADNs.

Reduced-Reference Learning for Target Localization in Deep Brain Stimulation.

This work proposes a supervised machine learning method for target localization in deep brain stimulation (DBS). DBS is a recognized treatment for essential tremor. The effects of DBS significantly depend on the precise implantation of electrodes. Recent research on diffusion tensor imaging shows that the optimal target for essential tremor is related to the dentato-rubro-thalamic tract (DRTT), thus DRTT targeting has become a promising direction. The tractography-based targeting is more accurate than conventional ones, but still too complicated for clinical scenarios, where only structural magnetic resonance imaging (sMRI) data is available. In order to improve efficiency and utility, we consider target localization as a non-linear regression problem in a reduced-reference learning framework, and solve it with convolutional neural networks (CNNs). The proposed method is an efficient two-step framework, and consists of two image-based networks: one for classification and the other for localization. We model the basic workflow as an image retrieval process and define relevant performance metrics. Using DRTT as pseudo groundtruths, we show that individualized tractography-based optimal targets can be inferred from sMRI data with high accuracy. For two datasets of 280×220/272×227 (0.7/0.8 mm slice thickness) sMRI input, our model achieves an average posterior localization error of 2.3/1.2 mm, and a median of 1.7/1.02 mm. The proposed framework is a novel application of reduced-reference learning, and a first attempt to localize DRTT from sMRI. It significantly outperforms existing methods using 3D-CNN, anatomical and DRTT atlas, and may serve as a new baseline for general target localization problems.

Randomizing Human Brain Function Representation for Brain Disease Diagnosis.

Resting-state fMRI (rs-fMRI) is an effective tool for quantifying functional connectivity (FC), which plays a crucial role in exploring various brain diseases. Due to the high dimensionality of fMRI data, FC is typically computed based on the region of interest (ROI), whose parcellation relies on a pre-defined atlas. However, utilizing the brain atlas poses several challenges including 1) subjective selection bias in choosing from various brain atlases, 2) parcellation of each subject's brain with the same atlas yet disregarding individual specificity; 3) lack of interaction between brain region parcellation and downstream ROI-based FC analysis. To address these limitations, we propose a novel randomizing strategy for generating brain function representation to facilitate neural disease diagnosis. Specifically, we randomly sample brain patches, thus avoiding ROI parcellations of the brain atlas. Then, we introduce a new brain function representation framework for the sampled patches. Each patch has its function description by referring to anchor patches, as well as the position description. Furthermore, we design an adaptive-selection-assisted Transformer network to optimize and integrate the function representations of all sampled patches within each brain for neural disease diagnosis. To validate our framework, we conduct extensive evaluations on three datasets, and the experimental results establish the effectiveness and generality of our proposed method, offering a promising avenue for advancing neural disease diagnosis beyond the confines of traditional atlas-based methods. Our code is available at https://github.com/mjliu2020/RandomFR.

Achieving Better Category Separability for Hyperspectral Image Classification: A Spatial-Spectral Approach.

The task of hyperspectral image (HSI) classification has attracted extensive attention. The rich spectral information in HSIs not only provides more detailed information but also brings a lot of redundant information. Redundant information makes spectral curves of different categories have similar trends, which leads to poor category separability. In this article, we achieve better category separability from the perspective of increasing the difference between categories and reducing the variation within category, thus improving the classification accuracy. Specifically, we propose the template spectrum-based processing module from spectral perspective, which can effectively expose the unique characteristics of different categories and reduce the difficulty of model mining key features. Second, we design an adaptive dual attention network from spatial perspective, where the target pixel can adaptively aggregate high-level features by evaluating the confidence of effective information in different receptive fields. Compared with the single adjacency scheme, the adaptive dual attention mechanism makes the ability of target pixel to combine spatial information to reduce variation more stable. Finally, we designed a dispersion loss from the classifier's perspective. By supervising the learnable parameters of the final classification layer, the loss makes the category standard eigenvectors learned by the model more dispersed, which improves the category separability and reduces the rate of misclassification. Experiments on three common datasets show that our proposed method is superior to the comparison method.

LGGNet: Learning From Local-Global-Graph Representations for Brain-Computer Interface.

Neuropsychological studies suggest that co-operative activities among different brain functional areas drive high-level cognitive processes. To learn the brain activities within and among different functional areas of the brain, we propose local-global-graph network (LGGNet), a novel neurologically inspired graph neural network (GNN), to learn local-global-graph (LGG) representations of electroencephalography (EEG) for brain-computer interface (BCI). The input layer of LGGNet comprises a series of temporal convolutions with multiscale 1-D convolutional kernels and kernel-level attentive fusion. It captures temporal dynamics of EEG which then serves as input to the proposed local- and global-graph-filtering layers. Using a defined neurophysiologically meaningful set of local and global graphs, LGGNet models the complex relations within and among functional areas of the brain. Under the robust nested cross-validation settings, the proposed method is evaluated on three publicly available datasets for four types of cognitive classification tasks, namely the attention, fatigue, emotion, and preference classification tasks. LGGNet is compared with state-of-the-art (SOTA) methods, such as DeepConvNet, EEGNet, R2G-STNN, TSception, regularized graph neural network (RGNN), attention-based multiscale convolutional neural network-dynamical graph convolutional network (AMCNN-DGCN), hierarchical recurrent neural network (HRNN), and GraphNet. The results show that LGGNet outperforms these methods, and the improvements are statistically significant ( ) in most cases. The results show that bringing neuroscience prior knowledge into neural network design yields an improvement of classification performance. The source code can be found at https://github.com/yi-ding-cs/LGG.

DCCD: Reducing Neural Network Redundancy via Distillation.

Deep neural models have achieved remarkable performance on various supervised and unsupervised learning tasks, but it is a challenge to deploy these large-size networks on resource-limited devices. As a representative type of model compression and acceleration methods, knowledge distillation (KD) solves this problem by transferring knowledge from heavy teachers to lightweight students. However, most distillation methods focus on imitating the responses of teacher networks but ignore the information redundancy of student networks. In this article, we propose a novel distillation framework difference-based channel contrastive distillation (DCCD), which introduces channel contrastive knowledge and dynamic difference knowledge into student networks for redundancy reduction. At the feature level, we construct an efficient contrastive objective that broadens student networks' feature expression space and preserves richer information in the feature extraction stage. At the final output level, more detailed knowledge is extracted from teacher networks by making a difference between multiview augmented responses of the same instance. We enhance student networks to be more sensitive to minor dynamic changes. With the improvement of two aspects of DCCD, the student network gains contrastive and difference knowledge and reduces its overfitting and redundancy. Finally, we achieve surprising results that the student approaches and even outperforms the teacher in test accuracy on CIFAR-100. We reduce the top-1 error to 28.16% on ImageNet classification and 24.15% for cross-model transfer with ResNet-18. Empirical experiments and ablation studies on popular datasets show that our proposed method can achieve state-of-the-art accuracy compared with other distillation methods.