Learning Regularization Research Articles

Optimizing High-Performance Predictive Modeling of the Medium-Speed WEDM Processing of Inconel 718

The purpose of this research was to create a predictive model for a medium-speed wire electrical discharge machine (WEDM) utilizing an artificial neural network (ANN). Medium-speed WEDM experiments were developed based on the I-optimal mixture design for machining, the Inconel 718 superalloy. During the experiment, the input parameters were the spark ontime, spark offtime, wire feed, and current, with the material removal rate (MRR) and surface roughness (Ra) selected as performance indicators. The ANN model was trained on experimental data and built using a feed-forward backpropagation neural network with a (4-8-2) structure and the Bayesian regularization (BR) learning approach. The model correctly predicted the relationship between the medium-speed WEDM’s primary process parameters and machining performance. An integrated ANN model and the Non-Dominated Sorting Genetic Algorithm-II (NSGA-II) were used to determine the ideal parameters for the MRR and Ra, resulting in a set of Pareto-optimal solutions. The confirmation experiment revealed that the mean prediction error between the experimental and ideal solutions had a maximum error percentage of 1% for the MRR and 2% for the Ra, which are within acceptable ranges. This showed that the best process–parameter combinations were better for the MRR and Ra.

Incidental learning of predictive temporal context within cortical representations of visual shape.

Abstract Incidental learning of spatio-temporal regularities and consistencies— also termed ‘statistical learning’— may be important for discovering the causal principles governing the world. We studied statistical learning of temporal structure simultaneously at two time-scales: the presentation of synthetic visual objects (3 s) and predictive temporal context (30 s) in the order of appearance of such objects. Visual objects were complex and rotated in three dimensions about varying axes. Observers viewed fifteen (15) objects recurring many times each, intermixed with other objects that appeared only once, while whole-brain BOLD activity was recorded. Over three successive days, observers grew familiar with the recurring objects and reliably distinguished them from others. As reported elsewhere (Kakaei & Braun, 2023), representational similarity analysis (RSA) of multivariate BOLD activity revealed 124 ‘object-selective’ brain parcels with selectivity for recurring objects, located mostly in the ventral occipitotemporal cortex and the parietal cortex. Here we extend RSA to the representation of predictive temporal context, specifically “temporal communities” formed by objects that tended to follow each other. After controlling for temporal proximity, we observed 27 ‘community-sensitive’ brain parcels, in which pairwise distances between multivariate responses reflected community structure, either positively (smaller distances within than between communities) or negatively (larger distances within). Among object-selective parcels, 11 parcels were positively community-sensitive in the primary visual cortex (2 parcels), the ventral occipital, lingual, or fusiform cortex (8 parcels), and the inferior temporal cortex (1 parcel). Among non-object-selective parcels, 12 parcels were negatively community-sensitive in the superior, middle, and medial frontal cortex (6 parcels), the insula (2 parcels), the putamen (1 parcel), and in the superior temporal or parietal cortex (3 parcels). We conclude that cortical representations of object shape and of predictive temporal context are largely coextensive along the ventral occipitotemporal cortex.

Statistical Learning of Incidental Perceptual Regularities Induces Sensory Conditioned Cortical Responses.

Statistical learning of sensory patterns can lead to predictive neural processes enhancing stimulus perception and enabling fast deviancy detection. Predictive processes have been extensively demonstrated when environmental statistical regularities are relevant to task execution. Preliminary evidence indicates that statistical learning can even occur independently of task relevance and top-down attention, although the temporal profile and neural mechanisms underlying sensory predictions and error signals induced by statistical learning of incidental sensory regularities remain unclear. In our study, we adopted an implicit sensory conditioning paradigm that elicited the generation of specific perceptual priors in relation to task-irrelevant audio-visual associations, while recording Electroencephalography (EEG). Our results showed that learning task-irrelevant associations between audio-visual stimuli resulted in anticipatory neural responses to predictive auditory stimuli conveying anticipatory signals of expected visual stimulus presence or absence. Moreover, we observed specific modulation of cortical responses to probabilistic visual stimulus presentation or omission. Pattern similarity analysis indicated that predictive auditory stimuli tended to resemble the response to expected visual stimulus presence or absence. Remarkably, Hierarchical Gaussian filter modeling estimating dynamic changes of prediction error signals in relation to differential probabilistic occurrences of audio-visual stimuli further demonstrated instantiation of predictive neural signals by showing distinct neural processing of prediction error in relation to violation of expected visual stimulus presence or absence. Overall, our findings indicated that statistical learning of non-salient and task-irrelevant perceptual regularities could induce the generation of neural priors at the time of predictive stimulus presentation, possibly conveying sensory-specific information about the predicted consecutive stimulus.

Utilizing Silhouette and Head Information for Improved Cloth-changing Person Re-Identification

In recent years, significant progress has been made in person re-identification (ReID). However, as the time span increases, the phenomenon of changing clothes often occur, and the performance of methods based on person re-identification may decrease in this case. In this article, we focus on solving the problem of cloth-changing person re-identification from a single RGB image, which is more challenging, and improving the accuracy and robustness of traditional methods for recognition. We propose a cloth-changing Feature Regularization learning framework by fusing Silhouette and Head information (SHFR), a two-stream framework which transfers the silhouette information and head information learned by auxiliary stream to the main stream and supplements features unrelated to clothing. Specifically, the main stream approach we adopt is the traditional cloth-changing person re-identification network. In the auxiliary stream, we use the DeepLabV3 semantic segmentation model to extract person silhouette features, and the FCHD fully convolutional head detection model to extract head information with advanced semantic information. We concatenate the head information and background information into the silhouette as input for BackBone to continue training the network. In order to utilize silhouette information and head information, we regularize the cloth-changing features on both main stream and auxiliary stream, allowing main stream model to pay more attention to features unrelated to clothing and improve the accuracy of model recognition. We conduct extensive experiments on our model on the PRCC dataset, and the results show that our method has highly competitive performance.

Electrophysiological markers of orthographic pattern learning in school-aged children with reading challenges: An ERP investigation

BackgroundPrevious studies suggested that children with reading difficulty have impaired statistical learning ability in extracting distributional orthographic regularities. However, the neural mechanisms underlying have not been fully investigated. AimsThe current study aimed to identify the electrophysiological markers and to examine the neural underpinnings of statistical learning of orthographic regularities in children with reading difficulties. Methods and proceduresUsing the event-related potentials (ERPs) and the orthographic learning task, 157 children were exposed to a sequence of artificial pseudocharacters with varying levels of positional and semantic consistency (low at 60 %, moderate at 80 %, and high at 100 %). Outcomes and resultsPoor readers elicited an increased N170 response in the low consistency and a lack of left-lateralized P300 effect when learning positional regularities of radicals. Similarly, larger N170 effects were observed in poor readers, while similar N400 effects were found in both poor and average readers when learning semantic regularities of radicals. Conclusions and implicationsOur findings indicate that poor readers may have trouble using statistical information for early-stage orthographic pattern extraction, yet they can identify semantic inconsistencies after sufficient exposure. These results deepen our understanding of the neural mechanisms involved in statistical learning for poor readers and aid in improving criteria for differentiating between typically developing children and those with reading challenges.

Noise-Generating and Imaging Mechanism Inspired Implicit Regularization Learning Network for Low Dose CT Reconstrution.

Low-dose computed tomography (LDCT) helps to reduce radiation risks in CT scanning while maintaining image quality, which involves a consistent pursuit of lower incident rays and higher reconstruction performance. Although deep learning approaches have achieved encouraging success in LDCT reconstruction, most of them treat the task as a general inverse problem in either the image domain or the dual (sinogram and image) domains. Such frameworks have not considered the original noise generation of the projection data and suffer from limited performance improvement for the LDCT task. In this paper, we propose a novel reconstruction model based on noise-generating and imaging mechanism in full-domain, which fully considers the statistical properties of intrinsic noises in LDCT and prior information in sinogram and image domains. To solve the model, we propose an optimization algorithm based on the proximal gradient technique. Specifically, we derive the approximate solutions of the integer programming problem on the projection data theoretically. Instead of hand-crafting the sinogram and image regularizers, we propose to unroll the optimization algorithm to be a deep network. The network implicitly learns the proximal operators of sinogram and image regularizers with two deep neural networks, providing a more interpretable and effective reconstruction procedure. Numerical results demonstrate our proposed method improvements of > 2.9 dB in peak signal to noise ratio, > 1.4% promotion in structural similarity metric, and > 9 HU decrements in root mean square error over current state-of-the-art LDCT methods.

Individual differences in auditory perception predict learning of non-adjacent tone sequences in 3-year-olds.

Auditory processing of speech and non-speech stimuli oftentimes involves the analysis and acquisition of non-adjacent sound patterns. Previous studies using speech material have demonstrated (i) children's early emerging ability to extract non-adjacent dependencies (NADs) and (ii) a relation between basic auditory perception and this ability. Yet, it is currently unclear whether children show similar sensitivities and similar perceptual influences for NADs in the non-linguistic domain. We conducted an event-related potential study with 3-year-old children using a sine-tone-based oddball task, which simultaneously tested for NAD learning and auditory perception by means of varying sound intensity. Standard stimuli were A × B sine-tone sequences, in which specific A elements predicted specific B elements after variable × elements. NAD deviants violated the dependency between A and B and intensity deviants were reduced in amplitude. Both elicited similar frontally distributed positivities, suggesting successful deviant detection. Crucially, there was a predictive relationship between the amplitude of the sound intensity discrimination effect and the amplitude of the NAD learning effect. These results are taken as evidence that NAD learning in the non-linguistic domain is functional in 3-year-olds and that basic auditory processes are related to the learning of higher-order auditory regularities also outside the linguistic domain.

Unraveling Temporal Dynamics of Multidimensional Statistical Learning in Implicit and Explicit Systems: An X-Way Hypothesis.

Statistical learning enables humans to involuntarily process and utilize different kinds of patterns from the environment. However, the cognitive mechanisms underlying the simultaneous acquisition of multiple regularities from different perceptual modalities remain unclear. A novel multidimensional serial reaction time task was developed to test 40 participants' ability to learn simple first-order and complex second-order relations between uni-modal visual and cross-modal audio-visual stimuli. Using the difference in reaction times between sequenced and random stimuli as the index of domain-general statistical learning, a significant difference and dissociation of learning occurred between the initial and final learning phases. Furthermore, we used a negative and positive occurrence-frequency-and-reaction-time correlation to indicate implicit and explicit learning, respectively, and found that learning simple uni-modal patterns involved an implicit-to-explicit segue, while acquiring complex cross-modal patterns required an explicit-to-implicit segue, resulting in a X-shape crossing of regularity learning. Thus, we propose an X-way hypothesis to elucidate the dynamic interplay between the implicit and explicit systems at two distinct stages when acquiring various regularities in a multidimensional probability space.

Visual linguistic statistical learning is traceable through neural entrainment.

The human brain can detect statistical regularities in the environment across a wide variety of contexts. The importance of this process is well-established not just in language acquisition but across different modalities; in addition, several neural correlates of statistical learning have been identified. A current technique for tracking the emergence of regularity learning and localizing its neural background is frequency tagging (FT). FT can detect neural entrainment not only to the frequency of stimulus presentation but also to that of a hidden structure. Auditory learning paradigms with linguistic and nonlinguistic stimuli, along with a visual paradigm using nonlinguistic stimuli, have already been tested with FT. To complete the picture, we conducted an FT experiment using written syllables as stimuli and a hidden triplet structure. Both behavioral and neural entrainment data showed evidence of structure learning. In addition, we localized two electrode clusters related to the process, which spread across the frontal and parieto-occipital areas, similar to previous findings. Accordingly, we conclude that fast-paced visual linguistic regularities can be acquired and are traceable through neural entrainment. In comparison with the literature, our findings support the view that statistical learning involves a domain-general network.

CariesXrays: Enhancing Caries Detection in Hospital-Scale Panoramic Dental X-rays via Feature Pyramid Contrastive Learning

Dental caries has been widely recognized as one of the most prevalent chronic diseases in the field of public health. Despite advancements in automated diagnosis across various medical domains, it remains a substantial challenge for dental caries detection due to its inherent variability and intricacies. To bridge this gap, we release a hospital-scale panoramic dental X-ray benchmark, namely “CariesXrays”, to facilitate the advancements in high-precision computer-aided diagnosis for dental caries. It comprises 6,000 panoramic dental X-ray images, with a total of 13,783 instances of dental caries, all meticulously annotated by dental professionals. In this paper, we propose a novel Feature Pyramid Contrastive Learning (FPCL) framework, that jointly incorporates feature pyramid learning and contrastive learning within a unified diagnostic paradigm for automated dental caries detection. Specifically, a robust dual-directional feature pyramid network (D2D-FPN) is designed to adaptively capture rich and informative contextual information from multi-level feature maps, thus enhancing the generalization ability of caries detection across different scales. Furthermore, our model is augmented with an effective proposals-prototype contrastive regularization learning (P2P-CRL) mechanism, which can flexibly bridge the semantic gaps among diverse dental caries with varying appearances, resulting in high-quality dental caries proposals. Extensive experiments on our newly-established CariesXrays benchmark demonstrate the potential of FPCL to make a significant social impact on caries diagnosis.

Distribution-free Bayesian regularized learning framework for semi-supervised learning

In machine learning it is often necessary to assume or know the distribution of the data, however it is difficult to do so in practical applications. Aiming to this problem, this work, we propose a novel distribution-free Bayesian regularized learning framework for semi-supervised learning, which is called Hessian regularized twin minimax probability extreme learning machine (HRTMPELM). In this framework, we attempt to construct two non-parallel hyperplanes by introducing the high separation probability assumption, such that each hyperplane separates samples from one class with maximum probability while moving away from samples from the other class. Subsidiently, the framework can be utilized to construct reasonable semi-supervised classifiers by using the information of the inherent geometric distribution of the samples through the Hessian regularization term. Additionally, the proposed framework controls the misclassification error of samples by minimizing the upper limit of the worst-case misclassification probability, and improves the generalization performance of the model by introducing the idea of regularization to avoid the occurrence of ill-posedness and overfitting problems. More importantly, the framework has no hyperparameters, making the learning process very simplified and efficient. Finally, a simple and reliable algorithm with globally optimal solutions via multivariate Chebyshev inequalities is designed for solving the proposed learning framework. Experiments on multiple datasets demonstrate the reliability and effectiveness of the proposed learning framework compared to other methods. Especially, we applied the framework to Ningxia wolfberry quality detection, which greatly enriches and facilitates the application of machine learning algorithms in the agricultural field.

ISP-IRLNet: Joint optimization of interpretable sampler and implicit regularization learning network for accerlerated MRI

Compressed Sensing Magnetic Resonance Imaging (CS-MRI) was proposed to accelerate data acquisition and reconstruct MR images from under-sampled data in k-space. However, the traditional approaches design the sampling patterns separately from the reconstruction process, which often leads to suboptimal reconstruction performance. To address this issue, we propose a joint optimization model dubbed as ISP-IRLNet to yield optimal recovery including an interpretable sampler and a reconstructor using the implicit regularization learning network. In particular, we introduce a probabilistic distribution model of the sampling incorporated into the prior of k-space locations, which provides a robust CS-MRI sample pattern with an explicit expression. To estimate the approximating gradient in backward propagation for more stable training, we employ a differentiable binarization strategy. In addition, we unroll the ADMM optimization for solving the regularized reconstruction model to be a deep network (i.e., IRLNet) with a learnable implicit regularization sub-network to learn a regularizer or prior term implicitly. The experiments on our collected brain and public knee datasets demonstrate that our method provides an optimal sample pattern and achieves superior image reconstruction performance compared with existing methods, especially at high acceleration rates.

The Role of Feedback in the Statistical Learning of Language-Like Regularities.

In language learning, learners engage with their environment, incorporating cues from different sources. However, in lab-based experiments, using artificial languages, many of the cues and features that are part of real-world language learning are stripped away. In three experiments, we investigated the role of positive, negative, and mixed feedback on the gradual learning of language-like statistical regularities within an active guessing game paradigm. In Experiment 1, participants received deterministic feedback (100%), whereas probabilistic feedback (i.e., 75% or 50%) was introduced in Experiment 2. Finally, Experiment 3 explored the impact of mixed probabilistic feedback (33% positive, 33% negative, 33% no feedback). The results showed that cross-situational learning of words was observed without feedback, but participants were able to learn structural regularities of the miniature language only when feedback was provided. Interestingly, the presence of positive feedback was particularly helpful for the learner, promoting more in-depth learning of the artificial language.

Excitatory cerebellar transcranial direct current stimulation boosts the leverage of prior knowledge for predicting actions.

The cerebellum causally supports social processing by generating internal models of social events based on statistical learning of behavioral regularities. However, whether the cerebellum is only involved in forming or also in using internal models for the prediction of forthcoming actions is still unclear. We used cerebellar transcranial Direct Current Stimulation (ctDCS) to modulate the performance of healthy adults in using previously learned expectations in an action prediction task. In a first learning phase of this task, participants were exposed to different levels of associations between specific actions and contextual elements, to induce the formation of either strongly or moderately informative expectations. In a following testing phase, which assessed the use of these expectations for predicting ambiguous (i.e. temporally occluded) actions, we delivered ctDCS. Results showed that anodic, compared to sham, ctDCS boosted the prediction of actions embedded in moderately, but not strongly, informative contexts. Since ctDCS was delivered during the testing phase, that is after expectations were established, our findings suggest that the cerebellum is causally involved in using internal models (and not just in generating them). This encourages the exploration of the clinical effects of ctDCS to compensate poor use of predictive internal models for social perception.

Online regularized learning algorithm for functional data

In recent years, functional linear models have attracted growing attention in statistics and machine learning for recovering the slope function or its functional predictor. This paper considers online regularized learning algorithm for functional linear models in a reproducing kernel Hilbert space. It provides convergence analysis of excess prediction error and estimation error with polynomially decaying step-size and constant step-size, respectively. Fast convergence rates can be derived via a capacity dependent analysis. Introducing an explicit regularization term extends the saturation boundary of unregularized online learning algorithms with polynomially decaying step-size and achieves fast convergence rates of estimation error without capacity assumption. In contrast, the latter remains an open problem for the unregularized online learning algorithm with decaying step-size. This paper also demonstrates competitive convergence rates of both prediction error and estimation error with constant step-size compared to existing literature.

A duality approach to regularized learning problems in Banach spaces

Regularized learning problems in Banach spaces, which often minimize the sum of a data fidelity term in one Banach norm and a regularization term in another Banach norm, is challenging to solve. We construct a direct sum space based on the Banach spaces for the fidelity term and the regularization term, and recast the objective function as the norm of a quotient space of the direct sum space. We then express the original regularized problem as an optimization problem in the dual space of the direct sum space. It is to find the maximum of a linear function on a convex polytope, which may be solved by linear programming. A solution of the original problem is then obtained by using related extremal properties of norming functionals from a solution of the dual problem. Numerical experiments demonstrate that the proposed duality approach is effective for solving the regularization learning problems.

Lexically-specific syntactic restrictions in second-language speakers

In two structural priming experiments, we investigated the representations of lexically-specific syntactic restrictions of English verbs for highly proficient and immersed second language (L2) speakers of English. We considered the interplay of two possible mechanisms: generalization from the first language (L1) and statistical learning within the L2 (both of abstract structure and of lexically-specific information). In both experiments, L2 speakers with either Germanic or Romance languages as L1 were primed to produce dispreferred double-object structures involving non-alternating dative verbs. Priming occurred from ungrammatical double-object primes involving different non-alternating verbs (Experiment 1) and from grammatical primes involving alternating verbs (Experiment 2), supporting abstract statistical learning within the L2. However, we found no differences between L1-Germanic speakers (who have the double-object structure in their L1) and L1-Romance speakers (who do not), inconsistent with the prediction for between-group differences of the L1-generalization account. Additionally, L2 speakers in Experiment 2 showed a lexical boost: There was stronger priming after (dispreferred) non-alternating same-verb double-object primes than after (grammatical) alternating different-verb primes. Such lexically-driven persistence was also shown by L1 English speakers (Ivanova, Pickering, McLean, Costa, & Branigan, 2012) and may underlie statistical learning of lexically-dependent structural regularities. We conclude that lexically-specific syntactic restrictions in highly proficient and immersed L2 speakers are shaped by statistical learning (both abstract and lexically-specific) within the L2, but not by generalization from the L1.

GMRLNet: A graph-based manifold regularization learning framework for placental insufficiency diagnosis on incomplete multimodal ultrasound data.

Multimodal analysis of placental ultrasound (US) and microflow imaging (MFI) could greatly aid in the early diagnosis and interventional treatment of placental insufficiency (PI), ensuring a normal pregnancy. Existing multimodal analysis methods have weaknesses in multimodal feature representation and modal knowledge definitions and fail on incomplete datasets with unpaired multimodal samples. To address these challenges and efficiently leverage the incomplete multimodal dataset for accurate PI diagnosis, we propose a novel graph-based manifold regularization learning (MRL) framework named GMRLNet. It takes US and MFI images as input and exploits their modality-shared and modality-specific information for optimal multimodal feature representation. Specifically, a graph convolutional-based shared and specific transfer network (GSSTN) is designed to explore intra-modal feature associations, thus decoupling each modal input into interpretable shared and specific spaces. For unimodal knowledge definitions, graph-based manifold knowledge is introduced to describe the sample-level feature representation, local inter-sample relations, and global data distribution of each modality. Then, an MRL paradigm is designed for inter-modal manifold knowledge transfer to obtain effective cross-modal feature representations. Furthermore, MRL transfers the knowledge between both paired and unpaired data for robust learning on incomplete datasets. Experiments were conducted on two clinical datasets to validate the PI classification performance and generalization of GMRLNet. State-of-the-art comparisons show the higher accuracy of GMRLNet on incomplete datasets. Our method achieves 0.913 AUC and 0.904 balanced accuracy (bACC) for paired US and MFI images, as well as 0.906 AUC and 0.888 bACC for unimodal US images, illustrating its application potential in PI CAD systems.

Playing hide and seek: Contextual regularity learning develops between 3 and 5 years of age

The ability to acquire contextual regularities is fundamental in everyday life because it helps us to navigate the environment, directing our attention where relevant events are more likely to occur. Sensitivity to spatial regularities has been largely reported from infancy. Nevertheless, it is currently unclear when children can use this rapidly acquired contextual knowledge to guide their behavior. Evidence of this ability is indeed mixed in school-aged children and, to date, it has never been explored in younger children and toddlers. The current study investigated the development of contextual regularity learning in children aged 3 to 5 years. To this aim, we designed a new contextual learning paradigm in which young children were presented with recurring configurations of bushes and were asked to guess behind which bush a cartoon monkey was hiding. In a series of two experiments, we manipulated the relevance of color and visuospatial cues for the underlying task goal and tested how this affected young children’s behavior. Our results bridge the gap between the infant and adult literatures, showing that sensitivity to spatial configurations persists from infancy to childhood, but it is only around the fifth year of life that children naturally start to integrate multiple cues to guide their behavior.

Statistical Learning of Chord-Transition Regularities in a Novel Equitempered Scale: An MMN Study

In music and language domains, it has been suggested that patterned transitions of sounds can be acquired implicitly through statistical learning. Previous studies have investigated the statistical learning of auditory regularities by recording early neural responses to a sequence of tones presented at high or low transition probabilities. However, it remains unclear whether the statistical learning of musical chord transitions is reflected in endogenous, regularity-dependent components of the event-related potential (ERP). The present study aimed to record the mismatch negativity (MMN) elicited by chord transitions that deviated from newly learned transitional regularities. Chords were generated in a novel 18 equal temperament pitch class scale to avoid interference from the existing tonal representations of the 12 equal temperament pitch class system. Thirty-six adults without professional musical training listened to a sequence of randomly inverted chords in which certain chords were presented with high (standard) or low (deviant) transition probabilities. An irrelevant timbre change detection task was assigned to make them attend to the sequence during the ERP recording. After that, a familiarity test was administered in which the participants were asked to choose the more familiar chord sequence out of two successive sequences. The results showed that deviant transitions elicited the MMN, although the participants could not recognize the standard transition beyond the level of chance. These findings suggest that humans can statistically learn new transitional regularities of chords in a novel musical scale, even though they did not recognize them explicitly. This study provides further evidence that music-syntactic regularities can be acquired implicitly through statistical learning.