Memory Units Research Articles

Invited: Mimicking Biological Synapse Activity with LDH-Based Memristor

Memristors, mimicking biological synapse response, are believed to have a great potential for use as an adaptive memory unit in future neuromorphic computing systems. Here we report on fabrication of two-dimensional memristor comprised of layered double hydroxide (LDH) material, covered with graphene layer, serving as a terminal for connecting to electrodes. We demonstrated nonvolatile resistive switching of LDH memristor. Device performance is studied upon replacing graphene semimetal with alternative 2D materials. Comprehensive characterization of the memristor materials was performed with correlated Raman, Electron, and Scanning Probe Microscopy methods.ACKNOWLEDGMENTThis work was supported by the US Department of Defense (Contract #W911QY2220006), 2DCC-MIP NSF cooperative agreement DMR-2039351. Work was performed at JSNN, a member of the Southeastern Nanotechnology Infrastructure Corridor (SENIC) and National Nanotechnology Coordinated Infrastructure (NNCI), which is supported by the NSF (ECCS-1542174)

Volatile and Nonvolatile Programmable Iontronic Memristor with Lithium Imbued TiOx for Neuromorphic Computing Applications.

We demonstrate a lithium (Li) imbued TiOx iontronic device that exhibits synapse-like short-term plasticity behavior without requiring a forming process beforehand or a compliance current during switching. A solid-state electrolyte lithium phosphorus oxynitride (LiPON) behaves as the ion source, and the embedding and releasing of Li ions inside the cathodic like TiOx renders volatile conductance responses from the device and offers a natural platform for hardware simulating neuron functionalities. Besides, these devices possess high uniformity and great endurance as no conductive filaments are present. Different short-term pulse-based phenomena, including paired pulse facilitation, post-tetanic potentiation, and spike rate-dependent plasticity, were observed with self-relaxation characteristics. Based on the voltage excitation period, the time scale of the volatile memory can be tuned. Temperature measurement reveals the ion displacement-induced conductance channels become frozen below 220 K. In addition, the volatile analog devices can be configured into nonvolatile memory units with multibit storage capabilities after an electroforming process. Therefore, on the same platform, we can configure volatile units as nonlinear dynamic reservoirs for performing neuromorphic training and the nonvolatile units as the weight storage layer. We proceed to use voice recognition as an example with the tunable time constant relationship and obtain 94.4% accuracy with a minimal training data set. Thus, this iontronic platform can effectively process and update temporal information for reservoir and neuromorphic computing paradigms.

Design of All-Optical D Flip Flop Memory Unit Based on Photonic Crystal.

This paper proposes a unique configuration for an all-optical D Flip Flop (D-FF) utilizing a quasi-square ring resonator (RR) and T-Splitter, as well as NOT and OR logic gates within a 2-dimensional square lattice photonic crystal (PC) structure. The components realizing the all-optical D-FF comprise of optical waveguides in a 2D square lattice PC of 45 × 23 silicon (Si) rods in a silica (SiO2) substrate. The utilization of these specific materials has facilitated the fabrication process of the design, diverging from alternative approaches that employ an air substrate, a method inherently unattainable in fabrication. The configuration underwent examination and simulation utilizing both plane-wave expansion (PWE) and finite-difference time-domain (FDTD) methodologies. The simulation outcomes demonstrate that the designed waveguides and RR effectively execute the operational principles of the D-FF by guiding light as intended. The suggested configuration holds promise as a logic block within all-optical arithmetic logic units (ALUs) designed for digital computing optical circuits. The design underwent optimization for operation within the C-band spectrum, particularly at 1550 nm. The outcomes reveal a distinct differentiation between logic states '1' and '0', enhancing robust decision-making on the receiver side and minimizing logic errors in the photonic decision circuit. The D-FF displays a contrast ratio (CR) of 4.77 dB, a stabilization time of 0.66 psec, and a footprint of 21 μm × 12 μm.

Utilization of genetic algorithm in tuning the hyper-parameters of hybrid NN-based side-slip angle estimators

This paper proposes a solution to enhance and compare different neural network (NN)-based side-slip angle estimators. The feed-forward neural networks (FFNNs), recurrent neural networks, long short-term memory units (LSTMs), and gated recurrent units are investigated. However, there is a lack in the selection criteria of the architectures’ hyper-parameters. Therefore, the genetic algorithm is integrated with the NN-based estimators to find the optimal hyper-parameters for the studied architectures. The tuned hyper-parameters in this work include the number of neurons, number of layers, activation function, optimizer type, and learning rate. The objective function of the optimization problem is minimizing the root-mean-square error (RMSE) on multiple testing data. The optimal models are further included in the design of a hybrid NN estimator with Kalman filter. In the hybrid estimators, the optimal NN estimators are used as virtual sensors to correct the prediction of the side-slip angle resulting from the mathematical lateral vehicle model. Eventually, the performance of the best selected model is evaluated in terms of different metrics; mean RMSE, mean error variance, mean training time, and mean estimation time. LSTMs are found to achieve the lowest mean RMSE while being tested on highly generalized data yielding the highest training and estimation time. However, FFNNs achieve the lowest RMSE while being tested on low generalized data and the lowest training and estimation time. Meanwhile, it is observed that the hybrid estimators achieved lower RMSE with great enhancement compared to the non-hybridized ones proving the effectiveness of the proposed approach and increasing the side-slip estimation generalization ability in unknown environments with high uncertainties, which are not covered by the training dataset for the NNs estimators.

Effect of Ultraviolet Radiation on Properties of Ge2Sb2Te5 Phase Change Films.

With the expanding utilization of space technology, the stability of electronic components' performance in radiation environments has garnered significant attention. In this study, we prepared Ge2Sb2Te5 phase change films and memory units on silicon substrates to explore the influence of ultraviolet (UV) radiation on their characteristics. The experimental findings revealed that UV irradiation at a power density of 450 mW/cm2 decreased the amorphous resistance and thermal stability of Ge2Sb2Te5 films, impeding their multistage storage performance. Nevertheless, the amorphous state could still undergo effective transformation into a crystalline state. Furthermore, UV irradiation triggered the photoelectric effect, narrowing the band gap and causing a redshift of the Raman peak in amorphous films. Remarkably, the surface properties of Ge2Sb2Te5 films remained unchanged under irradiation. The phase change memory device based on Ge2Sb2Te5 film retained its SET-RESET conversion capability at a pulse width of 100 ns post-UV irradiation, demonstrating resilience against UV radiation. This study offers the practical insights for the application of phase change memory in space radiation environments.

Very short-term wind power forecasting for real-time operation using hybrid deep learning model with optimization algorithm

This paper proposes a new hybrid deep learning model to enhance the accuracy of forecasting very short-term wind power generation. The proposed model comprises a convolutional layer, a long-short-term memory (LSTM) unit, and fully connected neural network. Convolution layer can automatically learn complicated features from the raw input, whereas the LSTM layers can retain useful information through which gradient information may flow over extended periods. To obtain the best performance from the forecasting model, a random search optimization technique has been developed for tuning hyper-parameters of the model developed. The 5 min datasets from the White Rock wind farm, Australia are used to investigate the effectiveness of the proposed model as wind farms are participating in spot electricity market. To compare the effectiveness, the proposed model is compared with the existing models, such as convolution neural network (CNN), LSTM, gated recurrent unit (GRU), bidirectional LSTM (BiLSTM), artificial neural network (ANN), and support vector machine (SVM). The root-mean-square error (RMSE), mean absolute error (MAE), and Theil’s inequality coefficient (TIC) are used to analyze and compare the performances of the predictive models. Based on RMSE and MAE, the proposed model exhibits a higher accuracy of approximately 23.79% and 28.63% compared to other forecasting methods, respectively.

Spiking neural networks for nonlinear regression of complex transient signals on sustainable neuromorphic processors

In recent years, spiking neural networks were introduced in science as the third generation of artificial neural networks leading to a tremendous energy saving on neuromorphic processors. This sustainable effect is due to the sparse nature of signal processing in-between spiking neurons leading to much less scalar multiplications as in second-generation networks. The spiking neuron’s efficiency is even more pronounced by their inherently recurrent nature being useful for recursive function approximations. We believe that there is a need for a general regression framework for SNNs to explore the high potential of neuromorphic computations. However, besides many classification studies with SNNs in the literature, nonlinear neuromorphic regression analysis represents a gap in research. Hence, we propose a general SNN approach for function approximation applicable for complex transient signal processing taking surrogate gradients due to the discontinuous spike representation into account. However, to pay attention to the need for high memory access during deep SNN network communications, additional spiking Legrendre Memory Units are introduced in the neuromorphic architecture. Path-dependencies and evolutions of signals can be tackled in this way. Furthermore, interfaces between real physical and binary spiking values are necessary. Following this intention, a hybrid approach is introduced, exhibiting an autoencoding strategy between dense and spiking layers. However, to verify the presented framework of nonlinear regression for a wide spectrum of scientific purposes, we see the need for obtaining realistic complex transient short-time signals by an extensive experimental set-up. Hence, a measurement technique for benchmark experiments is proposed with high-frequency oscillations measured by capacitive and piezoelectric sensors resulting in wave propagations and inelastic solid deformations to be predicted by the developed SNN regression analysis. Hence, the proposed nonlinear regression framework can be deployed to a wide range of scientific and technical applications.

Industrial Sustainable Development: The Development Trend of Programmable Logic Controller Technology

Programmable Logic Controllers (PLCs) are indispensable for current and future industrial development, especially in smart factories, smart home technology, automated production lines, and machinery manufacturing. This study presents the trends in PLC software and hardware development through a technology roadmap and offers relevant suggestions to help industries achieve sustainable development, enhance market competitiveness, and provide references for research. Through expert interviews and fuzzy Delphi analysis, this study points out that future PLC development needs to focus on editing interfaces, syntax, Central Processing Units, Memory Units, and Communication Modules. Specific recommendations include visualizing regional/global label settings and connection settings, adding Python, JAVA, LabVIEW, and Scratch syntax, improving instruction execution speed, expanding program and expansion capacities, and adopting dual-channel Ethernet and connections to external networks and wireless networks. Fuzzy hierarchical analysis shows that Communication Modules are the most important component, followed by Central Processing Units and syntax expansion, and, finally, program and expansion capacity enhancements. These suggestions aim to promote product innovation and social environment demand evaluation, enhance product competitiveness, and achieve sustainable development goals.

Non-volatile 2D MoS2/black phosphorus heterojunction photodiodes in the near- to mid-infrared region

Cutting-edge mid-wavelength infrared (MWIR) sensing technologies leverage infrared photodetectors, memory units, and computing units to enhance machine vision. Real-time processing and decision-making challenges emerge with the increasing number of intelligent pixels. However, current operations are limited to in-sensor computing capabilities for near-infrared technology, and high-performance MWIR detectors for multi-state switching functions are lacking. Here, we demonstrate a non-volatile MoS2/black phosphorus (BP) heterojunction MWIR photovoltaic detector featuring a semi-floating gate structure design, integrating near- to mid-infrared photodetection, memory and computing (PMC) functionalities. The PMC device exhibits the property of being able to store a stable responsivity, which varies linearly with the stored conductance state. Significantly, device weights (stable responsivity) can be programmed with power consumption as low as 1.8 fJ, and the blackbody peak responsivity can reach 1.68 A/W for the MWIR band. In the simulation of Faster Region with convolution neural network (CNN) based on the FLIR dataset, the PMC hardware responsivity weights can reach 89% mean Average Precision index of the feature extraction network software weights. This MWIR photovoltaic detector, with its versatile functionalities, holds significant promise for applications in advanced infrared object detection and recognition systems.

SEMANTIC SPACE OF THE UNIVERSAL MORAL AND ETHICAL CONCEPT ЖАЛІСТЬ/SYMPATHY

The paper deals with the study of the concept ЖАЛІСТЬ/SYMPATHY in Ukrainian and English languages which can be considered rather relevant for the present-day political, economic, social and educational situation in Ukraine because it is the universal, moral and ethical emotion concept which is utilised as the element of war professional propaganda via specific formats and technologies to deliver social and political impact and manipulation on the existing system of social and political views and outlooks. The concept was selected as the basic unit for the analysis because it is defined as the operational memory unit, the mental entity, the result of knowledge objectivisation. Moreover, as the concept functioning relates to specific national context and particular conditions of society existence, it can express its semantic potential and adapt it in accordance with the existing value priorities at a particular stage of society development and cause purposeful influence or even manipulation on its members. The comparison of the semantic spaces of the universal moral and ethical concept ЖАЛІСТЬ/SYMPATHY in Ukrainian and English languages has revealed the fact that this mental unit is determined as important in moral and ethical paradigms in the both languages because it it is stipulated for the basic human antithesis "good/evil". In Ukrainian it relates, to more extent, to emotional feature of moral, ethical value of both humans and any living creatures implying the desire to introduce not only love, pity, weal for others and even an element of empathy. In English it also expresses spontaneous irrational emotional response to any misfortune or disaster, but, at the same time, implies the pragmatic decision-making to render some practical assistance. The results of the study can be used for teaching semiotics, cognitive linguistics or PR technologies.

Comparative Analysis of Neural Network Architectures for Predicting Chronic Disease Indicators Using CDC’s Chronic Disease Indicators Dataset

This research evaluates the performance of three machine learning models—Neural Network (NN), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) using Long Short-Term Memory (LSTM) units—in predicting chronic disease indicators using the CDC's Chronic Disease Indicators (CDI) dataset. The study employs a comprehensive preprocessing pipeline and 5-fold cross-validation to ensure robustness and generalizability of the results. The CNN model outperformed both the NN and RNN models across all key performance metrics, achieving an accuracy of 0.6303, precision of 0.6445, recall of 0.6303, and F1 score of 0.5950. The superior performance of the CNN is attributed to its ability to capture spatial hierarchies and interactions within the structured dataset. The findings underscore the importance of selecting appropriate machine learning architectures based on the data characteristics. This research provides valuable insights for public health officials and policymakers to enhance chronic disease monitoring, early detection, and intervention strategies. Future work will explore hybrid models and advanced techniques to further improve predictive performance. This study highlights the potential of CNNs in public health informatics and sets a foundation for further research in this domain

Hardware implementation of backpropagation using progressive gradient descent for in situ training of multilayer neural networks.

Neural network training can be slow and energy-expensive due to the frequent transfer of weight data between digital memory and processing units. Neuromorphic systems can accelerate neural networks by performing multiply-accumulate operations in parallel using nonvolatile analog memory. However, executing the widely used backpropagation training algorithm in multilayer neural networks requires information-and therefore storage-of the partial derivatives of the weight values preventing suitable and scalable implementation in hardware. Here, we propose a hardware implementation of the backpropagation algorithm that progressively updates each layer using in situ stochastic gradient descent, avoiding this storage requirement. We experimentally demonstrate the in situ error calculation and the proposed progressive backpropagation method in a multilayer hardware-implemented neural network. We confirm identical learning characteristics and classification performance compared to conventional backpropagation in software. We show that our approach can be scaled to large and deep neural networks, enabling highly efficient training of advanced artificial intelligence computing systems.

Enhanced resistive switching behaviors of organic resistive random access memory devices by adding polyethyleneimine interlayer

Organic nonvolatile memory devices have garnered significant attention as next-generation electrical memory units owing to their potential for low-cost and straightforward fabrication through a solution process. In this study, we successfully fabricated fully solution-processed organic resistive random access memory (RRAM) devices using poly(3-hexylthiophene-2,5-diyl) (P3HT) and [6,6]-phenyl-C61-butyric acid methyl ester (PCBM) as the resistive switching (RS) layer, with poly(3,4-ethylene-dioxythiophene):poly(styrene sulfonate) (PEDOT:PSS) employed as the top electrode. Additionally, to enhance performance further, a polyethyleneimine (PEIE) interlayer was introduced between the bottom electrode and the P3HT:PCBM RS layer. The resulting organic RRAM devices with the PEIE interlayer exhibited bipolar resistive switching, with improved endurance increasing from 50 to 100 cycles, a retention time of 103 s, and a low SET voltage of 0.7 V. The organic RRAM devices featuring the PEIE interlayer demonstrated superior RS performance attributed to the higher Schottky barrier at the interface between the bottom electrode and the active switching layer, creating an asymmetric structure. I–V curve fitting confirmed that the potential switching mechanism involved Schottky emission in the high-resistance state and Ohmic conduction in the low-resistance state. Our findings suggest that organic RRAM devices with a PEIE interlayer hold promise for stable nonvolatile memory applications.

Boolean map and object reconcile as the unit of visual working memory.

The unit of visual working memory is a fundamental issue under debate in the fields of cognitive psychology and neuroscience, with some traditional research suggesting that it is an object, while other recent studies demonstrating that a Boolean map offers a better account. The controversy surrounding the unit of visual working memory often centers on the representation of objects consist of same dimensional features (e.g. bicolor objects). For 2 colors in a bicolor object, some behavioral studies have suggested that they need to be represented by separate units, while some other studies using electrophysiological measures have found that they can be represented within a single unit. This apparent conflict hints that Boolean map and object may reconcile as the unit of visual working memory. Adopting the contralateral delay activity as an electrophysiological marker of visual working memory, experiments 1 and 2 consistently found that the contralateral delay activity amplitude for memorizing bicolor circles at P7/P8 conformed the Boolean map-based storage throughout the whole maintenance, while the contralateral delay activity amplitude at P3/P4 just conformed the object-based storage during the early period. It suggests though Boolean map got stronger supporting evidence than object, they 2 may coexist as the unit of visual working memory.

Force Field X: A computational microscope to study genetic variation and organic crystals using theory and experiment.

Force Field X (FFX) is an open-source software package for atomic resolution modeling of genetic variants and organic crystals that leverages advanced potential energy functions and experimental data. FFX currently consists of nine modular packages with novel algorithms that include global optimization via a many-body expansion, acid-base chemistry using polarizable constant-pH molecular dynamics, estimation of free energy differences, generalized Kirkwood implicit solvent models, and many more. Applications of FFX focus on the use and development of a crystal structure prediction pipeline, biomolecular structure refinement against experimental datasets, and estimation of the thermodynamic effects of genetic variants on both proteins and nucleic acids. The use of Parallel Java and OpenMM combines to offer shared memory, message passing, and graphics processing unit parallelization for high performance simulations. Overall, the FFX platform serves as a computational microscope to study systems ranging from organic crystals to solvated biomolecular systems.

Burden of family caregivers in Alzheimer’s disease: The role of caregivers’ perception of cognitive impairment

Objectives This study examined the role of caregivers’ perception of cognitive impairment in burden of family caregivers in Alzheimer’s disease (AD). We hypothesized that the evaluation of cognitive impairment by family caregivers plays a pivotal role in burden. Methods The study included 110 dyads (person with AD and their caregiver) recruited from a Memory Unit in France. The cognitive impairment and depressive symptoms of person with AD were evaluated by a geriatrician using the Mini Mental State Examination (MMSE) and the Geriatric Depression Scale (GDS-15). Caregivers provided self-reports on the perception of cognitive impairment (IQCODE) of the care recipient, the caregiving burden (ZBI), depressive symptoms (GDS-15), and self-esteem (RSE). Descriptive analyses, comparison of different caregiver burden groups, and multinomial logistic regression analyses to understand correlates of caregiver burden were conducted with SPSS®, version 20. Results The findings show that the caregivers are on average 60 years old and the majority are women. They care for persons with AD, who are on average 82 years old and most of whom are women. Our results show that the duration of caregiving, depression of the caregiver, and caregivers’ perception of cognitive impairment contribute significantly to burden of caregiver. Discussion This study shows that it is necessary to adopt the caregiver-centered approach to support the dyad. The role of the caregivers’ perception of cognitive impairment in AD should be developed when supporting caregivers in suffering.

Diagnostic performance of plasma pTau217, pTau181, Aβ1-42 and Aβ1-40 in the LUMIPULSE automated platform for the detection of Alzheimer disease

BackgroundRecently developed blood markers for Alzheimer's disease (AD) detection have high accuracy but usually require ultra-sensitive analytic tools not commonly available in clinical laboratories, and their performance in clinical practice is unknown.MethodsWe analyzed plasma samples from 290 consecutive participants that underwent lumbar puncture in routine clinical practice in a specialized memory clinic (66 cognitively unimpaired, 130 participants with mild cognitive impairment, and 94 with dementia). Participants were classified as amyloid positive (A +) or negative (A-) according to CSF Aβ1–42/Aβ1–40 ratio. Plasma pTau217, pTau181, Aβ1–42 and Aβ1–40 were measured in the fully-automated LUMIPULSE platform. We used linear regression to compare plasma biomarkers concentrations between A + and A- groups, evaluated Spearman’s correlation between plasma and CSF and performed ROC analyses to assess their diagnostic accuracy to detect brain amyloidosis as determined by CSF Aβ1–42/Aβ1–40 ratio. We analyzed the concordance of pTau217 with CSF amyloidosis.ResultsPlasma pTau217 and pTau181 concentration were higher in A + than A- while the plasma Aβ1–42/Aβ1–40 ratio was lower in A + compared to A-. pTau181 and the Aβ1–42/Aβ1–40 ratio showed moderate correlation between plasma and CSF (Rho = 0.66 and 0.69, respectively). The areas under the ROC curve to discriminate A + from A- participants were 0.94 (95% CI 0.92–0.97) for pTau217, and 0.88 (95% CI 0.84–0.92) for both pTau181 and Aβ1–42/Aβ1–40. Chronic kidney disease (CKD) was related to increased plasma biomarker concentrations, but ratios were less affected. Plasma pTau217 had the highest fold change (× 3.2) and showed high predictive capability in discriminating A + from A-, having 4–7% misclassification rate. The global accuracy of plasma pTau217 using a two-threshold approach was robust in symptomatic groups, exceeding 90%.ConclusionThe evaluation of blood biomarkers on an automated platform exhibited high diagnostic accuracy for AD pathophysiology, and pTau217 showed excellent diagnostic accuracy to identify participants with AD in a consecutive sample representing the routine clinical practice in a specialized memory unit.

Multifunctional In-Memory Logics Based on a Dual-Gate Antiambipolar Transistor toward Non-von Neumann Computing Architecture.

In-memory computing may make it possible to realize non-von Neumann computing because the logic circuits are unified in the memory units. We investigated two types of in-memory logic operations, namely, two-input logic circuits and multifunctional artificial synapses. These were realized in a dual-gate antiambipolar transistor (AAT) with a ReS2/WSe2 heterojunction, in which polystyrene with a zinc phthalocyanine core (ZnPc-PS4) was incorporated as a memory layer. Here, reconfigurability is a key concept for both types of device operations and was achieved by merging the Λ-shaped transfer curve of the AAT and the nonvolatile memory effect of ZnPc-PS4. First, we achieved electrically reconfigurable two-input logic circuits. Versatile logic circuits such as AND, OR, NAND, NOR, and XOR circuits were demonstrated by taking advantage of the Λ-shaped transfer curve of the dual-gate AAT. Importantly, the nonvolatile memory function provided the electrical switching of the individual circuits between AND/OR, NAND/NOR, and XOR/NAND circuits with constant input signals. Second, the memory effect was applied to multifunctional artificial synapses. The inhibitory/excitatory and long-term potentiation/depression synaptic operations were electrically reconfigured simply by controlling one parameter (readout voltage), making three distinct responses possible even with the same presynaptic signals. These findings provide hints that may lead to the realization of new in-memory computing architectures beyond the current von Neumann computers.

InfoCAVB-MemoryFormer: Forecasting of wind and photovoltaic power through the interaction of data reconstruction and data augmentation

Rare or missing data pose significant challenges in the prediction of wind power (WP) and photovoltaic power (PV). Many methods address the data scarcity issue solely through augmentation techniques, often neglecting the impact of missing data on the augmentation process. When data augmentation is performed on missing datasets, the prediction accuracy cannot be further improved, and this is called as an extreme data scarcity problem. To solve this problem, we introduce two methods, called Information Maximizing Collaborative Adversarial Variational Bayes (InfoCAVB) and MemoryFormer, to achieve data augmentation based on missing data reconstruction. In this paper, the augmentation and reconstruction process are performed at the same time. When the reconstruction process is performed, InfoCAVB utilizes adversarial training to construct variational bayes that approximates the posterior distribution of real data to recover missing data. Meanwhile, in the augmentation process, InfoCAVB maximizes the mutual information between real data and reconstruction data to establish correspondences between specific dimensions of augmentation data and the features of real and reconstruction data, respectively. The advantage of InfoCAVB lies in incorporating data augmentation into data reconstruction through the information maximizing principle. Finally, we propose a MemoryFormer adapted for InfoCAVB to predict WP and PV, and the benefit of MemoryFormer lies in excavating the potential temporal correlations between reconstructed and augmented data. MemoryFormer embeds the distribution of real data into the generated data through memory units, ensuring consistent distribution during the training process of discrete reconstructed and augmented data. Experimental results indicate that the proposed InfoCAVB-MemoryFormer reduces the RMSE averages by 15.7%, 14.1%, and 5.92% for WP prediction and 28.23%, 22.67%, and 12.21% for PV prediction compared to other state-of-the-art model models, demonstrating the effectiveness of the proposed approach in extreme data scarcity scenarios.

Random Projection‐Based Locality‐Sensitive Hashing in a Memristor Crossbar Array with Stochasticity for Sparse Self‐Attention‐Based Transformer

AbstractSelf‐attention mechanism is critically central to the state‐of‐the‐art transformer models. Because the standard full self‐attention has quadratic complexity with respect to the input's length L, resulting in prohibitively large memory for very long sequences, sparse self‐attention enabled by random projection (RP)‐based locality‐sensitive hashing (LSH) has recently been proposed to reduce the complexity to O(L log L). However, in current digital computing hardware with a von Neumann architecture, RP, which is essentially a matrix multiplication operation, incurs unavoidable time and energy‐consuming data shuttling between off‐chip memory and processing units. In addition, it is known that digital computers simply cannot generate provably random numbers. With the emerging analog memristive technology, it is shown that it is feasible to harness the intrinsic device‐to‐device variability in the memristor crossbar array for implementing the RP matrix and perform RP‐LSH computation in memory. On this basis, sequence prediction tasks are performed with a sparse self‐attention‐based Transformer in a hybrid software‐hardware approach, achieving a testing accuracy over 70% with much less computational complexity. By further harnessing the cycle‐to‐cycle variability for multi‐round hashing, 12% increase in the testing accuracy is demonstrated. This work extends the range of applications of memristor crossbar arrays to the state‐of‐the‐art large language models (LLMs).