Incomplete Samples Research Articles

Tensorized topological graph learning for generalized incomplete multi-view clustering

The success of the current multi-view clustering lies in the default assumption of completeness on each view, while it is hardly satisfied for real-world applications. Incomplete multi-view clustering (IMC) studies clustering multi-view instances that are partially observed due to data corruption or sensor failure. Although making some progress, the current research suffers from the following obstacles: (1) Existing works are always built on pairwise similarity measurement, which cannot fully capture the topological structure of incomplete multi-view samples for precise clustering; (2) Due to the absence of partially-observed samples across multiple views, how could we recover and enhance the high-order similarities across different views becomes one of the main bottlenecks of IMC; (3) Current research mainly aims to handle specific dual-view IMC, which greatly limits the deployment in real-world applications. In this paper, we propose a novel Tensorized Topological Graph Learning (TTGL) for the generalized incomplete multi-view clustering, which jointly considers the topological graph construction, missing feature completion, and high-order uncertain correlation enhancement. Specifically, instead of using pairwise similarity measurement, we formulate a consensus topological graph construction module, which can coalesce affinity matrices to analyze the topological structure of incomplete multi-view data. Moreover, we build an iterative missing similarity imputation scheme for feature completion, which ensures the intrinsic similarity preservation on observed instances as well as an imputation on unobserved ones. Furthermore, a low-rank tensor structure is imposed to capture the high-order similarities via both feature and similarity enhancement and completion across different views. Our method can cluster incomplete data with an arbitrary number of views and any missing statuses. Extensive experiments on several benchmark datasets validate the effectiveness of our method when compared with a series of state-of-the-art IMC clustering algorithms. The source code of our TTGL is available at: https://github.com/DarrenZZhang/TTGL.

Effects of misrepresentative length samples on individual growth and stock condition estimates

Despite its importance in fisheries studies, there is insufficient understanding on the effect of sampling error or bias on individual growth and other stock indicators. We show the influence of sample length distributions on parameter estimates, illustrating with an example. For the brown swimming crab, we simulated length samples in five configurations and estimated parameters of von Bertalanffy (k, L∞L∞ , t0), asymptotic weight ( W∞W∞ ), weight-length relationship (a, b), growth performance (ϕ’) and condition factor (Kn). Parameter estimates were compared with baseline values using relative bias, standard error and root mean square error. The results show that the accuracy and bias of parameter estimates depend on the lengths sampled. For example, the bias and accuracy of L∞L∞ and W∞W∞ vary inversely with sampled length, whereas combining length segments yields smaller biases of k and t0 than those of L∞L∞ and W∞W∞ . In general, the accuracy of parameter estimates does not always depend on sampling the entire length range, and errors are not the same for all parameters. These results are useful to guide sampling when resources are scarce. We discuss potential reasons for incomplete length sample structure and offer recommendations to obtain best estimates for parameters of interest.

Statistics-based method for large-scale group decision-making with incomplete linguistic distribution fuzzy information: Incorporating reliability and entropy

Aiming at large-scale decision-makers (DMs) and the loss of decision sample information in large-scale group decision-making (LSGDM), a novel statistical estimation method incorporating the reliability and entropy of linguistic distribution assessment is proposed. First, classify the large-scale DMs into several subgroups according to their prior decision efficiency distribution. After clustering, collect the five-number summary of the incomplete decision sample information provided by the DMs in the subgroups. Second, estimate the mean, standard deviation, skewness and kurtosis of the decision sample via the Cornish–Fisher expansion. Then utilize the Bayes estimation to address the reliability of the subgroups, thereby obtaining the confidence interval, which is used to develop the interval linguistic distribution preference relation (ILDPR) for the subgroups. Moreover, combine the reliability and the entropy measure constructed by the above four estimators to determine subgroup weights. Furthermore, present the expectation and variance of the ILDPR to sort the alternatives. Finally, demonstrate the feasibility and validity of the proposed LSGDM method based on a case and a comparison.

DDLSNet: A Novel Deep Learning-Based System for Grading Funduscopic Images for Glaucomatous Damage

To report an image analysis pipeline, DDLSNet, consisting of a rim segmentation (RimNet) branch and a disc size classification (DiscNet) branch to automate estimation of the disc damage likelihood scale (DDLS). Retrospective observational. RimNet and DiscNet were developed with 1208 and 11536 optic disc photographs (ODPs), respectively. DDLSNet performance was evaluated on 120 ODPs from the RimNet test set, for which the DDLS scores were graded by clinicians. Reproducibility was evaluated on a group of 781 eyes, each with 2 ODPs taken within 4years apart. Disc damage likelihood scale calculation requires estimation of optic disc size, provided by DiscNet (VGG19 network), and the minimum rim-to-disc ratio (mRDR) or absent rim width (ARW), provided by RimNet (InceptionV3/LinkNet segmentation model). To build RimNet's dataset, glaucoma specialists marked optic disc rim and cup boundaries on ODPs. The "ground truth" mRDR or ARW was calculated. For DiscNet's dataset, corresponding OCT images provided "ground truth" disc size. Optic disc photographs were split into 80/10/10 for training, validation, and testing, respectively, for RimNet and DiscNet. DDLSNet estimation was tested against manual grading of DDLS by clinicians with the average score used as "ground truth." Reproducibility of DDLSNet grading was evaluated by repeating DDLS estimation on a dataset of nonprogressing paired ODPs taken at separate times. The main outcome measure was a weighted kappa score between clinicians and the DDLSNet pipeline with agreement defined as ± 1 DDLS score difference. RimNet achieved an mRDR mean absolute error (MAE) of 0.04 (± 0.03) and an ARW MAE of 48.9 (± 35.9) degrees when compared to clinician segmentations. DiscNet achieved 73% (95% confidence interval [CI]: 70%, 75%) classification accuracy. DDLSNet achieved an average weighted kappa agreement of 0.54 (95% CI: 0.40, 0.68) compared to clinicians. Average interclinician agreement was 0.52 (95% CI: 0.49, 0.56). Reproducibility testing demonstrated that 96% of ODP pairs had a difference of 1 DDLS score. DDLSNet achieved moderate agreement with clinicians for DDLS grading. This novel approach illustrates the feasibility of automated ODP grading for assessing glaucoma severity. Further improvements may be achieved by increasing the number of incomplete rims sample size, expanding the hyperparameter search, and increasing the agreement of clinicians grading ODPs.

The effects of ketamine on symptoms of depression and anxiety in real-world care settings: A retrospective controlled analysis

IntroductionKetamine intravenous therapy (KIT) appears effective for treating depression in controlled trials testing a short series of infusions. A rapidly proliferating number of clinics offer KIT for depression and anxiety, using protocols without a strong evidence basis. Controlled comparison of mood and anxiety from real-world KIT clinics, and the stability of outcomes, is lacking. MethodsWe performed a retrospective controlled analysis on patients treated with KIT in ten community clinics across the US, between 08/2017‐03/2020. Depression and anxiety symptoms were evaluated using the Quick Inventory of Depressive Symptomatology-Self Report 16-item (QIDS) and the Generalized Anxiety Disorder 7-item (GAD-7) scales, respectively. Comparison data sets from patients who did not undergo KIT were obtained from previously published real-world studies. ResultsOf 2758 patients treated, 714 and 836 met criteria for analysis of KIT induction and maintenance outcomes, respectively. Patients exhibited significant and concordant reduction in both anxiety and depression symptoms after induction (Cohen's d = -1.17 and d = -1.56, respectively). Compared to two external datasets of KIT-naive depressed patients or patients starting standard antidepressant therapy, KIT patients experienced a significantly greater reduction in depression symptoms at eight weeks (Cohen's d = -1.03 and d = -0.62 respectively). Furthermore, we identified a subpopulation of late-responders. During maintenance, up to a year post-induction, increases in symptoms were minimal. LimitationsDue to the retrospective nature of the analyses, interpreting this dataset is limited by incomplete patient information and sample attrition. ConclusionsKIT treatment elicited robust symptomatic relief that remained stable up to one year of follow-up.

A Machine Learning Methodology for Diagnosing Chronic Kidney Disease

Chronic kidney disease (CKD) is a global health problem with high morbidity and mortality rate, and it induces other diseases. Since there are no obvious symptoms during the early stages of CKD, patients often fail to notice the disease. Early detection of CKD enables patients to receive timely treatment to ameliorate the progression of this disease. Machine learning models can effectively aid clinicians achieve this goal due to their fast and accurate recognition performance. In this study, we propose a machine learning methodology for diagnosing CKD. The CKD data set was obtained from the University of California Irvine (UCI) machine learning repository, which has a large number of missing values. KNN imputation was used to fill in the missing values, which selects several complete samples with the most similar measurements to process the missing data for each incomplete sample. Missing values are usually seen in real-life medical situations because patients may miss some measurements for various reasons. After effectively filling out the incomplete data set, six machine learning algorithms (logistic regression, random forest, support vector machine, k-nearest neighbor, naive Bayes classifier and feed forward neural network) were used to establish models. Among these machine learning models, random forest achieved the best performance with 99.75% diagnosis accuracy. By analyzing the misjudgments generated by the established models, we proposed an integrated model that combines logistic regression and random forest by using perceptron, which could achieve an average accuracy of 99.83% after ten times of simulation. Hence, we speculated that this methodology could be applicable to more complicated clinical data for disease diagnosis.

Acral BRAF‐mutated tubular adenoma should be distinguished from HPV42‐related digital papillary adenocarcinoma

Acral <i>BRAF</i>‐mutated tubular adenoma should be distinguished from <scp>HPV42</scp>‐related digital papillary adenocarcinoma

Rapid detection of insect pest with CRISPR‐Cas13a‐Based Lateral Flow Strip: Using Locusta migratoria manilensis

AbstractThe Clustered Regularly Interspaced Short Palindromic Repeats‐associated (CRISPR‐Cas) system has been used as a powerful diagnostic tool because of its trans‐cleavage ability to accurately recognize and cleave specific nucleic acid targets. However, its application in insect pest detection has rarely been reported. The aim of the current study was to develop a new molecular identification method for insect based on CRISPR‐Cas13a system. In this study, we have developed a highly specific and sensitive detection system that combines polymerase chain reaction, CRISPR‐Cas13a system and lateral flow dipstick (PCR‐Cas13a‐LFD) to rapidly detect the presence of Locusta migratoria manilensis (Orthoptera: Acrididae) DNA and discriminate it from other species, by using material from different stages of development. A pair of specific primers and crRNA were designed based on the Gene DEAD‐box DDX10 mRNA sequence. For visual readout, a FAM‐RNA‐biotin reporter was used for detection. The CRISPR‐Cas13a‐based detection method utilizes the visual identification of L. migratoria manilensis at different developmental stages or with the use of incomplete samples. The test results could be distinguished within 2 h. The sensitivity of the PCR‐Cas13a‐LFD detection system reached 0.1 ng/μL, a 10‐fold increase compared with that of the PCR with electrophoresis. The LFD coupled with CRISPR‐Cas13a (Cas13a‐LFD) is promising for the portable, onsite and fast identification of insect pests. This study provides a reference for the development of a new method for insect identification.

A novel missing value imputation relying on K-means clustering and kernel-based weighting using grey relation (KWGI)

Data pre-processing is one of the crucial phases of data mining that enhances the efficiency of data mining techniques. One of the most important operations performed on data pre-processing is missing values imputation in incomplete datasets. This research presents a new imputation technique using K-means and samples weighting mechanism based on Grey relation (KWGI). The Grey-based K-means algorithm applicable to all samples of incomplete datasets clusters the similar samples, then an appropriate kernel function generates appropriate weights based on the Grey relation. The missing values estimation of the incomplete samples is done based on the weighted mean to reduce the impact of outlier and vague samples. In both clustering and imputation steps, a penalty mechanism has been considered to reduce the similarity of ambiguous samples with a high number of missing values, and consequently, increase the accuracy of clustering and imputation. The KWGI method has been applied on nine natural datasets with eight state-of-the-art and commonly used methods, namely CMIWD, KNNI, HotDeck, MeanI, KmeanI, RKmeanI, ICKmeanI, and FKMI. The imputation results are evaluated by the Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) criteria. In this study, the missing values are generated at two levels, namely sample and value, and the results are discussed in a wide range of missingness from low rate to high rate. Experimental results of the t-test show that the proposed method performs significantly better than all the other compared methods.

METHODOLOGY FOR CLASSIFYING MONEY LAUNDERING RISKS TYPOLOGIES

The method of classifying the typologies of money laundering and terrorist financing risks in big data from a variety of data sources of organizational systems is investigated. The main provisions of this technique are theoretically substantiated, reference models and calculation formulas are given. An algebraic approach for the synthesis of the method is proposed, a theoretical justification of the feasibility of its use is given. The algebraic approach to the synthesis of the methodology is implemented on the basis of category theory and the theory of algorithms, in particular the theory of Markov algorithms, which is considered outside of normal algorithms. The N-scheme of the Markov algorithm is considered, which was developed to replace its well-known g-scheme and provides processing of words in natural, artificial and combined languages, which can be represented in an ordered and extended Markov alphabet A±2, which takes into account many sources of big data. Methods for inputting incomplete, heterogeneous (heterogeneous) and non-representative input data samples are given. A method is proposed for converting input data samples into cycles of the Big Data Transfer Protocol. A reference model of the legal economic activity of the organizational system has been synthesized. This reference model is subjected to analysis and signs of illegal economic activity of mutually interacting organizational systems are highlighted. A reference model of the system for classifying the risks of laundering proceeds from crime and financing of terrorism has been developed. To combat money laundering, the N-scheme of the Markov algorithm for one data source and the Markov system, consisting of such schemes, for a variety of big data sources are theoretically substantiated. It is concluded that the synthesized Markov system provides a decomposition of words and their morphisms, contains a common data input control channel, and provides a classification of money laundering risks.

PHANGS-MUSE: Detection and Bayesian classification of ~40 000 ionised nebulae in nearby spiral galaxies

In this work, we present a new catalogue of &gt;40 000 ionised nebulae distributed across the 19 galaxies observed by the PHANGS-MUSE survey. The nebulae have been classified using a new model-comparison-based algorithm that exploits the odds ratio principle to assign a probabilistic classification to each nebula in the sample. The resulting catalogue is the largest catalogue containing complete spectral and spatial information for a variety of ionised nebulae available so far in the literature. We developed this new algorithm to address some of the main limitations of the traditional classification criteria, such as their binarity, the sharpness of the involved limits, and the limited amount of data they rely on for the classification. The analysis of the catalogue shows that the algorithm performs well when selecting H II regions. In fact, we can recover their luminosity function, and its properties are in line with what is available in the literature. We also identify a rather significant population of shock-ionised regions (mostly composed of supernova remnants), which is an order of magnitude larger than any other homogeneous catalogue of supernova remnants currently available in the literature. The number of supernova remnants we identify per galaxy is in line with results in our Galaxy and in other very nearby sources. However, limitations in the source detection algorithm result in an incomplete sample of planetary nebulae, even though their classification seems robust. Finally, we demonstrate how applying a correction for the contribution of the diffuse ionised gas to the nebulae’s spectra is essential to obtain a robust classification of the objects and how a correct measurement of the extinction using diffuse-ionised-gas-corrected line fluxes prompts the use of a higher theoretical Hα/Hβ ratio (3.03) than what is commonly used when recovering the E(B – V) via the Balmer decrement technique in massive star-forming galaxies.

A Statistical Study of a Large Sample of Changing-look Active Galactic Nuclei with Multi-frequency Radio Sky Surveys

Optically “changing-look” active galactic nuclei (AGNs) are a class of objects that exhibit appearance or disappearance of broad lines. Recent studies have shown that these “changing-look” activities very likely result from the accretion activities of the central supermassive black holes. Large variation of the accretion rate might cause jet ejection and fading events, thus lead to a certain variability of observational radio properties. From the published literature, 74 “changing-look” AGNs and 90 “changing-look” AGNs candidates identified in optical observations were collected. Based on this largest, complex, and incomplete sample to date, the radio properties of “changing-look” AGNs were investigated. The radio counterparts of 51 “changing-look” AGNs (including 21 candidates) were discovered from the all-sky surveys conducted by the Australian Square Kilometre Array Pathfinder (ASKAP) and Very Large Array (VLA) covering the frequency range of 0.9–3 GHz. This corresponds to a detection rate of about 41 percent, which is similar to general AGNs. The radio spectral indices between 0.9 and 1.4 GHz and between 1.4 and 3 GHz were calculated, which suggest that these “changing-look” AGNs tend to have flatter radio spectra than normal AGNs. This statistical result indicates that “changing-look” AGNs might host some young, compact, and faint radio jets.

Mechanical Equipment Performance Degradation Assessment Method Based on Improved Degradation Feature Extraction and Sample Completeness

Performance degradation assessment (PDA), as an important part of health management, is playing a crucial role in evaluating the degenerate state of mechanical equipment. To evaluate the performance degradation state more effectively by using monitoring data of mechanical equipment, a PDA method based on improved degradation feature extraction technology and considering sample completeness is proposed in this paper. Firstly, multiple degradation features, including statistical features and intrinsic energy features, are extracted to construct a high-dimensional feature set calculated by time-domain analysis and the EMD method. Then, a sensitive feature set, which has significant robustness, correlation, and monotonicity in the degenerate process, is reduced and processed by the improved PCA method to obtain the final health index. Secondly, the degradation assessment model is established by considering the completeness of the samples. As for the complete sample dataset, which has both normal and failure state data, a logistic regression model (LRM) is built to assess the performance degradation status. In addition, an SVDD model is established for the incomplete sample dataset, which only has normal state data. Finally, the reliability of the proposed method is verified by the public XJTU-SY dataset, and the comparative experiment further verifies the superiority of this method.

Highly-sensitive label-free deep profiling of N-glycans released from biomedically-relevant samples

Alterations of protein glycosylation can serve as sensitive and specific disease biomarkers. Labeling procedures for improved separation and detectability of oligosaccharides have several drawbacks, including incomplete derivatization, side-products, noticeable desialylation/defucosylation, sample loss, and interference with downstream analyses. Here, we develop a label-free workflow based on high sensitivity capillary zone electrophoresis-mass spectrometry (CZE-MS) for profiling of native underivatized released N-glycans. Our workflow provides a >45-fold increase in signal intensity compared to the conventional CZE-MS approaches used for N-glycan analysis. Qualitative and quantitative N-glycan profiling of purified human serum IgG, bovine serum fetuin, bovine pancreas ribonuclease B, blood-derived extracellular vesicle isolates, and total plasma results in the detection of >250, >400, >150, >310, and >520 N-glycans, respectively, using injected amounts equivalent to <25 ng of model protein and nL-levels of plasma-derived samples. Compared to reported results for biological samples of similar amounts and complexity, the number of identified N-glycans is increased up to ~15-fold, enabling highly sensitive analysis of sample amounts as low as sub-0.2 nL of plasma volume equivalents. Furthermore, highly sialylated N-glycans are identified and structurally characterized, and untreated sialic acid-linkage isomers are resolved in a single CZE-MS analysis.

Fault diagnosis for rotor based on multi-sensor information and progressive strategies

Fault diagnosis is an effective tool to ensure safe operation of machinery and avoid serious accidents. As most currently used fault diagnosis methods usually employ mapping relationship established by training samples and their labels to achieve classification of testing samples, it is difficult for them to achieve fault diagnosis under the condition of incomplete training sample types. In addition, previous studies usually focus on feature extraction of single-channel vibration signal, which cannot get complete fault feature information. To solve the above problems, a progressive fault diagnosis method is investigated in this paper. First, the preliminary fault detection for the rotor is performed by studying reconstruction error of a sparse auto-encoder. Second, if a fault exists in the rotor, the outlier detection is implemented by the support vector data description method. Finally, if there are no outlier samples, the well -trained support vector machine is used to confirm the type of fault samples and complete the diagnosis. The performance of the proposed method was verified using the data obtained from a rotor laboratory bench. The types of rotor states investigated include normal, contact-rubbing, unbalance and misalignment. The experimental results verify the effectiveness and superiority of the proposed method in reducing the incidents of fault omission and fault misunderstanding.

Comparing avian species richness estimates from structured and semi-structured citizen science data

Citizen science, including structured and semi-structured forms, has become a powerful tool to collect biodiversity data. However, semi-structured citizen science data have been criticized for higher variability in quality, including less information to adjust for imperfect detection and uneven duration that bias the estimates of species richness. Species richness estimators may quantify bias in estimates. Here, we test the effectiveness of Chao1 estimator in eBird (semi-structured) by comparing it to averaged species richness in Breeding Bird Survey Taiwan, BBS (structured) and quantifying bias. We then fit a power function to compare bias while controlling for differences in count duration. The Chao1 estimator increased the species richness estimates of eBird data from 56 to 69% of the average observed BBS and from 47 to 59% of the average estimated BBS. Effects of incomplete short duration samples and variability in detectability skills of observers can lead to biased estimates. Using the Chao1 estimator improved estimates of species richness from semi-structured and structured data, but the strong effect of singleton species on bias, especially in short duration counts, should be evaluated in advance to reduce the uncertainty of estimation processes.

A novel fault diagnosis scheme for a ballast system

The Back Propagation (BP) neural network model is one of the neural network models suitable for model recognition and classification. However, a single BP network model cannot fully tap the feature information hiding in the data and is vulnerable to the uncertainty associated with system parameter changes and also suffers from the effects of long diagnosis times, inaccurate and undesired diagnosis results in applying a large amount of high-dimensional feature data. In addition, the standard BP algorithm has limitations in pattern recognition and classification. For example, it adopts the quadratic cost function to train the neural network, which can lead to low convergence accuracy, low convergence rate, and so on. Compared with the quadratic cost function, the cross-entropy cost function can effectively improve the convergence accuracy and the convergence rate of the neural network. The Dempster–Shafer (D–S) evidence theory is a powerful tool for handling incomplete, imprecise, and uncertain information and merging it when it derives from multiple BP networks. In order to overcome the drawbacks of a single BP network, it is necessary to divide the high-dimensional feature data into several different categories of low-dimensional feature data and to input the same dimensional feature data into the same BP network model, as well as to design the information fusion algorithms based on D–S evidence theory and multiple improved BP networks with the cross-entropy cost function. For this reason, this paper proposes a novel information fusion diagnosis approach based on D–S evidence theory and multiple improved BP networks for fault diagnosis and classification of multiple devices in a ballast system subjected to extreme environmental conditions. The proposed diagnosis approach utilizes multiple neural networks to carry out local fault diagnosis, adopts the D–S evidence theory to fuse the results of local diagnosis and derive the decision results, and introduces an improved BP algorithm based on the cross-entropy cost function into the individual neural networks to enhance the convergence rate and diagnosis accuracy of the individual neural networks. This method can overcome the deficiencies of individual BP networks in multiple BP networks in their poor diagnostic ability to handle incomplete, imprecise, and uncertain sample information. According to the design idea of the proposed diagnosis approach, a new fusion modeling for fault diagnosis of multiple devices is constructed for a ballast system. The validity of the proposed diagnosis method and model is illustrated through both simulation studies and experimental tests.

Numerical Model Driving Multi-Domain Information Transfer Method for Bearing Fault Diagnosis

Given the complexity of the application scenarios of rolling bearing and the severe scarcity of fault samples, a solution to the issue of fault diagnosis under varying working conditions along with the absence of fault samples is required. A numerical model-driven cross-domain fault diagnosis method targeting variable working conditions is proposed based on the cross-Domain Nuisance Attribute Projection (cDNAP). Firstly, the simulation datasets consisting of multiple fault types under variable working conditions are constructed to solve the problem of incomplete fault samples. Secondly, the simulation datasets are expanded by means of generating adversarial network to ensure sufficient samples for subsequent model training. Finally, cDNAP is used to obtain the cross-domain simulation projection matrix, which eliminates the variance in the distribution of measured and simulated sample features under varying working conditions. The experimental results of cross-domain for variable working conditions show that the diagnostic accuracy reaches up to 99%. Compared with DANN, DSAN, and DAAN domain adversarial neural networks, the proposed method performs better in bearing fault diagnosis.

Missing data estimation based on the chaining technique in survey sampling

Abstract Sample surveys are often affected by missing observations and non-response caused by the respondents’ refusal or unwillingness to provide the requested information or due to their memory failure. In order to substitute the missing data, a procedure called imputation is applied, which uses the available data as a tool for the replacement of the missing values. Two auxiliary variables create a chain which is used to substitute the missing part of the sample. The aim of the paper is to present the application of the Chain-type factor estimator as a means of source imputation for the non-response units in an incomplete sample. The proposed strategies were found to be more efficient and bias-controllable than similar estimation procedures described in the relevant literature. These techniques could also be made nearly unbiased in relation to other selected parametric values. The findings are supported by a numerical study involving the use of a dataset, proving that the proposed techniques outperform other similar ones.

Recent Advances toward Efficient Calculation of Higher Nuclear Derivatives in Quantum Chemistry.

In this paper, we provide an overview of state-of-the-art techniques that are being developed for efficient calculation of second and higher nuclear derivatives of quantum mechanical (QM) energy. Calculations of nuclear Hessians and anharmonic terms incur high costs and memory and scale poorly with system size. Three emerging classes of methods─machine learning (ML), automatic differentiation (AD), and matrix completion (MC)─have demonstrated promise in overcoming these challenges. We illustrate studies that employ unsupervised ML methods to reduce the need for multiple Hessian calculations in dynamics simulations and those that utilize supervised ML to construct approximate potential energy surfaces and estimate Hessians and anharmonic terms at reduced cost. By extension, if electronic structure operations could be written in a manner similar to functions underlying ML methods, rapid differentiation or AD routines can be employed to inexpensively calculate higher arbitrary-order derivatives. While ML approaches are typically black-box, we describe methods such as compressed sensing (CS) and MC, which explicitly leverage problem-specific mathematical properties of higher derivatives such as sparsity and low-rank, to complete higher derivative information using only a small, incomplete sample. The three classes of methods facilitate reliable predictions of observables ranging from infrared spectra to thermal conductivity and constitute a promising way forward in accurately capturing otherwise intractable higher-order responses of QM energy to nuclear perturbations.