Stochastic Context-free Grammars Research Articles

Hidden Abstract Stack Markov Models with Learning Process

We present hidden abstract stack Markov models (HASMMs) with their learning process. The HASMMs we offer carry the more expressive nature of probabilistic context-free grammars (PCFGs) while allowing faster parameter fitting of hidden Markov models (HMMs). Both HMMs and PCFGs are widely utilized structured models, offering an effective formalism capable of describing diverse phenomena. PCFGs are better accommodated than HMMs such as for expressing natural language processing; however, HMMs outperform PCFGs for parameter fitting. We extend HMMs towards PCFGs for such applications, by associating each state of an HMM with an abstract stack, which can be thought of as the single-stack alphabet of pushdown automata (PDA). As a result, we leverage the expressive capabilities of PCFGs for such applications while mitigating the cubic complexity of parameter learning in the observation sequence length of PCFGs by adopting the bilinear complexity of HMMs.

Probabilistic grammars for modeling dynamical systems from coarse, noisy, and partial data

Ordinary differential equations (ODEs) are a widely used formalism for the mathematical modeling of dynamical systems, a task omnipresent in scientific domains. The paper introduces a novel method for inferring ODEs from data, which extends ProGED, a method for equation discovery that allows users to formalize domain-specific knowledge as probabilistic context-free grammars and use it for constraining the space of candidate equations. The extended method can discover ODEs from partial observations of dynamical systems, where only a subset of state variables can be observed. To evaluate the performance of the newly proposed method, we perform a systematic empirical comparison with alternative state-of-the-art methods for equation discovery and system identification from complete and partial observations. The comparison uses Dynobench, a set of ten dynamical systems that extends the standard Strogatz benchmark. We compare the ability of the considered methods to reconstruct the known ODEs from synthetic data simulated at different temporal resolutions. We also consider data with different levels of noise, i.e., signal-to-noise ratios. The improved ProGED compares favourably to state-of-the-art methods for inferring ODEs from data regarding reconstruction abilities and robustness to data coarseness, noise, and completeness.

Robustness of the random language model.

The random language model [Phys. Rev. Lett. 122, 128301 (2019)0031-900710.1103/PhysRevLett.122.128301] is an ensemble of stochastic context-free grammars, quantifying the syntax of human and computer languages. The model suggests a simple picture of first-language learning as a type of annealing in the vast space of potential languages. In its simplest formulation, it implies a single continuous transition to grammatical syntax, at which the symmetry among potential words and categories is spontaneously broken. Here this picture is scrutinized by considering its robustness against extensions of the original model, and trajectories through parameter space different from those originally considered. It is shown here that (i) the scenario is robust to explicit symmetry breaking, an inevitable component of learning in the real world, and (ii) the transition to grammatical syntax can be encountered by fixing the deep (hidden) structure while varying the surface (observable) properties. It is also argued that the transition becomes a sharp thermodynamic transition in an idealized limit. Moreover, comparison with human data on the clustering coefficient of syntax networks suggests that the observed transition is equivalent to that normally experienced by children at age 24 months. The results are discussed in light of the theory of first-language acquisition in linguistics, and recent successes in machine learning.

Estimation of Distribution Algorithm for Grammar-Guided Genetic Programming.

Genetic variation operators in grammar-guided genetic programming are fundamental to guide the evolutionary process in search and optimization problems. However, they show some limitations, mainly derived from an unbalanced exploration and local-search trade-off. This article presents an estimation of distribution algorithm for grammar-guided genetic programming to overcome this difficulty and thus increase the performance of the evolutionary algorithm. Our proposal employs an extended dynamic stochastic context-free grammar to encode and calculate the estimation of the distribution of the search space from some promising individuals in the population. Unlike traditional estimation of distribution algorithms, the proposed approach improves exploratory behavior by smoothing the estimated distribution model. Therefore, this algorithm is referred to as SEDA, smoothed estimation of distribution algorithm. Experiments have been conducted to compare overall performance using a typical genetic programming crossover operator, an incremental estimation of distribution algorithm, and the proposed approach after tuning their hyperparameters. These experiments involve challenging problems to test the local search and exploration features of the three evolutionary systems. The results show that grammar-guided genetic programming with SEDA achieves the most accurate solutions with an intermediate convergence speed.

Exploring the Frontier of Password Cracking: Methods, Effectiveness, and Defense Strategies

The manner of attacker's behavior cannot be underestimated; hackers use simple traditional attacks such as brute-force and dictionary attacks as well as sophisticated algorithms including: Markov models, probabilistic context-free grammars (PCFG), and generative adversarial networks (GANs). These are one of among the most advanced approaches which utilize artificial intelligence and machine learning to identify the patterns in passwords, guess them and crack them. Markov models calculate transitions from one character state to another, so they estimate password guesses as a probability which is sampled from the distribution. PCFGs further advance the concept by making use of context-specific inputs for producing the passwords, and as a result it is possible to come up with the candidates who are balanced and have the contextually valid characters. Years ago, the CPA attack was considered the ultimate approach for password cracking. Today, GANs have taken their place, implementing adversarial networks that use them as generators to generate valid password examples. It is evident from the research that abusing users' habits and context during password cracking has been proved that it can lead to a tremendous speed gain of the cracking process. The crackers exploit here patterns in the behavior and environmental features and tailor the cracking strategies. Besides that, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) are considered to be good options in password modeling as well, where the ongoing researches are devoted to the structures of neural network so that the guessing powers of networks can improve. These new approaches have demonstrated an improvement of at least 10-15% over the outdated ones, thus they are credible in forming the password cracking paradigm shift. Thus, for defeating such sophisticated threats, companies should take into account such robust passphrase policies, teach the user about safety of passwords and implementation of rigid access mechanisms. Educating people on cyber threats basics and development of the reasonable cybersecurity culture are the factors that provide the impact reduction of attacks based on users’ behavior and contextual information.

Confidence-based interactable neural-symbolic visual question answering

Visual question answering (VQA) task demands proficiency in processing multi-modal information, and the ability to reason effectively using the information. One promising method for this task is neural-symbolic (NS) learning, which leverages the strengths of both neural network (NN) learning and symbolic reasoning to achieve efficient VQA. However, current NS approaches do not account for the uncertain nature of NN learning and can only provide a single answer to a question without any indication of its confidence, thereby limiting their ability to handle incorrect reasoning. To address this limitation, we propose a confidence-based neural-symbolic (CBNS) approach, which evaluates the confidence of the NN inferences based on uncertainty quantification and makes confidence-based reasoning. The proposed approach comprises three main components: (1) a probabilistic question parser that generates multiple program candidates, each with a corresponding confidence evaluation; (2) a probabilistic scene perception module that provides object-based scene representation and confidence evaluations for each attribute of objects in an image; and (3) a confidence-based program executor that provides answers with confidence evaluations throughout the inference process by leveraging the confidence evaluations of the scene representation and programs. Additionally, we present a data augmentation method to improve the training efficiency of NS learning. The proposed approach allows user interactions and feedback on the weak links based on confidence evaluations. Experiments on CLEVR and GQA datasets demonstrate that the proposed approach was effective in identifying the correctness of predictions and led to a promising performance improvement with a significantly reduced computation cost.

Coordinated Motion Planning for Heterogeneous Autonomous Vehicles Based on Driving Behavior Primitives

Heterogeneous autonomous vehicle (HAV) coordinated motion planning must guide each vehicle out of the conflict zone based on the differences in vehicle platform characteristics. Decomposing complex driving tasks into primitives is an effective way to improve algorithm efficiency. Hence, the purpose of this paper is to complete the coordinated motion planning tasks through offline driving behavior primitive (DBP) library generation, online extension and selection of DBPs. The proposed algorithm applies dynamic movement primitives and singular value decomposition to learn driving behavior patterns from driving data, integrates them into a model-based optimization generation method as constraints, and builds a DBP library by fusing driving data and vehicle model. Based on the generated DBP library and primitive association probabilities learned from labeled driving segments via stochastic context-free grammar, the planning method completes the independent DBP extension of each vehicle in the conflict zone, generates an interaction DBP tree, and uses the mixed-integer linear programming algorithm to optimally select the primitives to be executed. This study demonstrates that the generated DBP library not only expands the types of primitives, but also distinguishes the characteristics of HAVs. We also present how to utilize the DBP libraries to obtain coordinated motion planning results with spatiotemporal information in the form of DBP extension and selection. The results obtained by real vehicle platforms and simulation show that the proposed method can accomplish coordinated motion planning tasks without relying on specific scene elements and highlight the unique motion characteristics of HAVs.

An Arabic Probabilistic Parser Based on a Property Grammar

The specificities of Arabic parsing, such as agglutination, vocalization, and the relatively order-free words in Arabic sentences, remain major issues to consider. To promote its robustness, such parseing should define different types of constraints. Property Grammar (PG) formalism verifies the satisfiability of the constraints directly on the units of the structure, thanks to its properties (or relations). In this context, we propose to build a probabilistic parser with syntactic properties, using a PG, and we measure the production rules in terms of different implicit information and in particular the syntactic properties. We experimented with our parser on the treebank ATB, using the parsing algorithm CYK, and we obtained encouraging results. Our method is also automatic for implementation of most property types. Its generalization for other languages or corpus domains (using treebanks) could be a good perspective. Its combination with pre-trained models of BERT may also make our parser faster.

Using language-specific input methods and pronunciation rules to improve the guesses of passwords

Password-based authentication has the advantage of easy implementation and no requirement for additional hardware; therefore, it will remain one of the primary identification methods in the near future. However, users tend to choose passwords that are easy to remember, so these passwords may be vulnerable to guessing attacks. Based on the idea that some users are influenced by their native language when choosing passwords, we propose an improved password-guessing method that adds grammars regarding Asian-language input procedures and pronunciation rules to the most updated Probabilistic Context-Free Grammar (PCFG) v4.1. The experimental results show that the improved PCFG v4.1 can increase the success rate of password cracking compared to PCFG v4.1. The improvements range from 2% to 14% for different password datasets. Additionally, we compare the proposed method to many other guessing methods; ours can achieve an excellent performance. Moreover, the characteristics of passwords that are not cracked are analyzed, and we suggest some criteria for more robust passwords.

A syntactic method for robot imitation learning of complex sequence task

PurposeAiming at the problems of weak generalization of robot imitation learning methods and higher accuracy requirements of low-level detectors, this study aims to propose an imitation learning method based on structural grammar.Design/methodology/approachThe paper proposes a hybrid training model based on artificial immune algorithm and the Baum–Welch algorithm to extract the action information of the demonstration activity to form the {action-object} sequence and extract the symbol description of the scene to form the symbol primitives sequence. Then, probabilistic context-free grammar is used to characterize and manipulate these sequences to form a grammar space. Minimum description length criteria are used to evaluate the quality of the grammar in the grammar space, and the improved beam search algorithm is used to find the optimal grammar.FindingsIt is found that the obtained general structure can parse the symbol primitive sequence containing noise and obtain the correct sequence, thereby guiding the robot to perform more complex and higher-order demonstration tasks.Practical implicationsUsing this strategy, the robot completes the fourth-order Hanoi tower task has been verified.Originality/valueAn imitation learning method for robots based on structural grammar is first proposed. The experimental results show that the method has strong generalization ability and good anti-interference performance.

Discriminative estimation of probabilistic context-free grammars for mathematical expression recognition and retrieval

We present a discriminative learning algorithm for the probabilistic estimation of two-dimensional probabilistic context-free grammars (2D-PCFG) for mathematical expressions recognition and retrieval. This algorithm is based on a generalization of the H-criterion as the objective function and the growth transformations as the optimization method. For the development of the discriminative estimation algorithm, the N-best interpretations provided by the 2D-PCFG have been considered. Experimental results are reported on two available datasets: Im2Latex and IBEM. The first experiment compares the proposed discriminative estimation method with the classic Viterbi-based estimation method. The second one studies the performance of the estimated models depending on the length of the mathematical expressions and the number of admissible errors in the metric used.

ParsingPhrase: Parsing-based automated quality phrase mining

Phrases represent independent semantics in natural language but usually have indeterminate lengths and different combinations. So, extracting meaningful phrases from unstructured texts will substantially reduce semantic ambiguity and lay the foundation for downstream natural language tasks. Most existing research obtains candidate phrases by N-grams, which includes meaningless word sequences and degrades algorithm performance. In this paper, we propose a novel phrase-mining algorithm, called ParsingPhrase, which effectively extracts combination phrases from text and improves phrase quality by syntactic features. It consists of three stages. Firstly, all sentences in texts are represented as parsing trees by PCFG (Probabilistic Context-Free Grammar). We propose PBMP (Parsing-Based Phrase mining) to obtain candidate phrases from those parsing trees. Then, we introduce a new phrase evaluation indicator, called Significance, that relies on the role of phrases to measure their importance. We integrate the Significance with conventional evaluation indexes for a more reasonable phrase evaluation. Finally, we optimize the phrase quality again by exploiting the optimal phrase composition features of sentences. To the best of our knowledge, it is the first work to employ parsing for combination phrase mining and evaluation, meanwhile offering a solution for syntactic disambiguation. Experiments on three real corpora demonstrate that the ParsingPhrase exceeds state-of-the-art baselines, is 7% higher in the candidate phrase conversion rate, and is 6% better in terms of Precision.

Dimensionally-consistent equation discovery through probabilistic attribute grammars

Equation discovery, also known as symbolic regression, is a machine learning task of inducing closed-form equations from data and background knowledge. The latter takes various forms. Domain-specific knowledge can constrain the space of candidate equations to those that make sense in the scientific or engineering domain of use. Cross-domain knowledge, on the other hand, imposes general rules for model acceptability, such as parsimony, understandability, or consistency of the equations with the dimensional units of the variables. In this paper, we propose using attribute grammars to ensure the induced equations' dimensional consistency. Attribute grammars are flexible enough to combine cross-domain knowledge on dimensional consistency with domain-specific knowledge expressed as a probabilistic context-free grammar. At the same time, we show that attribute grammars can be efficiently transformed into probabilistic context-free grammars for equation discovery with existing algorithms. Finally, we provide empirical evidence that attribute grammars ensuring dimensional consistency of equations can significantly improve the performance of equation discovery on the standard set of a hundred Feynman benchmarks.

Learning of Structurally Unambiguous Probabilistic Grammars

The problem of identifying a probabilistic context free grammar has two aspects: the first is determining the grammar's topology (the rules of the grammar) and the second is estimating probabilistic weights for each rule. Given the hardness results for learning context-free grammars in general, and probabilistic grammars in particular, most of the literature has concentrated on the second problem. In this work we address the first problem. We restrict attention to structurally unambiguous weighted context-free grammars (SUWCFG) and provide a query learning algorithm for \structurally unambiguous probabilistic context-free grammars (SUPCFG). We show that SUWCFG can be represented using \emph{co-linear multiplicity tree automata} (CMTA), and provide a polynomial learning algorithm that learns CMTAs. We show that the learned CMTA can be converted into a probabilistic grammar, thus providing a complete algorithm for learning a structurally unambiguous probabilistic context free grammar (both the grammar topology and the probabilistic weights) using structured membership queries and structured equivalence queries. A summarized version of this work was published at AAAI 21.

LiDetector : License Incompatibility Detection for Open Source Software

Open-source software (OSS) licenses dictate the conditions, which should be followed to reuse, distribute, and modify software. Apart from widely-used licenses such as the MIT License, developers are also allowed to customize their own licenses (called custom license), whose descriptions are more flexible. The presence of such various licenses imposes challenges to understand licenses and their compatibility. To avoid financial and legal risks, it is essential to ensure license compatibility when integrating third-party packages or reusing code accompanied with licenses. In this work, we propose LiDetector , an effective tool that extracts and interprets OSS licenses (including both official licenses and custom licenses), and detects license incompatibility among these licenses. Specifically, LiDetector introduces a learning-based method to automatically identify meaningful license terms from an arbitrary license, and employs Probabilistic Context-Free Grammar (PCFG) to infer rights and obligations for incompatibility detection. Experiments demonstrate that LiDetector outperforms existing methods with 93.28% precision for term identification, and 91.09% accuracy for right and obligation inference, and can effectively detect incompatibility with 10.06% FP rate and 2.56% FN rate. Furthermore, with LiDetector , our large-scale empirical study on 1,846 projects reveals that 72.91% of the projects are suffering from license incompatibility, including popular ones such as the MIT License and the Apache License. We highlighted lessons learned from perspectives of different stakeholders and made all related data and the replication package publicly available to facilitate follow-up research.

Exploring a diverse world of effector domains and amyloid signaling motifs in fungal NLR proteins.

NLR proteins are intracellular receptors constituting a conserved component of the innate immune system of cellular organisms. In fungi, NLRs are characterized by high diversity of architectures and presence of amyloid signaling. Here, we explore the diverse world of effector and signaling domains of fungal NLRs using state-of-the-art bioinformatic methods including MMseqs2 for fast clustering, probabilistic context-free grammars for sequence analysis, and AlphaFold2 deep neural networks for structure prediction. In addition to substantially improving the overall annotation, especially in basidiomycetes, the study identifies novel domains and reveals the structural similarity of MLKL-related HeLo- and Goodbye-like domains forming the most abundant superfamily of fungal NLR effectors. Moreover, compared to previous studies, we found several times more amyloid motif instances, including novel families, and validated aggregating and prion-forming properties of the most abundant of them in vitro and in vivo. Also, through an extensive in silico search, the NLR-associated amyloid signaling was identified in basidiomycetes. The emerging picture highlights similarities and differences in the NLR architectures and amyloid signaling in ascomycetes, basidiomycetes and other branches of life.

#Segments: A Dominant Factor of Password Security to Resist against Data-driven Guessing

Understanding which factors dominate password security is vital for users to create their secure passwords. Prior works generally consider the password length and the number of character classes as the dominant factors. However, creating secure passwords based on the above two factors becomes much more challenging than before due to the emergence of powerful data-driven guessing methods, e.g., the Probabilistic Context-free Grammars (PCFG) and its variations, Markov-based methods, and neural-network-based methods. In this paper, inspired by the segments used in PCFG, where a segment is a continuous string whose characters have a strong correlation, we conduct a comprehensive empirical analysis and find that the number of segments (# Segments for short) is a dominant factor of password security to resist against data-driven guessing. That is, the increase of # Segments generally leads to a significant improvement of password security. The observation helps us explore an optimised identification method for segments, referred to as re-segment, which reduces # Segments as much as possible to obtain accurate # Segments by leveraging five popular patterns (i.e., keyboard, abbreviation, leet, mixture, and component), to evaluate password security more accurately from an adversary’s viewpoint. Then we propose an efficient data-driven guessing method, referred to as ReSeg-PCFG, by leveraging re-segment based on the latest version of PCFG. Our study shows that ReSeg-PCFG outperforms the state-of-the-art data-driven guessing methods in almost all scenarios; e.g., it outperforms the latest version of PCFG by up to 79.34% at 1014 guesses, a commonly used threshold of off-line attacks.

Understanding digit‐only financial account passwords: ID card, structure, and security

Password-based authentication is widely used in website access. Current studies on password security have focused considerable attention on letter-only passwords or passwords that include both letters and digits. However, little research has been conducted on commercial and digit-only passwords. In this paper, we comprehensively evaluate digit-only passwords by analyzing approximately 130,000 passwords from the 12306 data set. We observe that approximately 20% of six-digit passwords belonging to Chinese users include the user's identity document (ID) card information. Most of the passwords are found to be related to the user's birth dates. Additionally, we demonstrate that male users are more likely to set passwords with digits extracted from their ID card numbers than female users. It is the first time that the relation between real-user passwords and ID card series numbers was considered. It focuses on actual attack methodologies and real-user passwords, which renders this study an explicit study on digit-only password security. We propose a password-encryption authentication model based on FF3-1 to implement secure storage of passwords. We experiment with ciphertext and plaintext passwords using three–three and four–two structures combined with probabilistic context-free grammars. The cracking success rates of ciphertext passwords are found to be 0.086% and 1.06%, and they can recover 59.83% and 33.2% of the passwords in the test set. Conversely, using plaintext passwords results in the recovery of 98% and 85% of passwords in the test set. Our results demonstrate that the use of encryption for password storage can significantly reduce the cracking success rate of attackers.

ALP: Data Augmentation Using Lexicalized PCFGs for Few-Shot Text Classification

Data augmentation has been an important ingredient for boosting performances of learned models. Prior data augmentation methods for few-shot text classification have led to great performance boosts. However, they have not been designed to capture the intricate compositional structure of natural language. As a result, they fail to generate samples with plausible and diverse sentence structures. Motivated by this, we present the data Augmentation using Lexicalized Probabilistic context-free grammars (ALP) that generates augmented samples with diverse syntactic structures with plausible grammar. The lexicalized PCFG parse trees consider both the constituents and dependencies to produce a syntactic frame that maximizes a variety of word choices in a syntactically preservable manner without specific domain experts. Experiments on few-shot text classification tasks demonstrate that ALP enhances many state-of-the-art classification methods. As a second contribution, we delve into the train-val splitting methodologies when a data augmentation method comes into play. We argue empirically that the traditional splitting of training and validation sets is sub-optimal compared to our novel augmentation-based splitting strategies that further expand the training split with the same number of labeled data. Taken together, our contributions on the data augmentation strategies yield a strong training recipe for few-shot text classification tasks.

LPG-PCFG: An Improved Probabilistic Context- Free Grammar to Hit Low-Probability Passwords.

With the development of the Internet, information security has attracted more attention. Identity authentication based on password authentication is the first line of defense; however, the password-generation model is widely used in offline password attacks and password strength evaluation. In real attack scenarios, high-probability passwords are easy to enumerate; extremely low-probability passwords usually lack semantic structure and, so, are tough to crack by applying statistical laws in machine learning models, but these passwords with lower probability have a large search space and certain semantic information. Improving the low-probability password hit rate in this interval is of great significance for improving the efficiency of offline attacks. However, obtaining a low-probability password is difficult under the current password-generation model. To solve this problem, we propose a low-probability generator–probabilistic context-free grammar (LPG–PCFG) based on PCFG. LPG–PCFG directionally increases the probability of low-probability passwords in the models’ distribution, which is designed to obtain a degeneration distribution that is friendly for generating low-probability passwords. By using the control variable method to fine-tune the degeneration of LPG–PCFG, we obtained the optimal combination of degeneration parameters. Compared with the non-degeneration PCFG model, LPG–PCFG generates a larger number of hits. When generating and times, the number of hits to low-probability passwords increases by and , respectively.