Basic Local Independence Model Research Articles

Assessment-based correct rates in learning spaces

The basic local independence model (BLIM) is the standard probabilistic model in knowledge structure theory. It assumes that the probability of a correct response to a problem is constant for all individuals that master the problem, and accordingly, for all individuals that do not master it, irrespective of the mastering of other problems. Recently published data on the problem correct rate as inferred from a response-based assessment of the mastering of the problem seem to contradict this assumption. The analysis presented in this paper, however, reveals that deviations from constancy in the observed direction are to be expected under the BLIM. They are mainly due to the inaccuracy inherent in any response-based assessment. The implications of these results for the empirical validation of the BLIM are discussed.

Empirical indistinguishability: From the knowledge structure to the skills.

Recent literature has pointed out that the basic local independence model (BLIM) when applied to some specific instances of knowledge structures presents identifiability issues. Furthermore, it has been shown that for such instances the model presents a stronger form of unidentifiability named empirical indistinguishability, which leads to the fact that the existence of certain knowledge states in such structures cannot be empirically tested. In this article the notion of indistinguishability is extended to skill maps and, more generally, to the competence-based knowledge space theory. Theoretical results are provided showing that skill maps can be empirically indistinguishable from one another. The most relevant consequence of this is that for some skills there is no empirical evidence to establish their existence. This result is strictly related to the type of probabilistic model investigated, which is essentially the BLIM. Alternative models may exist or can be developed in knowledge space theory for which this indistinguishability problem disappears.

Commentary on "Extending the Basic Local Independence Model to Polytomous Data" by Stefanutti, de Chiusole, Anselmi, and Spoto.

The Polytomous Local Independence Model (PoLIM) by Stefanutti, de Chiusole, Anselmi, and Spoto, is an extension of the Basic Local Independence Model (BLIM) to accommodate polytomous items. BLIM, a model for analyzing responses to binary items, is based on Knowledge Space Theory, a framework developed by cognitive scientists and mathematical psychologists for modeling human knowledge acquisition and representation. The purpose of this commentary is to show that PoLIM is simply a paraphrase of a DINA model in cognitive diagnosis for polytomous items. Specifically, BLIM is shown to be equivalent to the DINA model when the BLIM-items are conceived as binary single-attribute items, each with a distinct attribute; thus, PoLIM is equivalent to the DINA for polytomous single-attribute items, each with a distinct attribute.

Psychometric Data Analysis: A Size/fit Trade-off Evaluation Procedure for Knowledge Structures

A crucial problem in knowledge space theory, a modern psy chological test theory, is the derivation of a realistic knowledge structure representing the organization of knowledge in an information domain and examinee population under reference. Often, one is left with the problem of selecting among candidate competing knowledge structures. This article proposes a measure for the selection among competing knowledge structures. It is derived within an operational framework (prediction paradigm), and is partly based on the unitary method of proportional reduction in predictive error as advocated by the authors Guttman, Goodman, and Kruskal. In particular, this measure is designed to trade off the (descriptive) fit and size of a knowledge structure, which is of high interest in knowledge space theory. The proposed approach is compared with the Correlational Agreement Coef ficient, which has been recently discussed for the selection among competing surmise relations. Their performances as selection measures are compared in a simulation study using the fundamental basic local independence model in knowledge space theory

On the empirical indistinguishability of knowledge structures.

In recent years a number of articles have focused on the identifiability of the basic local independence model. The identifiability issue usually concerns two model parameter sets predicting an identical probability distribution on the response patterns. Both parameter sets are applied to the same knowledge structure. However, nothing is known about cases where different knowledge structures predict the same probability distribution. This situation is referred to as 'empirical indistinguishability' between two structures and is the main subject of the present paper. Empirical indistinguishability is a stronger form of unidentifiability, which involves not only the parameters, but also the structural and combinatorial properties of the model. In particular, as far as knowledge structures are concerned, a consequence of empirical indistinguishability is that the existence of certain knowledge states cannot be empirically established. Most importantly, it is shown that model identifiability cannot guarantee that a certain knowledge structure is empirically distinguishable from others. The theoretical findings are exemplified in a number of different empirical scenarios.

A Correct Response Model in knowledge structure theory

In knowledge space theory, the (latent) knowledge state of a student consists of the subset of test items that he masters in principle. Even at a given stage of apprenticeship, the student’s knowledge state may vary in a given collection of subsets. The collection of all possible states of all potential students forms a knowledge structure. In the modeling of student answer production, the probabilities governing the knowledge state are parameters. Moreover, for each item, two additional parameters capture the probabilities of careless errors and lucky guesses in the answers to the item. From all the latter parameters, the Correct Response Model (CRM) predicts the probability of a correct answer to any isolated item. From the same parameters, the Basic Local Independence Model (BLIM) predicts the probability of any possible pattern of correct responses. Here, a particular pattern records at a given time all the items to which a given student would provide correct answers. While general properties of the BLIM (such as identifiability of parameters) have been investigated, the simpler Correct Response Model still requires scrutiny. The present paper investigates the CRM as regards testability, identifiability and characterizability. It either provides explicit answers or points out serious difficulties garnered from various mathematical disciplines.

On the necessary and sufficient conditions for delineating forward- and backward-graded knowledge structures from skill maps

In recent years two classes of knowledge structures received particular attention. They are, respectively, forward-graded and backward-graded knowledge structures. Their importance is related to the fact that the basic local independence model applied to such structures turns out to be locally unidentifiable. Spoto et al. (2012) established some sufficient conditions for a skill multimap to delineate forward- and/or backward graded knowledge structures. Drawing upon some of the theoretical results contained in that article, the present theoretical note aims at providing necessary and sufficient conditions for a skill map to delineate a forward- and/or backward graded knowledge structure.

Modeling misconceptions in knowledge space theory

Building intelligent tutoring systems (ITSs) that are aware of the students’ misconceptions has been one of the ambitions for many of the approaches to computerized assessment of knowledge and learning. In the present article we extend knowledge space theory (KST) to mistakes and misconceptions. The proposed approach completes and extends the work initiated by J. Lukas (1997, Modellierung von Fehlkonzepten in einer algebraischen Wissensstruktur [Modeling misconceptions in an algebraic knowledge structure], Kognitionswissenschaft, 6(4), 196–204) on the application of information systems in KST for modeling misconceptions. The approach is divided into a deterministic and a probabilistic part. The notion of a “polytomous skill map” (PSM) represents the cornerstone of the deterministic part. Properties of the PSM are established that assure a consistent link between item responses on the one hand and the (correct or buggy) cognitive rules applied by an individual on the other hand. A weaker notion of information system (named a “cognitive rule system”) is proposed for representing two types of dependencies among cognitive rules: precedence and compatibility. The probabilistic part consists of an extension of the basic local independence model to more than two response alternatives. This model can be used for knowledge diagnoses, as well as for empirically testing the deterministic assumptions from a PSM. An empirical application of this probabilistic model to the responses of 331 university students is illustrated and discussed using two different PSMs.

Extending the Basic Local Independence Model to Polytomous Data

A probabilistic framework for the polytomous extension of knowledge space theory (KST) is proposed. It consists in a probabilistic model, called polytomous local independence model, that is developed as a generalization of the basic local independence model. The algorithms for computing “maximum likelihood” (ML) and “minimum discrepancy” (MD) estimates of the model parameters have been derived and tested in a simulation study. Results show that the algorithms differ in their capability of recovering the true parameter values. The ML algorithm correctly recovers the true values, regardless of the manipulated variables. This is not totally true for the MD algorithm. Finally, the model has been applied to a real polytomous data set collected in the area of psychological assessment. Results show that it can be successfully applied in practice, paving the way to a number of applications of KST outside the area of knowledge and learning assessment.

Representing probabilistic models of knowledge space theory by multinomial processing tree models

Knowledge Space Theory (KST) aims at modeling the hierarchical relations between items or skills in a learning process. For example, when studying mathematics in school, students first need to master the rules of summation before being able to learn multiplication. In KST, the knowledge states of individuals are represented by means of partially ordered latent classes. In probabilistic KST models, conditional probability parameters are introduced to model transitions from latent knowledge states to observed response patterns. Since these models account for discrete data by assuming a finite number of latent states, they can be represented by Multinomial Processing Tree (MPT) models (i.e., binary decision trees with parameters referring to the conditional probabilities of entering different states). We prove that standard probabilistic models of KST such as the Basic Local Independence Model (BLIM) and the Simple Learning Model (SLM) can be represented as specific instances of MPT models. Given this close link, MPT methods may be applied to address theoretical and practical issues in KST. By highlighting the MPT–KST link and its implications for modeling violations of local stochastic independence in Item Response Theory (IRT), we hope to facilitate an exchange of theoretical results, statistical methods, and software across these different domains of mathematical psychology and psychometrics.

BLIM’s identifiability and parameter invariance under backward and forward transformations

The basic local independence model (BLIM) is one of the most widely applied probabilistic models in knowledge space theory. It is known that the BLIM is not identifiable in general and that its identifiability strictly depends on the properties of the knowledge structure to which it is applied. If the knowledge structure is either forward- or backward-graded in one or more items, then the BLIM is not identifiable. In such cases, there exist continuous transformations of the model’s parameters, named forward and backward transformations, that keep constant the value of the model’s prediction function. Under certain constraints on the model’s parameters, some of the transformations might lose this property. The type of constraints considered in this article consist of fixing the probability of a knowledge state to a constant value. The theoretical results contained in the article shed light on the role of the different state probabilities in reducing the collection of transformations and thus restoring the identifiability of the model’s parameters.

On a Generalization of Local Independence in Item Response Theory Based on Knowledge Space Theory.

Knowledge space theory (KST) structures are introduced within item response theory (IRT) as a possible way to model local dependence between items. The aim of this paper is threefold: firstly, to generalize the usual characterization of local independence without introducing new parameters; secondly, to merge the information provided by the IRT and KST perspectives; and thirdly, to contribute to the literature that bridges continuous and discrete theories of assessment. In detail, connections are established between the KST simple learning model (SLM) and the IRT General Graded Response Model, and between the KST Basic Local Independence Model and IRT models in general. As a consequence, local independence is generalized to account for the existence of prerequisite relations between the items, IRT models become a subset of KST models, IRT likelihood functions can be generalized to broader families, and the issues of local dependence and dimensionality are partially disentangled. Models are discussed for both dichotomous and polytomous items and conclusions are drawn on their interpretation. Considerations on possible consequences in terms of model identifiability and estimation procedures are also provided.

Detecting and explaining BLIM’s unidentifiability: Forward and backward parameter transformation groups

The basic local independence model (BLIM) is a probabilistic model, widely applied in knowledge space theory. It has recently received attention concerning its identifiability. In particular it was shown that the BLIM is not identifiable for forward-graded (FG) and backward-graded (BG) knowledge structures, such as quasi-ordinal and learning spaces. These structures are widely and fruitfully applied in practice, providing one of the reasons for the investigations carried out in this article. Following up recent contributions on the BLIM identifiability, this article seeks to study the issue by examining groups of parameter transformations that, for FG and BG structures, keep constant the model’s prediction function, thus making it unidentifiable. This transformational approach is compared to the standard way of testing the local identifiability of a model, and its benefits are discussed.

Identifiability in probabilistic knowledge structures

The so-called basic local independence model (BLIM) constitutes the standard probabilistic model within the theory of knowledge structures. The present paper characterizes local identifiability of the BLIM through the rank of its Jacobian matrix. Within this framework, it reconsiders conditions known to give rise to non-identifiability, and presents some new cases. Together they completely cover the instances cropping up in the collection of BLIMs arising from all the possible knowledge structures on a three-item domain. The derived theoretical results, providing a full account of the trade-offs between parameters that occur in these situations, hold for arbitrary BLIMs, and are not limited to domains of particular cardinality. Moreover, it is shown that previously formulated conjectures on the encountered types of parameter trade-offs need not hold on the whole parameter space, but in general may only be true almost everywhere. If this indeed is the case, remains an open problem.

On the Link between Cognitive Diagnostic Models and Knowledge Space Theory.

The present work explores the connections between cognitive diagnostic models (CDM) and knowledge space theory (KST) and shows that these two quite distinct approaches overlap. It is proved that in fact the Multiple Strategy DINA (Deterministic Input Noisy AND-gate) model and the CBLIM, a competence-based extension of the basic local independence model (BLIM), are equivalent. To demonstrate the benefits that arise from integrating the two theoretical perspectives, it is shown that a fairly complete picture on the identifiability of these models emerges by combining results from both camps. The impact of the results is illustrated by an empirical example, and topics for further research are pointed out.

Naïve Tests of Basic Local Independence Model's Invariance.

The basic local independence model (BLIM) is a probabilistic model for knowledge structures, characterized by the property that lucky guess and careless error parameters of the items are independent of the knowledge states of the subjects. When fitting the BLIM to empirical data, a good fit can be obtained even when the invariance assumption is violated. Therefore, statistical tests are needed for detecting violations of this specific assumption. This work provides an extension to theoretical results obtained by de Chiusole, Stefanutti, Anselmi, and Robusto (2013), showing that statistical tests based on the partitioning of the empirical data set into two (or more) groups are not adequate for testing the BLIM's invariance assumption. A simulation study confirms the theoretical results.

Modeling missing data in knowledge space theory.

Missing data are a well known issue in statistical inference, because some responses may be missing, even when data are collected carefully. The problem that arises in these cases is how to deal with missing data. In this article, the missingness is analyzed in knowledge space theory, and in particular when the basic local independence model (BLIM) is applied to the data. Two extensions of the BLIM to missing data are proposed: The former, called ignorable missing BLIM (IMBLIM), assumes that missing data are missing completely at random; the latter, called missing BLIM (MissBLIM), introduces specific dependencies of the missing data on the knowledge states, thus assuming that the missing data are missing not at random. The IMBLIM and the MissBLIM modeled the missingness in a satisfactory way, in both a simulation study and an empirical application, depending on the process that generates the missingness: If the missing data-generating process is of type missing completely at random, then either IMBLIM or MissBLIM provide adequate fit to the data. However, if the pattern of missingness is functionally dependent upon unobservable features of the data (e.g., missing answers are more likely to be wrong), then only a correctly specified model of the missingness distribution provides an adequate fit to the data.

Considerations about the identification of forward- and backward-graded knowledge structures

The application of the basic local independence model (BLIM) to a knowledge structure (Q,K) that satisfies a particular kind of gradation (namely forward- or backward-gradedness) leads the model to be not identifiable. In the present article, we show that many important types of knowledge structures happen to be either forward- or backward-graded. This means that the application of the BLIM to these structures leads to unidentifiable models. No universal remedy for recovering identifiability is presently known. However, we propose a construction that consists in introducing an equally informative item for each item in Q. We conjecture that the BLIM based on the resulting knowledge structure is always identifiable. This conjecture is proven to be true for knowledge structures on small sets of items.

Minimum Discrepancy Estimation in Probabilistic Knowledge Structures

Practical applications of the theory of knowledge structures often rely on a probabilistic version, known as the basic local independence model. The paper outlines various procedures for estimating its parameters, including maximum likelihood (ML) via the expectation-maximization (EM) algorithm, the computationally efficient minimum discrepancy (MD) estimation as well as MDML, a hybrid method combining the two approaches.

Modeling Skill Dependence in Probabilistic Competence Structures

Abstract In knowledge space theory (KST) a competence structure is a set theoretical representation of the dependencies among a given set of skills. A probabilistic model for skill dependence is proposed which respects a precise correspondence requirement between set theoretical and probabilistic representations of skill dependence. An empirical application on integer subtraction problems at the primary school shows that the proposed model fits the data pretty well. Moreover, in a comparison with an unrestricted skill based version of the basic local independence model (BLIM), the proposed model fits better than this last, indicating that the restrictions implied by the correspondence requirement are not too strong.