Representational Theory Of Measurement Research Articles

Educational assessment without numbers

Psychometrics conceptualizes a person's proficiency (or ability, or competence), in a cognitive or educational domain, as a latent numerical quantity. Yet both conceptual and empirical studies have shown that the assumption of quantitative structure for such phenomena is unlikely to be tenable. A reason why most applications of psychometrics nevertheless continue to treat them as if they were numerical quantities may be that quantification is thought to be necessary to enable measurement. This is indeed true if one regards the task of measurement as the location of a measurand at a point on the real number line (the viewpoint adopted by, for example, the representational theory of measurement, the realist theory of measurement as the discovery of ratios, and Rasch measurement theory). But this is not the only philosophically respectable way of defining the notion of measurement. This paper suggests that van Fraassen's more expansive view of measurement as, in general, location in a logical space (which could be the real continuum, as in metrological applications in the physical sciences, but could be a different mathematical structure), provides a more appropriate conceptual framework for psychometrics. Taking educational measurement as a case study, it explores what that could look like in practice, drawing on fuzzy logic and mathematical order theory. It suggests that applying this approach to the assessment of intersubjectively constructed phenomena, such as a learner's proficiency in an inherently fuzzily-defined subject area, entails recognizing the theory-dependent nature of valid representations of such phenomena, which need not be conceived of structurally as values of quantities. Finally, some connections are made between this “qualitative mathematical” theorization of educational assessment, and the application of techniques from machine learning and artificial intelligence in this area.

Information Retrieval Evaluation Measures Defined on Some Axiomatic Models of Preferences

Information retrieval (IR) evaluation measures are essential for capturing the relevance of documents to topics and determining the task performance efficiency of retrieval systems. The study of IR evaluation measures through their formal properties enables a better understanding of their suitability for a specific task. Some works have modeled the effectiveness of retrieval measures with axioms, heuristics, or desirable properties, leading to order relationships on the set where they are defined. Each of these ordering structures constitutes an axiomatic model of preferences (AMP), which can be considered as an “ideal” scenario of retrieval. Based on lattice theory and on the representational theory of measurement, this work formally explores numeric, metric, and scale properties of some effectiveness measures defined on AMPs. In some of these scenarios, retrieval measures are completely determined from the scores of a subset of document rankings: join-irreducible elements. All the possible metrics and pseudometrics, defined on these structures are expressed in terms of the join-irreducible elements. The deduced scale properties of the precision, recall, F -measure, RBP , DCG , and AP confirm some recent results in the IR field.

The Conventionality of Real Valued Quantities

The original architects of the representational theory of measurement interpreted their formalism operationally and explicitly acknowledged that some aspects of their representations are conventional. In this paper we argue that the conventional elements of the representations afforded by the theory require careful scrutiny as one moves toward a more metaphysically robust interpretation by showing that there is a sense in which the very number system one uses to represent a physical quantity such as mass or length is conventional. This result undermines inferences which impute structure from the numerical representational structure to the quantity it is used to represent.

The Philosophical Significance of the Representational Theory of Measurement —RTM as Semantic Foundations

The Representational Theory of Measurement (RTM), especially the canonical three volume Foundations of Measurement by Krantz et al., is a landmark accomplishment in our understanding of measurement. Despite this, it has been far from easy to pinpoint what exactly we can learn about measurement from RTM, and who the target audience for RTM’s formal results should be. In what sense does RTM provide foundations of measurement, and what is the philosophical significance of such foundations? I argue that RTM provides semantic foundations of measurement, and that their philosophical significance lies in a shift towards structural representation.

Absolutely Zero Evidence

AbstractStatistical analysis is often used to evaluate the strength of evidence for or against scientific hypotheses. Here we consider evidence measurement from the point of view of representational measurement theory, focusing in particular on the 0-points of measurement scales. We argue that a properly calibrated evidence measure will need to count up from absolute 0, in a sense to be defined, and that this 0-point is likely to be something other than what one might have expected. This suggests the need for a new theory of statistical evidence in the context of which calibrated evidence measurement becomes tractable.

Analysis of the Scale Types and Measurement Units in Enterprise Architecture (EA) Measurement

Aim/Purpose: This study identifies the scale types and measurement units used in the measurement of enterprise architecture (EA) and analyzes the admissibility of the mathematical operations used. Background: The majority of measurement solutions proposed in the EA literature are based on researchers’ opinions and many with limited empirical validation and weak metrological properties. This means that the results generated by these solutions may not be reliable, trustworthy, or comparable, and may even lead to wrong investment decisions. While the literature proposes a number of EA measurement solutions, the designs of the mathematical operations used to measure EA have not yet been independently analyzed. It is imperative that the EA community works towards developing robust, reliable, and widely accepted measurement solutions. Only then can senior management make informed decisions about the allocation of resources for EA initiatives and ensure that their investment yields optimal results. Methodology: In previous research, we identified, through a systematic literature review, the EA measurement solutions proposed in the literature and classified them by EA entity types. In a subsequent study, we evaluated their metrology coverage from both a theoretical and empirical perspective. The metrology coverage was designed using a combination of the evaluation theory, best practices from the software measurement literature including the measurement context model, and representational theory of measurement to evaluate whether EA measurement solutions satisfy the metrology criteria. The research study reported here presents a more in-depth analysis of the mathematical operations within the proposed EA measurement solutions, and for each EA entity type, each mathematical operation used to measure EA was examined in terms of the scale types and measurement units of the inputs, their transformations through mathematical operations, the impact in terms of scale types, and measurement units of the proposed outputs. Contribution: This study adds to the body of knowledge on EA measurement by offering a metrology-based approach to analyze and design better EA measurement solutions that satisfy the validity of scale type transformations in mathematical operations and the use of explicit measurement units to allow measurement consistency for their usage in decision-making models. Findings: The findings from this study reveal that some important metrology and quantification issues have been overlooked in the design of EA measurement solutions proposed in the literature: a number of proposed EA mathematical operations produce numbers with unknown units and scale types, often the result of an aggregation of undetermined assumptions rather than explicit quantitative knowledge. The significance of such aggregation is uncertain, leading to numbers that have suffered information loss and lack clear meaning. It is also unclear if it is appropriate to add or multiply these numbers together. Such EA numbers are deemed to have low metrological quality and could potentially lead to incorrect decisions with serious and costly consequences. Recommendations for Practitioners: The results of the study provide valuable insights for professionals in the field of EA. Identifying the metrology limitations and weaknesses of existing EA measurement solutions may indicate, for instance, that practitioners should wait before using them until their design has been strengthened. In addition, practitioners can make informed choices and select solutions with a more robust metrology design. This, in turn, will benefit enterprise architects, software engineers, and other EA professionals in decision making, by enabling them to take into consideration factors more adequately such as cost, quality, risk, and value when assessing EA features. The study’s findings thus contribute to the development of more reliable and effective EA measurement solutions. Recommendation for Researchers: Researchers can use with greater confidence the EA measurement solutions with admissible mathematical operations and measurement units to develop new decision-making models. Other researchers can carry on research to address the weaknesses identified in this study and propose improved ones. Impact on Society: Developers, architects, and managers may be making inappropriate decisions based on seriously flawed EA measurement solutions proposed in the literature and providing undue confidence and a waste of resources when based on bad measurement design. Better quantitative tools will ultimately lead to better decision making in the EA domain, as in domains with a long history of rigor in the design of the measurement tools. Such advancements will benefit enterprise architects, software engineers, and other practitioners, by providing them with more meaningful measurements for informed decision making. Future Research: While the analysis described in this study has been explicitly applied to evaluating EA measurement solutions, researchers and practitioners in other domains can also examine measurement solutions proposed in their respective domains and design new ones.

Introduction: Measurement at the Crossroads

Introduction: Measurement at the Crossroads

Representation in measurement

The Representational Theory of Measurement (RTM) is the best known account of the kind of representation measurement requires. However, RTM has been challenged from various angles, with critics claiming e.g. that RTM fails to account for actual measurement practice and that it is ambiguous about the nature of measurable attributes. In this paper I use the critical literature on RTM to formulate Representation Minimalism – a characterization of what measurement-relevant representation requires at the minimum. I argue that Representation Minimalism avoids the main problems with RTM while acknowledging its usefulness as the formal foundation of representation in measurement.

Meinong Epistemologist of Measurement

Contemporary theories of measurement seem to have almost forgotten the epistemological questions linked to the psychological aspects of measurement. Indeed, the received approaches, i.e. the representational theory of measurement and the model-based account, are well developed from the mathematical and operational points of view; however, they do not consider sufficiently the reliability of the perceptual impressions involved in any measurement. In contrast, Gustav Theodor Fechner, one of the fathers of modern experimental psychology, addresses this question at length. His work stimulated an epistemological analysis of the quantification of qualities - a process necessary for empirical science - in an almost forgotten paper by Meinong. I present Meinong’s approach in much greater historical and epistemological detail than has been done in the literature thus far. Moreover, I show how Meinong’s approach can fruitfully enrich the received view on measurement theory.

Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales

Information Retrieval (IR) is a discipline deeply rooted in evaluation since its inception. Indeed, experimentally measuring and statistically validating the performance of IR systems are the only possible ways to compare systems and understand which are better than others and, ultimately, more effective and useful for end-users. Since the seminal paper by Stevens (1946), it is known that the properties of the measurement scales determine the operations you should or should not perform with values from those scales. For example, Stevens suggested that you can compute means and variances only when you are working with, at least, interval scales. It was recently shown that the most popular evaluation measures in IR are not interval-scaled. However, so far, there has been little or no investigation in IR on the impact and consequences of departing from scale assumptions. Taken to the extremes, it might even mean that decades of experimental IR research used potentially improper methods, which may have produced results needing further validation. However, it was unclear if and to what extent these findings apply to actual evaluations; this opened a debate in the community with researchers standing on opposite positions about whether this should be considered an issue (or not) and to what extent. In this paper, we first give an introduction to the representational measurement theory explaining why certain operations and significance tests are permissible only with scales of a certain level. For that, we introduce the notion of meaningfulness specifying the conditions under which the truth (or falsity) of a statement is invariant under permissible transformations of a scale. Furthermore, we show how the recall base and the length of the run may make comparison and aggregation across topics problematic. Then we propose a straightforward and powerful approach for turning an evaluation measure into an interval scale, and describe an experimental evaluation of the differences between the original measures and the interval-scaled ones. For all the regarded measures – namely Precision, Recall, Average Precision, (Normalized) Discounted Cumulative Gain, Rank-Biased Precision and Reciprocal Rank - we observe substantial effects, both on the order of average values and on the outcome of significance tests. For the latter, previously significant differences turn out to be insignificant, while insignificant ones become significant. The effect varies remarkably between the tests considered but on average, we observed a 25% change in the decision about which systems are significantly different and which are not. These experimental findings further support the idea that measurement scales matter and that departing from their assumptions has an impact. This not only suggests that, to the extent possible, it would be better to comply with such assumptions but it also urges us to clearly indicate when we depart from such assumptions and, carefully, point out the limitations of the conclusions we draw and under which conditions they are drawn.

Representational measurement theory: Is its number up?

Representational measurement theory was proposed initially to solve problems caused by disciplinary aspirations of 19th-century mathematicians, who wanted to construe their subject as independent of its applications in empirical science. Half a century later, S. S. Stevens seized the opportunity provided by representational theory’s reconstruction of measurement as numerical coding to rubber-stamp psychology’s own aspirations to be counted as a quantitative science. Patrick Suppes’ version of representational theory rectified defects in Stevens’ theory, making it explicit that representational theory entails that mathematical structure is already embedded in empirical systems. However, Suppes’ theory neglected the fact that attributes, not objects, are the focus of measurement and when that oversight is corrected, it follows that empirical systems sustaining measurement already instantiate positive real numbers. Thus, in measurement, real numbers are estimated, not assigned from without. Representational theory not only misrepresents measurement; it refutes itself.

MEASURING THE SOCIAL WORLD

Quantification is no longer a practice of natural science only but has become part of human sciences and everyday life as well. As a direct measurement, which fits the axiomatic of the representational theory of measurement, is mostly infeasible in the social sciences, indicators are used and frequently aggregated to indexes. The index scores can be used to construct a ranking of the units (HDI, PISA). Measurement level and meaning of the data remain often unknown (pseudo-metrical scaling). Furthermore many variables in the field of education are not quantifiable. In the course of globalization and international competition, an indicator system for educational quality measurement (PISA, TIMMS) and goal achievement (Education for All, Millenium Development Goals, Sustainable Development Goals), was established and exerts an increasing impact on national educational systems.The main concerns of indicator-based steering are its methodological limitations and the transformation from a descriptive information base to a normative control system.

How do interval scales help us with better understanding IR evaluation measures?

Evaluation measures are the basis for quantifying the performance of IR systems and the way in which their values can be processed to perform statistical analyses depends on the scales on which these measures are defined. For example, mean and variance should be computed only when relying on interval scales. In our previous work we defined a theory of IR evaluation measures, based on the representational theory of measurement, which allowed us to determine whether and when IR measures are interval scales. We found that common set-based retrieval measures—namely precision, recall, and F-measure—always are interval scales in the case of binary relevance while this does not happen in the multi-graded relevance case. In the case of rank-based retrieval measures—namely AP, gRBP, DCG, and ERR—only gRBP is an interval scale when we choose a specific value of the parameter p and define a specific total order among systems while all the other IR measures are not interval scales. In this work, we build on our previous findings and we carry out an extensive evaluation, based on standard TREC collections, to study how our theoretical findings impact on the experimental ones. In particular, we conduct a correlation analysis to study the relationship among the above-mentioned state-of-the-art evaluation measures and their scales. We study how the scales of evaluation measures impact on non parametric and parametric statistical tests for multiple comparisons of IR system performance. Finally, we analyse how incomplete information and pool downsampling affect different scales and evaluation measures.

The Complementarity of Psychometrics and the Representational Theory of Measurement

Psychometrics and the representational theory of measurement (RTM) are widely used in social scientific measurement. They are currently pursued largely in isolation from one another. I argue that despite their separation in practice, RTM and psychometrics are complementary approaches, because they can contribute in complementary ways to the establishment of what I argue is a crucial measurement property, namely, representational interpretability. Because RTM and psychometrics are complementary in the establishment of representational interpretability, the current separation of measurement approaches is unfounded. 1. Introduction2. Two Approaches to Measurement 2.1. Representational theory of measurement2.2. Psychometrics2.3. Representational interpretability3. Complementarity, Conceptually 3.1. Representational theory of measurement: Conditions of representational interpretability3.2. Psychometrics: Evidence of representational interpretability4. Complementarity in Action 4.1. What is the Rasch model?4.2. Rasch and conjoint measurement5. Conclusion: Critics and Fruits of Complementarity

Scoring the risk matrix

In systems engineering, the risk matrix is a popular protocol for binning risks into a collection of probability and consequence cells. In its traditional form, the risk matrix produces a ranking of cells according to their position in an ordered list. From this, management can distinguish whether a set of risks collected in one risk matrix cell has a higher priority than a set of risks collected in another cell. However, if decisions require measuring the relative differences between pairs of cells across their rank positions, then it is necessary to map them from their ordinal scale to an interval scale. This paper introduces methods from representational measurement theory to transform a rank ordered list of risk matrix cells into an interval measurement scale. The transformation produces a scored risk matrix. This allows relative differences among cells to be meaningfully compared, which broadens its use in management decisions. A scored risk matrix provides greater insights into the urgency of risks grouped within cells than is possible in a traditional risk matrix, while remaining within its ease and popularity of use.

Quantitative Data From Rating Scales: An Epistemological and Methodological Enquiry.

Rating scales are popular methods for generating quantitative data directly by persons rather than automated technologies. But scholars increasingly challenge their foundations. This article contributes epistemological and methodological analyses of the processes involved in person-generated quantification. They are crucial for measurement because data analyses can reveal information about study phenomena only if relevant properties were encoded systematically in the data. The Transdisciplinary Philosophy-of-Science Paradigm for Research on Individuals (TPS-Paradigm) is applied to explore psychological and social-science concepts of measurement and quantification, including representational measurement theory, psychometric theories and their precursors in psychophysics. These are compared to theories from metrology specifying object-dependence of measurement processes and subject-independence of outcomes as key criteria, which allow tracing data to the instances measured and the ways they were quantified. Separate histories notwithstanding, the article’s basic premise is that general principles of scientific measurement and quantification should apply to all sciences. It elaborates principles by which these metrological criteria can be implemented also in psychology and social sciences, while considering their research objects’ peculiarities. Application of these principles is illustrated by quantifications of individual-specific behaviors (‘personality’). The demands rating methods impose on data-generating persons are deconstructed and compared with the demands involved in other quantitative methods (e.g., ethological observations). These analyses highlight problematic requirements for raters. Rating methods sufficiently specify neither the empirical study phenomena nor the symbolic systems used as data nor rules of assignment between them. Instead, pronounced individual differences in raters’ interpretation and use of items and scales indicate considerable subjectivity in data generation. Together with recoding scale categories into numbers, this introduces a twofold break in the traceability of rating data, compromising interpretability of findings. These insights question common reliability and validity concepts for ratings and provide novel explanations for replicability problems. Specifically, rating methods standardize only data formats but not the actual data generation. Measurement requires data generation processes to be adapted to the study phenomena’s properties and the measurement-executing persons’ abilities and interpretations, rather than to numerical outcome formats facilitating statistical analyses. Researchers must finally investigate how people actually generate ratings to specify the representational systems underlying rating data.

Beyond the metrological viewpoint

The representational theory of measurement (RTM) has long been the central paradigm in the philosophy of measurement. Such is not the case anymore, partly under the influence of the critique according to which RTM offers too poor descriptions of the measurement procedures actually followed in science. This can be called the metrological critique of RTM. I claim that the critique is partly irrelevant. This is because, in general, RTM is not in the business of describing measurement procedures, be it in idealized form. To support this claim, I present various cases where RTM can be said to investigate measurement without providing any measurement procedure. Such limit cases lead to a better understanding of the RTM project. They also illustrate some of the questions which the philosophy of measurement can explore, when it is ready to go beyond the metrological viewpoint.

Helmholtz, Kaila, and the Representational Theory of Measurement

The problem of measurement can be reformulated as the problem of measurability: What are the conditions under which measurement becomes possible at all? And what is the ontological status of concre...

Conjoint measurement undone

According to classical measurement theory, fundamental measurement necessarily requires the operation of concatenation qua physical addition. Quantities which do not allow this operation are measurable only indirectly by means of derived measurement. Since only extensive quantities sustain the operation of physical addition, measurement in psychology has been considered problematic. In contrast, the theory of conjoint measurement, as developed in representational measurement theory, proposes that the operation of ordering is sufficient for establishing fundamental measurement. The validity of this view is questioned. The misconception about the advantages of conjoint measurement, it is argued, results from the failure to notice that magnitudes of derived quantities cannot be determined directly, i.e., without the help of associated quantitative indicators. This takes away the advantages conjoint measurement has over derived measurement, making it practically useless.

The Biased Balance: Observation, Formalism and Interpretation of a Dissymmetric Measuring Device

This paper studies a balance whose unobservable fulcrum is not necessarily located at the middle of its two pans. It presents three different models, showing how this lack of symmetry modifies the observation, the formalism and the interpretation of such a biased measuring device. It argues that the biased balance can be an interesting source of inspiration for broadening the representational theory of measurement.