Different Sets Of Items Research Articles

Measurement versus prediction in the construction of patient-reported outcome questionnaires: can we have our cake and eat it?

BackgroundTwo important goals when using questionnaires are (a) measurement: the questionnaire is constructed to assign numerical values that accurately represent the test taker’s attribute, and (b) prediction: the questionnaire is constructed to give an accurate forecast of an external criterion. Construction methods aimed at measurement prescribe that items should be reliable. In practice, this leads to questionnaires with high inter-item correlations. By contrast, construction methods aimed at prediction typically prescribe that items have a high correlation with the criterion and low inter-item correlations. The latter approach has often been said to produce a paradox concerning the relation between reliability and validity [1–3], because it is often assumed that good measurement is a prerequisite of good prediction.ObjectiveTo answer four questions: (1) Why are measurement-based methods suboptimal for questionnaires that are used for prediction? (2) How should one construct a questionnaire that is used for prediction? (3) Do questionnaire-construction methods that optimize measurement and prediction lead to the selection of different items in the questionnaire? (4) Is it possible to construct a questionnaire that can be used for both measurement and prediction?Illustrative exampleAn empirical data set consisting of scores of 242 respondents on questionnaire items measuring mental health is used to select items by means of two methods: a method that optimizes the predictive value of the scale (i.e., forecast a clinical diagnosis), and a method that optimizes the reliability of the scale. We show that for the two scales different sets of items are selected and that a scale constructed to meet the one goal does not show optimal performance with reference to the other goal.DiscussionThe answers are as follows: (1) Because measurement-based methods tend to maximize inter-item correlations by which predictive validity reduces. (2) Through selecting items that correlate highly with the criterion and lowly with the remaining items. (3) Yes, these methods may lead to different item selections. (4) For a single questionnaire: Yes, but it is problematic because reliability cannot be estimated accurately. For a test battery: Yes, but it is very costly. Implications for the construction of patient-reported outcome questionnaires are discussed.

Creating Parallel Forms to Support On-Demand Testing for Undergraduate Students in Psychology

On-demanding testing requires that multiple forms of an exam be administered to students in each testing session. But the use of multiple forms raises test security concern because of item exposure. One way to limit exposure is with parallel forms construction. Parallel forms are different versions of a test that measure the same content areas and have the same difficulty level but contain different sets of items. The purpose of our study is to describe and demonstrate how parallel forms can be created even from small test banks. We present three unique yet plausible test assembly problems. We also provide a solution for each problem using the results from a free, open-source, software add-in for Microsoft Excel called the Opensolver. Implications for test design and item development are discussed.

The role of feedback and differences between good and poor decoders in a repeated word reading paradigm in first grade

The direct, retention, and transfer effects of repeated word and pseudoword reading were studied in a pretest, training, posttest, retention design. First graders (48 good readers, 47 poor readers) read 25 CVC words and 25 CVC pseudowords in ten repeated word reading sessions, preceded and followed by a transfer task with a different set of items. Two weeks after training, trained items were assessed again in a retention test. Participants either received phonics feedback, in which each word was spelled out and repeated; word feedback, in which each word was repeated; or no feedback. During the training, both good and poor readers improved in accuracy and speed. The increase in speed was stronger for poor readers than for good readers. The good readers demonstrated a stronger increase for pseudowords than for words. This increase in speed was most prominent in the first four sessions. Two weeks after training, the levels of accuracy and speed were retained. Furthermore, transfer effects on speed were found for pseudowords in both groups of readers. Good readers performed most accurately during the training when they received no feedback while poor readers performed most accurately during the training with the help of phonics feedback. However, feedback did not differentiate for reading speed or for effects after the training. The effects of repeated word reading were found to be stronger for poor readers than for good readers. Moreover, these effects were found to be stronger for pseudowords than for words. This indicates that repeated word reading can be seen as an important trigger for the improvement of decoding skills.

The creation and validation of a listening vocabulary levels test

An important gap in the field of second language vocabulary assessment concerns the lack of validated tests measuring aural vocabulary knowledge. The primary purpose of this study is to introduce and provide preliminary validity evidence for the Listening Vocabulary Levels Test (LVLT), which has been designed as a diagnostic tool to measure knowledge of the first five 1000-word frequency levels and the Academic Word List (AWL). Quantitative analyses based on the Rasch model utilized several aspects of Messick’s validation framework. The findings indicated that (1) the items showed sufficient spread of difficulty, (2) the majority of the items displayed good fit to the Rasch model, (3) items and persons generally performed as predicted by a priori hypotheses, (4) the LVLT correlated with Parts 1 and 2 of the TOEIC listening test at .54, (5) the items displayed a high degree of unidimensionality, (6) the items showed a strong degree of measurement invariance with disattenuated Pearson correlations of .97 and .98 for person measures estimated with different sets of items, and (7) carelessness and guessing exerted only minor influences on test scores. Follow-up interviews and qualitative analyses indicated that the LVLT measures the intended construct of aural vocabulary knowledge, the format is easily understood, and the test has high face validity. This test fills an important gap in the field of second language vocabulary assessment by providing teachers and researchers with a way to assess aural vocabulary knowledge.

Evaluating system reliability using linear-exponential distribution function

Safety is a main criterion to design every system. Among various theories, which are applied to improve system safety, reliability theory is known as a powerful tool to reach higher safety levels in system design. In this paper, a compound of series and parallel systems is considered for reliability improvement. This system includes three items that two items are connected in parallel and their compound item is connected to the third item in series. It's assumed that the items are independent and their longevity follows linear-exponential distribution function. Reliability function of the mentioned system is formulated using linear-exponential distribution function. Then, three improvement methods will be applied to enhance system reliability. In each method, different sets of items will be considered for improvement and their reliability functions will be reformulated. A data analysis will be done in order to compare different improvement methods and a conclusion will be made based on the analyzed data.

Beyond the Score: Clinical Evaluation Through Formal Psychological Assessment

Questionnaires generate numerical scores, but endorsing different sets of items could produce the same score despite reflecting qualitatively different configurations of clinical features. Formal psychological assessment (FPA) attempts to overcome this by identifying the clinical features entailed by observed response patterns. This study illustrates an application of FPA to the cleaning subscale of a questionnaire assessing obsessive–compulsive symptoms and DSM–IV–TR diagnostic criteria for obsessive–compulsive disorder. A deterministic model of the items–criteria relationships was constructed by mapping each item to each diagnostic criterion. The resulting model was tested on a large community sample (N = 4,412). Results indicate that the theoretical model has adequate fit; item error rates and probabilities for each of the criteria are examined. Clinically relevant examples of the items–criteria relationships are discussed. Possible applications of FPA to personality assessment are also discussed, including long multidimensional questionnaires and questionnaires that use subtle item content.

Consumer-Based Brand Equity: Improving the Measurement - Empirical Evidence

Purpose: The present research aims to improve the measurement of consumer-based brand equity. Current measurement of consumer-based brand equity suffers from limitations, including: a lack of distinction between the dimensions brand awareness and brand associations, the use of non-discriminant indicators in the measurement scales and of student samples. Design/methodology/approach: Based on the recommendations of extant research, the scale constructed to measure consumer-based brand equity in this study included brand personality measures. Brand associations were measured using a different set of items. Unlike many of the previous studies that had used student samples, the present study used a sample of actual consumers from an Australian state capital city. Confirmatory factor analysis employing structural equations modeling was used to measure consumer-based brand equity in two product categories and across six brands. Findings: Results support the hypothesized four-dimension model of consumer-based brand equity across two product categories and six brands. Brand awareness and brand associations were found to be two distinct dimensions of brand equity as conceptualized in the marketing literature. The present study contributes to the understanding of consumer-based brand equity measurement by examining the dimensionality of this construct.

Maximum Clique Algorithm and Its Approximation for Uniform Test Form Assembly

Educational assessments occasionally require uniform test forms for which each test form comprises a different set of items, but the forms meet equivalent test specifications (i.e., qualities indicated by test information functions based on item response theory). We propose two maximum clique algorithms (MCA) for uniform test form assembly. The proposed methods can assemble uniform test forms with allowance of overlapping items among uniform test forms. First, we propose an exact method that maximizes the number of uniform test forms from an item pool. However, the exact method presents computational cost problems. To relax those problems, we propose an approximate method that maximizes the number of uniform test forms asymptotically. Accordingly, the proposed methods can use the item pool more efficiently than traditional methods can. We demonstrate the efficiency of the proposed methods using simulated and actual data.

Support for School-Based Obesity Prevention Efforts: Attitudes among Administrators at Nationally Representative Samples of US Elementary Schools

With the continued threat of childhood obesity, many public health intervention efforts focus on school settings. The current study sought to document administrator attitudes regarding obesity and interest in improving relevant school practices (i.e., nutrition and physical activity) in elementary schools. Mail-back surveys were used to gather data from public and private elementary schools during the 2006-2007, 2007-2008, 2008-2009, and 2010-2011 school years. In each year, a different set of items pertaining to administrator attitudes was included. Numbers of responding schools annually ranged from 259 to 336 private schools, and from 578 to 748 public schools. The vast majority of elementary school administrators (>90%) agreed that schools can play a role in addressing childhood obesity, physical education improves a variety of academic outcomes, and they were interested in improving practices at their school. Concern about childhood obesity and perceiving that schools can play a role in addressing obesity were both associated with more interest in improving school practices. However, only one-third of administrators agreed that parents were interested in participating in improving nutrition and physical activity practices, suggesting opportunities for efforts to improve collaboration. Administrators are generally very supportive of school-based efforts to improve nutrition and physical activity practices and see the value in doing so. Given the amount of time children spend in school, schools are an essential venue for efforts to address childhood obesity.

An Analysis of the Teachers’ Sense of Efficacy Scale within the Malaysian Context using the Rasch Measurement Model

Many concerns have been raised about the psychometric properties of instruments when used in different culture. Studies show that adaptation of instrument within a particular context sometimes has different meaning since a particular psychology construct is considered cultural specific. The purpose of the present study is to examine the psychometric properties of the Teachers’ Sense of Efficacy Scale (TSES) within the Malaysian educational context A total of 191 Malaysian in-service and 122 pre-service teachers participated in the study. A Rasch Measurement Model analysis is used to examine the psychometric properties of the scale, particularly the reliability and construct validity. Analyses showed somewhat similar variability with the in-service group demonstrated wider spread of item difficulty compared to the pre-service teachers. One important observation was that both group endorsed different set of items. Consistency of both item difficulty and teachers’ efficacy were high and threats to construct validity in terms of construct irrelevant-variance and construct under-representation were minimum. One worrying finding, however, was that the differential item function (DIF) analysis showed that 13 items (54.17%) behaved significantly different across groups of teachers. Even though several items shown to behave differently across both group of teachers, overall results are encouraging and seem to support the suitability of the TSES to assess teachers’ sense of efficacy within the Malaysian educational context.

The Role of Visual Information in Numerosity Estimation

Mainstream theory suggests that the approximate number system supports our non-symbolic number abilities (e.g. estimating or comparing different sets of items). It is argued that this system can extract number independently of the visual cues present in the stimulus (diameter, aggregate surface, etc.). However, in a recent report we argue that this might not be the case. We showed that participants combined information from different visual cues to derive their answers. While numerosity comparison requires a rough comparison of two sets of items (smaller versus larger), numerosity estimation requires a more precise mechanism. It could therefore be that numerosity estimation, in contrast to numerosity comparison, might rely on the approximate number system. To test this hypothesis, we conducted a numerosity estimation experiment. We controlled for the visual cues according to current standards: each single visual property was not informative about numerosity. Nevertheless, the results reveal that participants were influenced by the visual properties of the dot arrays. They gave a larger estimate when the dot arrays consisted of dots with, on average, a smaller diameter, aggregate surface or density but a larger convex hull. The reliance on visual cues to estimate numerosity suggests that the existence of an approximate number system that can extract numerosity independently of the visual cues is unlikely. Instead, we propose that humans estimate numerosity by weighing the different visual cues present in the stimuli.

Cache Replacement in Mobile Adhoc Networks

Caching of frequently accessed data in Mobile Adhoc NETwork (MANET) is a technique that can improve data access performance and availability. In the situation of sufficient cache space one may think of keeping all the data items in the cache. However, this has overhead in terms of multiple copies of same data distributed all over the network and there are chances of inconsistency of data among different nodes. The main objective of cooperative caching is the optimum utilization of cache by storing different data items in the cache of each Mobile Node (MN). By doing this, different sets of items are kept in the neighboring MNs. As and when some MN requires a data item which is not available with itself it may ask the neighboring nodes for the same. In this manner cooperative caching environment is created in MANETs. Cache replacement is required when certain items are to be stored in the cache, but MN does not have the sufficient space to store the incoming item. This paper proposes a Profit Based Replacement (PBR) where profit value is calculated for every item in the cache. Simulation experiments demonstrate that the PBR improves the cooperative caching performance in comparison to other conventional techniques.

Developing a Placement Exam for Spanish Heritage Language Learners: Item Analysis and Learner Characteristics

This paper illustrates a method of item analysis used to identify discriminating multiple-choice items in placement data. The data come from two rounds of pilots given to both SHL students and Spanish as a Second Language (SSL) students. In the first round, 104 items were administered to 507 students. After discarding poor items, the second round presented 64 items to 330 students. Both graphical and statistical item analyses were employed. Graphical analysis involved an examination of trace-line graphs of each item. A fine-grained statistical analysis was conducted using point-biserial correlation coefficients. Both of these methods were useful and contributed to measure reliability. Different sets of items were selected for each learner group: 31 items for SHL participants and 21 for SSL participants. These items are currently being used in a preliminary online placement exam; after taking a biographical questionnaire, students are piped to either the SHL exam or the SSL exam. Finally, this paper examines characteristics of SHL students found in these data and finds that regional characteristics should be considered in item creation in terms of answer variability and possible distinction between SHL and SSL students.

Item response theory and the measurement of deprivation: evidence from Luxembourg data

Item response theory (IRT) has recently been proposed as a framework to measure deprivation. It allows a latent measure of deprivation to be derived from a set of dichotomous items indicating deprivation, and the determinants of deprivation to be analysed. We investigate further the use of IRT models in the field of deprivation measurement. First, the paper emphasises the importance of item selection and the Mokken Scale Procedure is applied to select the items to be included in the scale of deprivation. Second, we apply the one- and the two-parameter probit IRT models for dichotomous items to two different sets of items, in order to highlight different empirical results. Finally, we introduce a graphical tool, the Item Characteristic Curve (ICC), and analyse the determinants of deprivation in Luxembourg. The empirical illustration is based on the fourth wave of the “Liewen zu Letzebuerg” Luxembourg socioeconomic panel (PSELL-3).

Corporate social responsibility organizational identification and motivation

PurposeThe purpose of this paper is to explore the impact of corporate social responsibility (CSR) engagement on employee motivation, job satisfaction and organizational identification as well as employee citizenship in voluntary community activities.Design/methodology/approachEmployees (n=224) of a major airline carrier participated in the study based on a 54‐item questionnaire, containing four different sets of items related to volunteering, motivation, job satisfaction and organizational identification. The employee sample consisted of two sub‐samples drawn randomly from the company pool of employees, differentiating between active participants in the company's CSR programs (APs) and non participants (NAPs).FindingsSignificant differences were found between APs and NAPs on organizational identification and motivation, but not for job satisfaction. In addition, positive significant correlations between organizational identification, volunteering, job satisfaction, and motivation were obtained. These results are interpreted within the broader context that ties social identity theory (SIT) and organizational identification increase.Practical implicationsThe paper contributes to the understanding of the interrelations between CSR and other organizational behavior constructs. Practitioners can learn from this study how to increase job satisfaction and organizational identification. Both are extremely important for an organization's sustainability.Originality/valueThis is a first attempt to investigate the relationship between CSR, organizational identification and motivation, comparing two groups from the same organization. The paper discusses the questions: “Are there potential gains at the intra‐organizational level in terms of enhanced motivation and organizational attitudes on the part of employees?” and “Does volunteering or active participation in CSR yield greater benefits for involved employees in terms of their motivation, job satisfaction and identification?”.

IRTEQ: Software for Linking and Equating with Item Response Theory

An essential requirement for testing programs where every examinee does not see the same set of items is the capability to link different sets of items together. Such a situation can occur when mul...

An ERP analysis of recognition and categorization decisions in a prototype-distortion task.

BackgroundTheories of categorization make different predictions about the underlying processes used to represent categories. Episodic theories suggest that categories are represented in memory by storing previously encountered exemplars in memory. Prototype theories suggest that categories are represented in the form of a prototype independently of memory. A number of studies that show dissociations between categorization and recognition are often cited as evidence for the prototype account. These dissociations have compared recognition judgements made to one set of items to categorization judgements to a different set of items making a clear interpretation difficult. Instead of using different stimuli for different tests this experiment compares the processes by which participants make decisions about category membership in a prototype-distortion task and with recognition decisions about the same set of stimuli by examining the Event Related Potentials (ERPs) associated with them.MethodSixty-three participants were asked to make categorization or recognition decisions about stimuli that either formed an artificial category or that were category non-members. We examined the ERP components associated with both kinds of decision for pre-exposed and control participants.ConclusionIn contrast to studies using different items we observed no behavioural differences between the two kinds of decision; participants were equally able to distinguish category members from non-members, regardless of whether they were performing a recognition or categorisation judgement. Interestingly, this did not interact with prior-exposure. However, the ERP data demonstrated that the early visual evoked response that discriminated category members from non-members was modulated by which judgement participants performed and whether they had been pre-exposed to category members. We conclude from this that any differences between categorization and recognition reflect differences in the information that participants focus on in the stimuli to make the judgements at test, rather than any differences in encoding or process.

Accounting for Item Variance in Large-scale Databases

GENERAL COMMENTARY article Front. Psychol., 24 November 2010Sec. Language Sciences https://doi.org/10.3389/fpsyg.2010.00200

The conception of phonological awareness in educational psychology

Significant research efforts in the field of cognitive, developmental and educational psychology over the past several decades have been focused on the impact of phonological awareness on children’s early literacy development. A substantial body of research evidence from different countries indicates that there is a powerful relationship between preschool phonological awareness and subsequent literacy achievement in alphabetic orthography. There is little research focused on the understanding of phonological awareness in Lithuania, so the aim of this article is to present a theoretical overview of the main areas of the field and to discuss them. First of all, there is a consensus on the definition of phonological awareness, and phonological awareness is broadly defined as a conscious awareness of separate linguistic units in speech and the ability to carry out mental operations on these units of speech. Unfortunately, the relationship between phonological awareness and its many operationalizations is ambiguous, resulting in both theoretical and practical difficulties. Second, there is still no consensus on the structure of phonological awareness. The question is whether the different sets of items intended to measure phonological awareness reflect a single underlying latent ability or several related abilities. Third, research has identified the general sequence of phonological awareness development as universal across languages, and the transparency of a language influences the rate of normal development of phonological awareness. The development of phonological awareness appears to progress from more global phonological characteristics of words to those representing smaller units, i.e. from the syllable level to the phonemic level. A substantial body of research evidence from English indicates that there is a powerful relationship between preschool phonological awareness and subsequent literacy achievements. However, the pattern of findings from research in transparent writing systems with simpler and more consistent letter–sound relationships than English tends to be inconsistent and sometimes contradictory. Some have reported findings more similar to those in English with a strong predictive effect of preschool phonological awareness on early literacy performance. Others have reported that preschool phonological awareness skills are either less important or irrelevant for future literacy attainment.

A Rasch-based validation of the Vocabulary Size Test

The primary purpose of this study was to provide preliminary validity evidence for a 140-item form of the Vocabulary Size Test, which is designed to measure written receptive knowledge of the first 14,000 words of English. Nineteen native speakers of English and 178 native speakers of Japanese participated in the study. Analyses based on the Rasch model were focused on several aspects of Messick’s validation framework. The findings indicated that (1) the items and examinees generally performed as predicted by a priori hypotheses, (2) the overwhelming majority of the items displayed good fit to the Rasch model, (3) the items displayed a high degree of unidimensionality with the Rasch model accounting for 85.6% of the variance, (4) the items showed a strong degree of measurement invariance with disattenuated Pearson correlations for person measures estimated with different sets of items of 0.91 and 0.96, and (5) various combinations of items provided precise measurement for this sample of examinees as indicated by Rasch reliability indices >0.96. The Vocabulary Size Test provides teachers and researchers with a new instrument that greatly extends the range of measurement provided by other measures of written receptive vocabulary size.