Traditional Psychometric Techniques Research Articles

Machine learning item selection for short scale construction: A proof-of-concept using the SIMS

ObjectiveThis proof-of-concept paper provides evidence to support machine learning (ML) as a valid alternative to traditional psychometric techniques in the development of short forms of longer parent psychological tests. ML comprises a variety of feature selection techniques that can be efficiently applied to identify the set of items that best replicates the characteristics of the original test. MethodsIn the present study, we integrated a dataset of 329 participants from published and unpublished datasets used in previous research on the Structured Inventory of Malingered Symptomatology (SIMS) to develop a short version of the scale. The SIMS is a multi-axial self-report questionnaire and a highly efficient psychometric measure of symptom validity, which is frequently applied in forensic settings. Results State-of-the-art ML item selection techniques achieved a 72% reduction in length while capturing 92% of the variance of the original SIMS. The new SIMS short form now consists of 21 items. ConclusionsThe results suggest that the proposed ML-based item selection technique represents a promising alternative to standard psychometric correlation-based methods (i.e. item selection, item response theory), especially when selection techniques (e.g. wrapper) are employed that evaluate global, rather than local, item value.

Introducing Machine Learning to Detect Personality Faking-Good in a Male Sample: A New Model Based on Minnesota Multiphasic Personality Inventory-2 Restructured Form Scales and Reaction Times.

Background and Purpose. The use of machine learning (ML) models in the detection of malingering has yielded encouraging results, showing promising accuracy levels. We investigated the possible application of this methodology when trained on behavioral features, such as response time (RT) and time pressure, to identify faking behavior in self-report personality questionnaires. To do so, we reintroduced the article of Roma et al. (2018), which highlighted that RTs and time pressure are useful variables in the detection of faking; we then extended the number of participants and applied an ML analysis. Materials and Methods. The sample was composed of 175 subjects, of whom all were graduates (having completed at least 17 years of instruction), male, and Caucasian. Subjects were randomly assigned to four groups: honest speeded, faking-good speeded, honest unspeeded, and faking-good unspeeded. A software version of the Minnesota Multiphasic Personality Inventory-2 Restructured Form (MMPI-2-RF) was administered. Results. Results indicated that ML algorithms reached very high accuracies (around 95%) in detecting malingerers when subjects are instructed to respond under time pressure. The classifiers’ performance was lower when the subjects responded with no time restriction to the MMPI-2-RF items, with accuracies ranging from 75% to 85%. Further analysis demonstrated that T-scores of validity scales are ineffective to detect fakers when participants were not under temporal pressure (accuracies 55–65%), whereas temporal features resulted to be more useful (accuracies 70–75%). By contrast, temporal features and T-scores of validity scales are equally effective in detecting fakers when subjects are under time pressure (accuracies higher than 90%). Discussion. To conclude, results demonstrated that ML techniques are extremely valuable and reach high performance in detecting fakers in self-report personality questionnaires over more the traditional psychometric techniques. Validity scales MMPI-2-RF manual criteria are very poor in identifying under-reported profiles. Moreover, temporal measures are useful tools in distinguishing honest from dishonest responders, especially in a no time pressure condition. Indeed, time pressure brings out malingerers in clearer way than does no time pressure condition.

Revised Hammersmith Scale for spinal muscular atrophy: A SMA specific clinical outcome assessment tool.

Recent translational research developments in Spinal Muscular Atrophy (SMA), outcome measure design and demands from regulatory authorities require that clinical outcome assessments are ‘fit for purpose’. An international collaboration (SMA REACH UK, Italian SMA Network and PNCRN USA) undertook an iterative process to address discontinuity in the recorded performance of the Hammersmith Functional Motor Scale Expanded and developed a revised functional scale using Rasch analysis, traditional psychometric techniques and the application of clinical sensibility via expert panels. Specifically, we intended to develop a psychometrically and clinically robust functional clinician rated outcome measure to assess physical abilities in weak SMA type 2 through to strong ambulant SMA type 3 patients. The final scale, the Revised Hammersmith Scale (RHS) for SMA, consisting of 36 items and two timed tests, was piloted in 138 patients with type 2 and 3 SMA in an observational cross-sectional multi-centre study across the three national networks. Rasch analysis demonstrated very good fit of all 36 items to the construct of motor performance, good reliability with a high Person Separation Index PSI 0.98, logical and hierarchical scoring in 27/36 items and excellent targeting with minimal ceiling. The RHS differentiated between clinically different groups: SMA type, World Health Organisation (WHO) categories, ambulatory status, and SMA type combined with ambulatory status (all p < 0.001). Construct and concurrent validity was also confirmed with a strong significant positive correlation with the WHO motor milestones rs = 0.860, p < 0.001. We conclude that the RHS is a psychometrically sound and versatile clinical outcome assessment to test the broad range of physical abilities of patients with type 2 and 3 SMA. Further longitudinal testing of the scale with regards change in scores over 6 and 12 months are required prior to its adoption in clinical trials.

Assessing fatigue in persons with cancer: an instrument development and testing study.

Cancer-related fatigue (CRF) is a highly prevalent, subjective experience of patients with and survivors of cancer. Effective assessment of the attributes of CRF from the patient's perspective is essential. The current study developed a measure of CRF from the patient's perspective and determined its psychometric properties for patients with cancer undergoing chemotherapy. Chemotherapy outpatients in a free-standing metropolitan area cancer clinic participated in the current multiphase study. In the instrument development phase, 42 items were generated from a qualitative study with 10 semistructured individual interviews and 6 daily fatigue diaries. These items were revised based on content evaluation by 20 cancer patient content experts. In the instrument analysis phase, the 30 content-validated items were pilot tested by 30 patients with breast carcinoma using cognitive interview techniques. The psychometric properties of the resulting 16-item, 5-point CRF rating scale were tested with 82 patients with breast carcinoma. Principal axis factoring gave a one-factor solution accounting for 59% variance. Coefficient alpha reliability was 0.95 for the final 15-item scale. Convergent validity with the Schwartz Cancer Fatigue Scale was 0.84, concurrent validity with the Geriatric Depression Scale was 0.61, and predictive validity with the Cancer-Related Fatigue Distress Scale was 0.83. This newly developed instrument to assess the subjective fatigue symptoms of patients with cancer demonstrated effective use of diary and interview methods in instrument development and both cognitive interviewing and traditional psychometric techniques in instrument analysis. The instrument has promising psychometric properties, but confirmatory testing is needed.

The Homelife Interview from the Project on Human Developmentin Chicago Neighborhoods: Assessment of Parenting and Home Environment for 3- to 15-Year-Olds

Objective. This study reports on the development of the Project on Human Development in Chicago Neighborhoods Homelife Interview. Traditional psychometric techniques and approaches based on item response theory were used to construct scales from 136 items assessing parental warmth and responsivity, provision of learning activities, parental supervision and monitoring, parental communication skills, routines, and quality of physical environment. Design. A majority of items were taken from 4 versions of the Home Observation and Measurement of the Environment Inventory (Bradley et al., 2000; Caldwell & Bradley, 1984). Data were drawn from 2,685 households (17% European American, 34% African American, and 45% Latin American), randomly chosen from 80 neighborhoods varying in racial/ethnic and socioeconomic composition. Results. A reduced set of 8 scales is proposed-5 scales spanning the entire age range from 3 to 15 years and based on interviewer observation (Parental Warmth, Parental Lack of Hostility, Parental Verbal Skills, Internal Environment, and External Environment), and 3 scales based on parent report, tapping similar constructs for 3- to 6-year-olds and 9- to 15-year-olds but with age-specific items (Developmental Stimulation, Access to Reading Materials, and Outings/Activities). No reliable scales were found for parental monitoring and routines. Conclusions. Results of test reliability functions indicate that a majority of scales function as "risk assessments," distinguishing between families providing lower levels of developmental support and families providing adequate levels of such support.

Information Processing and Testing

A conceptual framework for identifying, developing, and categorizing approaches to ]predicting success in academic or occupational programs is presented. A comparison is made of testing approaches that vary with respect to the measures used as predictors, measurement techniques, and the use of various task and test administration manipulations. Information-processing methods and measures are compared to traditional ability and style dimensions and traditional psychometric techniques. Processes most likely to be important predictors and the manner in which these processes might most profitably be measured are considered along dimensions including predictive validity, diagnostic/prescriptive power, routine diagnostic utility, and the extent to which meaningful comparisons can be made across methods as well as between performance during testing and learning outside the testing situation.

Repetitive Psychometric Measures: Memory-for-Faces

interviews, and psychomotor or psychophysical measures to assess many of the attributes which are best measured by traditional psychometric techniques. Recently, consuuction was begun on a battery of brief Repetitive Psychometric Measures (RPM) designed specifically for making multiple repetitive measurements on the same individual for extended periods of time (Moran & Mefferd, 1959). The first six tests of the battery provided 20 alternate forms equated empirically for difficulty level. These followed the format of tests in French's (1954) kit of reference aptitude and achievement factors. The present report describes a seventh brief repetitive test, Memory-for-Faces. Unlike the ocher six tests, Memory-for-Faces was not selected because it represented a distinct factor. However, the test has face validity as a measure of recognition memory, an ability not measured directly by the other six tests. Moreover, comparable measures of the abiliry to recognize faces have been employed in attempts to measure social intelligence (Gilliland & Burke, 1926; Hunt, 1928; Thorndike & Stein, 1937; Kraess & Witryol, 1955), a personality correlate which would be of value to measure in conjunction with the various abilities measured by the RPM. However, it is not the purpose of the present report to establish correlates of Memory-for-Faces, but rather to present a progress report on the construction and preliminary standardization work with the test. The report also may be used as a preliminary manual by investigators who might wish to use the repetitive measure in exploratory research.