Development of diagnostic instrument for studying reading for adults

The Programme for the International Assessment of Adult Competencies (PIAAC) assesses key skills in literacy, numeracy and problem-solving in technology-rich environments, as well as their relationship to other social outcomes for adults. PIAAC’s developers claimed to better account for adults’ literacy practices than earlier international studies such as the International Assessment of Literacy Study (IALS). Through the sociocultural lens of literacy as social practice, the authors explore the tension between PIAAC’s cognitive orientation and its attempt to factor in meaningful literacy practices. Specifically, they analyse PIAAC’s conceptualisation of literacy practices as instantiated in the background questionnaire given to adult participants. They conclude that PIAAC’s conceptualisation does not align well with current theoretical understandings of literacy practice, as evidenced by (1) its conflation of several key literacy terms, including text, genre and practice, and (2) its erasure of context, purpose and social interaction from literate practice. Thus, the authors found considerable room for improvement in the assessment of adults’ actual literacy practices.

This paper describes the main features of PIAAC-L, the German longitudinal follow-up to PIAAC (Programme for the International Assessment of Adult Competencies), and presents the PIAAC-L data. PIAAC-L was a collaborative study by three large-scale surveys in Germany, PIAAC, the National Educational Panel Study (NEPS), and the German Socio-Economic Panel (SOEP). Respondents from the German PIAAC sample (2011/2012) and the adult members of their households were interviewed over three waves (2014, 2015, 2016). PIAAC-L combined design features and instruments from PIAAC, NEPS, and the SOEP and included a re-assessment of basic cognitive skills. Literacy and numeracy were measured with instruments from PIAAC and NEPS and the assessment was extended to include cohabiting spouses/partners of PIAAC respondents. Interviewer-administered person and household questionnaires covered a broad range of content. The PIAAC-L data, which are available to researchers for secondary analyses, allow to explore cognitive skills over time and factors related to their acquisition and maintenance. In the German context, the study is of interest because it combined expertise and content from three national large-scale surveys.

The OECD Programme for the International Assessment of Adult Competencies (PIAAC) was the first computer-based large-scale assessment to provide anonymised log file data from the cognitive assessment together with extensive online documentation and a data analysis support tool. The goal of the chapter is to familiarise researchers with how to access, understand, and analyse PIAAC log file data for their research purposes. After providing some conceptual background on the multiple uses of log file data and how to infer states of information processing from log file data, previous research using PIAAC log file data is reviewed. Then, the accessibility, structure, and documentation of the PIAAC log file data are described in detail, as well as how to use the PIAAC LogDataAnalyzer to extract predefined process indicators and how to create new process indicators based on the raw log data export.

PurposeThis paper aims to provides a brief review of the dashboard literature, an account of the development of performance dashboards for field data collection at Westat, and more specifically for the first cycle of the Programme for the International Assessment of Adult Competencies (PIAAC). A brief concluding section offers suggestions for improvements in survey dashboards for PIAAC’s next cycle.Design/methodology/approachTo manage field work at Westat, the authors create views into various paradata databases and present them in a dashboard, showing key performance indicators at a glance. Users can drill down from the dashboard into underlying databases to investigate potential problems. The US PIAAC dashboard is a monitoring system that supports daily review of many activities. For example, it provides overnight alerts to the field supervisor when global positioning system (GPS) data from an interviewer’s smartphone shows the interview occurred far from the respondent’s home.FindingsPerformance dashboards may represent best practice for monitoring field activities. Paradata sources and systems vary greatly across the PIAAC countries, but a multitude of process data exists in every country and can be used to create quality indicators and a monitoring system. PIAAC can establish standards/guidelines to improve visualization of quality metrics and management data, regardless of the local survey infrastructure.Originality/valueThe core of the paper is a case study of the experiences on the US PIAAC implementation of dashboards to monitor survey quality, production and costs, with special attention to the issue of fabrication.

Skeptics of Programme for International Student Assessment (PISA) and Trend for International Math and Science Study (TIMSS) argue that while US elementary and high school students are behind their peers in other nations, the US workforce is still excellent because of the high quality post-secondary educational institutions in the US. However, the Programme for the International Assessment of Adult Competencies (PIAAC) indicates that US adults are in fact far behind their international peers in literacy, numeracy, and technology-based problem solving. Through the use of data mining, this study explored the possible association between PIAAC scores and several constructs. Since the US, Canada, and New Zealand were considered culturally similar nations, according to cluster analysis, patterns between PIAAC scores and selected constructs were analyzed by a variety of big data analytical methods, including cluster analysis, bootstrap forest, boosted tree, and data visualization. Given that PIAAC used multiple computerized adaptive testing, the consequential plausible values were randomly selected when the ensemble approach was used. Additionally, model comparison was utilized to decide between bagging and boosting in order to select the optimal model for each sample. In these samples, cultural engagement, readiness to learn, and social trust, respectively emerged as strong predictors for learning outcomes as they were assessed by PIAAC.

The present paper investigates the power of personality to predict important life outcomes in the context of the Programme for the International Assessment of Adult Competencies (PIAAC). On the most global level, personality can be described by the Big Five dimensions, extraversion, agreeableness, conscientiousness, emotional stability, and openness to experience. These five dimensions were assessed in the German PIAAC longitudinal study (N = 4122) and can thus be directly related to the central competence and outcome indicators measured in PIAAC. In a first step, we report the relationships between the Big Five dimensions and the basic competencies literacy and numeracy. In a second step, we investigate the extent to which the five personality dimensions can contribute to explaining six important life outcomes, above and beyond competencies and sociodemographic characteristics. Our results indicate that personality is substantially related to all six life outcomes. The portion of variance explained by personality was similar to, and sometimes larger than, that explained by competencies. After adjusting for competencies, personality was incrementally predictive of life satisfaction and health, in particular, and, to a lesser extent, of educational attainment, employment status, and income. The only outcome of which personality was not incrementally predictive over and above competencies was participation in continuing education. Overall, these findings highlight the merit of including measures for the Big Five personality domains in upcoming cycles of PIAAC.

The Programme for the International Assessment of Adult Competencies (PIAAC) of the Organisation for Economic Co-operation and Development (OECD) has transformed international research and policy debates on the assessment of adult skills. Although research using PIAAC data is accumulating, little is known about how these data are used and what they contribute to developing the various disciplines interested in adult skills. In this study, a data-driven approach was used to examine PIAAC-based international research to date. Drawing on a comprehensive analysis of 880 publications, the review found that the field of PIAAC research is young and geographically diverse, with dominant contributions from the United States and Germany. While PIAAC research relies on a broad pool of researchers with high collaboration rates, only a quarter of publications involve international collaboration. The analyses also revealed that the development of the field is based on four interrelated disciplines (education, sociology, psychology and economics) and three differentiated historical paths: theoretical and methodological approaches to the measurement of adult skills, cognitive skills and problem solving in technology-rich environments at the workplace, and the role of adult literacy skills for societal and economic development. Moreover, the PIAAC literature addresses a broad range of topics, including cognitive, non-cognitive and basic skills (e.g. literacy and numeracy), human capital, occupational mismatch, migration, “returns to skills”, informal learning and large-scale assessment methodologies. Implications for further development of PIAAC research for users of PIAAC data, data-providing institutions and policymakers are discussed.

This chapter gives an overview of the most important features of the Programme for the International Assessment of Adult Competencies (PIAAC) survey as it pertains to two main goals. First, only a well-designed survey will lead to accurate and comparable test scores across different countries and languages both within and across assessment cycles. Second, only an understanding of its complex survey design will lead to proper use of the PIAAC data in secondary analyses and meaningful interpretation of results by psychometricians, data analysts, scientists, and policymakers. The chapter begins with a brief introduction to the PIAAC survey followed by an overview of the background questionnaire and the cognitive measures. The cognitive measures are then compared to what was assessed in previous international adult surveys. Key features of the assessment design are discussed followed by a section describing what could be done to improve future PIAAC cycles.

This paper opens a series of publications based on the Program for the International Assessment of Adult Competencies (PIAAC) data. Among the many tasks PIAAC is aimed on the identification and measurement of differences between citizens and countries in the area of key competencies, evaluation of the impact of skills on individual economic and social achievement, the effectiveness of various national educational and training systems in providing necessary skills as well as the creation of the conditions for lifelong learning. PIAAC draws the first “profile” and pathbreaking results of Russian adults’ key competencies on literacy, numeracy and problem solving in technology-rich environment. The central part of the paper is dedicated to a general analysis of the role of competencies and competence-based approach, a description of the research method, sampling and tools used (including test samples) in the PIAAC, and particularly in Russian Federation. Finally, the paper deals with the first results of the adult competencies research in Russian Federation. The results on identified key competencies are presented in accordance with various age groups, and include gender and qualification level differences. IAAC is a long-term research project. Russia participated in the programme in 2009–2013. Over 5000 adults aged between 16 and 25 from 25 regions and 94 localities participated in the programme in Russia. The total sample included 3800 respondents; it is representative for the whole country except Moscow and Moscow Oblast. The average points Russia obtained in reading and mathematical literacy are comparable to those obtained in OECD countries. Nearly half of working-age adults in Russia either don’t have any computer experience or are incompetent users. As compared to OECD citizens, computer skills of Russian adults are more often restricted to doing only simple tasks (like logging in and out of their email boxes). As the level of education grows, so does the level of competencies of adult population in Russia. Similarly, competencies seem to accumulate with ageing, until the peak age of 45–49 years is reached. The discussion of the first PIAAC results and brief statement of research questions for future studies are presented.

A tension raised in recent scholarship is that between numeracy as a social practice and numeracy as a functional skill set. Such frameworks for conceptualizing numeracy pose a challenge to assessment because what individuals do with numeracy is not the same as what individuals can do (or express) in an assessment setting. This study builds on work related to numeracy assessment through a validity examination of a portion of a well-known assessment: the OECD’s Programme for the International Assessment of Adult Competencies (PIAAC). In following a path set out by standards for assessment, I ask: What does the PIAAC numeracy assessment claim to measure? What are the intended uses of the assessment? How are we to interpret scores with those uses in mind? And to what degree do evidence and theory support interpretations for those uses? The main finding from this work is that while score interpretations from the PIAAC numeracy assessment may be valid for the use of describing proficiency distributions for specific groups, the construct of interest—numerate behavior—is not what is measured. Moreover, evidence distinguishing what is measured from other constructs, such as the OECD’s conception of literacy, is largely absent. This study contributes to existing literature on numeracy assessment by providing sources of evidence to consider in making judgments about validity for an assessment. It also suggests that, as scholars, we carefully hedge the ways that we talk about large-scale assessments, and in relation, what individuals can or cannot do based on results from such assessments.

Given the rapidly changing job market in Canada and globally, there have been increasing calls to address the transferable skills gap between higher education graduates and the skills needed for the many new and changing jobs across the labour market. To investigate which fields of study in higher education in Canada produce graduates who go on to use more transferable skills on the job, we created an index of transferable skills use at work from several background questionnaire variables available in the Programme for the International Assessment of Adult Competencies (PIAAC) in Canada survey. A series of four least squares linear regression models were used to examine the impacts of variables such as field of study and occupation type on this transferable skills index. Teacher training and education programs were found to have the highest scores on the transferable skills index, suggesting that these programs (and other professional programs) should be considered as a source of inspiration for how all programs can promote transferable skill development in their students. We also found a connection between transferable skill use and management roles within the workplace, suggesting that transferable skills are important factors in promotion to management roles.

The Survey of Adult Skills, a product of the OECD Programme for the International Assessment of Adult Competencies (PIAAC), used computers as the main assessment deliver platform. This enabled the Programme to collect data not only on whether respondents were able to solve specific tasks, but also on how they approached the problems at hand and how much time they spent on them. This paper draws on this information to characterise individuals’ problem-solving strategies using the longest common subsequence (LCS) method, a sequence-mining technique commonly used in natural language processing and biostatistics. The LCS is used to compare the action sequences followed by PIAAC respondents to a set of “optimal” predefined sequences identified by test developers and subject matter experts. This approach allows studying problem-solving behaviours across multiple assessment items.

This paper uses data from the Program for the International Assessment of Adult Competencies (PIAAC) to analyze the relationship between self-reported health (SRH) and literacy and numeracy proficiency for immigrants compared to U.S.-born respondents and for Hispanic versus Asian immigrants. The research questions were: (1) Are literacy and numeracy scores associated with adults’ SRH? (2) Are associations between SRH and literacy and numeracy proficiency moderated by immigrant status? (3) Among immigrants, are literacy and numeracy scores more strongly associated with SRH for Hispanics versus Asians? Immigrants had significantly lower literacy and numeracy scores, yet reported better health than U.S.-born respondents. Ordinal logistic regression analyses showed that literacy and numeracy were both positively related to SRH for immigrants and U.S.-born adults, and should therefore be viewed as part of the growing evidence that literacy is an independent and significant social determinant of health. Second, U.S.-born and immigrant adults accrued similarly positive health benefits from stronger literacy and numeracy skills. Third, although Hispanic immigrants were more disadvantaged than Asian immigrants on almost all socioeconomic characteristics and had significantly lower literacy and numeracy scores and worse SRH than Asian immigrants, both Hispanic and Asian immigrants experienced similar positive health returns from literacy and numeracy proficiency. These findings underscore the potential health benefits of providing adult basic education instruction, particularly for immigrants with the least formal schooling and fewest socioeconomic resources.

The OECD Survey of Adult Skills is the jewel in the crown of its Programme for the International Assessment of Adult Competencies (PIAAC). This paper argues that the findings and policy lessons from the project to date justify the high hopes which were placed in PIAAC when detailed planning for the project began in 2003. First, it presents a brief recap of PIAAC and its two predecessor international skills surveys. Second, it outlines the main themes which have been investigated to date using data from PIAAC. Third, the main findings and policy lessons drawn from PIAAC are highlighted. Finally, looking forward to the second cycle of PIAAC, for which planning is now underway, the paper suggests some priority areas for improvement to the survey design in order to add to its analytical usefulness and enhance its utility to policy makers.

BackgroundIn Germany, three large-scale surveys–the Level One Study (LEO), the Programme for the International Assessment of Adult Competencies (PIAAC), and the National Educational Panel Study (NEPS)–provide complementary data on adults’ literacy skills that can be harnessed to study adults with low literacy. To ensure that research on low-literate adults using these surveys arrives at valid and robust conclusions, it is imperative to ascertain the comparability of the three surveys’ low-literacy samples. Towards that end, in the present study, we comprehensively assess the comparability of adults with low literacy across these surveys with regard to their sociodemographic and socioeconomic characteristics.MethodsWe used data from LEO, PIAAC, and NEPS. We identified features of the sample representation and measurement of (low) literacy as potential causes for variations in the low-literacy samples across the surveys. We then compared the low-literacy samples with regard to their sociodemographic and socioeconomic characteristics and performed logistic regressions to compare the relative importance of these characteristics as correlates of low literacy.ResultsThe key insight our study provides is that–despite different sample representations and measurement approaches–the low-literacy samples in the three surveys are largely comparable in terms of their socioeconomic and sociodemographic characteristics. Although there were small differences between the surveys with regard to the distribution of gender, educational attainment, and the proportion of non-native speakers within the group of low-literate adults, results revealed that both the prevalence of low literacy and its correlates were largely robust across LEO, PIAAC, and NEPS. Across all three surveys, lower educational attainment emerged as the most significant correlate of low literacy, followed by a non-German language background, unemployment and low occupational status.ConclusionsOur study provides evidence that all three surveys can be used for investigating adults with low literacy. The small differences between the low-literacy samples across the three surveys appear to be associated with sample representation and certain assessment features that should be kept in mind when using the surveys for research and policy purposes. Nevertheless, our study showed that we do not compare apples with oranges when dealing with low-literate adults across different large-scale surveys.