International Assessment Of Adult Competencies Research Articles

Unsupervised Anomaly Detection in Sequential Process Data

Abstract: In this study, we present three types of unsupervised anomaly detection to identify anomalous test-takers based on their action sequences in problem-solving tasks. The first method relies on the use of the Isolation Forest algorithm to detect anomalous test-takers based on raw action sequences extracted from process data. The second method transforms raw action sequences into contextual embeddings using the Bidirectional Encoder Representations from Transformers (BERT) model and then applies the Isolation Forest algorithm to detect anomalous test-takers. The third method follows the same procedure as the second method, but it includes an intermediary step of dimensionality reduction for the contextual embeddings before applying the Isolation Forest algorithm for detecting anomalous cases. To compare the outcomes of the three methods, we analyze the log files from test-takers in the US sample ( n = 2,021) who completed the problem-solving in technology-rich environments (PSTRE) section of the Programme for the International Assessment of Adult Competencies (PIAAC) 2012 assessment. The results indicated that different groups of test-takers were flagged as anomalous depending on the representation (raw action sequences vs. contextual embeddings) and dimensionality of action sequences. Also, when the contextual embeddings were used, a larger number of test-takers were flagged by the Isolation Forest algorithm, indicating the sensitivity of this algorithm to the dimensionality of input data.

An engagement-aware predictive model to evaluate problem-solving performance from the study of adult skills' (PIAAC 2012) process data

The benefits of incorporating process information in a large-scale assessment with the complex micro-level evidence from the examinees (i.e., process log data) are well documented in the research across large-scale assessments and learning analytics. This study introduces a deep-learning-based approach to predictive modeling of the examinee’s performance in sequential, interactive problem-solving tasks from a large-scale assessment of adults' educational competencies. The current methods disambiguate problem-solving behaviors using network analysis to inform the examinee's performance in a series of problem-solving tasks. The unique contribution of this framework lies in the introduction of an “effort-aware” system. The system considers the information regarding the examinee’s task-engagement level to accurately predict their task performance. The study demonstrates the potential to introduce a high-performing deep learning model to learning analytics and examinee performance modeling in a large-scale problem-solving task environment collected from the OECD Programme for the International Assessment of Adult Competencies (PIAAC 2012) test in multiple countries, including the United States, South Korea, and the United Kingdom. Our findings indicated a close relationship between the examinee's engagement level and their problem-solving skills as well as the importance of modeling them together to have a better measure of students’ problem-solving performance.

An Area-Level Hierarchical Bayes Bivariate Threefold Linear Model for US State and County Indicators of Adult Skills in Literacy and Numeracy

Significant enhancements have been made in small area estimation (SAE) methodology because there has been an increased demand for reliable estimates through SAE over the past decades. This article describes the advanced statistical methodology used to produce state and county model-based estimates of average scores and various proficiency levels of adults for all states and counties, using data from the first cycle of the US Program for the International Assessment of Adult Competencies and the American Community Survey. Challenges and issues are discussed especially in light of the small number of sample counties with survey data. Each major stage of the estimation process is discussed, including the approach for generating design-based survey estimates and modelled variances, identifying a set of predictor variables (available and measured consistently for all counties), the hierarchical Bayes linear threefold models, including bivariate models for proportions, and the diagnostics and evaluation.

Alignment of University Competencies With Global Skill Measures

This study analysed two internationally recognised measurements of professional skills: the Programme for the International Assessment of Adult Competencies (PIAAC) and the Digital Competence Framework for Citizens (DigComp 2.2). This analysis aimed to establish a baseline and reference point for a comprehensive understanding of these two scales with the objective of exploring the most relevant competencies and assessing their potential benefits in fostering a technology-driven environment in higher education institutions in Saudi Arabia. The research yielded significant findings regarding the essential competencies that university students should possess, drawing insights from the two assessments and identifying gaps in existing manuals. The research outcomes include valuable insights and best practices that can empower a new generation of university learners to effectively engage with the requirements of Education 4.0.

A Model-Based Approach to the Disentanglement and Differential Treatment of Engaged and Disengaged Item Omissions

Item omissions in large-scale assessments may occur for various reasons, ranging from disengagement to not being capable of solving the item and giving up. Current response-time-based classification approaches allow researchers to implement different treatments of item omissions presumably going back to different mechanisms. These approaches, however, are limited in that they require a clear-cut decision on the underlying missingness mechanism and do not allow to take the uncertainty in classification into account. We present a response-time-based model-based mixture modeling approach that overcomes this limitation. The approach (a) facilitates disentangling item omissions stemming from disengagement from those going back to solution behavior, (b) considers the uncertainty in omission classification, (c) allows for omission mechanisms to vary on the item-by-examinee level, (d) supports investigating person and item characteristics associated with different types of omission behavior, and (e) gives researchers flexibility in deciding on how to handle different types of omissions. The approach exhibits good parameter recovery under realistic research conditions. We illustrate the approach on data from the Programme for the International Assessment of Adult Competencies 2012 and compare it against previous classification approaches for item omissions.

Organizational Capital, Skills and Productivity

This paper studies how industries' investment in organizational capital (OC) and workforce skills relate to productivity, building on OECD estimates of OC, output data from the OECD Structural Analysis (STAN) database, and skill indicators from the OECD Programme for the International Assessment of Adult Competencies (PIAAC). The paper finds that, at the industry level, workers' numeracy and Science, Technology, Engineering and Mathematics (STEM) skills correlate positively with productivity, and that the positive correlation of STEM skills with productivity is generally larger for OC workers. Moreover, dispersion in Information and Communication Technology (ICT) skills harms industry performance. In fact, a gap between the ICT skills of OC and non‐OC workers seems to trigger a “lost in translation” type of mechanism, whereby communication and information flows become less fluid and impinge upon the economic performance of sectors, correlating negatively with productivity.

Language distance and labor market integration of migrants: Gendered perspective.

This paper examines the distinct effects of linguistics distance and language literacy on the labor market integration of migrant men and women. Using data from the Programme for International Assessment of Adult Competencies (PIAAC) 2018 in 16 countries of destination mainly from Europe and more than 110 languages of origin, we assess migrant labor force participation, employment, working hours, and occupational prestige. The study finds that linguistics distance of the first language studied has a significant negative association with labor force participation, employment, and working hours of migrant women, even after controlling for their abilities in their destination language, education, and cultural distance between the country of origin and destination. In contrast, linguistics distance is only negatively associated with migrant men's working hours. This suggests that linguistic distance serves as a proxy for cultural aspects, which are not captured by cultural distance and hence shape the labor market integration of migrant women due to cultural factors rather than human capital. We suggest that the gender aspect of the effect of language proximity is essential in understanding the intersectional position of migrant women in the labor force.

EDUCATION, ADULT LITERACY SKILLS, AND JOB AUTOMATION RISK IN THE OLDER US WORKFORCE

Abstract Despite the overall benefits of technological advancement, ongoing job automation has disrupted the U.S. workforce. Of particular concern is that increasing levels of job replacement—driven by job automation—is likely to be a risk factor for economic uncertainty, both for workers and the labor market. Older workers with lower educational attainment and basic adult literacy are known to face challenges in learning new knowledge and skills, which are currently required to be competitive in a technology-driven labor market. Thus, this study aimed to examine the relationship between educational attainment, literacy proficiency, and previously published job automation risk by occupations. Using the 2012/2014 Program for International Assessment of Adult Competencies (PIAAC) data, we analyzed a nationally representative sample of middle-aged and older workers (age 45-74 years; n = 1,484). The results of the linear regression indicated that higher educational attainment (college or higher vs. less than high school: b = -14.11, p < 0.05) and greater literacy proficiency (score 0-500 points: b = -0.06, p < 0.05) were associated with lower job automation risks (0-100 % replacement in the next decades) among middle-aged and older U.S. workers, accounting for the covariates (R-squared = 0.18). In alignment with the federal Workforce Innovation and Opportunity Act, these findings offer empirical support for policies and interventions that are purposefully designed to boost the human capital (i.e., knowledge and skills) of middle-aged and older workers through adult education and literacy training to lower their job automation risk, thereby increasing their level of job security.

Are general skills important for vocationally educated?

In this paper, we examine the effect of general skills on wages for vocationally educated workers with a qualification at upper secondary level. While general skills are considered crucial for labour market success of workers in general, it is not clear whether this also holds for the vocationally educated workers. We use the recently developed concept of effective skills to identify the relationship between general skills and wages for this group. Using data from the Programme for the International Assessment of Adult Competencies (PIAAC) in 25 countries with advanced economies, we show that general skills strongly affect wages of vocationally educated workers and are just as important a factor as they are for generally educated workers. For vocationally educated males, the effect is especially salient for older workers (aged 45 and above). For vocationally educated females, general skills are most important in the beginning of their career (aged 20 to 35) and at prime age (aged 36 to 45).

Associations between education, information-processing skills, and job automation risk in the United States

Job automation is a topical issue in a technology-driven labor market. However, greater amounts of human capital (e.g., often measured by education, and information-processing skills, including adult literacy) are linked with job security. A knowledgeable and skilled labor force better resists unemployment and/or rebounds from job disruption brought on by job automation. Therefore, the purpose of this study was to advance understanding of the association between educational attainment and literacy, and job automation risk. Using the 2012/2014/2017 Program for the International Assessment of Adult Competencies (PIAAC) data, survey-weighted linear regression was used to model the risk of job automation as a function of education, and literacy proficiency. Higher educational attainment (college or higher vs. less than high school: b = −18.23, p < .05) and greater literacy proficiency (score 0–500 points: b = −.038, p < .05) were associated with a decrease in job automation risk among the U.S. workforce.

Routinization and Employment: Evidence for Latin America

We study changes in employment by occupations characterized by different degree of exposure to routinization in the six largest Latin American economies over the last two decades. We combine our own indicators of routine task content based on information from the Programme for the International Assessment of Adult Competencies (PIACC) with labor market microdata from harmonized national household surveys. We find that the increase in jobs was decreasing in the automatability of the tasks typically performed in each occupation, and increasing in the initial wage, a pattern more consistent with the traditional skill-biased technological change than with the polarization hypothesis.

Identifying literacy and numeracy skill mismatch in OECD countries using the job analysis method

Abstract The Programme for the International Assessment of Adult Competencies (PIAAC) is currently the most important data source that provides information on the key skills possessed by workers, including literacy and numeracy. However, to assess skill mismatch, we also need information on the required skills in those domains, measured in the same metric and scale. In this article, we use the Job Analysis Method (JAM) to determine the required skill levels of literacy and numeracy for all four-digit International Standard Classification of Occupations (ISCO) 2008 unit groups. JAM is often considered the ‘gold standard’ in estimating required levels but has never been used in the context of PIAAC. This article thus presents the first results on the prevalence of skill mismatches using JAM comparing different OECD (Organisation for Economic Co-operation and Development) countries and different occupations. We compare our results with alternative methods and show some advantages of using JAM. We discuss some of the limitations as well.

Reliance on Social Networks and Health Professionals for Health Information in the U.S. Adult Population.

The subpopulation of adults depends on non-online health information sources including their social networks and health professionals, to the exclusion of online sources. In view of the digital divide and health information disparities, the roles of race/ethnicity and digital skills are yet to be explored. A nationally representative sample of 6,830 adults from the Program for the International Assessment of Adult Competencies (PIAAC) was analyzed, using binary logistic regression. Black adults and adults with higher digital skills were less likely to be reliant on non-online health information sources, compared to White adults and those with lower digital skills, respectively. Differences in non-online health information source reliance by race/ethnicity and digital skills might be further nuanced by the relevant demographic and socioeconomic characteristics. Increasing digital skills may expand one's health information sources to include reliable online sources and empower adults to promote their health.

Social functioning and personal development among individuals with low literacy skills; the role of active labour market policy

Abstract Even as education becomes increasingly important for functioning in society, and many welfare states have taken responsibility for providing education, many individuals have insufficient skill levels to fully participate in society. This paper investigates the relationship between literacy skills and basic functioning and participation in society, focusing on the role of the welfare state, and whether individuals with low literacy skills are better off in terms of labour market outcomes, quality of life, digital participation and adult learning in countries with higher investments in active labour market policies (ALMPs), and three underlying spending categories: 1) public employment services, 2) training and 3) private sector employment incentives. Through multi-level analysis of 25 Economic Co-operation and Development (OECD) countries and 139,449 individuals, using individual-level data from the Programme for the International Assessment of Adult Competencies (PIAAC) and country-level data from the OECD, our results show that while low literacy is associated with less favourable conditions related to all outcome variables investigated, ALMPs do not always moderate these negative associations. This is especially true for labour market participation, health and on-the-job training. However, higher ALMP spending is associated with more favourable conditions among low-literate individuals when it comes to job satisfaction, digital participation and life-long learning.

Innovative approaches in the management of an educational institution

The purpose of this study is to investigate the possibility of using digital and other types of innovations in the management of educational institutions in the country. The methods of content analysis of documents and scientific papers, institutional, system-structural approaches, and other means of scientific and theoretical analysis were used in the study. The investigation of the main areas of improving the management of educational organisations in Kazakhstan, various management decisions and attempts to implement state programme measures endows this study with socioeconomic and applied research character. This study shows the shortcomings of the management plan on the development of school curricula for mathematics, natural sciences, and humanities using the results of representatives (schoolchildren) of educational organisations of Kazakhstan passing the international assessment and programme verification (Programme for International Student Assessment, Programme for the International Assessment of Adult Competencies, and Trends in Mathematics and Science Study). The issues of the importance of advanced training of teachers of higher educational institutions, the reform of the entire education system, the problems of technical and technological provision of educational institutions in rural areas of the Turkestan region, digitalisation of the entire learning process were raised. The issues of material and financial security of educational activities in the region and the development of an inclusive education system were also considered. Based on the research of foreign researchers, the peculiarities of the introduction of innovations in education and the informatisation of education were considered.

Determinants of Adult Education and Training Participation in the United States: A Machine Learning Approach

In response to recent shifts in the global economy, there has been a growing academic interest in adult education and training (AET), enabling adults to meet the ever-changing demands of the workforce. However, empirical findings offer nuanced evidence on the most influential factors among various aspects. This study aims to reexamine the previously highlighted determinants of job-related AET participation using the random forest classifiers technique. The data is drawn from the 2017 U.S. Program for the International Assessment of Adult Competencies, where we selected 1,334 respondents with work experience in the last 12 months. Our findings suggest that age and skills use at work were found to be the most important factors for formal AET, whereas skills use at work and organization size were the most significant factors for nonformal AET. Our results emphasize the critical role of skills utilization and organizational support in working adults’ participation in AET.

Investigating Planning and Non-Targeted Exploration in PIAAC 2012: Validating Their Measures Based on Process Data and Investigating Their Relationships with Problem-Solving Competency.

Problem-solving is a critical aspect of intelligence that has become increasingly important in modern society. Mapping out the determinants of success in problem-solving helps understand the underlying cognitive processes involved. This article focuses on two key cognitive processes in problem-solving: non-targeted exploration and planning. We generalize previously defined indicators of planning and non-targeted exploration across tasks in the 2012 Programme for the International Assessment of Adult Competencies and examine the internal construct validity of the indicators using confirmatory factor analysis. We also investigate the relationships between problem-solving competency, planning, and non-targeted exploration, along with the specific dependence between indicators from the same task. The results suggest that (a) the planning indicator across tasks provides evidence of internal construct validity; (b) the non-targeted exploration indicator provides weaker evidence of internal construct validity; (c) overall, non-targeted exploration is strongly related to problem-solving competency, whereas planning and problem-solving competencies are weakly negatively related; and (d) such relationships vary substantially across tasks, emphasizing the importance of accounting for the dependency of measures from the same task. Our findings deepen our understanding of problem-solving processes and can support the use of digital tools in educational practice and validate task design by comparing the task-specific relationships with the desired design.

The Learning Democracy: The Relationship Between Adult Informal Learning, Lifelong Learning Mindset, and Civic Engagement

Using structural regression analyses, data from the U.S. 2017 Program on International Assessment of Adult Competencies (PIAAC) background survey were examined to test the effect of informal learning (observed through everyday activities, such as reading the newspaper or using the Internet to find information about issues) on civic engagement (observed through volunteerism, political self-efficacy, and social trust). Results showed a positive relationship between informal learning and civic engagement with ICT-related activities having the strongest effect. This effect was mediated by presence of a lifelong learning mindset (LLM) observed through traits related to learning motivation and persistence. Although a majority of respondents indicated agreement with LLM attitudes and behaviors, they also reported low frequencies of volunteerism and less agreement with political self-efficacy and social trust statements. The need for governments to provide meaningful opportunities for citizens to contribute their knowledge to the community, thus enabling a learning democracy, is discussed in light of these findings.

Alternatives to Weighted Item Fit Statistics for Establishing Measurement Invariance in Many Groups

Evaluating scale comparability in international large-scale assessments depends on measurement invariance (MI). The root mean square deviation (RMSD) is a standard method for establishing MI in several programs, such as the Programme for International Student Assessment and the Programme for the International Assessment of Adult Competencies. Previous research showed that the RMSD was unable to detect departures from MI when the latent trait distribution was far from item difficulty. In this study, we developed three alternative approaches to the original RMSD: equal, item information, and b-norm weighted RMSDs. Specifically, we considered the item-centered normalized weight distributions to compute the item characteristic curve difference in the RMSD procedure more efficiently. We further compared all methods’ performance via a simulation study and the item information and b-norm weighted RMSDs showed the most promising results. An empirical example is demonstrated, and implications for researchers are discussed.

The Role of the Welfare State for NEETs: Exploring the Association Between Public Social Spending and NEET in European Countries

This study explores the role of the welfare state in reducing young people not being in education, employment, or training (NEET)s across 15 European countries. Using data from the Survey of Adult Skills in the Program for the International Assessment of Adult Competencies (PIAAC) in combination with the Social Expenditure Database, we conducted cross-national analysis to reveal whether the increase in public social spending moderates the risk of being NEET at a young adult age, especially for socially disadvantaged young people. Our results highlighted that the rise of one percentage point of public social spending per gross domestic product, including social expenditure on education, active labor market, and unemployment, is significantly associated with decreasing the odds of being NEET. Furthermore, these social expenditures appeared to lower the NEET risk given socially disadvantaged backgrounds suggesting that young people with low educational levels, whose parents have low educational attainment levels, non-immigrant families, and females are likely to benefit given the robust social protection system. These findings suggest that public social spending may be an effective investment in promoting the social involvement of young people from socially disadvantaged background.