Differential Bundle Functioning Research Articles

Differential Bundle Functioning of National Examinations Council Mathematics Test Items: An Exploratory Structural Equation Modelling Approach

A differential bundle function (DBF) is a situation in which examinees who are of the same ability but are from different groups are required to answer groups of items differently. DBF undermines the validity of the instrument if inadequately considered. The study examines the dimensionality of the 2017 NECO Mathematics items, determines the effect of DBF on 2017 Mathematics items concerning sex, and investigates the effect of DBF on 2017 Mathematics items concerning school ownership. This study explores Exploratory Structural Equation Modelling (ESEM), which permits the cross-loading of items that are not allowed in other models. The ex-post facto research design was adopted using secondary data, while six bundles were generated via the specification table. The population for the study comprised all 1,034,629 Senior School three students. A total of 815,104 students were selected using the simple random technique. The instrument for the study was 2017 NECO Mathematics paper three with a Cronbach's alpha of 0.82, and data were analysed using Mplus 7.4. The results revealed that the 2017 NECO Mathematics is multidimensional and items in the bundles possess construct validity as they functioned differentially to examinees' sex and school type. We recommend ESEM has a better approach to examining DBF on 2017 NECO Mathematics test items.

Effects of multimedia on psychometric characteristics of cognitive tests: A comparison between technology-based and paper-based modalities

The study aims to investigate the effects of delivery modalities on psychometric characteristics and student performance on cognitive tests. A first study assessed the inductive reasoning ability of 715 students under the supervision of teachers. A second study examined 731 students’ performance on the application of the control-of-variables strategy in basic physics but without teacher supervision due to the COVID-19 pandemic. Rasch measurement showed that the online format fitted to the data better in the unidimensional model across two conditions. Under teacher supervision, paper-based testing was better than online testing in terms of reliability and total scores, but contradictory findings were found in turn without teacher supervision. Although measurement invariance was confirmed between two versions at item level, the differential bundle functioning analysis supported the online groups on the item bundles constructed of figure-related materials. Response time was also discussed as an advantage of technology-based assessment for test development.

An Analysis of Differential Bundle Functioning in Multidimensional Tests Using the SIBTEST Procedure

This study aims to analyze the differential bundle functioning in multi-dimensional tests with a specific purpose to detect this effect through differentiating the location of the item with DIF in the test, the correlation between the dimensions, the sample size, and the ratio of reference to focal group size. The first 10 items of the test that is comprised of 30 items were acknowledged as the bundle. The data in line with the parameters were generated via SAS program as two categories (1-0) and multidimensional through an extended 2PL model. Differential bundle functioning was detected via the SIBTEST procedure. The results of the study were interpreted according to the criteria of the power rate and the type I error. When the results were reviewed, the analysis of the bundle revealed that the more the correlation between the two dimensions increased, relatively the less the power rates became. It was observed that the power rates, which were obtained according to two different sample sizes in the study, increased as the sample size increased. Another result as to the SIBTEST's power for detecting DIF was the highest when the ratio of reference to focal group size was equal. According to the results of the type I error rate, the error rate was observed to be relatively decreasing as the correlation between the dimensions increased and it was observed to be increasing as the sample size increased. Also, the highest error rate was obtained when the ratio of the samples was equal.

Detecting Differential Item Functioning Using SIBTEST, MH, LR and IRT Methods

In this study, differential item functioning (DIF) and differential bundle functioning (DBF) analyses of the Academic Staff and Postgraduate Education Entrance Examination Quantitative Ability Tests were carried out. Mantel-Haenszel, logistic regression, SIBTEST, Item Response Theory-Likelihood Ratio and BILOG-MG DIF Algorithm methods were used for DIF analyses. SIBTEST was the method used for DBF analyses. Data sets for the study came from an earlier application of the examination. Gender DIF analyses showed that eleven items showed DIF. Four of the items favored male applicants, where seven of them favored female applicants. In order to investigate the sources of DIF, we consulted experts. In general, the items which could be solved using routine algorithmic operations and which are presented in the algebraic, abstract format showed DIF in favor of females. The “real-life” word problems favored males. According to DBF analyses, the operations item group favored females and the word problems item group favored males.

Test fairness: Examining differential functioning of the reading comprehension section of the GSEEE in China

This study investigated differential item functioning (DIF), differential bundle functioning (DBF), and differential test functioning (DTF) across gender of the reading comprehension section of the Graduate School Entrance English Exam in China. The datasets included 10,000 test-takers’ item-level responses to 6 five-item testlets. Both DIF and DBF were examined by using poly-simultaneous item bias test and item-response-theory-likelihood-ratio test, and DTF was investigated with multi-group confirmatory factor analyses (MG-CFA). The results indicated that although none of the 30 items exhibited statistically and practically significant DIF across gender at the item level, 2 testlets were consistently identified as having significant DBF at the testlet level by the two procedures. Nonetheless, DBF does not manifest itself at the overall test score level to produce DTF based on MG-CFA. This suggests that the relationship between item-level DIF and test-level DTF is a complicated issue with the mediating effect of testlets in testlet-based language assessment.

An Investigation of Item Bias of English Test: The Case of 2016 Year Undergraduate Placement Exam in Turkey

The purpose of this study is to determine whether English test items of Undergraduate Placement Exam (UPE) in 2016 contain differential item functioning (DIF) and differential bundle functioning (DBF) in terms of gender and school type and examine the possible sources of bias of DIF items. Mantel Haenszel (MH), Simultaneous Item Bias Test (SIBTEST) and Multiple Indicator and Multiple Causes (MIMIC) methods were used for DIF analyses. DBF analyses were conducted by MIMIC and SIBTEST methods. Expert opinions were consulted to determine the sources of bias. Data set of the study consisted of responses of 59818 students to 2016 UPE English test. As a result of the analyses carried out on 60 items, it was seen that one item in translation subtest contained DIF favoring male students. In school type based analyses, it was concluded that there were nine DIF items in vocabulary and grammar knowledge subtest, six DIF items in reading comprehension subtest and four DIF items in translation subtest. Experts stated that one item containing DIF by gender was unbiased, and evidence of bias was found in thirteen of nineteen items that contained DIF by school type. According to DBF analyses, it was found that some item bundles contained DBF with respect to gender and school type. As a result of research, it was discovered that there were differences with regard to the number of DIF items identified by three methods and the level of DIF that the items contained; however, methods were consistent in detecting uniform DIF.

Investigating the Effects of Differential Item Functioning on Proficiency Classification.

This study provides a comprehensive evaluation of the effects of differential item functioning (DIF) on proficiency classification. Using Monte Carlo simulation, item- and test-level DIF magnitudes were varied systematically to investigate their impact on proficiency classification at multiple decision points. Findings from this study clearly show that the presence of DIF affects proficiency classification not by lowering the overall correct classification rates but by affecting classification error rates differently for reference and focal group members. The study also reveals that multiple items with low levels of DIF can be particularly problematic. They can do similar damage to proficiency classification as high-level DIF items with the same cumulative magnitudes but are much harder to detect with current DIF and differential bundle functioning (DBF) techniques. Finally, how DIF affects proficiency classification errors at multiple cut scores is fully described and discussed.

Understanding the Country's Underachievement in International Assessment: Differential Item and Bundle Functioning Approach

In educational context, Differential Item Functioning (Dif) and Differential Bundle Functioning (Dbf) are usually used in assessing item bias. In present study, it is demonstrated how results of Dif and Dbf analyses can help in better understanding the reasons behind one country’s underachievement in international assessments. In Timss 2011 mathematics assessment, Croatian fourth graders achieved a mean result that was below the international average, while Slovenian and Serbian students achieved mean results that were above the international average. The results of Dif and Dbf analyses indicate that large portion of items exhibiting differences in favour of Slovenian and Serbian students are related to topics that are not taught in Croatian schools during first four grades. The described methodology can be used in investigating other countries’ (under)achievement in international assessments.

Differential Performance on National Exams

The purpose of this study is to evaluate two methodological perspectives of test fairness using a national Secondary School Certificate (SSC) examinations. SSC is a suit of multi-subject national qualification tests at Grade 10 level in South Asian countries, such as Bangladesh, India, and Pakistan. Because it is a high-stakes test, the fairness of SSC tests is a major concern among public and educational policy planners. This study is a first attempt to investigate test fairness of the national SSC examination of Pakistan using two independent differential item functioning (DIF) and differential bundle functioning (DBF) procedures. The SSC was evaluated for possible gender bias using multiple-choice tests in three core subjects, namely, English, Mathematics, and Physics. The study was conducted in two phases using explanatory item response model (EIRM) and Simultaneous Item Bias Test (SIBTEST). In Phase 1, test items were studied for DIF, and items with severe DIF were flagged in each subject. In Phase 2, the item bundles were analyzed for DBF. Three items were detected with large DIF, one for each subject, and one item bundle was detected with a negligible DBF. Taken together, the results demonstrate that there is no major threat to the validity of the interpretation of examinees’ test scores on the SSC examination. The outcome from this study provided evidence for test fairness, which will enhance test development practices at the national examination authorities.

Assessing the Effect of Language Demand in Bundles of Math Word Problems

Differential bundle functioning (DBF) analyses were conducted to determine whether seventh and eighth grade second language learners (SLLs) had lower probabilities of answering bundles of math word problems correctly that had heavy language demands, when compared to non-SLLs of equal math proficiency. Math word problems on each of four test forms (two at Grade 7 and two at Grade 8) were bundled together if they used the passive voice, conditional clause, relative clause, or a combination of any two. The results showed that the average total scores for SLLs was significantly lower than that for non-SLLs on each test form. However, only two bundles (passive voice and conditional clause at Grade 7 on Form 2) indicated statistically significant DBF against SLLs in favor of matched non-SLLs. An additional step was taken to determine whether the two bundles that showed statistically significant DBF against SLLs had biased the mean total scores for this group.The Walker, Zhang, Banks, and Cappaert (2012) procedure established for this purpose showed that this was not the case. Implications of the results are provided as well as suggestions for future research.

A Synthesis of the Peer‐Reviewed Differential Bundle Functioning Research

The purpose of this article was to present a synthesis of the peer‐reviewed differential bundle functioning (DBF) research that has been conducted to date. A total of 16 studies were synthesized according to the following characteristics: tests used and learner groups, organizing principles used for developing bundles, DBF detection methods used, and types of bundles that indicated statistically significant DBF in the hypothesized direction on multiple occasions. The article concludes with a list of suggestions to individuals who conduct DBF research. For example, effect size guidelines should be established for interpreting the amount of DBF in bundles of items assessed with simultaneous item bias test (SIBTEST), given that it is the most commonly used DBF procedure. This would reduce our reliance on statistical significance testing. General effect size guidelines are needed as well as guidelines for special circumstances like small sample cases. Other useful suggestions are offered as well.

Exploring differential bundle functioning in mathematics by gender: the effect of hierarchical modelling

Researchers interested in exploring substantive group differences are increasingly attending to bundles of items (or testlets): the aim is to understand how gender differences, for instance, are explained by differential performances on different types or bundles of items, hence differential bundle functioning (DBF). Some previous work has modelled hierarchies in data in this context or considered item responses within persons, but here we model the bundles themselves as explanatory variables at the item level potentially explaining significant intra-class correlation due to gender differences in item difficulty, and thus explaining variation at the second item level. In this study, we analyse DBF using single- and two-level models (the latter modelling random item effects, which models responses at Level 1 and items at Level 2) in a high-stakes National Mathematics test. The models show comparable regression coefficients but the statistical significances of the two-level models are smaller due to the larger values of the estimated standard errors. We discuss the contrasting relevance of this effect for test developers and gender researchers.

Validation of a mobility item bank for older patients in primary care

BackgroundTo develop and validate an item bank to measure mobility in older people in primary care and to analyse differential item functioning (DIF) and differential bundle functioning (DBF) by sex.MethodsA pool of 48 mobility items was administered by interview to 593 older people attending primary health care practices. The pool contained four domains based on the International Classification of Functioning: changing and maintaining body position, carrying, lifting and pushing, walking and going up and down stairs.ResultsThe Late Life Mobility item bank consisted of 35 items, and measured with a reliability of 0.90 or more across the full spectrum of mobility, except at the higher end of better functioning. No evidence was found of non-uniform DIF but uniform DIF was observed, mainly for items in the changing and maintaining body position and carrying, lifting and pushing domains. The walking domain did not display DBF, but the other three domains did, principally the carrying, lifting and pushing items.ConclusionsDuring the design and validation of an item bank to measure mobility in older people, we found that strength (carrying, lifting and pushing) items formed a secondary dimension that produced DBF. More research is needed to determine how best to include strength items in a mobility measure, or whether it would be more appropriate to design separate measures for each construct.

IS THERE ANY INTERACTION BETWEEN BACKGROUND KNOWLEDGE AND LANGUAGE PROFICIENCY THAT AFFECTS TOEFL IBT® READING PERFORMANCE?

ABSTRACTThis study investigated the effect of the interaction between test takers' background knowledge and language proficiency on their performance on the TOEFL iBT® reading section. Test takers with the target content background knowledge (the focal groups) and those without (the reference groups) were identified for each of the 5 selected passages based on their self‐identified academic and cultural backgrounds. The test takers were further classified into high and low proficiency groups based on their TOEFL iBT scores. Differential functioning was investigated at the item, item bundle, and passage levels. The results suggested that background knowledge interacted with language proficiency on certain items, which could be attributed to idiosyncratic passage and item characteristics (i.e., characteristics that were specific to a particular passage or item). Only 1 of the 5 passages investigated showed intermediate differential bundle functioning, favoring the focal group for both the high and low proficiency groups. There was no differential functioning at the passage level. This research sheds new light on our understanding of the effects of background knowledge and its interaction with language proficiency in the context of second language reading comprehension. It also has significant practical implications for test developers in advancing fair assessments.

Are Inferential Reading Items More Susceptible to Cultural Bias Than Literal Reading Items?

The purpose of this article is to illustrate a seven-step process for determining whether inferential reading items were more susceptible to cultural bias than literal reading items. The seven-step process was demonstrated using multiple-choice data from the reading portion of a reading/language arts test for fifth and seventh grade Hispanic, Black, and White examinees. The process began at the broadest level of analyzing bundles of items for differential bundle functioning and finished at the narrowest level of analyzing individual items for differential distractor functioning. Some evidence was found to indicate that inferential items are more susceptible to cultural bias than literal items. Implications of the results are discussed, and suggestions for item writers and test developers are given.

Differential Bundle Functioning Analysis of the Self-Description Questionnaire Self-Concept Scale for Kenyan Female and Male Students Using the MIMIC Model

Understanding adolescent self-concept is of great concern for educators, mental health professionals, and parents, as research consistently demonstrates that low self-concept is related to a number of problem behaviors and poor outcomes. Thus, accurate measurements of self-concept are key, and the validity of such measurements, including the popular self-description questionnaire (SDQ), rests on the assumption that instruments used in this regard perform similarly for different groups of individuals (e.g., males and females). When this comparability across groups cannot be guaranteed, resultant scores for individuals from different groups are not comparable so that the validity and utility of these scores is called into question. Over the years, a great deal of literature has grown up around the assessment of such comparability in the framework of differential item functioning (DIF) and differential bundle functioning (DBF). The current study uses a novel approach, in the form of the MIMIC model, to ascertain whether there exists gender-based DBF in the SDQ for Kenyan high school students. Results show that DBF was present for both mathematics and academics self-esteem measures. Implications of these results and of the MIMIC model as a tool for DBF assessment are discussed.

The MIMIC Model as a Tool for Differential Bundle Functioning Detection

Increasingly, researchers interested in identifying potentially biased test items are encouraged to use a confirmatory, rather than exploratory, approach. One such method for confirmatory testing is rooted in differential bundle functioning (DBF), where hypotheses regarding potential differential item functioning (DIF) for sets of items (bundles) are developed based on the substantive nature of the items and expected differences in group performance. Most often, analyses of these bundles for DBF have been conducted using simultaneous item bias test (SIBTEST). The goal of the current study was to introduce an alternative methodology for DBF detection based on the multiple indicators multiple cause (MIMIC) model and to compare this alternative with the traditional SIBTEST-based approach using a Monte Carlo simulation study. The results of this study showed that the MIMIC model performed as well as SIBTEST in ideal conditions, and better when the reference and focal groups had different means on the primary latent trait being measured. In addition, the MIMIC model was more accurate at detecting the presence of DBF for two bundles simultaneously than was SIBTEST. The discussion focuses on recommendations for practitioners.

Establishing Effect Size Guidelines for Interpreting the Results of Differential Bundle Functioning Analyses Using SIBTEST

The purpose of this simulation study was to establish general effect size guidelines for interpreting the results of differential bundle functioning (DBF) analyses using simultaneous item bias test (SIBTEST). Three factors were manipulated: number of items in a bundle, test length, and magnitude of uniform differential item functioning (DIF) against the focal group in each item in a bundle. A secondary purpose was to validate the current effect size guidelines for interpreting the results of single-item DIF analyses using SIBTEST. The results of this study clearly demonstrate that ability estimation bias can only be attributed to DIF or DBF when a large number of items in a bundle are functioning differentially against focal examinees in a small way or a small number of items are functioning differentially against focal examinees in a large way. In either of these situations, the presence of DIF or DBF should be a cause for concern because it would lead one to erroneously believe that distinct groups differ in ability when in fact they do not.

Distinguishing Differential Testlet Functioning From Differential Bundle Functioning Using the Multilevel Measurement Model

This study extends the multilevel measurement model to handle testlet-based dependencies. A flexible two-level testlet response model (the MMMT-2 model) for dichotomous items is introduced that permits assessment of differential testlet functioning (DTLF). A distinction is made between this study’s conceptualization of DTLF and that of differential bundle functioning (DBF) with an emphasis on the practical utility of decomposing differential functioning into item- and testlet-specific components. A small-scale simulation study is also conducted to assess estimation of the MMMT-2 model and its measures of DIF and DTLF when compared with SIBTEST’s estimation of DIF and DBF. Results support use of the MMMT-2 model for assessing impact, DIF, and DTLF for tests that include testlet-based dichotomous items with equal item discriminations.

Exploration of the Validity of Gender Differences in Mathematics Assessment Using Differential Bundle Functioning

The aims of this study are (a) to examine the sources of differential functioning by gender via differential bundle functioning (DBF) in mathematics assessment and (b) to use DBF to explore whether the differential functioning displayed is construct-relevant or construct-irrelevant. Three qualitatively different areas, namely curriculum domains, test modality, and problem presentation, were investigated using the Roussos and Stout (1996) multidimensionality-based differential item functioning analysis paradigm with an adaptation of the Logistic Regression approach to model DBF. An empirical dataset from a mathematics national examination of 11-year-old pupils in England was analyzed. We argue that DBF found in curriculum domains may be construct-relevant of the measure, whereas problem presentation and test modality include arguably construct-irrelevant and so may indicate gender bias.