Bookmark Method Research Articles

Empirically Deriving Cut Scores in the Positive Behavioral Interventions and Supports (PBIS) Tiered Fidelity Inventory (TFI) Through a Bookmarking Process

We aimed to identify empirically valid cut scores on the positive behavioral interventions and supports (PBIS) Tiered Fidelity Inventory (TFI) through an expert panel process known as bookmarking. The TFI is a measurement tool to evaluate the fidelity of implementation of PBIS. In the bookmark method, experts reviewed all TFI items and item scores by difficulty in order estimated by an item response theory (IRT) model of TFI data. Over two sessions involving data review and discussion, panelists individually identified which corresponding cut scores in each TFI tier scale indicated minimally adequate fidelity in the TFI, leading to corresponding scores of 83.3% for TFI Tier 1, 84.6% for TFI Tier 2, and 82.4% for TFI Tier 3. Then, the proposed cut scores were compared with the original cut scores of 70% for percentage of schools meeting fidelity and the association between the cut scores and school discipline outcomes using an extant sample. The results of descriptive statistics and multiple t-tests suggested that the bookmark cut score in TFI Tier 1 was reachable and more strongly associated with improved school discipline outcomes. Findings for TFI Tier 2 and Tier 3 were less conclusive.

Standard Setting: A Review of Methods

In the field of education and assessment, cutoff scores are often used to distinguish between levels of achievement on a particular test or assessment. Standard setting is the process of establishing these cutoff scores, which are used to determine passing or failing grades, determine eligibility for certain programs, or identify individuals who may need additional support or intervention. Arbitrary cutoff scores are frequently utilized in many nations and institutions, and most commonly is the 60 % threshold. However, are students scoring around 60% considered competent and proficient? Is this passing threshold acceptable in professions where graduates will be treating and managing patients like Medicine and Dentistry? When implementing standard-setting techniques and when determining the right cutoff point, all of these issues need to be taken into consideration. Rather than using an arbitrary cutoff score of 60% as the passing level, it must be based on a methodical approach that adheres to a scientific technique with a history of validity and reliability. There are many methods which can be used to determine the cut off scores for different examinations. These can be based on judgements of borderline test takers, and these include for examples: Angoff, Nedelsky, and yes and no methods. In general, these methods depend on judges to determine characteristics of certain students who are defined as borderline, where judges evaluate test items before the examination, and assign numerical values for each item based on their belief whether these borderline students can answer these question items or not. Other methods may depend on judgements of examinees. These may include: borderline group, or contrasting group methods. These methods depend on the actual information about individual test takers including their test score and their level of knowledge and skills. Other methods to mention include: bookmark method, body of work, and compromise method.

중학교 수리적성 영역 검사에 대한 Modified-Angoff와 Bookmark 준거설정 방법 비교

본 연구의 목적은 두 가지 준거설정 방법을 비교 분석하여 학교 현장에서의 적용 가능성을 탐색하고자 하는 것으로, 다양한 준거설정 방법들 중에서 현재까지 보편적으로 사용되고 있는 Modified-Angoff 방법과 문항반응이론을 적용한 Bookmark 방법을 사용하여 분할점수를 산출 및 비교하였다. 검사자료는 ‘종합진로직업적성검사(CACV) 중학생용(성태제 외, 2016)’을 사용하였으며 성취수준은 기초미달, 기초, 숙달의 3개 범주로 분류하였다. Modified-Angoff 방법과 Bookmark 방법으로 준거설정을 실시한 결과, Modified-Angoff 방법의 분할점수가 더 높았으며, 1차 평정에 비해 2차 평정에서 준거설정자 간 점수의 표준편차가 줄어들었다. 또한 준거설정자 내 일치도는 Modified-Angoff 방법에서, 준거설정자 간 일치도는 Bookmark 방법에서 더 높았다. 준거설정 전반에 대한 설문 결과, Modified-Angoff 방법과 Bookmark 방법에 대한 선호도가 유사하였으며, 준거설정자의 자격 요건에 관한 질문에는 현장 경력이 있는 교사를 가장 적합하게 여기는 것으로 나타났다. 결론적으로 학교 교사들을 준거설정자로 선정하여 준거를 설정함에 있어서 Modified-Angoff 방법과 Bookmark 방법이 실제적으로 적용하기에 용이하였BR다. 따라서 국내외 교육 분야에서 보편적으로 사용되는 Modified-Angoff 방법과 문항반응이론에 기초한 Bookmark 방법을 사용하고, 두 방법에 의한 결과를 비교하여 볼 것을 권장한다. 특히 Bookmark 방법은 문항별로 준거를 설정하는 것이 아니라 문항순서집에 의하여 준거점수를 결정하므로 준거설정자 간의 점수 차이가 적어 보다 안정적인 준거를 설정할 수 있다. 단, Bookmark 방법을 원활히 적용하기 위해서는 문항반응이론에 대한 기초적인 이해가 전제되어야 한다.

Application of the Bookmark method: setting standard for the ninth-grade mathematics achievement test in China

The purpose of this paper is to apply the Bookmark method to the standard setting. Based on the Rasch Model in item response theory, a ninth-grade mathematics achievement test in china has been taken as an example of the standard setting, and 2 cut scores have been established to distinguish students into different performance levels eventually, namely basic and proficient cut scores. In addition, based on the use of generalizability theory, the standard error of the cut scores and the practical standard error are used as indicators to explore the effect that panelists and the standard setting rounds have made on the precision of Bookmark standard setting results through a mixed design of (p: g) × r. Result shows that the cut scores of basic and proficient were respectively 52.25 and 67.53. Besides, increasing the number of panelists in the group or standard setting rounds will reduce the standard error of the cut scores and the practical standard error. In addition, practical standard error is a necessary reference index when applying generalizability theory to analyze the cut scores established by Bookmark method, while the standard error of cut scores also has a great reference value.

The bookmark method to establish minimum performance standards for intensive care flight paramedics performing helicopter winch rescue

We aimed to use the bookmark method to establish minimum standards for selecting intensive care flight paramedics. Two subject matter expert (SME) focus groups reviewed nine videos of decreasing duration showing land and water helicopter rescue tasks. Focus Group 1 (FG1, n = 10) viewed videos individually off-site, whilst Focus Group 2 (FG2, n = 9) attended a face-to-face session. All SMEs selected the video they judged as the appropriate pace for the task then reviewed feedback. For both groups, the process was repeated until 80% agreement was achieved, or three rounds were completed (whichever occurred first). FG1 and FG2 achieved agreement after two rounds for the land task. For the water task, FG1 did not reach agreement. FG2 reached consensus after two rounds. The selected task durations were similar. The bookmark method is valuable to determine performance standards for performing winch rescue, and SMEs are more likely to reach consensus when face-to-face.

Evidence for Validity and Reliability, and Development of Performance Standards and Cut-Scores for Job-Related Tests of Physical Aptitude for Structural Firefighters.

This multi-part study aimed to revise an existing battery of physical aptitude tests for firefighter applicants. Test validity and reliability were evaluated and performance thresholds were determined. In Part I, 49 structural firefighters rated the similarity between the physical demands of the tests and corresponding work activities. In Part II, 23 participants completed the tests on 3 separate days. In Part III, cut-scores were determined using the Bookmark method by an expert panel of 25 firefighter supervisors. Analysis revealed high levels of validity and reliability. The expert panel provided invaluable direction through a combination of independent and group work, leading to consensus on acceptable completion times. Rigorous processes established scientific credibility for the revised battery of tests. Expert knowledge from firefighter supervisors contributed to determining cut-scores following established scientific methods.

Low Literacy is not Set in Stone

In most industrialized societies, around 10–15% of adults are only able to read at basic levels, a phenomenon commonly termed ‘low literacy’. This study addresses two questions about the development of low literacy during adulthood: (1) How stable is low literacy in adults, that is, what share of adults experiences ascents from, or descents into, low literacy? (2) What risk and protective factors predict such ascents and descents, as well as changes in literacy within the subgroup of low-literate adults? We use large-scale data from the German National Educational Panel Study (NEPS), which provides repeated measures of adults’ literacy spanning up to six years. We identified low-literate adults based on a cut score derived using the bookmark method. We found that ascents from low literacy to higher literacy were more prevalent than descents from higher literacy to low literacy. Almost a third of low-literate adults ascended from low literacy to higher literacy after four to six years. Reading practices emerged as the most important protective factor against descents from higher literacy and predicted gains in literacy within the group of low-literate adults. Overall, our study suggests that low literacy is not an immutable condition in adulthood, but has the potential to change over an individual’s lifetime, meaning that gains in literacy over time are indeed possible among low-literate adults.

Comparison of the validity of bookmark and Angoff standard setting methods in medical performance tests

BackgroundOne of the main processes of determining the ability level at which a student should pass an assessment is standard setting. The current study aimed to compare the validity of Angoff and bookmark methods in standard-setting.Method190 individuals with an M.Sc. degree in laboratory science participated in the study. A test with 32 items, designed by a group of experts, was used to assess the laboratory skills of the participants. Moreover, two groups each containing 12 content specialists in laboratory sciences, voluntarily participated in the application of the Angoff and bookmark methods. To assess the process validity, a 5-item questionnaire was asked from two groups of panelists. To investigate the internal validity, the classification agreement was calculated using the kappa and Fleiss’s Kappa coefficient. External validity was assessed by using five indices (correlation with criterion score, specificity, sensitivity, and positive and negative predictive values of correlation test with criterion score).ResultsThe results showed that the obtained cut-scores was 17.67 for Angoff and 18.8 for bookmark. The average total of items related to the quality of the execution process was 4.25 for the Angoff group and 4.79 for the bookmark group. Pass rates pass rates percentages for the Angoff and bookmark group were 55.78 and 41.36, respectively. Correlations of passing/failing, between employer ratings and test scores were 0.69 and 0.88 for Angoff and bookmark methods, respectively.ConclusionBased on the results, it can be concluded that the process and internal validities of the bookmark method were higher than the Angoff method. For evaluation of the external validity (concordance of the cut score with the criterion score), all five external validity indices supported the bookmark method.

Evaluating Panelists’ Understanding of Standard Setting Data

AbstractSince the No Child Left Behind Act (No Child Left Behind [NCLB], 2001) was enacted, the Bookmark method has been used in many state standard setting studies (Karantonis and Sireci; Zieky, Perie, and Livingston). The purpose of the current study is to evaluate the criticism that when panelists are presented with data during the Bookmark standard setting process, these data are often misunderstood. We collected survey responses from eight panels of teachers who worked on an alternate assessment standard setting workshop. We found that although many panelists understood these data, others misunderstood them. For example, when panelists reviewed panel judgment statistics, some extrapolated beyond what these data represent. Our results include themes describing the types of misconceptions we observed, and the need for training and evaluation related to understanding and use of data used in standard setting. We share some suggestions for consideration when implementing the Bookmark method.

It's Not Just Angoff: Misperceptions of Hard and Easy Items in Bookmark‐Type Ratings

AbstractA common belief is that the Bookmark method is a cognitively simpler standard‐setting method than the modified Angoff method. However, a limited amount of research has investigated panelist's ability to perform well the Bookmark method, and whether some of the challenges panelists face with the Angoff method may also be present in the Bookmark method. This article presents results from three experiments where panelists were asked to give Bookmark‐type ratings to separate items into groups based on item difficulty data. Results of the experiments showed, consistent with results often observed with the Angoff method, that panelists typically and paradoxically perceived hard items to be too easy and easy items to be too hard. These perceptions were reflected in panelists often placing their Bookmarks too early for hard items and often placing their Bookmarks too late for easy items. The article concludes with a discussion of what these results imply for educators and policymakers using the Bookmark standard‐setting method.

Examination of Student Growth Using Gain Score and Categorical Growth Models

In this study, gain score, and categorical growth models were used to examine the role of student (gender and socioeconomic level) and school characteristics (school size and school resources) in the student growth on comprehension skills in language. The participants of this study were 2,416 sixth-grade students in 2011 who became seventh-grade students in 2012. The data was collected through two achievement tests, student and school questionnaires. Two achievement tests were calibrated using the Rasch Model and were scaled using the concurrent estimation method. Moreover, the cut-off scores of these tests were determined by using the bookmark method. Students’ growth was modelled with the gain score and categorical growth models. All data was analyzed using multilevel models. Results showed that some students did not achieve sufficient gains to advance to higher performance levels. Although some schools’ average gains were higher, their performance was still not significant enough in terms of tests’ standards. Moreover, the analyses demonstrated that the student gain scores and growth categories varied significantly among the schools. In addition, the study was able to determine student and school characteristics that have an impact on the students' gain scores and categorical growth. Given the different aspects gained about students’ performance with these models, it is recommended to utilize different growth models in schools.

Some Problems With the Analytical Argument in Support of RP67 in the Context of the Bookmark Standard Setting Method.

The choice of response probability in the bookmark method has been shown to affect outcomes in important ways. These findings have implications for the validity of the bookmark method because panelists' inability to internally adjust when given different response probabilities suggests that they are not performing the intended judgment task. In response to the concerns these findings raise, proponents of the bookmark method argue that such concerns can be addressed by using a response probability of .67. A crucial part of their argument includes the often-repeated claim that the .67 value corresponds with the maximum information for a correct response, which is believed to be beneficial in some way. In this article, it is shown that this claim is mistaken; that the formula upon which the .67 result is based is incorrect; that (for the relevant measurement model) there is no difference between the information for a correct response, for an incorrect response, or for the item overall; and, more generally, that the "maximize information" approach is based on the wrong likelihood function altogether.

Proceedings of Contributed Presentations 10th World Federation of Chiropractic/Association of Chiropractic Colleges Education Conference

Proceedings of Contributed Presentations 10th World Federation of Chiropractic/Association of Chiropractic Colleges Education Conference

Validation of the health literacy scale for diabetes as a criterion-referenced test with standard setting procedures

ObjectiveTo develop and examine the psychometric properties of the Korean Health Literacy Scale for Diabetes Mellitus (KHLS-DM), and to establish reasonable cutoff scores. MethodsInitially, 299 items were generated to measure diabetes-related words, numeracy, and information utilization. Content validity assessment and preliminary tests were conducted. After exploratory factor analysis (EFA) and Rasch analysis, the remaining 65 items were administered to a quota sample of 500 diabetic patients aged 40–74 years. The items were narrowed down to 58 items based on an item fit index. To obtain cutoff scores, Jaeger’s method and the Bookmark method were employed. ResultsA confirmatory factor analysis (CFA) was performed, and a three-factor model was supported (χ2/df = 3.891, CFI = 0.92, TLI = 0.91, RMSEA = 0.04). The overall scores ranged from 0 to 58, and two cutoff points were established. The scale exhibited good internal consistency (Cronbach’s α = 0.83). ConclusionsThe KHLS-DM is a reliable and valid measure with cutoff points to classify individuals into three groups (adequate, marginal, and inadequate). Practice implicationsThe standard setting may be useful for researchers to validate health literacy measures in other countries and populations.

Criterion task work rates in Royal Australian Air Force personnel do not align with the acceptable work rate determined using the Bookmark method

Introduction: Contemporary employment standards literature has advocated an item-mapping approach termed the Bookmark method to establish cut-scores for critical and physically demanding tasks. This investigation explored the application of the Bookmark method as it relates to (i) the determination of an acceptable work rate versus actual work rate during a task, and (ii) the influence of the order in which videoed work samples are presented on the selection of an acceptable work rate.

Evaluating different standard-setting methods in an ESL placement testing context

In language programs, it is crucial to place incoming students into appropriate levels to ensure that course curriculum and materials are well targeted to their learning needs. Deciding how and where to set cutscores on placement tests is thus of central importance to programs, but previous studies in educational measurement disagree as to which standard-setting method (or methods) should be employed in different contexts. Furthermore, the results of different standard-setting methods rarely converge on a single set of cutscores, and standard-setting procedures within language program placement testing contexts specifically have been relatively understudied. This study aims to compare and evaluate three different standard-setting procedures – the Bookmark method (a test-centered approach), the Borderline group method (an examinee-centered approach), and cluster analysis (a statistical approach) – and to discuss the ways in which they do and do not provide valid and reliable information regarding placement cut-offs for an intensive English program at a large Midwestern university in the USA. As predicted, the cutscores derived from the different methods did not converge on a single solution, necessitating a means of judging between divergent results. We discuss methods of evaluating cutscores, explicate the advantages and limitations associated with each standard-setting method, recommend against using statistical approaches for most English for academic purposes (EAP) placement contexts, and demonstrate how specific psychometric qualities of the exam can affect the results obtained using those methods. Recommendations for standard setting, exam development, and cutscore use are discussed.

The Issue of Range Restriction in Bookmark Standard Setting

This article uses data from a large‐scale assessment program to illustrate the potential issue of range restriction with the Bookmark method in the context of trying to set cut scores to closely align with a set of college and career readiness benchmarks. Analyses indicated that range restriction issues existed across different response probability (RP) values and item response theory (IRT) models if one were to apply the Bookmark procedure using intact test forms. Results also suggested that range restriction may still be present if one had access to additional data from an item bank. This demonstration critically highlights challenges that may exist in some practical applications of the Bookmark method due items not being designed to cover the full range of examinee abilities.

Establishment of performance standards and a cut-score for the Canadian Forces Firefighter Physical Fitness Maintenance Evaluation (FF PFME)

The Bookmark method for setting cut-scores was used to re-set the cut-score for the Canadian Forces Firefighter Physical Fitness Maintenance Evaluation (FF PFME). The time required to complete 10 tasks that together simulate a first-response firefighting emergency was accepted as a measure of work capacity. A panel of 25 Canadian Forces firefighter supervisors set cut-scores in three rounds. Each round involved independent evaluation of nine video work samples, where the times systematically increased from 400 seconds to 560 seconds. Results for Round 1 were discussed before moving to Round 2 and results for Round 2 were discussed before moving to Round 3. Accounting for the variability among panel members at the end of Round 3, a cut-score of 481 seconds (mean Round 3 plus 2 SEM) was recommended. Firefighters who complete the FF PFME in 481 seconds or less have the physical capacity to complete first-response firefighting work.

Application of the Rasch Model and the Bookmark Method in Setting Cut Scores in Mathematics

Application of the Rasch Model and the Bookmark Method in Setting Cut Scores in Mathematics

Comparing Yes/No Angoff and Bookmark Standard Setting Methods in the Context of English Assessment

The Yes/No Angoff and Bookmark method for setting standards on educational assessment are currently two of the most popular standard-setting methods. However, there is no research into the comparability of these two methods in the context of language assessment. This study compared results from the Yes/No Angoff and Bookmark methods as applied to a Taiwanese national sixth-grade English assessment composed only of multiple-choice items. Thirty-two panelists implemented both methods to establish three cut scores. The results show there are discrepancies in the final mean cut scores between the two methods, with some large differences. Evaluation data collected from standard-setting panelists indicated similar levels of confidence in setting cut scores, and each standard-setting procedure has its own strengths and weaknesses. Implications of this study and future research directions are presented at the end of this study.