Questionable Validity Research Articles

Development of Kodular Based Interactive Teaching Materials in Meeting Organizing Materials in SMK Negeri 2 Buduran

This study describes the development of kodular-based interactive teaching materials in the essential competencies of holding meetings and making notes, analyzing the feasibility of teaching materials, analyzing student responses, and analyzing the effectiveness of using teaching materials to improve student learning outcomes at SMK Negeri 2 Buduran. This research is development research that refers to the 4D model: define, design, develop, and disseminate. Students from class XII OTKP 1 as the experimental class and class XII OTKP 2 as the control class were used as subjects and samples in this study. Material expert validation sheets, language, graphs, student response questionnaires, and student learning outcomes tests were all used as part of the data collection instrument. Expert validation of questions and materials, language, graphics, item analysis, N-Gain analysis, completeness of student learning outcomes, homogeneity test, normality test, and hypothesis testing are data analysis techniques used. The developed interactive teaching materials were scored in the appropriate category, with 87.6% material experts, 90.6 % linguists, and 94.2 % graphic experts, with the overall criteria assessed in the very decent range. Meanwhile, students' responses to the developed interactive teaching materials got a %age of 90.26% with excellent criteria. It can be concluded that the interactive teaching materials developed are effective as supporting teaching materials to improve student learning outcomes of SMK Negeri 2 Buduran in class XII OTKP.

Moving toward standardization: physician reporting of sleep studies.

Lastra AC, Ingram D, Park J, etal. Moving toward standardization: physician reporting of sleep studies. J Clin Sleep Med. 2023;19(3):595-603.

TEST THE VALIDITY AND RELIABILITY OF ARABIC LEARNING QUESTIONS

This study aims to determine the validity and reliability of Arabic language test items for grade II students of an elementary school without a pesantren (Islamic Boarding School) background in Bekasi City. The research method used is descriptive with a qualitative approach. Data collection techniques used in this research were non-test instruments in the form of observation and test instruments in the form of 10 multiple-choice Arabic language questions, adapted to teaching materials at the existing learning level. Data analysis techniques include reducing, presenting, and drawing conclusions from existing data. Based on the data found, the level of validity of the Arabic language test questions from question numbers 1 to 10 provided 60% is in the valid category and 40% is not. While the level of reliability based on Cronbach's Alpha testing technique is at 0.82 which indicates very reliable. It is hoped that the findings of this research have positive implications for teacher knowledge.

Penerapan aplikasi quizizz sebagai media penilaian hasil belajar ips pembelajaran daring kelas iv sekolah dasar

<p class="Abstract">This study aims to describe the application of the Quizizz Application as a medium for assessing social studies learning outcomes in online learning according to the assessment procedure in the cognitive domain of fourth-grade students at SDN Sondakan No. 11 Academic Year 2021/2022. This research is a qualitative descriptive study. The research data source comes from interview transcripts and questionnaires for fourth-grade teacher, the fourth-grade student questionnaire, and document studies. Data collection techniques were carried out through interviews, questionnaires, and document studies. The data validity test technique used is technique triangulation and source triangulation. This study uses the data analysis model of Miles and Huberman which consists of data reduction, data presentation, and conclusion. The results of this study indicate that the implementation of the assessment using the Quizizz assessment media carried out by teacher at the planning stage of the assessment has not made a question grid and validation questions due to time constraints and the making of scoring guidelines are made at the assessment processing stage. At the stage of reporting and utilizing the results of the assessment, the teacher has not implemented remedial and enrichment due to the lack of time, then the teacher also has not made maximum use of the features contained in Quizizz, especially in the implementation of feedback.</p>

Development of a Chemical Literacy Assessment on Colloid (CLAC) Instrument to Measure Chemical Literacy

The 21st century requires four specific skills namely literacy, inventive thinking, effective communication, and high productivity. Literacy includes basic literacy, scientific literacy, economic literacy, technological literacy, visual literacy, information literacy, and multicultural literacy. The purpose of this research is to produce a chemical literacy assessment instrument on colloidal system material that is suitable for use as a chemistry measurement tool for high school students who meet the requirements of feasibility validity, reliability, and level of difficulty. This research is a form of development research or R&D (Research and Development). The development model used in this research is the Oriondo and Dallo Antonio instrument development model, which has five stages: 1) planning the test, 2) trying out the test, 3) establishing empirical validity (establishing test validity), 4) determining reliability (establishing test reliability), 5) interpretation of scores (interpreting the test scores). The data collection instruments used in this study were questionnaire validation sheets, question validation sheets, and development results questions. The reliability value of the person from the score obtained in the trial was 0.55 indicating that the reliability of the person or students was in the medium category. Reliability value item was 0.48 in the medium category. Overall, the average reliability value or Cronbach Alpha is 0.69 in the sufficient category. So it can be concluded that the score obtained from the assessment instrument is reliable because it has fulfilled a minimum coefficient value of 0.60.

Test-Items Analysis of English Teacher-Made Test

The primary purpose of this study is to find out information about the quality of item analysis on English teacher-made tests related to the difficulty level, item discrimination level, validity, and reliability of the final semester assessment for the twelfth-grade student of SMAN 8 Pontianak in the academic year 2020/2021. The researcher conducted this study using descriptive quantitative analysis with the data obtained from the English teacher-made test, consisting of 40 items with 257 test-takers of twelfth-grade students. The researcher analyzed the test items by using the combinations of Master TAP and SPSS Version 16 applications. Based on the difficulty level, the analysis results show that 22 items were categorized as easy, 14 items as medium, and four items as difficult level. Moreover, the item level of item discrimination shows that 9 items are categorized as poor, 18 items as enough, and 13 items as good index level. The analysis results also show that 23 items as valid questions, while the other 17 are invalid. On the other hand, the questions' reliability level showed satisfactory results, with the K-R20 result of 0.835. The results of the analysis showed that there were some questions that had already of good quality, but some problematic items were also found after being analyzed. Therefore, it is necessary for the teacher to conduct an evaluation of the problematic items so that the assessment process will be better in the future.
 The primary purpose of this study is to find out information about the quality of item analysis on English teacher-made tests related to the difficulty level, item discrimination level, validity, and reliability of the final semester assessment for the twelfth-grade student of SMAN 8 Pontianak in the academic year 2020/2021. The researcher conducted this study using descriptive quantitative analysis with the data obtained from the English teacher-made test, consisting of 40 items with 257 test-takers of twelfth-grade students. The researcher analyzed the test items by using the combinations of Master TAP and SPSS Version 16 applications. Based on the difficulty level, the analysis results show that 22 items were categorized as easy, 14 items as medium, and four items as difficult level. Moreover, the item level of item discrimination shows that 9 items are categorized as poor, 18 items as enough, and 13 items as good index level. The analysis results also show that 23 items as valid questions, while the other 17 are invalid. On the other hand, the questions' reliability level showed satisfactory results, with the K-R20 result of 0.835. The results of the analysis showed that there were some questions that had already of good quality, but some problematic items were also found after being analyzed. Therefore, it is necessary for the teacher to conduct an evaluation of the problematic items so that the assessment process will be better in the future.

IMPACT OF EXPLICIT SCIENTIFIC INQUIRY INSTRUCTION HYBRID MODE WITH SOCIOSCIENTIFIC ISSUE CONTEXT ON STUDENTS’ CRITICAL THINKING SKILLS IN CHEMICAL KINETICS

This study aimed to explore whether Explicit Scientific Inquiry Instruction (ESII) Hybrid Mode in the context of Socioscientific Issues could improve the students' critical thinking skills. The research design was a one group pretest-posttest design. The research sample was 31 science eleventh graders at SMA Mojosari for the academic year 2021/2022. The data was collected using a critical thinking skills test on Chemical Kinetics which consists of 27 valid questions with a Cronbach Alpha reliability of 0.756. A paired-samples t-test showed that the Explicit Scientific Inquiry Instruction (ESII) Hybrid Mode in the Socioscientific Issues context improved students’ critical thinking skills (p = 0.00 < 0.05). The improvement was strong as its N-gain score was 0.58 (upper–medium category) and d-effect size was 1.40 (much larger than typical category). This shows that statistically ESII hybrid mode in the context of SSI can improve students' critical thinking skills. Therefore, this strategy can improve students' critical thinking skills in the instruction that allows students to interpret, analyze, explain, evaluate, and draw conclusions

MOODLE-BASED LEARNING MEDIA DEVELOPMENT OF FLEX MODEL IN IMPROVING MATHEMATICAL HARD SKILLS OF HIGH SCHOOL STUDENTS

The development of the flex media model looks like the beginning of the problems in the classroom, especially in the new normal. Students experience difficulties when given math problems. Students are less able to solve these problems, especially in understanding justifying mathematical ideas and arithmetic skills, and are less interested and enthusiastic when learning. This study aimed to develop a Moodle-based flex model learning media to improve students' mathematical hard skills with three-dimensional material. The study was used to research and develop the four-D model. The instruments used included validation, questionnaires, pre, and post-test questions, small-scale trials of 10 students, and large-scale trials with a purposive sampling technique of 30 class XII students. Data collection techniques include (1) validation questionnaires of media and material experts, (2) practicality questionnaires (student responses), and (3) pre-posttest questions. Material and media expert validator test with very decent results. The results of the student response questionnaire were reasonably practical regarding statements related to the practicality of the Moodle-based flex model media. The N-gain average score indicated a "moderate" improvement category. Based on the study results, the developed media meets valid, practical, and effective criteria for improving students' hard skills

Positron emission tomography studies in adult patients with attention-deficit/hyperactivity disorder.

Attention-deficit/hyperactivity disorder (ADHD) is a neurodevelopmental disorder characterized by inattention, motor hyperactivity, impulsivity, and psychosocial as well as cognitive dysfunction. Although characteristic clinical manifestations have been described, no definitive biomarkers to diagnose ADHD have been established. In this review article, we summarize positron emission tomography (PET) studies conducted in adult patients with ADHD. We found that, although, disturbances of dopamine, serotonin, and norepinephrine functions have been implicated in ADHD, no characteristic findings have been identified from PET studies in patients with ADHD. Several previous PET studies on the central dopaminergic transmission-related ligands in patients with ADHD have shown altered binding of dopamine markers in the basal ganglia. However, no consistent results were observed in the binding characteristics for dopamine transporters and receptors. Findings from PET studies with ligands related to serotonin and norepinephrine pathways showed either unclear clinical significance or low replicability. Therefore, whether alterations of monoamine function may be involved in the pathophysiological mechanism remains to be clarified. The limitations of previous PET studies include their small sample sizes, focus on several kinds of existing ligands, and a questionable validity of the diagnosis (lack of biological diagnostic criteria). To determine the characteristic findings for diagnosing ADHD, further research is needed, and particularly, studies that evaluate new active ligands with specific binding to monoamine pathways should be undertaken.

To Baptize or Not to Baptize: A Practical Guide for Clergy by Sarah Hinlicky Wilson

Reviewed by: To Baptize or Not to Baptize: A Practical Guide for Clergy by Sarah Hinlicky Wilson Mark Nygard To Baptize or Not to Baptize: A Practical Guide for Clergy. By Sarah Hinlicky Wilson. Thornbush Press, 2021. 185 pp. Most seasoned pastors can tell wild and wooly baptismal stories where the context was, to put it mildly, daunting. I remember my first call in North Dakota when grandparents asked me to baptize a new granddaughter but the father was dead set against it. In my second call in Cameroon a woman of traditional tribal religion could not articulate Christian faith in any way but earnestly wanted to join her friends in baptism the next morning. During my third call in Senegal a catechumen advised me that if he were to receive baptism, his father-in-law would simply take his wife away from him. Baptism is certainly a flashpoint in ministry, as any pastor on the planet can testify. Consequently, it would seem that this book should receive wide interest. Based on a dozen years of teaching seminars on Luther and Lutheranism in Wittenberg, Germany, under the auspices of the Lutheran World Federation, it reflects the actual questions and experiences of hundreds of Lutheran pastors “from places as far distant as Greenland and Senegal, El Salvador and Myanmar” (1). The book has two parts. Part I reviews the doctrine of baptism by examining the New Testament witness—“Baptism is the beginning of life in the Spirit and the first act of association with the church” (15); pondering what baptism does—“Baptism saves,” the stark [End Page 450] two-word paragraph beginning the entire section (18); debating infant baptism versus believer’s baptism—“In the New Testament, acts of prayer and vicarious faith appear again and again in the Gospels” (27); affirming pastoral authority to baptize—“Hopefully you are so intoxicated by the gospel that the very prospect of withholding baptism fills you with repugnance. And yet . . .” (32); and making a distinction between the validity of baptism and its effectiveness— “No amount of doubt, denial, or even outright rejection can undo a baptism” (40). Most Lutherans and their ecumenical partners would not only recognize themselves in this discussion, but rise to its vigorous expression. Part II brings the discussion rudely down to earth by presenting 49 case studies of problematic baptismal situations using three considerations: questions of validity, questions of integrity of witness, and questions of safety or permission. For example, does one baptize someone who was not baptized in the name of the Father and of the Son and of the Holy Spirit, but in the name of the Creator, Redeemer, and Sanctifier, or maybe in the name of Jesus (the question of validity)? How about those who are using it to help them get a job, or tourists who would find it so meaningful to do it again in the Jordan River (risk of integrity of witness)? What about a teen in the youth group whose parents forbid it, a Jew seeking political protection through it, or a hospitalized person whose perilous medical condition makes human proximity problematic (the safety/permission issue)? In each case Wilson’s presentation follows the same pattern: “The Situation,” “Considerations,” and an answer, “To Baptize or Not to Baptize.” That very pattern will be off-putting to some. Wilson herself warns, “Chances are pretty good that at least one, and probably more than one, case in this book will make you uncomfortable because you landed on the other side of the judgment offered here” (7). But she makes her cases with humility, “as a pastor among pastors” (8), and the theological reflection she elicits is renewing. Frankly I enjoyed the book immensely, and I heartily recommend it as lighter minis-terial reading for pastors and accessible theology for lay discussion. [End Page 451] Mark Nygard St. Paul, Minnesota Copyright © 2022 Johns Hopkins University Press and Lutheran Quarterly, Inc

The questionable validity of attention bias variability: Evidence from two conceptually unrelated cognitive tasks.

Attention bias variability is thought to measure fluctuations in attention towards and away from threat-related information and is elevated in affective disorders. However, recent evidence suggests that attention bias variability may quantify general reaction time variability rather than attention bias behavior per se. The current study calculated "attention bias variability" from two conceptually unrelated cognitive tasks: the dot-probe task (measuring attentional bias) and the arrow flanker task (measuring cognitive control). Attention bias variability measures were correlated across these unrelated tasks. Yet, when general reaction time variability was controlled, attention bias variability across tasks was no longer correlated. In addition, the reliability of attention bias variability measures decreased when controlling for general reaction time variability. Finally, although attention bias variability calculated from the dot-probe task initially correlated with anxious symptoms, this association was no longer significant when controlling for general reaction time variability. Our sample was comprised of high trait anxious individuals. Replication in clinical samples is warranted. These findings collectively provide strong empirical evidence that attention bias variability is not a valid measure of attention-related behavior, but reflective of general reaction time variability more broadly.

Pengembangan Soal Hots Pada Materi Sistem Pencernaan Manusia Kelas VIII SMP/MTs

The problem in this research is motivated by data about daily assessment questions that still use low-level abilities (LOTS), while 21st century learning requires students to have higher-order thinking skills. This research aims to develop and determine the theoretical and empirical validity of the HOTS questions on the human digestive system for class VIII SMP/MTs. The research conducted is a type of research development, this research produces HOTS questions on the Human Digestive System. The development of this HOTS question refers to the ADDIE model development procedure which consists of five stages, (analyze), (design), (development), (implementation) and (evaluation). This development research produces a product HOTS questions on the material of the human digestive system class VIII, which consists of 8 questions. The results of the average score percentage of theoretical validity by material experts, construction experts, and science teachers are 95.3%, 94.7%, and 98.7%, very valid categories. and empirical validity which contains the validity of the items with 8 questions declared valid, with an average score of 0.597, Reliability resulting in an average score of 0.627 being declared reliable, Distractor producing an average score of 18.7% with poor criteria, and discriminating power produce an average score of 0.35% with sufficient criteria. This shows that the product of HOTS questions that have been developed is feasible to use.

The Effect of Auditory, Intellectually, And Repetition (Air) Model on Students’ Reading Comprehension at Eleventh Grade of SMAN 8 South Bengkulu

The purpose of this study was to determine whether there was an effect of the Auditory, Intellectually, And Repetition (Air) Model on Reading Comprehension in understanding the explanation text. This study used a Quasi-Experimental research design. In this study, the sample size was 48 students, including 24 students in the experimental class (XI IPS 2) and 24 students in the control class (XI IPS 1) at SMAN 8 South Bengkulu taken by purposive sampling. The experimental class has given the treatment using the Auditory, Intellectually, And Repetition AIR Model, while the control class did not use the AIR Model. The researchers collect data by administering tests twice, namely a pretest and a post-test. Prior to administering the post-test, the researcher administered four treatments in the experimental and control classrooms using different teaching models. The instrument employed in this study was a set of 20 valid multiple-choice questions. The data from the pre-test and post-test were examined using the t-test. The learning outcomes were as follows: First, the experimental class's mean score was 37,29, whereas the control class was 40,83. And for the post-test findings reveal that the experimental class has an average value of 81.25 while the control class has an average value of 70.42. Second, the sample score of the independent T-test shows the significant value (2 tailed) is 0,000 <0.05. In other words, Ho was rejected and Ha was accepted. In brief, Auditory, Intellectually, And Repetition (AIR) Model can be one of effective ways in teaching reading for eleventh grade students.

E-Book Development of Static Fluid Contents to Improve Students' Numeracy Literacy Competence

This study aims to develop an E-Book on Static Fluid Content in Tutoring as an educational partner to improve students' numeracy literacy competence. E-Book development is carried out using the 4-D development model which consists of defining, designing, developing, and disseminating stages. The research instruments used were E-Book validation sheets and questions that were tested on material expert lecturers and media expert lecturers from Surabaya State University. The validation results show that the E-Book developed has a very valid quality, with a validity percentage of 97%. Aspects of presentation feasibility, content feasibility and linguistic feasibility in the E-Book as a whole meet the criteria of "Very Valid" (85-100%) based on the validity interpretation table. In addition, the validity of the pre-test and post-test questions in the E-Book also proved high with a validity percentage of 89%. The questions measure students' ability to identify facts, formulate mathematical problems, use mathematical concepts, facts, procedures, and reasoning in the context of static fluids, carry out calculations based on certain rules or formulas, and interpret numeracy to solve problems. With the results of a high percentage of validity, this Static Fluid E-Book is reliable and feasible to use as teaching material media in improving students' numeracy literacy competencies. Interactive features in the E-Book, such as "Go Watch", "Mind Map", "Go Information", "Go Concept", "Problem Drilling", and "Reflection", provide interesting and diverse learning experiences for students in understanding static fluid concepts.

Counseling Group and Games Education in Developing English Language for Students’ Elementary School

The Article analyzed Counseling Group and Games Education in Developing English Language for Students in elementary school. The researcher used the steps for a literature review to obtain scientific evidence (Snyder, 2019; Peters et al., 2020), that is, of counseling groups and games education in developing the English language for students in elementary school. Hence, the researchers have visited several scientific publications on several applications, for instance, Eric Publications, Google Book, Taylor, and France, and many others. In comparison, the review process involves several stages. First, giving coded the data, evaluated and looked for relevance, and interpreted it in depth to conclude if the findings were relevant in answering valid questions. Based on the review, many kinds of publications showed that Innovative games could be in the form of educational games. Educational games are created using computer technology by taking into account the peculiarities, language development, and theories of acquiring children's language so that children are not easily bored in recognizing or learning English vocabulary. Therefore, English educational games can be used to develop English vocabulary.

Validity of Simple Self-Reported Periodontal Status Questions

The aim of this research was to evaluate the validity of the questions developed by Thai periodontists on self-reported periodontal status and symptoms in identifying severe periodontitis amongst adults in Thailand. Registered Electricity Generating Authority of Thailand (EGAT) employees completed medical examinations and full-mouth periodontal examinations. They also were interviewed using a self-reported questionnaire that was developed by Thai periodontists. The questions pertained to their periodontal status and symptoms comprising swollen gums, bad breath, loose teeth, bleeding on brushing, painful gums, and pus or abscesses. The participants were categorised as having nonsevere and severe periodontitis according to the Centers for Disease Control and Prevention in partnership with the American Academy of Periodontology (CDC/AAP) case definitions. Self-reported periodontal status and symptom results were compared with actual periodontal status. The area under the receiver operating characteristic curve (AUROCC), sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated. A total of 1393 EGAT employees participated in this study. The questions on self-reported periodontal status and all symptoms poorly identified patients with severe periodontitis, with an AUROCC of 0.52 to 0.60. The sensitivity was 5.0% to 40.2%; however, the specificity was good, at 73.2% to 99.1%. The validity of the self-reported questions was comparable amongst the sex, education, and income subgroups. This study demonstrated that self-reported periodontal status and symptoms were inadequate in identifying patients with severe periodontal disease.

PENGEMBANGAN TES SUBJEKTIF BERBASIS HIGHER ORDER THINKING SKILL (HOTS) PADA MATERI POLA BILANGAN

This research is to develop a subjective test instrument based on Higher Order Thinking Skills (HOTS) on number pattern. The research method used in this research is research and development (RD). This study uses the ADDIE model which consists of five stages: including analysis, design, development, implementation, and evaluation (Sugiyono, 2017). The results of the CVR test of 3 items showed that all questions met the appropriate criteria but were recommended for improvement. A total of 3 questions include material, construction, and language aspects. Then a limited field trial was conducted on 20 class VIII students of SMPN 2 Lembang. 3 valid questions used for wide field trials on 30 class VIII students of SMPN 2 Lembang. In the limited field test the reliability of the test is 0.615, while the broad field test reaches 0.642. Based on the discriminatory power, in the limited field test and the wide field test, the discriminatory power of the questions in both situations is quite good. Based on the level of difficulty in the limited field test questions, 3 moderate questions were found, while in the wide field test, 1 easy question and 2 medium questions.

"Simultaneously Vague and Oddly Specific": Understanding Autistic People's Experiences of Decision Making and Research Questionnaires.

Autism researchers often use questionnaires to gather the views and experiences of autistic people. However, questionnaires may not always be designed in accessible ways. In addition, answering questions within a questionnaire involves decision making, which some autistic people have reported finding difficult. Therefore, this exploratory study aimed to enhance our understanding of autistic people's experiences of decision making, and to analyze their feedback on questionnaire measures to further understand decision making within the research context. One hundred seventeen participants completed an online questionnaire. In the questionnaire, they answered an open question about what affected their ability to make decisions. They then completed four questionnaire measures and after each one provided feedback. We used content analysis to categorize participants' qualitative answers. Participants discussed how their internal state, other people, the quality and quantity of information, pressure on choosing a response, external distractions, and lack of time all affected their decision making. Feedback on the questionnaires highlighted how questions needed context, often questions themselves were unclear and difficult to understand, that there were issues with Likert scales, and how measures could have questionable validity for autistic people. Autism researchers need to consider how they can make their research as accessible as possible for autistic people. Our study highlights how decision making is not a straight-forward process, and researchers have a role in ensuring they give their participants clear and contextualized information. Involving autistic people in the design of research is a potential way of improving the quality of research.

The illusion of validity: how effort inflates the perceived validity of interview questions

ABSTRACT Interviewers are often confident in the validity of their interview questions. What drives this confidence and is it justified? In three studies, we found that question creators judged their own interview questions as more valid than when the same questions are judged by an evaluator. We also found that effort expenditure inflated the perceived validity of interview questions but not question quality. Question creators’ perceptions of validity were primarily driven by their self-confidence, and not the question quality. As an intervention, we nudged participants into holding more favourable attitudes towards better questions (i.e., structured questions) by allowing them to choose a subset of them from a pre-written list. Together, we found that while effort expenditure was responsible for the illusion of validity when evaluating unstructured (i.e., low-quality) questions, the same mechanism could also be used to improve interviewers’ acceptance of structured questions. Implications for structured interviews and the scientist-practitioner gap are discussed.

Clinical questions in primary care: Where to find the answers - a cross-sectional study.

Clinicians raise at least one question for every two patients they see, but search for an answer to less than half of these questions, and rarely use evidence-based resources. One barrier to evidence-based practice is doubt that the search would yield an answer, and we found insufficient evidence to refute this concern. This study aims to identify what proportion of clinical questions in primary care can be answered with online evidence-based practice resources, and what proportion of these can be answered with pre-appraised evidence. Cross-sectional study in two primary care practices. The inclusion criteria were family doctors, generalists and residents working in 2 selected practices. We collected a total of 238 questions from 19 family medicine specialists, 9 family medicine residents and 3 generalist doctors. Doctors were asked to record any clinical question that arose during 4 days of appointments. The primary outcome was the proportion of clinical questions answered with online evidence-based practice resources. The secondary outcome was the level of evidence needed to reach to find the answers (clinical summaries, systematic guidelines, systematic reviews or primary studies), according to Haynes' pre-appraised evidence pyramid model. 191 of the 206 valid clinical questions could be answered with online evidence-based practice resources (92.7% [95% CI 88.3%-95.9%]). Most of these questions (90.8% CI 95% 85.9%-94.4%) were successfully answered using clinical summaries (BMJ Best Practice, DynaMed or UpToDate), with a median search time of 4 minutes (range 1-16.5). Contrary to clinician's beliefs, the majority of clinical questions can be answered with online evidence-based practice resources, and most of them with pre-appraised evidence. This study could encourage family doctors to increase the use of clinical summaries. Furthermore, these results highlight the importance of teaching how to search for and apply pre-appraised evidence.