Reliability Of Code Research Articles

Ortaokul Seviyesinde Çevre Okuryazarlık Bileşenleri İle İlgili Ölçek Geliştirme Çalışması

– The purpose of this research is to improve ‘Environmental Literacy Scale’ related to the components of environmental literacy for secondary school students. In this research, survey model is used. In order to specify the validity and reliability of the tests and scales, pilot scheme are done with 258 secondary school students. As a result of the pilot scheme, “Enviromental Knowledge Test” (EKT) consists of 19 multiple choice questions, “Enviromental Affect Scale” (EAS) consists of 15 questions in type of five point likert scale, “Enviromental Behavior Scale” (EBS) consists of 12 questions in type of seven point likert scale, and “Cognitive Skill Interview Form” (CSIF) consists of 1 closed ended and 2 open ended questions are prepared. The reliability analysis are made by using Kuder Richardson 20 (KR-20) internal consistency for knowledge test, Cronbach Alpha (α) for item discrimination, item difficulty index and affect and behaviour scales, coder reliability for internal coefficient of consistence and interview form. As a result of the validity and reliability studies, it is seen that “Environmental Literacy Scale” is valid and reliable

The Impact of Coder Reliability on Reconstructing Archaeological Settlement Patterns from Satellite Imagery: a Case Study from South Africa

AbstractA large archaeological remote sensing project is underway to digitize and classify the pre‐colonial stone walled structures (SWS) on Google Earth satellite imagery in the southern part of Gauteng Province, South Africa. Over 7000 such SWS have been digitized in a study area of some 8000 km2. Several research assistants have been involved in classifying the structures. The problem is that different analysts may assign the same SWS to different types and even digitize their outline differently no matter how well they have been trained. Such inter‐analyst variability is a common problem in many fields. In order to minimize its impact, a thorough study of coder reliability in classification of remotely sensed Iron Age SWS has been initiated. The results show unacceptably high variability in the classification of individual SWS. Several contributing factors have been identified. Surprisingly, at the regional level, relatively high levels of inter‐analyst agreement are seen in the same data. The reason probably has to do with strong agreement on the classification of the most diagnostic structures. This may suffice to produce the replicable results at the regional level. Copyright © 2015 John Wiley & Sons, Ltd.

Developing and Assessing Intercoder Reliability in Studies of Group Interaction

Although group interaction is a rich source of data, it offers many challenges for analysis. We discuss how pretesting can solve many coding and reliability issues before the actual data collection and analysis. We also consider how coding schemes can be developed, how coders can be trained, and how coder reliability can be ascertained. We examine the properties of different estimates of intercoder reliability and detail how these relate to decisions about the data.

Coder Reliability and Misclassification in the Human Coding of Party Manifestos

The Comparative Manifesto Project (CMP) provides the only time series of estimated party policy positions in political science and has been extensively used in a wide variety of applications. Recent work (e.g., Benoit, Laver, and Mikhaylov 2009; Klingemann et al. 2006) focuses on nonsystematic sources of error in these estimates that arise from the text generation process. Our concern here, by contrast, is with error that arises during the text coding process since nearly all manifestos are coded only once by a single coder. First, we discuss reliability and misclassification in the context of hand-coded content analysis methods. Second, we report results of a coding experiment that used trained human coders to code sample manifestos provided by the CMP, allowing us to estimate the reliability of both coders and coding categories. Third, we compare our test codings to the published CMP “gold standard” codings of the test documents to assess accuracy and produce empirical estimates of a misclassification matrix for each coding category. Finally, we demonstrate the effect of coding misclassification on the CMP's most widely used index, its left-right scale. Our findings indicate that misclassification is a serious and systemic problem with the current CMP data set and coding process, suggesting the CMP scheme should be significantly simplified to address reliability issues.

Comparing models of borderline personality disorder: Mothers’ experience, self-protective strategies, and dispositional representations

This study compared aspects of the functioning of mothers with borderline personality disorder (BPD) to those of mothers without psychiatric disorder using two different conceptualizations of attachment theory. The Adult Attachment Interviews (AAIs) of 32 mothers were classified using both the Main and Goldwyn method (M&G) and the Dynamic-Maturational Model method (DMM). We found that mothers with BPD recalled more danger, reported more negative effects of danger, and gave evidence of more unresolved psychological trauma tied to danger than other mothers. We also found that the DMM classifications discriminated between the two groups of mothers better than the M&G classifications. Using the DMM method, the AAIs of BPD mothers were more complex, extreme, and had more indicators of rapid shifts in arousal than those of other mothers. Representations drawn from the AAI, using either classificatory method, did not match the representations of the mother's child drawn from the Working Model of the Child Interview; mothers with very anxious DMM classifications were paired with secure-balanced child representations. We propose that the DMM offers greater clinical utility, conceptual coherence, empirical validity, and coder reliability than the M&G.

The Curious Case of Academic Reviewing

through on her book's suggestion of ironical distancing, but examines in earnest a system she knows well. Literary evocation here is subordinated to formal analysis of interview transcripts, including qualitative analysis through analytic matrices and content analysis using software that standardizes codes and checks inter coder reliability. The quantification yields tables compar ing how three disciplinary clusters, namely the humanities, history, and social sciences, invoke various standards of excellence, including clarity, originality, and signifi cance, as well as in how they construe the ideal of diversity. The tables show sur prising similarity among fields, such as English literature and economics, whose forms of writing and analysis are sharply discrepant. Had her topic extended beyond the humanities and social sciences to chem

Doppler imaging of stellar surface structure

Aims. We present the first Doppler images of the bright RS CVn-type binary ζ And. The star is a magnetically active Kl giant with its rotation synchronized to the 17.8-day orbital period. Our revised lithium abundance of log n = 1.2 places ζ And in the vicinity of Li-rich RGB stars but it is nevertheless a Li-normal chromospherically active binary star. The star seems to undergo its first standard dredge-up dilution. Methods. Four consecutive Doppler images were obtained from a continuous 67-night observing run at NSO-McMath in 1996/97. An additional single image was obtained from a continuous 19-night run at KPNO in 1997/98. These unique data allow to compute a small time series of the evolution of the star's surface structure. All line-profile inversions are done with a modified TempMap version that takes into account the non-spherical shape of the star. Representative test reconstructions are performed and demonstrate the code's reliability and robustness. Results. High and low-latitude spot activity was recovered together with an asymmetric polar cap-like feature. The latter dominated the first half of the two-month time series in 1996/97. The second half showed mostly medium-to-high latitude activity and only a fainter polar spot. The coolest areas were restored with a temperature contrast of about 1000 ± 200 K. Some weaker features at equatorial latitudes were also recovered but these could be partially spurious and appear blurred due to imperfect phase coverage. We use our line profiles to reconstruct an average non-sphericity of R pole /R point = 0.96 which would, if not taken into account, mimic a temperature difference pole-to-equator of ≈ 220 K, especially at the phases of quadrature. Finally, we apply two different methods for restoring surface differential rotation and found a weak solar-type rotation law with a shear ΔΩ ≈ 0.95°/day (α = ΔΩ/Ω eq = +0.049 ± 0.003), i.e. roughly a factor of four weaker at a rotation rate roughly 1.5 times faster than the Sun's.

Public Journalism and Non-Elite Actors and Sources

Public journalism is grounded in its desire to strengthen public life. Through a news work norm of community-involvement, by listening and acting upon a community's collective concerns, public journalism activities aspire to deliver news that engages people in public life. (1) It is evident that community involvement refers to a two-way process: A newspaper focuses its coverage on the community by identifying its problems, issues and stake in public policy. The paper then converses with citizens to mobilize the community as a whole to participate in problem-solving to realize its civic work. As such, community involvement is manifest in several aspects of news work. Foremost among them are reporters' selecting sources that emphasize the average citizen and news that features the ordinary citizen as an active player in the news. Massey (2) suggested community connections would be forged through bottom-up framing of news realities and community issues. He found that there was no evidence of greater news prominence for non-elites. Others reported similar results. (3) Public journalism, by focusing on citizens, emphasizes the use of ordinary people both as news sources and as players in the news. Unlike previous studies, in this one a distinction is made between sources and actors. A source, by having information attributed to him or her, is given a voice. An actor, on the other hand, may or may not possess a voice. For example, in a news story about a hit-and-run accident victim written from a police report, no information would be explicitly attributed to the victim in the form of a direct or indirect quote. In such a case, the source becomes a non-elite actor, but not a non-elite source. This content analysis study evaluated the use of non-elite and elite actors and sources in two daily newspapers: the public journalism newspaper St. Louis Post-Dispatch and the traditional, non-public journalism The Washington Post. It sought to answer the question: Does the practice of public journalism inspire editors and reporters to use more non-elite sources and actors than elite ones? Method The unit of analysis was the news item. Only local spot news and news blurbs or shorts on the front page and metro section were analyzed. Based on stratified random, four different constructed weeks, from June 15,1998, to June 15,1999, were used in the analysis of the two papers. (4) For this study, these five operational definitions were used: non-elite source and actor, elite source and actor, and not applicable (5) In total, 766 stories from The Washington Post (416) and the St. Louis Post-Dispatch (350) were analyzed. A coder reliability test on 65 stories ranged from +.84 to +.93. (6) Results and Discussion The Post-Dispatch used more non-elite sources (13 to 8) and actors (231 to 184) than The Washington Post. What is striking, however, was the significant disparity between the number of non-elite sources and non-elite actors found in the two newspapers. Overall, only 21 non-elite sources were used compared with 415 non-elite actors. In the Post-Dispatch, the ratio of non-elite actors was 1 to 18 compared to 1 to 23 in The Washington Post. Both newspapers were more predisposed toward non-elite actors than non-elite sources. In terms of non-elite sources versus elite sources, the differences between the papers also were statistically significant (chi square=8.81; p=.01). The implication is that even though the Post-Dispatch demonstrated a stronger use of non-elite sources and actors compared to the traditional Washington Post, the results for non-elite sources, when examined in the overall context of non-elite sources and actors, are less extraordinary. Although the public journalism paper used a higher proportion of non-elite sources (1 for ever 5.3 elites), the finding may not contradict what other researchers (6) have reported: Elite rather than non-elite sources dominate daily news in the public journalism newspaper. …

Reliability of central coding of patient reasons for encounter in general practice, using the International Classification of Primary Care

Objective to assess the accuracy, intra- and inter-coder reliability of secondary, centralised coding of patient reasons for encounter (RFEs) with the International Classification of Primary Care (ICPC). Design Almost 150,000 RFEs were secondarily coded with ICPC in a centralised central coding situation by trained coders. Efforts were made to gain high coder reliability. A random sample of 2,369 RFEs was independently assessed for coding accuracy. A further sample of 162 RFEs were tested for inter-coder reliability by comparison with a gold standard and for intra-coder reliability by matched comparison of double coded RFEs. Setting primary health care. Subjects doctor_patient contact records. Main outcome measures Accuracy was qualitatively assessed as: absent; incorrect; acceptable but could be improved; correct. Inter- and intra-coder reliability: mean percentage correct scores calculated at ICPC chapter level, at individual rubric level and within each ICPC chapter. Results Only 1.8% of RFEs were missing, incorrect or needed improvement. Inter-coder reliability at ICPC chapter level was 91.7% and at rubric level, 81.8%. Intra-coder reliability was 96.2% at chapter level and 90.0% at rubric level. Reliability varied with ICPC chapter. Conclusion High coder reliability can be gained with ICPC in a central, secondary coding environment but training, an improved index and coding rules are required.

Reliability of an Idiographic Q-Sort Measure of Defense Mechanisms

Despite the important insights the concept of defense mechanisms may offer to our understanding of human behavior, no standardized definitions of defense mechanisms have been universally accepted. Inconsistencies in the definition and conceptualization of defense mechanisms has limited the practical utility of research involving these constructs. In addition, lack of interrater reliability, use of anecdotal evidence, and reliance on self-reports has retarded their investigation. Conducting methodologically rigorous investigations with a psychometrically sound instrument is the first step in addressing some of the issues concerning defense mechanisms and their theoretical postulates. This study was conducted to determine the reliability associated with the Defense-Q, an observer-based Q-sort measure of defense mechanisms. Thirty participants who had undergone an interpersonally stressful interview (the Type A Structured Interview; Rosenman, 1978) were rated by 11 trained coders, both for their use of the 25 defense mechanisms and for their ego strength. Reliability was assessed using Cronbach's alpha. Individual defense mechanisms demonstrated reliability ranging from .28 (undoing) to .92 (humor), with an average reliability of .73. Coder reliability ranged from .63 to .76, with an average of .69. These results indicate that defense mechanisms can be reliably assessed by the Defense-Q. Reliability of the Defense-Q is compared to existing observational measures of defense mechanisms.

Quantifying tissue damage from childhood injury: the minor injury severity scale.

Described an instrument designed to quantify the extent of tissue severity damage following a minor childhood injury event. Few tools exist to allow objective assessment of injury and those which do exist are oriented toward major injuries necessitating medical intervention. The Minor Injury Severity Scale (MISS) does not require specialized medical experience to apply. It yields a 0-7 score which indexes objective parameters such as depth and length of lesion for 22 different kinds of injuries. The data reported here suggest good coder reliability, excellent test-retest stability, and acceptable correspondence between mothers' and children's reports. Data are presented on divergent validity to show that tissue damage is related to but is not the same as injury outcomes such as pain, fear, and disability. Limitations of the scale and challenges to future research are also discussed.

The New Nutrition Facts Label in the Print Media: A content Analysis

A content analysis was conducted to evaluate the coverage of the new food labels in the print media from December 1, 1992, to August 30, 1993. We used newspaper, magazine, and health newsletter indexes to identify 59 newspaper articles, 16 magazine articles, and 7 health newsletter articles for examination. Articles were evaluated by four trained coders using a pretested coding form addressing 35 aspects of the nutrition label coverage. Twenty percent of articles were double-coded with at least 80% coder reliability. Analysis of the data indicated that health newsletters covered the topic in the most detail, followed by magazines and then newspapers. Ten of 59 (17%) newspaper articles named and defined the term percent daily value, whereas 6 of 7 (86%) health newsletter articles and 9 of 16 (56%) magazine articles provided this information. Analysis of quotes in the articles indicated that more than half of the quotes were from government and industry officials. In contrast, quotes from college and university faculty represented only 5% of total quotes and quotes from dietitians represented less than 8% of total quotes. Coders identified several errors resulting from oversimplification of complex concepts. These findings suggest that dietitians need to increase their exposure with the media and help the media translate complex nutrition labeling information to the public. J Am Diet Assoc. 1995; 95:348-351.

Reliability of Validity Generalization Data Bases

This paper addresses the usefulness of reporting coder reliability in validity generalization studies. The Principles for the Validation and Use of Personnel Selection Instruments of the Society for Industrial and Organizational Psychology state that given the results of meta-analytic studies, validities generalize far more than previously believed; however, users of validity generalization results are required to report the reliability of data entering validity generalization analyses. In response to this concern, reliability coefficients were computed on the validity and sample size between two studies (i.e., data bases) of the Wonderlic Personnel Test and the Otis Test of General Mental Ability. These variables, validity, and sample size, were investigated since these are the crucial components in validity generalization analysis. Results indicated that the correlation between the validities of the two studies was .99 and the correlation between the sample sizes of the two studies was 1.00. To illustrate further the reliability of coding in validity generalization research, separate meta-analyses were conducted on the validity of these tests on each of the two data bases. When correcting only for sampling error, the results indicated that the separate meta-analyses yielded identical results, M = .24, SD = .09. These results show that concerns about the reliability of validity generalization data bases are unwarranted and that independent investigators coding the same data, record the same values and obtain the same results.

Physician Communication with Children and Parents

The way physicians communicate with patients has been shown to affect physiologic measurements, adherence to therapeutic regimens, and satisfaction with medical care. The purpose of this study was to document the content of medical interviews in routine pediatric visits and to identify demographic and situational characteristics that influenced the extent of communication between doctor and child. One hundred fifteen office visits to 49 physicians were videotaped and analyzed. Children studied were 4 to 14 years old with a mean age of 8.5 years. Verbal transactions were coded according to direction of communication, transaction type, and content category. Coder reliability for this system was 0.84. A considerable amount of the total communication, 45.5%, was between doctor and child. Doctors interacted differently with parents and children. More information about the current problem was obtained from children; physicians provided feedback primarily to parents. Parents received 4.4. times as much information as children about the nature and prognosis of a condition. The extent to which doctors talked to children in "substantive" areas was primarily associated with a child's age (r = .52, P less than .001) but was partly influenced by family size (r = .20, P less than .05) and family utilization (r = .22, P less than .02). Race, socioeconomic status, type of problem, and previous encounter with the examining physician did not alter communication patterns. Boys were given more information than girls (6.5% vs 4.0%, P less than .01). We suggest a theoretical framework for future investigation and teaching that identifies the child as an active participant in the medical process.

Methodological Issues in Follow‐Up and Replication Studies

Methodological problems in follow‐up and replication studies and in using projective tests in research are discussed with particular reference to three fear‐of‐success studies: the original research by Horner in 1965, a replication in 1972, and a follow‐up of the original 1965 subjects carried out in 1974. The importance of each of the following is stressed: obtaining responses from the full original sample in follow‐up work; matching the settings in which projective tests are administered; establishing coder reliability between the data sets that are to be compared; and matching the stimulus value of the cue, which may not always mean using the same cue.

A model of properties of compliance‐gaining strategies

This paper intends to induce a set of properties that unify and distinguish compliancegaining strategies and to determine whether coders can reliably classify messages on the basis of the proposed properties. The first goal was accomplished by deriving codified strategies from open‐ended responses of subjects to persuasive situations. Properties that reflected differences in the strategies were induced. The second goal had three coders content‐analyze the original responses in terms of the derived properties. Measures of unitizing and coder reliability and content validity were assessed. In addition, information concerning representational validity was presented. The approach taken in this paper provides us with an assessment of the state of affairs found in a compliance‐gaining strategy.

The Home Observation Assessment Method (HOAM): Real‐Time Naturalistic Observation of Families in Their Homes

The Home Observation Assessment Method (HOAM) is a new method developed to carry out objective coding of family interaction over extended time periods in a home setting. It is a computer-compatible coding system that permits on-line data reduction of interactional variables emphasizing contextual and structural dimensions of family behavior. To date, the HOAM has been applied to the study of 31 families observed for a total of over 250 sessions. Initial analysis of data from these sessions indicates that coder reliability is high and that the HOAM successfully measures dimensions of family behavior independent of architectural aspects of the home setting. Preliminary findings also indicate that although families differ dramatically along the interactional dimensions measured by the HOAM, all families studied spent remarkably little time in decision-making behavior, suggesting that behavior in the home is predominantly "maintenance-oriented."

Changing attitudes toward women in Egypt: the role of fiction in women's magazines

In general, it can be said that in almost all cultures and throughout recorded history, women have been socialised to accept the more traditional and conservative views and to engage in a clearly specified and limited number of activities, usually associated with the household. Social change in a particular era or a specific geographical region, therefore, can most fruitfully be studied through an examination of changes in the role and status of, and attitudes toward, women. Needless to say, it is not sufficient to examine the legal and constitutional formulations to determine the complexity of societal attitudes toward men and women in a particular country. Often, in fact, changes in the law come about as the end result of a lengthy, if not bitter, campaign in the various media of communications. In this paper, we shall examine what attitudes the fiction writers in the only women's magazine in Egypt, Hawwa' (Eve), are hoping to inculcate in their readers. The methodology used is simple and was based, with some modification, on content analysis techniques previously used in a similar study which compared the image of women in the U.S. and Latin America.' All 136 short stories which appeared in the 52 issues of Hawwa' in 1972 (92 Egyptianauthored and 44 translated from West European authors) were analysed. Basically, any and all images and stereotypes of men and women which appeared in the stories were recorded to indicate whether the attitudes toward them were positive, negative or neutral. All stories were read at least once by an Egyptian female graduate assistant and the author to check for coder reliability. Where differences of interpretation arose, they were resolved through discussion.2