Criteria-Based Content Analysis Research Articles

Can credibility criteria be assessed reliably? A meta-analysis of criteria-based content analysis.

This meta-analysis synthesizes research on interrater reliability of Criteria-Based Content Analysis (CBCA). CBCA is an important component of Statement Validity Assessment (SVA), a forensic procedure used in many countries to evaluate whether statements (e.g., of sexual abuse) are based on experienced or fabricated events. CBCA contains 19 verbal content criteria, which are frequently adapted for research on detecting deception. A total of k = 82 hypothesis tests revealed acceptable interrater reliabilities for most CBCA criteria, as measured with various indices (except Cohen's kappa). However, results were largely heterogeneous, necessitating moderator analyses. Blocking analyses and meta-regression analyses on Pearson's r resulted in significant moderators for research paradigm, intensity of rater training, type of rating scale used, and the frequency of occurrence (base rates) for some CBCA criteria. The use of CBCA summary scores is discouraged. Implications for research vs. field settings, for future research and for forensic practice in the United States and Europe are discussed. (PsycINFO Database Record

�� ���� � ����� ��������������� ������ ����������� ������������ ���

The present study addresses the problem of how the two US presidential candidates Donald Trump and Hillary Clinton use statements judged to be false by the Politifact site while delivering their campaign speeches. Two corpora of Clinton’s and Trump’s alleged lies were compiled. Each corpus contained 16 statements judged to be false or ridiculously untrue (‘pants on fire’) by the Pulitzer Prize Winner site Politifact. Some statements were accompanied by the video recordings where they appeared; others had no video recordings affiliated because they are either tweets or their events had not been recorded on Youtube or elsewhere. The present research made use of CBCA (Criteria-based Content Analysis) but as a stepping stone for building a new model of detecting lies in political discourse to suit the characteristics of campaign discourse. This furnished the qualitative dimension of the research. As for the quantitative dimension, data were analyzed using software, namely LIWC (Linguistic Inquiry & Word Count), and also focused on the content analysis of the deception cues that can be matched with the results obtained from computerized findings. When VSA (Voice Stress Analysis) was required, Praat was used. Statistical analyses were occasionally applied to reach highly accurate results. The study concluded that the New Model (NM) is not context-sensitive, being a quantitative one, and is thus numerically oriented in its decisions. Moreover, when qualitative analysis intervenes, especially in examining Politifact rulings, context plays a crucial role in passing judgements on deceptive vs. non-deceptive discourse.

Supplemental Material for Can Credibility Criteria Be Assessed Reliably? A Meta-Analysis of Criteria-Based Content Analysis

Supplemental Material for Can Credibility Criteria Be Assessed Reliably? A Meta-Analysis of Criteria-Based Content Analysis

Attributes of True and Deceptive Statements Made in Evaluations of Criminal Defendants

ABSTRACTCourt-ordered psychological evaluations of criminal defendants generally include two types: mental status (competence or sanity) versus risk assessment. When defendants are deceptive in mental status evaluations, lies of commission, in which symptoms of psychopathology are feigned (malingering), are most likely. When defendants are deceptive in risk assessments, lies of omission, in which antisocial impulses are concealed or denied, are most likely. Four research conditions, consisting of the true and false statements of two men accused of sex offenses and two men claiming an insanity defense, were rated by participants on a test of truthfulness containing eight items derived from Criterion-Based Content Analysis and Reality Monitoring. All items, as well as the collective test score, significantly differentiated between true and false statements. Further, the lie of commission was rated as significantly more false than the lie of omission, and a confession was rated as more truthful than a true statement involving only exculpatory information, which is in accordance with the theory of cognitive load. Factor analyses provided support for the construct validity of the test of truthfulness and two subscales. Further, the data suggests that statements judged true are characterized by the presence of attributes of truthfulness, while statements judged false are characterized by the absence of attributes of truthfulness along with the presence of attributes of untruthfulness.

A call to improve the validity of criterion-based content analysis (CBCA): Results from a field-based study including 60 children's statements of sexual abuse

The growing awareness of sexually-abused children has led to a major shift: previously considered untrustworthy, children are now regarded as competent in providing medico-legal evidence. Professionals undertaking the challenging task of assessing the child's credibility need to rely upon approved evaluation methods. The Criteria-Based Content Analysis is a tool developed to assess the truthfulness of a child's verbal statement. This field-based study explores its validity and its limitations.Three independent experts rated the verbatim statements of 60 real-life alleged victims of sexual abuse. The CBCA scoring and final assessment of credibility were linked to the outcomes: confirmed or unconfirmed allegation of sexual abuse.Inter-rater reliability coefficient was 0.74. The average overall accuracy rate corresponding to confirmed and unconfirmed cases was 75%. Among the confirmed allegations, the accuracy rate reached 90%, whereas the probability of discriminating the true negative cases within the unconfirmed cases was lower than chance level. Of all the 19 criteria, items 6 “Reproduction of conversation” and 12 “Accounts of subjective mental state” were the strongest predictors of genuine accounts. A significant association between age and CBCA scores was noted, the effect of age on CBCA scores was strongest in the unconfirmed cases.Although some may argue that the validity of the CBCA is reasonably acceptable, results from this field study are less convincing. Increasing the diagnostic accuracy of the CBCA by adding new criteria, so as to raise the percentage of correct classifications in the confirmed accounts as well as in the unconfirmed accounts, would represent a major improvement.

Validity of content-based techniques to distinguish true and fabricated statements: A meta-analysis.

Within the scope of judicial decisions, approaches to distinguish between true and fabricated statements have been of particular importance since ancient times. Although methods focusing on "prototypical" deceptive behavior (e.g., psychophysiological phenomena, nonverbal cues) have largely been rejected with regard to validity, content-based techniques constitute a promising approach and are well established within the applied forensic context. The basic idea of this approach is that experience-based and nonexperience-based statements differ in their content-related quality. In order to test the validity of the most prominent content-based techniques, criteria-based content analysis (CBCA) and reality monitoring (RM), we conducted a comprehensive meta-analysis on English- and German-language studies. Based on a variety of decision criteria, 56 studies were included revealing an overall effect size of g = 1.03 (95% confidence interval [0.78, 1.27], Q = 420.06, p < .001, I2 = 92.48%, N = 3,429). There was no significant difference in the effectiveness of CBCA and RM. Additionally, we investigated a number of moderator variables, such as characteristics of participants, statements, and judgment procedures, as well as general study characteristics. Results showed that the application of all CBCA criteria outperformed any incomplete CBCA criteria set. Furthermore, statement classification based on discriminant functions revealed higher discrimination rates than decisions based on sum scores. Finally, unpublished studies showed higher effect sizes than studies published in peer-reviewed journals. All results are discussed in terms of their significance for future research (e.g., developing standardized decision rules) and practical application (e.g., user training, applying complete criteria set). (PsycINFO Database Record

Criteria-Based Content Analysis (CBCA) reality criteria in adults: A meta-analytic review

Background/Objective: Criteria-Based Content Analysis (CBCA) is the tool most extensively used worldwide for evaluating the veracity of a testimony. CBCA, initially designed for evaluating the testimonies of victims of child sexual abuse, has been empirically validated. Moreover, CBCA has been generalized to adult populations and other contexts though this generalization has not been endorsed by the scientific literature. Method: Thus, a meta-analysis was performed to assess the Undeutsch Hypothesis and the CBCA checklist of criteria in discerning in adults between memories of self-experienced real-life events and fabricated or fictitious memories. Results: Though the results corroborated the Undeutsch Hypothesis, and CBCA as a valid technique, the results were not generalizable, and the self-deprecation and pardoning the perpetrator criteria failed to discriminate between both memories. The technique can be complemented with additional reality criteria. The study of moderators revealed discriminating efficacy was significantly higher in filed studies on sexual offences and intimate partner violence. Conclusions: The findings are discussed in terms of their implications as well as the limitations and conditions for applying these results to forensic settings.

Inter-rater Reliability of Criteria-Based Content Analysis of Children's Statements of Abuse.

The evaluation of children's statements of sexual abuse cases in forensic cases is critically important and must and reliable. Criteria-based content analysis (CBCA) is the main component of the statement validity assessment (SVA), which is the most frequently used approach in this setting. This study investigated the inter-rater reliability (IRR) of CBCA in a forensic context. Three independent raters evaluated the transcripts of 95 statements of sexual abuse. IRR was calculated for each criterion, total score, and overall evaluation. The IRR was variable for the criteria, with several being unsatisfactory. But high IRR was found for the total CBCA scores (Kendall's W=0.84) and for overall evaluation (Kendall's W=0.65). Despite some shortcomings, SVA remains a robust method to be used in the comprehensive evaluation of children's statements of sexual abuse in the forensic setting. However, the low IRR of some CBCA criteria could justify some technical improvements.

Detecting truth in suspect interviews: the effect of use of evidence (early and gradual) and time delay on Criteria-Based Content Analysis, Reality Monitoring and inconsistency within suspect statements

The strategic use of evidence in interviews with suspects has been shown to increase the ability of interviewers to accurately and consistently distinguish truthful from deceptive accounts. The present study considers the effect of early and gradual revelation of evidence by the interviewer, and the effect of shorter and longer delay on the verbal quality of truth-teller and liar statements within a mock crime paradigm. It was hypothesised that gradual disclosure of evidence (1) in terms of inconsistencies (a) within statements and (b) between statements and such evidence and (2) of the criteria of Criteria-Based Content Analysis (CBCA) and of Reality Monitoring (RM) would emphasise differences in the verbal quality of truth-teller and liar statements. Forty-two high school students took part in the study. The use of statement-evidence and within-statement inconsistency appears to be a robust cue to deception across interview style and delay. This indicates that gradual disclosure in interviews may increase interviewer accuracy in veracity decisions by eliciting statement inconsistencies. However, gradual revelation and delay affected the ability of CBCA and RM criteria to distinguish the veracity of suspect statements.

Оценка достоверности свидетельских показаний несовершеннолетних

The article presents a review of the English language publications concerning the history and the current state of differential psychological assessment of validity of testimonies produced by child and adolescent victims of crimes. The topicality of the problem in Russia is high due to the tendency of Russian specialists to use methodical means and instruments developed abroad in this sphere for forensic assessments of witness testimony veracity. A system of Statement Validity Analysis (SVA) by means of Criteria-Based Content Analysis (CBCA) and Validity Checklist is described. The results of laboratory and field studies of validity of CBCA criteria on the basis of child and adult witnesses are discussed. The data display a good differentiating capacity of the method, however, a high level of error probability. The researchers recommend implementation of SVA in the criminal investigation process, but not in the forensic assessment. New perspective developments in the field of methods for differentiation of witness statements based on the real experience and fictional are noted. The conclusion is drawn that empirical studies and a special work for adaptation and development of new approaches should precede their implementation into Russian criminal investigation and forensic assessment practice

Attributes of Truthful Versus Deceitful Statements in the Evaluation of Accused Child Molesters

The ability to detect deception, in everyday social interactions and psychological evaluations, can literally mean the difference between life and death. Beyond physiological and nonverbal techniques for detecting deception, research has focused on criteria designed to evaluate the content of verbal statements to distinguish between true or actually experienced events versus internally manufactured or fabricated events. Criteria from two techniques that have received empirical support, criteria-based content analysis and reality monitoring, were used to create an 11-item Deception Detection Checklist (DDCL). In this study, 130 college undergraduates used the DDCL to rate the exculpatory statements of two accused child molesters: one truthful, the other untruthful. The 11 items composing the DDCL, as well as a measure of perceived truthfulness, were all scored on 7-point Likert-type scales. Nine of the 11 items on the DDCL significantly differentiated between the true and untrue statements in the predicted direction. Overall scores on the DDCL indicated that the false statement was rated as significantly more deceptive than the true statement. The DDCL possessed good reliability, and a series of factor analyses provided strong support for the construct validity of the measure. The 7 psychometrically strongest items from the DDCL included variables assessing the extent to which statements included clarity of detail, spatial details, temporal details, and contextual details, as well as the relevance, reconstructability, and realism of the statement. These results indicate that subjects were able to use this measure to reliably differentiate between true and false statements made by accused child molesters.

Is This Testimony Truthful, Fabricated, or Based on False Memory?

In 1989, Steller and Köhnken presented a systematic compilation of content characteristics for distinguishing between truthful and fabricated testimonies (criteria-based content analysis – CBCA) designed to be applied within a more comprehensive overall diagnostic procedure known as statement validity assessment (SVA; Steller, 1989 ). The subsequent 25 years have seen a marked increase in knowledge about the distinction between experience-based and non-experience-based statements. This supports the SVA approach and permits a better explanation of the underlying processes. The rationale of CBCA is that a true statement differs in content quality from a fabricated account because (a) a truth teller can draw on an episodic autobiographical representation containing a multitude of details, whereas a liar has to relate to scripts containing only general details of an event; and (b) a liar is busier with strategic self-presentation than a truth teller. The present article proposes a modified model of content characteristics that pays greater attention than before to these underlying processes. SVA takes into account that content quality is influenced not only by the veracity of a statement but also by other (personal and contextual) variables that need to be considered in the individual credibility assessment. Theoretical analyses and empirical research do not indicate comparable qualitative differences between true statements and those based on false memories. Witnesses giving testimony based on false memories do not fabricate false statements actively, and they make no effort to conceal a deception; they are not deceiving but mistaken. In these cases, a noncritical application of content criteria can lead to false results. To examine the hypothesis that a statement is based on a false memory, it is necessary to focus on the way in which the statement has emerged and evolved.

Using an Example Statement Increases Information but Does Not Increase Accuracy of CBCA, RM, and SCAN

AbstractVerbal credibility assessment methods are frequently used in the criminal justice system to investigate the truthfulness of statements. Three of these methods are Criteria Based Content Analysis (CBCA), Reality Monitoring (RM), and Scientific Content Analysis (SCAN). The aim of this study is twofold. First, we investigated the diagnostic accuracy of CBCA, RM, and especially SCAN. Second, we tested whether giving the interviewee an example of a detailed statement can enhance the diagnostic accuracy of these verbal credibility methods. To test the latter, two groups of participants were requested to write down one true and one fabricated statement about a negative event. Prior to this request, one group received a detailed example statement, whereas the other group received no additional information. Results showed that CBCA and RM scores differed between true and fabricated statements, whereas SCAN scores did not. Giving a detailed example statement did not lead to better discrimination between truth tellers and liars for any of the methods but did lead to the participants producing significantly longer statements. The implications of these findings are discussed. Copyright © 2013 John Wiley &amp; Sons, Ltd.

Validity of Criteria-Based Content Analysis (CBCA) at trial in free-narrative interviews

Objective The reliability of child witness testimony in sexual abuse cases is often controversial, and few tools are available. Criteria-Based Content Analysis (CBCA) is a widely used instrument for evaluating psychological credibility in cases of suspected child sexual abuse. Only few studies have evaluated CBCA scores in children suspected of being sexually abused. We designed this study to investigate the reliability of CBCA in discriminating allegations of child sexual abuse during court hearings, by comparing CBCA results with the court's final, unappealable sentence. We then investigated whether CBCA scores correlated with age, and whether some criteria were better than others in distinguishing cases of confirmed and unconfirmed abuse. Methods From a pool of 487 child sexual abuse cases, confirmed and unconfirmed cases were selected using various criteria including child IQ ≥ 70, agreement between the final trial outcome and the opinion of 3 experts, presence of at least 1 independent validating informative component in cases of confirmed abuse, and absence of suggestive questions during the child's testimonies. This screening yielded a study sample of 60 confirmed and 49 unconfirmed cases. The 14 item version of CBCA was applied to child witness testimony by 2 expert raters. Results Of the 14 criteria tested, 12 achieved satisfactory inter-rater agreement (Maxwell's Random Error). Analyses of covariance, with case group (confirmed vs. unconfirmed) and gender as independent variables and age as a covariate, showed no main effect of gender. Analyses of the interaction showed that the simple effects of abuse were significant in both sex. Nine CBCA criteria were satisfied more often among confirmed than unconfirmed cases; seven criteria increased with age. Conclusion CBCA scores distinguish between confirmed and unconfirmed cases. The criteria that distinguish best between the 2 groups are Quantity of Details, Interactions, and Subjective Experience. CBCA scores correlate positively with age, and independently from abuse; all the criteria test except 2 (Unusual Details and Misunderstood Details) increase with age. The agreement rate could be increased by merging criteria Unusual and Superfluous details that achieve a low inter-rater agreement when investigated separately. Practice implication Given its ability to distinguish between confirmed and unconfirmed cases of suspected child abuse, the CBCA could be a useful tool for expert opinion. Because our strict selection criteria make it difficult to generalize our results, further studies should investigate whether the CBCA is equally useful in the cases we excluded from our study (for example mental retardation).

Offenders’ uncoerced false confessions: A new application of statement analysis?

Purpose. Deception detection research has mainly studied denials and distortions given by students. This study examined true and false confessions as told by offenders. It was hypothesized that the statement analytic techniques Criteria‐Based Content Analysis (CBCA) and Reality Monitoring (RM) would discriminate truths and lies.Methods. Truthful and deceptive confessions to crime were given by 30 offenders (both women and men) in a within‐subject design. The participants were in prison at the time of data collection, and told the truth about a crime they had committed and been sentenced for. In addition, they made up a lie about a different crime after a few minutes of preparation. The transcribed statements were scored for CBCA and RM criteria.Results. Results showed that neither total CBCA nor total RM scores differentiated between lies and truths. Some individual CBCA criteria, however, showed differences: more self‐deprecations and doubts about own testimony in the told lies, and more unexpected complications in the truths.Conclusions. The results are discussed in relation to statement analysis of offenders’ accounts, individual CBCA criteria, as well as the development of criminal experience and familiarity with the event and setting. Implications for triers of fact and suggestions for future research are considered.

Careful calculation or a leap of faith? A field study of the translation of CBCA ratings to final credibility judgements

This field experiment investigated the influence of Criteria-Based Content Analysis (CBCA) ratings on ultimate decision accuracy regarding the credibility of children's statements of sexual abuse. Following a selection procedure, based on case facts independent of statement quality, 21 truthful accounts and 10 fabricated accounts of 6- to 17-year olds were analysed. Two experts rated the presence of the CBCA criteria and made overall credibility judgements for each statement. Rater one achieved an overall hit rate of 84% (95% for truthful statements and 60% for fabricated statements) and rater two a hit rate of 81% (81% for both truthful and fabricated statements) but the raters did not always agree. The CBCA criteria appeared more often in the truthful statements compared to the fabricated statements. Additional factors that influenced raters' credibility judgements, besides CBCA scores, are discussed.

Fantasy proneness as a confounder of verbal lie detection tools

AbstractThis study investigated whether high fantasy‐prone individuals have superior storytelling abilities. It also explored whether this trait is related to specific linguistic features (i.e. self‐references, cognitive complexity, and emotional words). Participants high (n = 30) and low (n = 30) on a fantasy proneness scale were instructed to write down a true and a fabricated story about an aversive situation in which they had been the victim. Stories were then examined using two verbal lie detection approaches: criteria‐based content analysis (CBCA) and linguistic inquiry and word count (LIWC). Irrespective of the truth status of the stories, independent observers rated stories of high fantasy‐prone individuals as being richer in all nine CBCA elements than those of low fantasy‐prone individuals. Furthermore, overall, high fantasy‐prone people used more self‐references in their stories compared with low fantasy‐prone individuals. High fantasy prones' fabricated stories scored higher on various truth indices than authentic stories of low fantasy prones. Thus, high fantasy‐prone people are good in creating a sense of authenticity, even when they fabricate stories. Forensic experts should bear this in mind when they employ verbal lie detection tools. Copyright © 2010 John Wiley &amp; Sons, Ltd.

Criteria‐based content analysis of true and suggested accounts of events

AbstractWorldwide, the criteria‐based content analysis (CBCA) is probably the most widely used veracity assessment technique for discriminating between accounts of true and fabricated events. In this study, two experiments examined the effectiveness of the CBCA for discriminating between accounts of true events and suggested events believed to be true. In Experiment 1, CBCA‐trained judges evaluated participants' accounts of true and suggestively planted childhood events. In Experiment 2, judges analysed accounts of recent events that were experimentally manipulated to be a (a) true experience, (b) false experience believed to be true and (c) deliberately fabricated experience. In both experiments CBCA scores were significantly higher for accounts of true events than suggested events. However, this difference was not significant for participants classified as experiencing ‘full’ memories for the suggested event. Self‐report memory measures supported the findings of the CBCA analyses. Taken together these results suggest that the CBCA discriminative power is greatly constrained. Copyright © 2008 John Wiley &amp; Sons, Ltd.

T06-P-06 Quality of the interview and the reliability of accounts of presumably sexually-abused minors

Objects The present study aims at verifying the quality of interviews carried out by different professional figures on closed hearings of presumably abused minors. Moreover, we tried to find a relationship between the quality of the interview and the efficacy of Criteria-Based Content Analysis (CBCA) for the assessment of reliability. Materials and method 19 interviews of closed hearings of supposed sexually-abused minors have been analysed. Interviews were audio-recorded and transcribed in full. Transcripts were then analysed to verify how they satisfied the requirements for the Cognitive Interview and the Step-Wise Interview. A technique for content analysis applied to testimony and hearing of under-age victims (CBCA) was then performed on the transcripts. All evaluations were carried out by three independent judges specifically trained in using this instrument. Interviewers were classified on the basis of their professions and, therefore, on the basis of their official studying and professional profile. Results Results showed a low compliance with international protocols of interviews. When interviews were carried out in conformity to protocols suggested in the specialised literature, results showed a greater inter-rater's agreement in CBCA scores. Discussion In this study, the interviewers’professional qualifications do not seem to guarantee the necessary quality of interview according to the criteria for CI and Step Wise Interview.

Cues to deception and ability to detect lies as a function of police interview styles.

In Experiment 1, we examined whether three interview styles used by the police, accusatory, information-gathering and behaviour analysis, reveal verbal cues to deceit, measured with the Criteria-Based Content Analysis (CBCA) and Reality Monitoring (RM) methods. A total of 120 mock suspects told the truth or lied about a staged event and were interviewed by a police officer employing one of these three interview styles. The results showed that accusatory interviews, which typically result in suspects making short denials, contained the fewest verbal cues to deceit. Moreover, RM distinguished between truth tellers and liars better than CBCA. Finally, manual RM coding resulted in more verbal cues to deception than automatic coding of the RM criteria utilising the Linguistic Inquiry and Word Count (LIWC) software programme. In Experiment 2, we examined the effects of the three police interview styles on the ability to detect deception. Sixty-eight police officers watched some of the videotaped interviews of Experiment 1 and made veracity and confidence judgements. Accuracy scores did not differ between the three interview styles; however, watching accusatory interviews resulted in more false accusations (accusing truth tellers of lying) than watching information-gathering interviews. Furthermore, only in accusatory interviews, judgements of mendacity were associated with higher confidence. We discuss the possible danger of conducting accusatory interviews.