Fair Levels Of Agreement Research Articles

Automated remote speech-based testing of individuals with cognitive decline: Bayesian agreement of transcription accuracy.

We investigated the agreement between automated and gold-standard manual transcriptions of telephone chatbot-based semantic verbal fluency testing. We examined 78 cases from the Screening over Speech in Unselected Populations for Clinical Trials in AD (PROSPECT-AD) study, including cognitively normal individuals and individuals with subjective cognitive decline, mild cognitive impairment, and dementia. We used Bayesian Bland-Altman analysis of word count and the qualitative features of semantic cluster size, cluster switches, and word frequencies. We found high levels of agreement for word count, with a 93% probability of a newly observed difference being below the minimally important difference. The qualitative features had fair levels of agreement. Word count reached high levels of discrimination between cognitively impaired and unimpaired individuals, regardless of transcription mode. Our results support the use of automated speech recognition particularly for the assessment of quantitative speech features, even when using data from telephone calls with cognitively impaired individuals in their homes. High levels of agreement were found between automated and gold-standard manual transcriptions of telephone chatbot-based semantic verbal fluency testing, particularly for word count.The qualitative features had fair levels of agreement.Word count reached high levels of discrimination between cognitively impaired and unimpaired individuals, regardless of transcription mode.Automated speech recognition for the assessment of quantitative and qualitative speech features, even when using data from telephone calls with cognitively impaired individuals in their homes, seems feasible and reliable.

Multiple perspectives on mental health outcome: needs for care and service satisfaction assessed by staff, patients and family members

Community-based mental health care requires the involvement of staff, patients, and their family members when both planning intervention programmes and evaluating mental health outcomes. The present study aimed to compare the perceptions of these three groups on two important subjective mental health outcome measures--needs for care and service satisfaction--to identify potential areas of discrepancy. The sample consisted of patients with a DSM diagnosis of psychosis and attending either outpatient or day centres operating in a community-based care system. Staff, patients and family members were assessed by using the CAN and the VSSS to evaluate, respectively, needs for care and service satisfaction. Kappa statistics were computed to assess agreement in the three groups. Patients identified significantly fewer basic (e.g. daytime activities, food, accommodation) and functioning needs (e.g. self-care, looking after home, etc.) than staff or family members. Only fair levels of agreement were found in the three groups (average kappa was 0.48 for staff and patients, 0.54 for staff and family members, and 0.45 for patients and relatives), with patients and family members showing more areas of discrepancies in both needs and service satisfaction. These findings provide further support for the idea that mental health services should routinely involve patients and their relatives when planning and evaluating psychiatric intervention and that this policy is a premise for developing a partnership care model.

Cleft Audit Protocol for Speech (CAPS‐A): a comprehensive training package for speech analysis

The previous literature has largely focused on speech analysis systems and ignored process issues, such as the nature of adequate speech samples, data acquisition, recording and playback. Although there has been recognition of the need for training on tools used in speech analysis associated with cleft palate, little attention has been paid to this issue. To design, execute, and evaluate a training programme for speech and language therapists on the systematic and reliable use of the Cleft Audit Protocol for Speech-Augmented (CAPS-A), addressing issues of standardized speech samples, data acquisition, recording, playback, and listening guidelines. Thirty-six specialist speech and language therapists undertook the training programme over four days. This consisted of two days' training on the CAPS-A tool followed by a third day, making independent ratings and transcriptions on ten new cases which had been previously recorded during routine audit data collection. This task was repeated on day 4, a minimum of one month later. Ratings were made using the CAPS-A record form with the CAPS-A definition table. An analysis was made of the speech and language therapists' CAPS-A ratings at occasion 1 and occasion 2 and the intra- and inter-rater reliability calculated. Trained therapists showed consistency in individual judgements on specific sections of the tool. Intraclass correlation coefficients were calculated for each section with good agreement on eight of 13 sections. There were only fair levels of agreement on anterior oral cleft speech characteristics, non-cleft errors/immaturities and voice. This was explained, at least in part, by their low prevalence which affects the calculation of the intraclass correlation coefficient statistic. Speech and language therapists benefited from training on the CAPS-A, focusing on specific aspects of speech using definitions of parameters and scalar points, in order to apply the tool systematically and reliably. Ratings are enhanced by ensuring a high degree of attention to the nature of the data, standardizing the speech sample, data acquisition, the listening process together with the use of high-quality recording and playback equipment. In addition, a method is proposed for maintaining listening skills following training as part of an individual's continuing education.

The Reliability of the Clinical Tests and Questions Recommended in International Guidelines for Low Back Pain

An intertester reliability study of the questions and tests recommended in guidelines for the management of low back pain (LBP). This study undertook a reliability study to evaluate the reliability of the items of the LBP clinical examination with a large sample of LBP patients. A crucial part of the diagnostic triage process, recommended by many national and international guidelines for the management of LBP, is the clinical examination. The questions and tests used in this process have never been rigorously evaluated for their intertester reliability in first contact clinicians who are not medically trained. Patients, referred to physiotherapy departments across the United Kingdom with LBP (n = 301) were recruited in a sample of convenience. The 50 questions and physical tests were administered by a physiotherapist and then repeated by another physiotherapist within the same day. Data were analyzed using kappa and weighted kappa correlation coefficients (kappa). Confidence intervals (95% CIs) were calculated. Eighty-six percent (n = 43) of the questions and test demonstrated kappa of 0.41 (fair agreement) or above. Five questions and 2 physical tests (prone knee bend and myotomal assessment) demonstrated agreement of only slight levels. CIs were generally narrow and the uncertainty regarding the kappa coefficients demonstrated correspondingly low. This study has rigorously evaluated the intertester reliability of the clinical examination process of the diagnostic triage. These clinicians generally demonstrated fair agreement when testing features of the "nerve root," "yellow" and "red flag" presentations recommended in international guidelines for the management of LBP and nonspecific LBP. However, reliance on single tests with only fair levels of agreement may be unwise. Further work is required to investigate the validity of the tests.

Testing a tool to assess perineal trauma

There is a need for an objective method of assessing perineal trauma. This would improve documentation and help clinicians decide whether or not a perineal tear should be sutured. This article describes the evaluation of a perineal assessment tool involving a convenience sample of 52 double assessments of postnatal perineal trauma. Comments made during and after the assessment showed midwives generally found the tool easy to understand and use. However, whilst inter-rater reliability in terms of concordance for each of the 6 components of the assessment tool was good (ranging from 68% to 96%), Cohen's Kappa for each set of scores was showed only low or fair levels of agreement (values for Kappa ranged from 0.081 to 0.559). One of the main reasons for these low scores was the limited range of trauma used to test the assessment tool. Further work is planned to carry out standardized reliability testing of the tool using a wider range of trauma.

Assessment of invasive growth pattern and lymphocytic infiltration in colorectal cancer.

A total of 122 specimens of colorectal cancer were re-assessed in relation to the reporting of invasive growth pattern (expanding vs. infiltrating) and presence or absence of peritumoral lymphocytic infiltrate as used in the Jass prognostic classification. Jass agreed with 69% of cases reported as infiltrating and 90% of reported as expanding. This parameter was distributed similarly amongst Dukes B and C cases in the original assessment (P = 0.27), whereas in the reviewed data infiltrating cases were more likely to be staged as Dukes C (P = 0.04). Jass agreed with 44% of lymphocyte present and 94% of lymphocyte absent assessments. The original lymphocyte assessments showed no significant differences in distribution between Dukes A and B cases (P = 0.12) or B and C cases (P = 0.75), whereas the reviewed data showed significant differences for A vs. B (P = 0.015) and B vs. C cases (P = 0.0025). Criteria for assessment were circulated to eight observers who revisited 20 of the cases in which there was disagreement. Consensus agreement with Jass was achieved in nine of 10 cases for invasive growth pattern and seven of 10 cases for lymphocyte infiltration (with two being evenly split). Most observers showed at least fair levels of agreement with Jass and some achieved excellent levels of agreement. This study indicates that assessment of criteria used in the Jass prognostic system for colorectal cancer is less than optimal in routine practice, but is improved through the provision of simple guidelines.