Interjudge Agreement Research Articles

Design and validation of a Primary Reading Proficiency Test (PCL-P)

Reading proficiency is one of the most important skills to be developed by students at school. However, the data on reading proficiency performance in Spanish schoolchildren show a downward trend. Thus, to measure this performance, a reading proficiency assessment test has been developed for Primary School students (PCL-P). Based on a multilevel model (executive, functional, instrumental, and epistemic), the test content is evaluated by 5 pedagogical and linguistic experts (inter-judge agreement: Fleiss' Kappa was 0.88). After that, it is applied to a sample of 216 students (6–12 years old) of the autonomous community of the Basque Country. Cronbach's alpha was 0.83. On the other hand, the exploratory factor analysis (principal components method and Varimax rotation) results in 5 factors explaining 59% of the variance: (1) word and text recognition and structural understanding; (2) evaluation and critical reflection of information for the creation of new ideas; (3) advanced/deep reading strategies; (4) phonological awareness; and (5) structural understanding of language and detection of key elements. In conclusion, the PCL-P is a valid instrument for the assessment of reading proficiency in Primary.

Methodological Proposal for the Adaptation of the Living with Long-Term Conditions Scale to the Family Caregiver.

(1) Background: Living with long-term conditions affects both patients and family caregivers. To obtain a more complete overview of this phenomenon, a measurement instrument is needed that includes both perspectives. The aim is to adapt a scale to family caregivers of individuals with long-term conditions. (2) Methods: A methodological proposal is presented that illustrates the adaptation of the EC-PC scale to the family caregiver. Three phases are proposed: adaptation of the items, panel of experts, and pre-test. (3) Results: In the adaptation phase, the items from the original EC-PC were modified to adapt them to the family caregiver, and new items were added associated with the differences in living with LTC from the perspective of family caregivers. In the panel of experts phase, a universal agreement was reached related to the clarity, relevance, and essentiality of the items included. In the pre-test phase, the content of the scale was verified quantitatively and qualitatively. (4) Conclusions: The content of the items of version 5 of the EC-PC-Family showed a high index of inter-judge agreement. When a phenomenon affects both patients and their environment, such as living with LTC, it is necessary to include both perspectives in the measurement tools.

Evaluation of the interjudge agreement of the ShOM protocol in children with Down syndrome and Obstructive Sleep Apnea

Evaluation of the interjudge agreement of the ShOM protocol in children with Down syndrome and Obstructive Sleep Apnea

Desenvolvimento e validação de conteúdo do Instrumento de Rastreio da Comunicação - IRC-36

ABSTRACT Purpose: to develop and validate the content of the communication screening instrument for children aged 0 to 36 months. Methods: an instrument comprising nine categories organized into five domains, according to developmental milestones per age group. The research was conducted in two stages. The first included item construction and theoretical content validation by seven judges with expertise in child language. The second included the empirical validation by the target population, in which three speech-language-hearing pathologists applied the instrument to 30 parents in a school in Recife, PE, Brazil. First-stage data analysis focused primarily on verifying agreement between the judges. In the second stage, the examiners suggested adjusting some items, and the children's performances were stratified. Results: a high level of interjudge agreement was observed in the first stage (CVI > 0.90). In the second stage, the performances stratified into quartiles confirmed the instrument’s feasibility for discriminating normally developing children from those at risk of a communication disorder or needing attention. Conclusion: the process ensured the instrument's content validity, with further validation steps being recommended.

Experiencias de personas adultas con discapacidad intelectual sobre el derecho a la vida independiente: Un estudio de caso

In the field of intellectual disability, the development of an independent life has been configured as a human and civil right that allows these people to articulate desired life plans. Despite the advances, at the international level, research on aspects related to independent living in people with intellectual disabilities continues to be very scarce. Generally, the approaches to the study of this construct are based on broader investigations dedicated to evaluating the levels of quality of life or self-determination of this population in residential services or sheltered housing. Precisely, this study analyzes, through a case study, the meanings that an advisory group made up of eight adults with intellectual disabilities has on the concept of independent living in the Chilean context in order, on the one hand, to identify relevant elements for the definition of the construct from the perceptions and experiences of this population; and on the other, to propose guidelines at various levels that allow progress in the realization of this right. To collect the data, a semi-structured group interview was designed whose questions were grouped around four themes: previous conceptions of the participants about the construct of independent life, satisfaction with life and possibilities of the context for the development of personal independence, role of family, friendship and partner in independent life, and opportunities of the work context for the achievement of personal independence. During the interview, another strategy was incorporated in which visual language gained more relevance. Specifically, a set of ten moodboards were used that allowed participants to delve into the different topics addressed. The results of the code validation process were calculated using Krippendorff's alpha statistics (.954) and Cohen's kappa (.953), which showed a high level of inter-judge agreement. As a result of this process, categories (N = 6) and subcategories (N = 17) were established. The results show how various needs, barriers and facilitators for the development of independent life are identified from the interviews of the interviewees. Among the main needs is the performance of a paid work activity or the establishment of solid affective relationships. With regard to the barriers that hinder personal independence, the most notorious are related to the difficulty of being financially independent or to the lack of accessibility in the work environment. Regarding the elements that facilitate the development of an independent life, the role of the couple stands out as a constant figure of support for independence and personal empowerment. These results suggest some guidelines for planning interventions on this construct. Among others, they should seek the acquisition of skills that strengthen the self-care capacity of this group; should promote increased opportunities for making meaningful choices in the workplace, personal and community; and they must adapt to the spaces in which people with intellectual disabilities operate in adulthood. On the other hand, it is assumed that one of the main limitations of this research is that the results are not generalizable. However, this study has some strengths. Among others, it contributes to enriching the bulk of research on independent living, this being an aspect barely addressed in the literature. It also complements the results of other research regarding the elements that can condition the development of skills for personal independence. This allows for the design of interventions that provide opportunities for independent living based on the felt needs of these people. In addition, it raises future lines of research related, for example, to the importance of the informed choice of the residential environment or to the study of the possibilities of the digital world as an incident phenomenon in the acquisition of skills for personal independence. https://doi.org/10.16888/interd.2023.40.2.18

Proposal of requirements for the development of a training simulator for the auditory-perceptual judgment of voice.

to identify a set of requirements for the development of an auditory-perceptual training simulator (APT) based on the experience of professors who provide APT. This is a cross-sectional, descriptive study with a quantitative approach. Twenty-two professors answered an online questionnaire containing 31 items related to APT, involving items about the professional profile, conditions for APT in undergraduate and postgraduate courses in Speech Therapy, APT structure, and evaluation of the APT effect. it was observed that there is a variation in APT procedures performed in Brazil. The main requirements indicated by the respondents for the APT involve the use of synthesized voices in the initial moments, followed by human voices later; the use of speech tasks with sustained vowels and connected speech; the insertion of complementary information such as gender, age, the profession of the speaker and the spectrography of the vocal signal; training with a minimum time of six hours; the evaluation of the training effect by comparing intra- and inter-judge agreement before and after training; the addition of the parameters of general degree of vocal deviation, roughness, breathiness, and strain; the use of validated continuous and numerical scales; and offering it from the second year of the undergraduate program. although there is variability in the response of experts, a minimum set of requirements indicated for performing APT with new judges was identified.

Proposição de requisitos para o desenvolvimento de um simulador de treinamento para julgamento perceptivo-auditivo da voz

ABSTRACT Purpose to identify a set of requirements for the development of an auditory-perceptual training simulator (APT) based on the experience of professors who provide APT. Methods This is a cross-sectional, descriptive study with a quantitative approach. Twenty-two professors answered an online questionnaire containing 31 items related to APT, involving items about the professional profile, conditions for APT in undergraduate and postgraduate courses in Speech Therapy, APT structure, and evaluation of the APT effect. Result it was observed that there is a variation in APT procedures performed in Brazil. The main requirements indicated by the respondents for the APT involve the use of synthesized voices in the initial moments, followed by human voices later; the use of speech tasks with sustained vowels and connected speech; the insertion of complementary information such as gender, age, the profession of the speaker and the spectrography of the vocal signal; training with a minimum time of six hours; the evaluation of the training effect by comparing intra- and inter-judge agreement before and after training; the addition of the parameters of general degree of vocal deviation, roughness, breathiness, and strain; the use of validated continuous and numerical scales; and offering it from the second year of the undergraduate program. Conclusion although there is variability in the response of experts, a minimum set of requirements indicated for performing APT with new judges was identified.

Saúde bucal na infância: construção e validação de instrumento sobre conhecimento, atitude e prática de cuidadores

Abstract The objective of this article is to develop and validate a KAP (knowledge, attitude, and practice) instrument for caregivers of children up to 36 months of age monitored by the Family Health Strategy. This methodological study was conducted in three stages: an integrative review, preparation of the initial version, and content validation by 29 judges. The instrument was validated for content and appearance. The Content Validity Index (CVI) and the FINN and Gwe-AC1 coefficients were calculated to assess inter-judge agreement. The overall CVI values of the 39-item instrument were: Clarity (0.91) and Relevance (0.95). The final version was obtained through 19 knowledge, 10 attitude, and 10 practice questions on caries, diet, oral hygiene, fluoride, breastfeeding, artificial feeding, milk teeth, and the need to take the baby to the dentist. The instrument produced can be used because it has the potential for use depending on the more global assessment of its psychometric properties.

Adaptation and validation of two annotation scales for assessing social skills in a corpus of multimodal collaborative interactions.

Behavioral observation scales are important for understanding and assessing social skills. In the context of collaborative problem-solving (CPS) skills, considered essential in the 21st century, there are no validated scales in French that can be adapted to different CPS tasks. The aim of this study is to adapt and validate, by annotating a new video corpus of dyadic interactions that we have collected, two observational scales allowing us to qualitatively assess CPS skills: the Social Performance Rating Scale (SPRS) and the Social Skills of Collaboration Scale (SSC). The construct validity of these two scales was assessed by exploratory factor analysis and inter-item correlations. We also checked inter-judge agreement using inter-class correlation coefficients. Internal consistency was determined using Cronbach's alpha and convergent and divergent validity by assessing correlations between the two scales and measures of depression and alexithymia. Finally, the discriminative properties of the two scales were analyzed by comparing the scores obtained by a group of anxious individuals and a non-anxious control group. The results show that our two scales have excellent inter-item correlations. Internal consistency is excellent (alpha SPRS =0.90; SSC = 0.93). Inter-rater agreement ranged from moderate to high. Finally, convergent validity was significant with the alexithymia scale, as was divergent validity with the depression scale. Anxious individuals had lower scores on both scales than non-anxious individuals. Both scales show good psychometric properties for assessing social skills relevant to different collaborative tasks. They also identify individuals with difficulties in social interaction. Thus, they could allow monitoring the effectiveness of training social skills useful in CPS.

Reliability of data on percent consonants correct and its associated quality indicator in the Swedish cleft lip and palate registry.

Data in national health care quality registries must be valid and reliable in order to enable open comparisons of results. To assess the reliability of data on percent consonants correct (PCC) and its associated quality indicator ≥86% correct consonants in the Swedish quality registry for patients born with cleft lip and palate (CLP) registry. Six independent speech-language pathologists re-assessed the audio recordings of 96 five-year-olds with PCC data in the CLP registry. Target consonants of a single-word picture-naming test were phonetically transcribed, and PCC was calculated. The reliability of PCC data was assessed with the intraclass correlation coefficient (ICC). The reliability of the quality indicator ≥86% correct consonants was assessed with point-by-point percentage agreement and Cohen's kappa. Intra- and inter-judge agreement for PCC was excellent with ICCs above 0.9, and so was the agreement of data from the CLP registry and the six judges' re-assessments. The percentage agreement between all judges and the CLP registry for the quality indicator ≥86% correct consonants was poor (67%). However, in 88% of the cases, results from four judges and the CLP registry agreed, corresponding to good agreement. The mean of all kappa values for six judges and the CLP registry corresponded to good agreement (0.72). The results indicate the PCC data in the CLP registry and the quality indicator ≥86% correct consonants to be reliable. When differences in outcome between treatment centres are detected, the raw data collected should always be re-examined before drawing definitive conclusions.

Validation of a Questionnaire of Food Education Content on School Catering Websites in Spain.

The aim of this study was to design and validate a questionnaire on quality and food education on catering company websites. For the validation of the questionnaire, its reliability, feasibility and content validity characteristics were determined. For content validity, a panel of experts was used and the overall and item-wise Content Validity Index (CVI) of the experts’ responses was calculated. Reliability was determined by the inter-judge agreement method with the analysis of 30 websites using the intraclass correlation coefficient (ICC) and the Bland and Altman plot. Adequate content validity was verified by obtaining a very high CVI (above 0.80) in the analysis of the expert panel’s responses and excellent inter-observer agreement with a very high global ICC (above 0.90) value in the determination of test-retest reliability. The questionnaire is a valid instrument for the evaluation of the quality of school catering websites and their content in food education, as it has good feasibility, high content validity and excellent reliability.

Promoting remote workers' psychological health: Effective management practices during the COVID‐19 crisis

The aim of this study was to identify specific management practices that promote the psychological health of remote workers in the context of the COVID‐19 crisis. A two‐round Delphi study was conducted among 28 teleworkers and 22 managers. A list of 60 specific management practices was presented and participants had to identify whether each one could be used in the current remote working context and, if so, how useful it was to promote psychological health at work. Results indicate that most specific management practices usually used in a face‐to‐face setting can also be used in a remote context (85%). Practices that show consideration, establishing work structure, and allowing flexibility were also identified as the most useful to promote remote workers' psychological health during the pandemic. This study contributes to the advancement of knowledge about specific management practices, remote working, and crisis management. It also suggests specific practices that managers can adopt to promote the psychological health of their employees during a period of crisis, even while managing from a distance.

A SCALE DESIGN OF PERSONAL COMPETENCES ASSOCIATED WITH VIRTUES IN UNIVERSITY STUDENTS - AN APPROXIMATION STUDY

Integral formation is undoubtedly one of the great pretensions of higher education in the 21st century. By integral education we understand that which is directed at the whole person, and which aims at the harmonious development of all their dimensions and faculties. This education, which places the student at the centre, making them the protagonist of his learning, is designed so that each student discovers and carries out their own life project. A relevant contribution to integral education is the development of cardinal virtues (prudence, justice, fortitude and temperance) through competency-based learning. This study presents the design of a measurement instrument on personal competences, based on cardinal virtues and whose purpose is to measure the degree of development of the undergraduate's integral formation. Accordingly, each item of the questionnaire measures the self-perception of development of some of the transversal competences linked to cardinal virtues. The competences measured are critical thinking, decision-making, resilience, proactivity, self-improvement, emotional management, servant leadership and teamwork. The questionnaire was subjected to expert validation. The main results confirm that the questionnaire reaches an acceptable level of interjudge agreement, although minor changes suggested by experts are introduced. Thus, it can be concluded that this study presents a relevant questionnaire to measure the self-perception of the development of some competences associated with cardinal virtues that contributes to the integral formation of undergraduates. In this sense, this questionnaire may be useful for university community since it allows to test among students whether the curricula are acceptable in terms of integral formation.

Effectiveness of Problem Based Learning Device Development to Improve Generic Science Skills

This development research aims to develop an effective problem-based physics learning device model to improve students' generic science skills. The type of research used is Research and Development (R&D) using a 4D model. This 4D model consists of four stages, namely Define, Design, Develop and Disseminate. The effectiveness of the learning device was obtained based on the analysis of the observation sheet on the implementation of the lesson plans from three observers was then determined using the calculation of the Interjudge Agreement (IJA) and the N-Gain test. Based on the results of the research that has been carried out, the results of the IJA calculation analysis show that the implementation of learning is above 75% with the N-Gain test results of 0.54 are in the medium category, while the average value for the pretest value is 36.91 and the average posttest value 68.75. So it can be said that the problem based model of physics learning tools used is effective in improving the generic science skills of students

Use of Automated Thematic Annotations for Small Data Sets in a Psychotherapeutic Context: Systematic Review of Machine Learning Algorithms.

BackgroundA growing body of literature has detailed the use of qualitative analyses to measure the therapeutic processes and intrinsic effectiveness of psychotherapies, which yield small databases. Nonetheless, these approaches have several limitations and machine learning algorithms are needed.ObjectiveThe objective of this study is to conduct a systematic review of the use of machine learning for automated text classification for small data sets in the fields of psychiatry, psychology, and social sciences. This review will identify available algorithms and assess if automated classification of textual entities is comparable to the classification done by human evaluators.MethodsA systematic search was performed in the electronic databases of Medline, Web of Science, PsycNet (PsycINFO), and Google Scholar from their inception dates to 2021. The fields of psychiatry, psychology, and social sciences were selected as they include a vast array of textual entities in the domain of mental health that can be reviewed. Additional records identified through cross-referencing were used to find other studies.ResultsThis literature search identified 5442 articles that were eligible for our study after the removal of duplicates. Following abstract screening, 114 full articles were assessed in their entirety, of which 107 were excluded. The remaining 7 studies were analyzed. Classification algorithms such as naive Bayes, decision tree, and support vector machine classifiers were identified. Support vector machine is the most used algorithm and best performing as per the identified articles. Prediction classification scores for the identified algorithms ranged from 53%-91% for the classification of textual entities in 4-7 categories. In addition, 3 of the 7 studies reported an interjudge agreement statistic; these were consistent with agreement statistics for text classification done by human evaluators.ConclusionsA systematic review of available machine learning algorithms for automated text classification for small data sets in several fields (psychiatry, psychology, and social sciences) was conducted. We compared automated classification with classification done by human evaluators. Our results show that it is possible to automatically classify textual entities of a transcript based solely on small databases. Future studies are nevertheless needed to assess whether such algorithms can be implemented in the context of psychotherapies.

Stigma Attached to Smoking Pregnant Women: A Qualitative Insight in the General French Population.

Cigarette consumption during pregnancy has major health consequences for women and unborn children. The stigma of smoking during pregnancy might hinder mothers-to-be's access to adequate healthcare and smoking cessation, especially in disadvantaged groups. This qualitative study was designed to describe extensively the public stigma associated with smoking during pregnancy. Participants were French adults recruited from the general population through social networks (N = 100). They were asked to answer three pairs of open-ended questions regarding cognitions, emotions, and behaviors elicited in the general population by pregnant smoking women. An inductive thematic analysis was performed and interjudge agreement was computed on 30% of the corpus analyzed deductively. Finally, independence (chi-square) between themes and gender, education, parenthood, and smoking status was tested. Themes (n = 25) were defined regarding cognitions (n = 9, eg, irresponsible, thoughtless, and unmindful), emotions (n = 8, eg, anger and disgust), and behaviors, (n = 8, eg, inform and persuade, and moralize and blame). Global interjudge agreement was strong (κ = .8). No difference was observed in themes according to gender, parental status, or education, indicating a heterogenous awareness of stigma. However, some differences were observed according to smoking status (χ 2 = 69.59, p = .02) (eg, nonsmokers more frequently stressed immorality). The stigma associated with smoking during pregnancy includes various components that might be measured and targeted in interventions to improve access to adequate healthcare and smoking cessation in this specific population. This qualitative study explores the stigma that the general French population attaches to pregnant women who smoke. Themes regarding cognitions (eg, irresponsible, thoughtless, and unmindful), emotions (eg, anger and disgust), and behaviors (eg, inform and persuade, and moralize and blame) were identified. These themes could guide further research regarding scale development and antistigma interventions to support smoking cessation.

Inclusion of Gender Views for the Evaluation and Mitigation of Urban Vulnerability: A Case Study in Castellón

The inclusion of gender views in every field and, especially, in urbanism, has become crucial for urban planning. Considering both men’s and women’s interests in an integrated gender equality perspective provides better results that improve the quality of public spaces and engenders a more sustainable society. However, to realize such benefits, efforts are required not only to understand the needs and preferences of urban residents but also to shape policies and develop strategies to mitigate vulnerability with population involvement. In order to help decision makers at the urban level evaluate vulnerability with the inclusion of gender views, this study proposes a model that incorporates the specificities of urban fabric users that face adverse conditions. The model is based on a structured and standardized checklist of key topics that could be applied to any urban development. From this checklist, a list of categories, subcategories, and indicators were proposed and validated using the inter-judge agreement technique. To illustrate this model, this paper presents the case study of Castellón (Spain) in which deprived neighborhoods were analyzed, updating a previous model intended only to detect vulnerability. The results help link policy making to social vulnerability and indicate strategies to reach inclusive neighborhoods via a gender equality approach.

An Inclusive Model for Assessing Age-Friendly Urban Environments in Vulnerable Areas

Population aging is becoming a major challenge in many countries. This paper deals with the elderly’s specific needs in the public open space as it can play a significant role in their social inclusion and could be especially relevant in deprived areas. The main goal is to build a model to evaluate the vulnerability of the public space by focusing on the elderly’s needs, using indicators. A previous analysis of the scientific and policy-oriented literature and of the technical standards and regulations linked with accessibility and social aspects that affect the elderly in urban areas was performed to identify the main dimensions for evaluation. The interjudge agreement technique was applied to validate the indicators with a panel of experts in technical and social disciplines. The model was applied to a vulnerable area in Castellón (East Spain), based on indicators adapted to the specific context features. The agreement level reached by experts was used to weight the indicators. The application of the model permitted the vulnerability in the suggested dimensions to be estimated and a global integrated index of vulnerability in the area to be calculated. It could assist in urban planning decision making toward age-friendly and, therefore, inclusive cities.

Consultas virtuales en Traumatología y Cirugía Ortopédica

ObjectiveThe objective of this study is to analyse the current system of virtual consultations between the levels of Primary and Specialised Care in the field of Traumatology and Orthopaedic Surgery (TOS) in our healthcare area. Material and methodA retrospective observational study was carried out on 90 consecutive patients who had a non-face-to-face consultation between 3 January 2017 and 10 February 2017 and subsequently a face-to-face consultation. All the patients belonged to the same healthcare area attached to the Nuestra Señora de Candelaria University Hospital. The data on the diagnostic orientation, medical history provided and complementary tests were evaluated by 2 observers, one with training in Family and Community Medicine and the other with specialised training in TOS, and compared with those obtained in the final face-to-face assessment. ResultsThe results showed a low inter-judge agreement regarding the diagnostic orientation, anamnesis, exploration and complementary tests provided in the virtual consultation request. It was considered that only 59% for one observer (Family and Community Medicine) and 47.7% for the other (specialised care) had sufficient information for decision-making. Furthermore, 35.2% required more than one face-to-face assessment consultation until diagnosis and in 45.5% it was necessary to request new complementary tests. In 30.7%, there was no concordance in the suggested and final diagnosis. In 51.9%, no therapeutic action other than that carried out by Primary Care was carried out and 34.1% of the patients were referred to the Rehabilitation department. ConclusionsThe current model of virtual consultations in TOS does not seem adequate to respond to this new healthcare model. The number of unnecessary referrals is very high despite the previous virtual assessment by a specialist in TOS. The Family and Community Medicine specialist should have more diagnostic resources and coordination between Primary and Specialised Care is necessary to determine, in the area of TOS, the type of consultations and conditions for which this system should be implemented to obtain adequate coordination and improve communication between both levels of care.

Social media for talent selection? a validity test of inter-judge agreement and behavioral prediction

Individuals have a large amount of personal information on social media (SM), which provides companies with new opportunities for talent selection. However, researchers’ understanding of the effectiveness of assessments based on SM is relatively ambiguous, and the conclusions of empirical studies remain controversial. The Realistic Accuracy Model provides theoretical and methodological support for the application of SM information in zero-acquaintance contexts. Accordingly, we collected and matched 160 sets of Chinese SM assessment (other-assessment) and employee self-assessment data. Through a two-step data analysis, we conducted a consistency check and verification of behavioural predictions. The results suggested that in terms of general suitability, as well as knowledge, skills, abilities, and other characteristics (KSAOs), the other-assessments and self-assessments were consistent. Furthermore, the general suitability and KSAOs of the other-assessments were predictive of behavioural intention (i.e., openness to change). This study empirically tested the accuracy of SM talent assessments, and finally, the research limitations and future trends were discussed.