Multilingual Evaluation Research Articles

BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining

The current research direction in generative models, such as the recently developed GPT4, aims to find relevant knowledge information for multimodal and multilingual inputs to provide answers. Under these research circumstances, the demand for multilingual evaluation of visual question answering (VQA) tasks, a representative task of multimodal systems, has increased. Accordingly, we propose a bilingual outside-knowledge VQA (BOK-VQA) dataset in this study that can be extended to multilingualism. The proposed data include 17K images, 17K question-answer pairs for both Korean and English and 280K instances of knowledge information related to question-answer content. We also present a framework that can effectively inject knowledge information into a VQA system by pretraining the knowledge information of BOK-VQA data in the form of graph embeddings. Finally, through in-depth analysis, we demonstrated the actual effect of the knowledge information contained in the constructed training data on VQA.

Multilingual evaluation of interpretable biomarkers to represent language and speech patterns in Parkinson's disease.

Motor impairments are only one aspect of Parkinson's disease (PD), which also include cognitive and linguistic impairments. Speech-derived interpretable biomarkers may help clinicians diagnose PD at earlier stages and monitor the disorder's evolution over time. This study focuses on the multilingual evaluation of a composite array of biomarkers that facilitate PD evaluation from speech. Hypokinetic dysarthria, a motor speech disorder associated with PD, has been extensively analyzed in previously published studies on automatic PD evaluation, with a relative lack of inquiry into language and task variability. In this study, we explore certain acoustic, linguistic, and cognitive information encoded within the speech of several cohorts with PD. A total of 24 biomarkers were analyzed from American English, Italian, Castilian Spanish, Colombian Spanish, German, and Czech by conducting a statistical analysis to evaluate which biomarkers best differentiate people with PD from healthy participants. The study leverages conceptual robustness as a criterion in which a biomarker behaves the same, independent of the language. Hence, we propose a set of speech-based biomarkers that can effectively help evaluate PD while being language-independent. In short, the best acoustic and cognitive biomarkers permitting discrimination between experimental groups across languages were fundamental frequency standard deviation, pause time, pause percentage, silence duration, and speech rhythm standard deviation. Linguistic biomarkers representing the length of the narratives and the number of nouns and auxiliaries also provided discrimination between groups. Altogether, in addition to being significant, these biomarkers satisfied the robustness requirements.

A global portrait of expressed mental health signals towards COVID-19 in social media space.

Globally, the COVID-19 pandemic has induced a mental health crisis. Social media data offer a unique opportunity to track the mental health signals of a given population and quantify their negativity towards COVID-19. To date, however, we know little about how negative sentiments differ across countries and how these relate to the shifting policy landscape experienced through the pandemic. Using 2.1 billion individual-level geotagged tweets posted between 1 February 2020 and 31 March 2021, we track, monitor and map the shifts in negativity across 217 countries and unpack its relationship with COVID-19 policies. Findings reveal that there are important geographic, demographic, and socioeconomic disparities of negativity across continents, different levels of a nation's income, population density, and the level of COVID-19 infection. Countries with more stringent policies were associated with lower levels of negativity, a relationship that weakened in later phases of the pandemic. This study provides the first global and multilingual evaluation of the public's real-time mental health signals to COVID-19 at a large spatial and temporal scale. We offer an empirical framework to monitor mental health signals globally, helping international authorizations, including the United Nations and World Health Organization, to design smart country-specific mental health initiatives in response to the ongoing pandemic and future public emergencies.

Multilingual evaluation of pre-processing for BERT-based sentiment analysis of tweets

Social media offer a big amount of information, to exploit in many fields of research. However, while methods for Natural Language Processing are being developed with good results when applied to well-formed datasets made of written text with a clear syntax, these sources present text written in informal language, unstructured syntax, and with peculiar symbols; therefore, particular approaches are required for text processing in this case. In this paper, the task of sentiment analysis of tweets is regarded. In particular, in order to avoid noise constituted by some web constructs like URLs and mentions and by other text fragments, and to exploit information hidden in symbols like emoticons, emojis and hashtags, the pre-processing of tweets is analyzed. More in detail, a number of experiments, performed by a state-of-the-art classification model (BERT), are designed, to evaluate many currently available operations for pre-processing tweets, in terms of the statistical significance of their influence on sentiment analysis performances. Moreover, available data in two languages are considered, i.e., English and Italian, in order to also evaluate dependence on the language. Results allow to individuate the most convenient strategy to pre-process tweets, and thus to improve the state of the art in both languages for the considered task of sentiment analysis.

P70 A Mixed Methods, Multilingual Evaluation of the Neighborhood Produce Market Participant Experience

ObjectiveTo evaluate the participant experience of the Neighborhood Produce Market (NPM), a mobile fresh food pantry, in a small Mid-Atlantic city.Use of Theory or ResearchA participatory approach used to implement a mixed-methods, cross-sectional study.Target AudienceAdult NPM participants with children in the household and primary language of English, Spanish, and/or Arabic.Program DescriptionThrough a collaboration of food bank, university, and hospital community outreach partners, NPM offers fresh, free produce and nutrition education with food samples to families at risk of food insecurity. Since its inception in 2017, NPM has served over 300 families and 500 children in 20 neighborhoods.Evaluation MethodsFour phases informed by collaborative partners with assistance from Spanish and Arabic interpreters/translators: 16 interviews to inform survey development; survey development with closed- and open-ended items and testing; survey translation and testing; and survey data collection using Qualtrics. Content analysis and descriptive statistics conducted for interview and survey data, respectively.ResultsAmong interviewees (14 females; 2 males), half of the interviews were conducted in English, 31% in Spanish, and 19% in Arabic. Major themes categorized by benefits, impact on family fruit and vegetable intake, food access norms, and suggestions. Survey completed by 40 participants (34 females; 6 males) in Spanish (40%), Arabic (32%), English (28%). Over 95% reported NPM connected people in the communities and had a positive impact on physical or financial health of families. Most respondents (97%) were introduced to new foods, with 80% reporting their children tried new foods. Over 85% reported they and their household members ate more fresh vegetables due to the NPM. Only 11% reported being uncomfortable getting food from the NPM. Suggestions included marketing, accessibility, and variety/quantity of foods.ConclusionsThere may be benefits to using a mobile fresh food pantry model like NPM for multilingual communities. Findings may be useful to nutrition educators to meet participant fruit and vegetable needs in similar communities. To evaluate the participant experience of the Neighborhood Produce Market (NPM), a mobile fresh food pantry, in a small Mid-Atlantic city. A participatory approach used to implement a mixed-methods, cross-sectional study. Adult NPM participants with children in the household and primary language of English, Spanish, and/or Arabic. Through a collaboration of food bank, university, and hospital community outreach partners, NPM offers fresh, free produce and nutrition education with food samples to families at risk of food insecurity. Since its inception in 2017, NPM has served over 300 families and 500 children in 20 neighborhoods. Four phases informed by collaborative partners with assistance from Spanish and Arabic interpreters/translators: 16 interviews to inform survey development; survey development with closed- and open-ended items and testing; survey translation and testing; and survey data collection using Qualtrics. Content analysis and descriptive statistics conducted for interview and survey data, respectively. Among interviewees (14 females; 2 males), half of the interviews were conducted in English, 31% in Spanish, and 19% in Arabic. Major themes categorized by benefits, impact on family fruit and vegetable intake, food access norms, and suggestions. Survey completed by 40 participants (34 females; 6 males) in Spanish (40%), Arabic (32%), English (28%). Over 95% reported NPM connected people in the communities and had a positive impact on physical or financial health of families. Most respondents (97%) were introduced to new foods, with 80% reporting their children tried new foods. Over 85% reported they and their household members ate more fresh vegetables due to the NPM. Only 11% reported being uncomfortable getting food from the NPM. Suggestions included marketing, accessibility, and variety/quantity of foods. There may be benefits to using a mobile fresh food pantry model like NPM for multilingual communities. Findings may be useful to nutrition educators to meet participant fruit and vegetable needs in similar communities.

Çok Dilli Metin Analizinde Alan Bağımlı Değerlendirme Verisinin Oluşturulması

Web enables to retrieve concise information about specific entities including people, organizations, movies and their features. Additionally, large amount of Web resources generally lies on a unstructured form and it tackles to find critical information for specific entities. Text analysis approaches such as Named Entity Recognizer and Entity Linking aim to identify entities and link them to relevant entities in the given knowledge base. To evaluate these approaches, there are a vast amount of general purpose benchmark datasets. However, it is difficult to evaluate domain-specific approaches due to lack of evaluation datasets for specific domains. This study presents WeDGeM that is a multilingual evaluation set generator for specific domains exploiting Wikipedia category pages and DBpedia hierarchy. Also, Wikipedia disambiguation pages are used to adjust the ambiguity level of the generated texts. Based on this generated test data, a use case for well-known Entity Linking systems supporting Turkish texts are evaluated in the movie domain.

Multilingual extension and evaluation of a poetry generator

AbstractPoetry generation is a specific kind of natural language generation where several sources of knowledge are typically exploited to handle features on different levels, such as syntax, semantics, form or aesthetics. But although this task has been addressed by several researchers, and targeted different languages, all known systems have focused on a limited purpose and a single language. This article describes the effort of adapting the same architecture to generate poetry in three different languages – Portuguese, Spanish and English. An existing architecture is first described and complemented with the adaptations required for each language, including the linguistic resources used for handling morphology, syntax, semantics and metric scansion. An automatic evaluation was designed in such a way that it would be applicable to the target languages. It covered three relevant aspects of the generated poems, namely: the presence of poetic features, the variation of the linguistic structure and the semantic connection to a given topic. The automatic measures applied for the second and third aspect can be seen as novel in the evaluation of poetry. Overall, poems were successfully generated in the three languages addressed. Despite minor differences in different languages or seed words, poems revealed to have a regular metre, frequent rhymes, to exhibit an interesting degree of variation, and to be semantically-associated with the initially given seeds.

Multilingual evaluation of voice disability index using pitch rate

Multilingual evaluation of voice disability index using pitch rate

Health information quality on the internet in gynecological oncology: a multilingual evaluation.

Oncological internet information quality is considered variable, but no comprehensive analysis of gynecological malig- nancies exists. The present authors' objectives were to compare the quality of common malignancy websites and to assess for language or disease differences; and secondly, to perform a quality comparison between medical and layperson terminology. World Health Organization (WHO) Health on the Net (HON) principles may be applied to websites using an automated toolbar function. Using a search engine (www.Google.com) 8,400 websites were assessed using keywords 'endometrial, 'uterine', 'cervical', 'ovarian', 'vaginal', 'vulvar', plus 'cancer', in English, French, German, and Spanish; repeated for alternate terms e.g. 'cervix', 'womb'. Searches for "vaginal' 'uterine', 'cervical', and 'endometrial' each returned millions of websites. The total percentage of all assessed HON-accredited sites was notably low across all search terms (median 15%; range 3-19%). Significant differences by malignancy type (p < 0.0001), language (p < 0.0001), and tertiles (thirds) of the first 150 websites returned (p < 0.0001). French language had most accredited websites. Using alternate terms demonstrated significant differences (p < 0.001) in accredited websites for most gynecological cancers. Internet data on gynecological malignancies is overwhelming. Further, a lack of validation of the majority of gynecological oncologic sites should be appreciated with discrepancies in quality and number of websites across diseases, languages, and also between medical and layperson terms. Physicians should encourage and more importantly their professional bodies should participate in the development of informative, ethical, and reliable health websites on the internet and direct patients to them.

Multilingual Evaluation of the DSR, DSDV and AODV Routing Protocols in Mobile Ad Hoc Networks

Multilingual Evaluation of the DSR, DSDV and AODV Routing Protocols in Mobile Ad Hoc Networks

A multilingual evaluation of current health information on the Internet for the treatments of benign prostatic hyperplasia.

PurposeTo compare the quality of current Internet information on benign prostatic hyperplasia (BPH) and its surgical and medical managements across four Western languages and a comparative analysis of website sponsors. BPH Internet information quality is particularly relevant in an era of expanding, minimally invasive and surgical therapies. However, no comprehensive analysis exists.MethodsWorld Health Organization Health on the Net (HON) principles may be applied to websites using an automated toolbar function. Using a search engine (www.google.com), 9,000 websites were assessed using keywords related to BPH and its medical and surgical treatment in English, French, German, and Spanish. The first 150 websites in each language had HON principles measured whilst a further analysis of site sponsorship was undertaken.ResultsVery few BPH websites had greater than ten per cent HON accredited with significant differences (P<0.001) based on terms used for BPH, its medical and surgical management. Tertiles (thirds) of the first 150 websites returned differences in accredited websites (P<0.0001). English language had most accredited websites. Odds ratios for different terms returning accredited websites also were significantly different across terms (P<0.001). Websites were largely commercially sponsored.ConclusionsA lack of validation of most BPH sites should be appreciated with discrepancies in quality and number of websites across diseases, languages and also between medical and alternate terms. Physicians should participate in and encourage the development of informative, ethical and reliable health websites on the Internet and direct patients to them.

Preliminary psychometric properties of the CFTIndex in Greece: the perspective of physical education

The promotion of creativity has been proved a necessity in the education process. The research on creativity in physical education though is very limited worldwide and there is a lack of multilingual evaluation instruments. In the current study, the Creativity Fostering Teacher Index (CFTIndex) was translated and culturally adapted in Greek and further checked for its stability and internal consistency. The final version of the Greek CFTIndex (Gr-CFTIndex) was assessed for items stability, sub-scales and total instrument internal consistency using a sample of 220 primary physical educators. Items stability was measured with Kendall's τc and was found to have a moderate positive significant correlation (+0.3 < τc > +0.8, p < .01). Exploratory factor analysis yielded six factors and their Cronbach alpha coefficients varied from .695 to .897. The total internal consistency of the Gr-CFTIndex was high (α = .92). All sub-scales’ inter-correlations were positive and significant.

At the intersection of text and talk: On the reproduction and transformation of language in the multi-lingual evaluation of multi-lingual texts

Reported speech is the most important feature in the reproduction and transformation of language as a living phenomenon. Speakers generally draw on prosodic means to distinguish reported from reporting discourse. In multi-lingual contexts, change of language and translations add resources and constraints on setting reported speech apart. This study analyzes the phenomenon of reported speech in a multi-lingual context where the multi-lingual source texts are available in the setting. Prosody and (deictic, iconic) gestures are important means to set apart both direct and indirect reported speech.

Assessing Internet Health Information on Female Pelvic Floor Disorders

While health information on the Internet on female pelvic floor disorders is abundant, to our knowledge there has been no comprehensive evaluation of the quality of this material. We assessed health information on the Internet using Health on the Net Foundation principles as they pertain to female pelvic floor disorders. Web sites were searched using the key words urinary incontinence, vaginal, uterine or pelvic organ prolapse and overactive bladder. The first 150 websites identified for each female pelvic disorder were assessed for Health on the Net Foundation certification using an automated toolbar function. The specific sponsorship of each web site was also recorded. All searched domains for female pelvic floor disorders revealed that most websites were not certified. The certification rate for urinary incontinence, pelvic organ prolapse and overactive bladder was 29%, 27% and 27%, respectively. Overall websites were sponsored by for profit commercial entities (44%), urologist/specialty societies (30%), government/educational organizations (16%), nonprofit organizations (9%) and law firms (1%). Nonprofit organizations (40%) and government/educational organization (38%) sponsored websites were more likely to be certified than those sponsored by commercial entities (25%) and urologist/specialty societies (23%) (p=0.036). The Internet has become a heavily used resource for the distribution and acquisition of health information. Availability notwithstanding, certification and validation are lacking for most sites related to information on female pelvic floor disorders. As we move forward, improvement in the quality of information on the Internet is imperative along with proper guidance to patients using the Internet as a reference.

Multilingual and location evaluation of search engines for websites and searched for keywords

Search engines are competing to be seen as universal, consistent and language independent. In principle, users searching for information through the Internet should get consistent information regardless of the language of the words they are searching for and regardless of the language of the matching or the relevant documents. Nevertheless, the language should affect the sequence or the ranking of the retrieved results. In this project, several tools are built to evaluate words and statements in several languages. Results are evaluated and compared for possible correlation. Another tool is built to crawl websites from different languages and locations in order to measure several aspects of those websites. Results from both studies showed that while it seems that popular search engines are making very good progress toward building search engines that are language and location independent, however, there are some limitations and situations where search for results can be biased toward the popularity of the website language and/or location.

CLEF 2010 conference on multilingual and multimodal information access evaluation

CLEF 2010 conference on multilingual and multimodal information access evaluation

Homocysteine and cognitive function in the Sacramento Area Latino Study on Aging

Elevated plasma homocysteine (hyperhomocysteinemia), an independent risk factor for vascular disease, has been reported to be inversely correlated with objective measures of cognitive function in patients with Alzheimer disease and in community-dwelling older adults. We evaluated the cross-sectional relation between total plasma homocysteine concentration and cognitive function in elderly Latinos (aged > or = 60 y; n = 1789) participating in the Sacramento Area Latino Study on Aging. Global cognitive function was assessed by using the Modified Mini-Mental State Examination, and specific cognitive functions were assessed by using 6 instruments developed for cross-cultural and multilingual neuropsychological evaluation of older persons. Associations between the cognitive function scores and total plasma homocysteine concentrations were then measured by multiple regression analysis with control for potential confounding by nutrient status (red blood cell folate, plasma vitamin B-12), kidney function (serum creatinine), demographic variables (age, sex, education, acculturation), and depressive symptoms. Modest inverse associations were found between homocysteine concentrations and several indexes of cognitive function, including the global Modified Mini-Mental State Examination assessment and the picture-association, verbal attention-span, and pattern-recognition tests (P < or = 0.05). Demographic variables, particularly age and education, were more strongly associated with cognitive function scores than was homocysteine. Homocysteine is a modest independent predictor of cognitive function in community-dwelling elderly Latinos. Reducing plasma homocysteine concentrations by administering B-vitamin supplements may provide some protection against cognitive decline in this and other elderly populations, but the effect may be limited.

Babel in document design: the evaluation of multilingual texts

The aim of the study is to analyze the process of document design in a multilingual setting. In order to evaluate translation quality, a theoretical perspective is formulated as a basis for criteria for a good translation. In this perspective, the target text is considered an autonomous document. Two sets of criteria are distinguished: correctness errors and functional errors. The tools that were used to assess translation quality were expert analysis and reader focused evaluation. For both tools, a multilingual evaluation team was formed with the highest possible expertise in the target languages, in linguistics, and in usability. In this case study, the process of evaluation and the results are described.