Macro-averaged F1-score Research Articles

Inverse similarity and reliable negative samples for drug side-effect prediction

BackgroundIn silico prediction of potential drug side-effects is of crucial importance for drug development, since wet experimental identification of drug side-effects is expensive and time-consuming. Existing computational methods mainly focus on leveraging validated drug side-effect relations for the prediction. The performance is severely impeded by the lack of reliable negative training data. Thus, a method to select reliable negative samples becomes vital in the performance improvement.MethodsMost of the existing computational prediction methods are essentially based on the assumption that similar drugs are inclined to share the same side-effects, which has given rise to remarkable performance. It is also rational to assume an inverse proposition that dissimilar drugs are less likely to share the same side-effects. Based on this inverse similarity hypothesis, we proposed a novel method to select highly-reliable negative samples for side-effect prediction. The first step of our method is to build a drug similarity integration framework to measure the similarity between drugs from different perspectives. This step integrates drug chemical structures, drug target proteins, drug substituents, and drug therapeutic information as features into a unified framework. Then, a similarity score between each candidate negative drug and validated positive drugs is calculated using the similarity integration framework. Those candidate negative drugs with lower similarity scores are preferentially selected as negative samples. Finally, both the validated positive drugs and the selected highly-reliable negative samples are used for predictions.ResultsThe performance of the proposed method was evaluated on simulative side-effect prediction of 917 DrugBank drugs, comparing with four machine-learning algorithms. Extensive experiments show that the drug similarity integration framework has superior capability in capturing drug features, achieving much better performance than those based on a single type of drug property. Besides, the four machine-learning algorithms achieved significant improvement in macro-averaging F1-score (e.g., SVM from 0.655 to 0.898), macro-averaging precision (e.g., RBF from 0.592 to 0.828) and macro-averaging recall (e.g., KNN from 0.651 to 0.772) complimentarily attributed to the highly-reliable negative samples selected by the proposed method.ConclusionsThe results suggest that the inverse similarity hypothesis and the integration of different drug properties are valuable for side-effect prediction. The selection of highly-reliable negative samples can also make significant contributions to the performance improvement.

A multitask bi-directional RNN model for named entity recognition on Chinese electronic medical records

BackgroundElectronic Medical Record (EMR) comprises patients’ medical information gathered by medical stuff for providing better health care. Named Entity Recognition (NER) is a sub-field of information extraction aimed at identifying specific entity terms such as disease, test, symptom, genes etc. NER can be a relief for healthcare providers and medical specialists to extract useful information automatically and avoid unnecessary and unrelated information in EMR. However, limited resources of available EMR pose a great challenge for mining entity terms. Therefore, a multitask bi-directional RNN model is proposed here as a potential solution of data augmentation to enhance NER performance with limited data.MethodsA multitask bi-directional RNN model is proposed for extracting entity terms from Chinese EMR. The proposed model can be divided into a shared layer and a task specific layer. Firstly, vector representation of each word is obtained as a concatenation of word embedding and character embedding. Then Bi-directional RNN is used to extract context information from sentence. After that, all these layers are shared by two different task layers, namely the parts-of-speech tagging task layer and the named entity recognition task layer. These two tasks layers are trained alternatively so that the knowledge learned from named entity recognition task can be enhanced by the knowledge gained from parts-of-speech tagging task.ResultsThe performance of our proposed model has been evaluated in terms of micro average F-score, macro average F-score and accuracy. It is observed that the proposed model outperforms the baseline model in all cases. For instance, experimental results conducted on the discharge summaries show that the micro average F-score and the macro average F-score are improved by 2.41% point and 4.16% point, respectively, and the overall accuracy is improved by 5.66% point.ConclusionsIn this paper, a novel multitask bi-directional RNN model is proposed for improving the performance of named entity recognition in EMR. Evaluation results using real datasets demonstrate the effectiveness of the proposed model.

A Text Mining Pipeline Using Active and Deep Learning Aimed at Curating Information in Computational Neuroscience

The curation of neuroscience entities is crucial to ongoing efforts in neuroinformatics and computational neuroscience, such as those being deployed in the context of continuing large-scale brain modelling projects. However, manually sifting through thousands of articles for new information about modelled entities is a painstaking and low-reward task. Text mining can be used to help a curator extract relevant information from this literature in a systematic way. We propose the application of text mining methods for the neuroscience literature. Specifically, two computational neuroscientists annotated a corpus of entities pertinent to neuroscience using active learning techniques to enable swift, targeted annotation. We then trained machine learning models to recognise the entities that have been identified. The entities covered are Neuron Types, Brain Regions, Experimental Values, Units, Ion Currents, Channels, and Conductances and Model organisms. We tested a traditional rule-based approach, a conditional random field and a model using deep learning named entity recognition, finding that the deep learning model was superior. Our final results show that we can detect a range of named entities of interest to the neuroscientist with a macro average precision, recall and F1 score of 0.866, 0.817 and 0.837 respectively. The contributions of this work are as follows: 1) We provide a set of Named Entity Recognition (NER) tools that are capable of detecting neuroscience entities with performance above or similar to prior work. 2) We propose a methodology for training NER tools for neuroscience that requires very little training data to get strong performance. This can be adapted for any sub-domain within neuroscience. 3) We provide a small corpus with annotations for multiple entity types, as well as annotation guidelines to help others reproduce our experiments.

A Metrical Analysis of Medieval German Poetry Using Supervised Learning

Middle High German (MHG) epic poetry presents a unique solution to the linguistic changes underpinning the transition from classical Latin poetry, based on syllable length, into later vernacular rhythmic poetry, based on phonological stress. The predominating pattern in MHG verse is the alternation between stressed and unstressed syllables, but syllable length also plays a crucial role. There are a total of eight possible metrical values. Single or half mora syllables can carry any one of three types of stress, resulting in six combinations. The seventh value is a double mora, i.e., a long stressed syllable. The eighth value is an elided syllable. We construct a supervised Conditional Random Fields (CRF) model to predict the metrical value of syllables, and subsequently investigate medieval German poets’ use of semantic and sonorous emphasis through meter. The features used are: 1) the syllable’s position within the line, 2) the syllable’s length in characters, 3) the syllable’s characters, 4) elision (last two characters of previous syllable and first two characters of focal syllable), 5) syllable weight, and 6) word boundaries. Additional metrical rules are enforced and marginal probabilities are calculated to yield the most likely legal scansion of a line. The model achieves a macro average F-score of .925 on internal cross-validation and .909 on held-out testing data. We determine that trochaic alternation with a one syllable anacrusis and words carrying clear stress assignment are the easiest for the model to scan. Lines with multiple double morae of syllables with few characters are the most difficult. We then rank all the epic poetry in the Mittelhochdeutsche Begriffsdatenbank (MHDBDB) by the difficulty of the meter. Finally, we investigate the double mora, which MHG poets used to draw attention to chosen concepts. We conclude that poets generally chose to use the double mora to emphasize highly sonorant words.

On Hierarchical Text Language-Identification Algorithms

Text on the Internet is written in different languages and scripts that can be divided into different language groups. Most of the errors in language identification occur with similar languages. To improve the performance of short-text language identification, we propose four different levels of hierarchical language identification methods and conducted comparative tests in this paper. The efficiency of the algorithms was evaluated on sentences from 97 languages, and its macro-averaged F1-score reached in four-stage language identification was 0.9799. The experimental results verified that, after script identification, language group identification and similar language group identification, the performance of the language identification algorithm improved with each stage. Notably, the language identification accuracy between similar languages improved substantially. We also investigated how foreign content in a language affects language identification.

Extensive Experimental Evaluation of Self-Organizing Maps for Automatic Classification of a Multi-Class Multi-Label Corpus

This paper aims at bridging the gap between feature selection and feature space size by utilizing both square and non-square self-organizing maps under different configuration scenarios for classifying a multi-class multi-label corpus, the Reuters Mod Apte’ split. The selection of non-square maps is based on a heuristic process for finding the suitable size for the self-organizing map. Vector construction is based on a simple, yet effective procedure aiming at transforming the vectors from multi-label to uni-label. The proposed solution improves classification efficiency not only in terms of accuracy but also in computational resources needed and time for training. Extensive experiments were conducted, using different configurations regarding map and vector sizes, and training cycles, also utilizing context words, to assess their impact in the classifier’s performance. Furthermore, an intelligent algorithm for label selection is being proposed, aiming to show that the neighboring nodes on the map affect the selection of labels for a specific node. According to the experiments conducted, our approach achieves 10% increase in Macro-Average F1 scores, $30\times $ decrease in vector dimensionality, and approximately $34\times $ smaller maps when compared to the baseline scenario.

Predicting and Grouping Digitized Paintings by Style using Unsupervised Feature Learning.

To create an system to aid in the analysis of art history by classifying and grouping digitized paintings based on stylistic features automatically learned without prior knowledge. 6,776 digitized paintings from 8 different artistic styles (Art Nouveau, Baroque, Expressionism, Impressionism, Realism, Romanticism, Renaissance, and Post-Impressionism) were utilized to classify (predict) and cluster (group) paintings according to style. The method of unsupervised feature learning with K-means (UFLK), inspired by deep learning, was utilized to extract features from the paintings. These features were then used in: (1) a support vector machine algorithm to classify the style of new test paintings based on a training set of paintings having known style labels; and (2) a spectral clustering algorithm to group the paintings into distinct style groups (anonymously, without employing any known style labels). Classification performance was determined by accuracy and F-score. Clustering performance was determined by: 1) the ability to recover the original stylistic groupings (using a cost analysis of all possible combinations of 8 group label assignments); 2) F-score; and 3) a reliability analysis. The latter analysis used two novel ways to determine the distribution of the null-hypothesis: (a) a uniform distribution projected onto the principal components of the original data; and (b) a randomized, weighted adjacency matrix. The ability to gain insights into art was tested by a semantic analysis of the clustering results. For this purpose, we represented the featural characteristics of each painting by an N-dimensional feature vector, and plotted the distance between vector endpoints (i.e., similarity between paintings). Then, we color-coded the endpoints with the assigned lowest-cost style labels. The scatterplot was visually inspected for separation of the paintings, where the amount of separation between color clusters provides semantic information on the interrelatedness between styles. The UFLK-extracted features resembled the edges/lines/colors in the paintings. For feature-based classification of paintings, the macro-averaged F-score was 0.469. Classification accuracy and F-score were similar/higher compared to other classification methods using more complex feature learning models (e.g., convolutional neural networks, a supervised algorithm). The clustering via UFLK-extracted features yielded 8 unlabeled style groupings. In 6 of 8 clusters, the most common true painting style matched the cluster style assigned by cost analysis. The clustering had an F-score of 0.212. (There are no comparison methods for clustering paintings.) For the semantic analysis, the featural characteristics of Baroque and Art Nouveau were found to be similar, indicating a relationship between these styles. The UFLK method can extract features from digitized paintings. We were able to extract characteristics of art without any prior information about the nature of the features or the stylistic designation of the paintings. The methods herein may provide art researchers with the latest computational techniques for the documentation, interpretation, and forensics of art. The tools could assist the preservation of culturally sensitive works of art for future generations, and provide new insights into works of art and the artists who created them.

An automated and robust image processing algorithm for glaucoma diagnosis from fundus images using novel blood vessel tracking and bend point detection

Glaucoma is an ocular disease which can cause irreversible blindness. The disease is currently identified using specialized equipment operated by optometrists manually. The proposed work aims to provide an efficient imaging solution which can help in automating the process of Glaucoma diagnosis using computer vision techniques from digital fundus images. The proposed method segments the optic disc using a geometrical feature based strategic framework which improves the detection accuracy and makes the algorithm invariant to illumination and noise. Corner thresholding and point contour joining based novel methods are proposed to construct smooth contours of Optic Disc. Based on a clinical approach as used by ophthalmologist, the proposed algorithm tracks blood vessels inside the disc region and identifies the points at which first vessel bend from the optic disc boundary and connects them to obtain the contours of Optic Cup. The proposed method has been compared with the ground truth marked by the medical experts and the similarity parameters, used to determine the performance of the proposed method, have yield a high similarity of segmentation. The proposed method has achieved a macro-averaged f-score of 0.9485 and accuracy of 97.01% in correctly classifying fundus images. The proposed method is clinically significant and can be used for Glaucoma screening over a large population which will work in a real time.

위키피디아 기반 개체명 사전 반자동 구축 방법

개체명은 다양한 자연어처리 연구 및 서비스에 중요한 정보로 이용된다. 개체명 인식의 성능을 향상시키기 위한 여러 연구에서 개체명 사전을 이용한 자질이 개체명 인식 성능에 큰 영향을 준다는 것을 보이고 있다. 그러나 개체명 사전을 구축하는 것은 매우 시간 소모적이고, 인력 소모적인 작업이다. 이를 완화하기 위해서 본 논문에서는 개체명 사전을 반자동으로 구축하는 방법을 제안한다. 제안 시스템은 능동학습을 이용하여 위키피디아 분류정보로 구성된 가상 문서를 개체명 범주 당 하나씩 생성한다. 그리고 잘 알려진 정보검색 모델인 BM25를 이용하여 위키피디아 엔트리와 가상문서 사이의 유사도를 계산한다. 마지막으로 유사도를 바탕으로 각 위키피디아 엔트리를 개체명 범주로 분류한다. 서로 다른 3종류의 개체명 범주 집합에서 실험한 결과, 제안 시스템은 매크로 평균 F1-점수 0.9028, 마이크로 평균 F1-점수 0.9554이라는 높은 성능을 보였다.

Sparse Modeling of Magnitude and Phase-Derived Spectra for Playing Technique Classification

Computational modeling of musical timbre is important for a variety of music information retrieval applications. While considerable progress has been made to recognize musical genres and instruments, relatively little attention has been paid to modeling playing techniques, which affect timbre in more subtle ways. In this paper, we contribute to this area of research by systematically evaluating various audio features and processing methods for multi-class playing technique classification, considering up to nine distinct playing techniques of bowed string instruments. Specifically, a collection of 6,759 chamber-recorded single notes of four bowed string instruments and a collection of 33 real-world solo violin recordings are used in the evaluation. Our evaluation shows that using sparse features extracted from the magnitude spectra and phase derivatives including group delay function (GDF) and instantaneous frequency deviation (IFD) leads to significantly better performance than using a combination of state-of-the-art temporal, spectral, cepstral and harmonic feature descriptors. For playing technique classification of violin singe notes, the former approach attains 0.915 macro-average F-score under a tenfold cross validation setting, while the latter only attains 0.835. Moreover, sparse modeling of magnitude and phase-derived spectra also performs well for single-note joint instrument-technique classification (F-score 0.770) and for playing technique classification of real-world violin solos (F-score 0.547). We find that phase information is particularly important in discriminating playing techniques with subtle differences, such as playing with different bowing positions (i.e., normal, sul tasto, and sul ponticello). A systematic investigation of the effect of parameters such as window sizes, hop factors, window types for phase-derived features is also reported to provide more insights.

Aara’– a system for mining the polarity of Saudi public opinion through e-newspaper comments

Aara’ is a system for mining opinion polarity through the pool of comments that readers write anonymously at the online edition of Saudi newspapers. We use a nave Bayes classifier with a revised n-gram approach to extract the public opinion polarity, which is expressed in Arabic, classifying it into four categories. For training we manually marked the comments as belonging to one of the categories. All the words in the documents of the training set were removed except those with explicit connotations. After the training the words designated as vocabulary were classified into one of the categories. Our system carries out polarity classification over informal colloquial Arabic that is unstructured and with a reasonable proportion of spelling errors. The result of testing our system showed a macro-averaged precision of 86.5%, while the macro-averaged F-score was 84.5%. The accuracy of the system is 82%.

Minimally Supervised Novel Relation Extraction Using a Latent Relational Mapping

The World Wide Web includes semantic relations of numerous types that exist among different entities. Extracting the relations that exist between two entities is an important step in various Web-related tasks such as information retrieval (IR), information extraction, and social network extraction. A supervised relation extraction system that is trained to extract a particular relation type (source relation) might not accurately extract a new type of a relation (target relation) for which it has not been trained. However, it is costly to create training data manually for every new relation type that one might want to extract. We propose a method to adapt an existing relation extraction system to extract new relation types with minimum supervision. Our proposed method comprises two stages: learning a lower dimensional projection between different relations, and learning a relational classifier for the target relation type with instance sampling. First, to represent a semantic relation that exists between two entities, we extract lexical and syntactic patterns from contexts in which those two entities co-occur. Then, we construct a bipartite graph between relation-specific (RS) and relation-independent (RI) patterns. Spectral clustering is performed on the bipartite graph to compute a lower dimensional projection. Second, we train a classifier for the target relation type using a small number of labeled instances. To account for the lack of target relation training instances, we present a one-sided under sampling method. We evaluate the proposed method using a data set that contains 2,000 instances for 20 different relation types. Our experimental results show that the proposed method achieves a statistically significant macroaverage F-score of 62.77. Moreover, the proposed method outperforms numerous baselines and a previously proposed weakly supervised relation extraction method.

Clustering Generalised Instances Set Approaches for Text Classification

This paper introduces three new text classification methods: Clustering-Based Generalised Instances Set (CB-GIS), Multilevel Clustering-Based Generalised Instances Set (MLC_GIS) and Multilevel Clustering-Based, k Nearest Neighbours (MLC-kNN). These new methods aim to unify the strengths and overcome the drawbacks of the three similarity-based text classification methods, namely, kNN, centroid-based and GIS. The new methods utilise a clustering technique called spherical K-means to represent each class by a representative set of generalised instances to be used later in the classification. The CB-GIS method applies a flat clustering method while MLC-GIS and MLC-kNN apply multilevel clustering. Extensive experiments have been conducted to evaluate the new methods and compare them with kNN, centroid-based and GIS classifiers on the Reuters-21578(10) benchmark dataset. The evaluation has been performed in terms of the classification performance and the classification efficiency. The experimental results show that the top-performing classification method is the MLC-kNN classifier, followed by the MLC-GIS and CB-GIS classifiers. According to the best micro-averaged F1 scores, the new methods (CB-GIS, MLC-CIS, MLC-kNN) have improvements of 4.48%, 4.65% and 4.76% over kNN, 1.84%, 1.92% and 2.12% over the centroid-based and 5.26%, 5.34% and 5.45% over GIS respectively. With respect to the best macro-averaged F1 scores, the new methods (CB-GIS, MLC-CIS, MLC-kNN) have improvements of 10.29%, 10.19% and 10.45% over kNN, respectively, 0.1%, 0.03% and 0.29% over the centroid-based and 3.75%, 3.68% and 3.94% over GIS respectively.

An interactive and user-centered computer system to predict physician’s disease judgments in discharge summaries

Purpose: This article describes a formative natural language processing (NLP) system that is grounded in user-centered design, simplification, and transparency of function. Methods: The NLP system was tasked to classify diseases within patient discharge summaries and is evaluated against clinician judgment during the 2008 i2b2 Shared Task competition. Text classification is performed by interactive, fully supervised learning using rule-based processes and support vector machines (SVMs). Results: The macro-averaged F-score for textual (t) and intuitive (i) classification were 0.614(t) and 0.629(i), while micro-averaged F-scores were recorded at 0.966(t) and 0.954(i) for the competition. These results were comparable to the top 10 performing systems. Discussion: The results of this study indicate that an interactive training method, de novo knowledge base with no external data sources, and simplified text mining processes can achieve a comparably high performance in classifying health-related texts. Further research is needed to determine if the user-centered advantages of a NLP system translate into real world benefits.