Standard Conditional Random Fields Research Articles

A hybrid Transformer approach for Chinese NER with features augmentation

Named entity recognition (NER) plays an important role in many downstream tasks of natural language processing, such as knowledge extraction and information retrieval. NER of Chinese is more challenging than that of English due to lack of the explicit word boundary. Features augmentation is a potential way to improve NER model of Chinese. Pre-trained models can implicitly preserve prior knowledge with additional features. This paper proposes a hybrid Transformer approach, which first utilize the fused additional features embeddings (e.g. char embeddings, bigram embeddings, lattice embeddings and BERT embeddings) as distributed representations to augment the representation ability of model. In addition, a new training strategy named DF strategy is proposed to efficiently fine-tune Bidirectional Encoder Representations from Transformers (BERT) and other embeddings in balance. Then, the proposed model can perceive the relations of features by introducing relative position embeddings to an additional adapted Transformer encoder. Lastly, a standard Conditional Random Field is used to alleviate the obvious tag errors. The proposed model is applied to four representative Chinese datasets to investigate its performance. Experiments results show that the proposed model outperforms the other popular models in terms of accuracy. The proposed BL-BTC model can effectively improve the recognition performance of formal and informal texts.

Mirrored conditional random field model for object recognition in indoor environments

Traditional object recognition algorithms are based on a commonly adopted closed-set hypothesis, assuming that the knowledge given in training is complete. However, real situations are often open and nonstatic, in which case the models only obtain incomplete knowledge during the training phase. This paper proposes a new type of conditional random field (CRF) model to solve a special case of incomplete knowledge, in which the visual appearance of certain objects changes significantly between training and testing, and as a result, certain unary features (features of individual objects) extracted from red green blue-depth (RGB-D) images are no longer reliable. Mirror nodes are introduced into the architecture based on the standard CRF model to build the mirrored conditional random field (Mirror-CRF) model, which integrates two types of object nodes: original nodes and mirror nodes. The mirror nodes have no unary features, only pairwise features, which describe relationships between two objects and are more reliable than unary features for object recognition in the case of appearance variation. The experimental results show that the Mirror-CRF model reduces the influence of significant changes in the appearance of certain objects and improves the object recognition ability under the condition of incomplete knowledge.

Ontology-based conditional random fields for object recognition

Object recognition is a cornerstone task in autonomous and/or assistance systems like robots, autonomous vehicles, or those assisting to visually impaired, aiming to achieve a certain level of understanding of their surroundings. Probabilistic models, such as Conditional Random Fields (CRFs), have been successfully applied to this end given their ability to exploit contextual and situation information, e.g. a bowl is typically found in a cabinet and not in a night-stand. In this work we propose to evolve CRFs into Ontology-based Conditional Random Fields (obCRFs), which define a multi-level structure where each level assigns a category with different granularity to the same set of objects. For example, a level could assign to an object the category appliance or furniture, while the next one could categorize it into the tv, microwave, cabinet, or table types. In this way, general categorizations can guide the classification into more specialized ones (and vice versa), improving recognition success, and mitigating the CRFs limitations when modeling a high number of object categories (shared, in general, by Machine Learning techniques). To set the categories in each level we propose to mimic the hierarchical structure of ontologies, where categories are naturally codified following a subsumption ordering. This leads us to the second advantage of obCRFs: the multi-labeling of objects provides a richer understanding of the scene, which can be leveraged for accomplishing high-level tasks (e.g. object search or scheduling). Our approach has been tested with scenes from two state-of-the-art datasets: Robot@Home and Cornell-RGBD, outperforming the results provided by standard CRFs.

Named Entity Recognition with Word Embeddings and Wikipedia Categories for a Low-Resource Language

In this article, we propose a word embedding--based named entity recognition (NER) approach. NER is commonly approached as a sequence labeling task with the application of methods such as conditional random field (CRF). However, for low-resource languages without the presence of sufficiently large training data, methods such as CRF do not perform well. In our work, we make use of the proximity of the vector embeddings of words to approach the NER problem. The hypothesis is that word vectors belonging to the same name category, such as a person’s name, occur in close vicinity in the abstract vector space of the embedded words. Assuming that this clustering hypothesis is true, we apply a standard classification approach on the vectors of words to learn a decision boundary between the NER classes. Our NER experiments are conducted on a morphologically rich and low-resource language, namely Bengali. Our approach significantly outperforms standard baseline CRF approaches that use cluster labels of word embeddings and gazetteers constructed from Wikipedia. Further, we propose an unsupervised approach (that uses an automatically created named entity (NE) gazetteer from Wikipedia in the absence of training data). For a low-resource language, the word vectors obtained from Wikipedia are not sufficient to train a classifier. As a result, we propose to make use of the distance measure between the vector embeddings of words to expand the set of Wikipedia training examples with additional NEs extracted from a monolingual corpus that yield significant improvement in the unsupervised NER performance. In fact, our expansion method performs better than the traditional CRF-based (supervised) approach (i.e., F-score of 65.4% vs. 64.2%). Finally, we compare our proposed approach to the official submission for the IJCNLP-2008 Bengali NER shared task and achieve an overall improvement of F-score 11.26% with respect to the best official system.

Beyond linear chain

Information Extraction (IE) is a field at the crossroads of IR and NLP that studies methods for extracting information from text in such a way that this information can be used to populate a structured information repository. The main methods by means of which IE has been tackled rely on supervised learning;the best-performing such methods belong to the class of probabilistic graphical models, and, in particular, to the class of Conditional Random Fields (CRFs). In this thesis we investigate two major aspects related to textual IE via CRFs: (a) the creation of CRFs models that can outperform the commonly adopted linear- chain CRFs, and the creation of methods for ensuring the quality of training data and for assessing the impact of training data quality on the accuracy of CRFs systems for IE. We start by facing the task of IE from medical documents written in the Italian language. We propose two novel approaches: (i) a cascaded, two-stage method composed by two layers of CRFs, and (ii) a confidence-weighted ensemble method that combines standard linear-chain CRFs and the proposed two-stage method. Both the proposed models are shown to outperform a standard linear-chain CRFs system. We then investigate aspect-oriented sentence-level opinion mining from product reviews, that consists in predicting, for all sentences in the review, whether the sentence expresses a positive, neutral, or negative opinion (or no opinion at all) about a specific aspect of the product. We propose a set of increasingly powerful models based on CRFs, including a hierarchical multi-label CRFs scheme that jointly models the overall opinion expressed in a product review and the set of aspect-specific opinions expressed in each of its sentences. The proposed CRFs models are shown to obtain better results than linear-chain CRFs. We then study the impact that the quality of training data has on the accuracy of an IE system via experiments performed on a dataset in which inter-coder agreement data are available. Finally, we investigate active learning techniques for a type of semi-supervised CRFs specifically devised for partially labeled sequences. We show that margin-based strategies always obtain the best results on the four tasks we have tested them on.

Conditional ordinal random fields for structured ordinal-valued label prediction

Predicting labels of structured data such as sequences or images is a very important problem in statistical machine learning and data mining. The conditional random field (CRF) is perhaps one of the most successful approaches for structured label prediction via conditional probabilistic modeling. In such models, it is traditionally assumed that each label is a random variable from a nominal category set (e.g., class categories) where all categories are symmetric and unrelated from one another. In this paper we consider a different situation of ordinal-valued labels where each label category bears a particular meaning of preference or order. This setup fits many interesting problems/datasets for which one is interested in predicting labels that represent certain degrees of intensity or relevance. We propose a fairly intuitive and principled CRF-like model that can effectively deal with the ordinal-scale labels within an underlying correlation structure. Unlike standard log-linear CRFs, learning the proposed model incurs non-convex optimization. However, the new model can be learned accurately using efficient gradient search. We demonstrate the improved prediction performance achieved by the proposed model on several intriguing sequence/image label prediction tasks.

Large margin cost-sensitive learning of conditional random fields

We tackle the structured output classification problem using the Conditional Random Fields (CRFs). Unlike the standard 0/1 loss case, we consider a cost-sensitive learning setting where we are given a non-0/1 misclassification cost matrix at the individual output level. Although the task of cost-sensitive classification has many interesting practical applications that retain domain-specific scales in the output space (e.g., hierarchical or ordinal scale), most CRF learning algorithms are unable to effectively deal with the cost-sensitive scenarios as they merely assume a nominal scale (hence 0/1 loss) in the output space. In this paper, we incorporate the cost-sensitive loss into the large margin learning framework. By large margin learning, the proposed algorithm inherits most benefits from the SVM-like margin-based classifiers, such as the provable generalization error bounds. Moreover, the soft-max approximation employed in our approach yields a convex optimization similar to the standard CRF learning with only slight modification in the potential functions. We also provide the theoretical cost-sensitive generalization error bound. We demonstrate the improved prediction performance of the proposed method over the existing approaches in a diverse set of sequence/image structured prediction problems that often arise in pattern recognition and computer vision domains.

Minimum tag error for discriminative training of conditional random fields

This paper proposes a new criterion called minimum tag error (MTE) for discriminative training of conditional random fields (CRFs). The new criterion, which is a smoothed approximation to the sentence labeling error, aims to maximize an average of transcription tagging accuracies of all possible sentences, weighted by their probabilities. Corpora from the second international Chinese word segmentation bakeoff (Bakeoff 2005) are used to test the effectiveness of this new training criterion. The experimental results have demonstrated that the proposed minimum tag error criterion can reliably improve the initial performance of supervised conditional random fields. In particular, the recall rate of out-of-vocabulary words ( R oov) is significantly improved compared with that obtained using standard conditional random fields. Furthermore, the new training method has the advantage of robustness to segmentation across all datasets.