Conditional Random Field Model Research Articles

Neural Architecture Comparison for Bibliographic Reference Segmentation: An Empirical Study

In the realm of digital libraries, efficiently managing and accessing scientific publications necessitates automated bibliographic reference segmentation. This study addresses the challenge of accurately segmenting bibliographic references, a task complicated by the varied formats and styles of references. Focusing on the empirical evaluation of Conditional Random Fields (CRF), Bidirectional Long Short-Term Memory with CRF (BiLSTM + CRF), and Transformer Encoder with CRF (Transformer + CRF) architectures, this research employs Byte Pair Encoding and Character Embeddings for vector representation. The models underwent training on the extensive Giant corpus and subsequent evaluation on the Cora Corpus to ensure a balanced and rigorous comparison, maintaining uniformity across embedding layers, normalization techniques, and Dropout strategies. Results indicate that the BiLSTM + CRF architecture outperforms its counterparts by adeptly handling the syntactic structures prevalent in bibliographic data, achieving an F1-Score of 0.96. This outcome highlights the necessity of aligning model architecture with the specific syntactic demands of bibliographic reference segmentation tasks. Consequently, the study establishes the BiLSTM + CRF model as a superior approach within the current state-of-the-art, offering a robust solution for the challenges faced in digital library management and scholarly communication.

Regularized Loss With Hyperparameter Estimation for Weakly Supervised Single Class Segmentation.

We propose a new image level weakly supervised segmentation approach for datasets with a single object class of interest. Our approach is based on a regularized loss function inspired by the classical Conditional Random Field (CRF) modeling. Our loss models properties of generic objects, and we use it to guide CNN towards segments that are more likely to correspond to the object, thus avoiding the need for pixel precise annotations. Training CNN with regularized loss is a difficult task for gradient descent. We develop an annealing algorithm which is crucial for a successful training. Furthermore, we develop an approach for hyperparameter setting for the most important components of our regularized loss. This is far from trivial, since there is no pixel precise ground truth for guidance. The advantage of our method is that we use a standard CNN architecture and an easy to interpret loss function, derived from classical CRF models. Furthermore, we apply the same loss function for any task/dataset. We first evaluate our approach for salient object segmentation and co-segmentation. These tasks naturally involve one object class of interest. Then we adapt our approach to image level weakly supervised multi-class semantic segmentation. We obtain state-of-the-art results.

Quantifying the relative importance of biotic and abiotic factors in landscape-based models of stream fish distributions

AbstractLotic fish species distributions are frequently predicted using remotely sensed habitat variables that characterize the adjacent landscape and serve as proxies for instream habitat. Recent advancements in statistical methodology, however, allow for leveraging fish assemblage data when predicting distributions. This is important because assemblage composition likely provides better information about instream habitat compared to landscape-derived metrics and therefore may improve predictions. To better understand the value of using multi-species fish data in species distribution modeling, we fit two conditional random fields (CRF) models to quantify the relative importance of fish assemblage co-occurrence, landscape-derived habitat variables, and interactions between these two predictor groups (i.e., effects of co-occurrence could be context-dependent) at over 1200 stream catchments in Pennsylvania, USA. We first compared predictive performance of CRF models against traditionally used single-species logistic regressions (generalized linear models; GLMs) and found that inclusion of fish assemblage data often improved predictive performance. The multi-species CRF models performed significantly better at predicting occurrence for 63% of species with an average percent increase in AUC of 25% compared to GLMs. Furthermore, the CRF identified species co-occurrences as more informative, and thus relatively more important, at predicting occurrence than the other effect types. The CRF also suggested that allowing these biotic effects to be context-dependent was important for predicting occurrence of many species. These findings illustrate the value of fish assemblage data for landscape-scale species distribution modeling and leveraging this information can improve predictions and inferences to help inform the management and conservation of freshwater fishes.

Graphical models for identifying pore-forming proteins.

Pore-forming toxins (PFTs) are proteins that form lesions in biological membranes. Better understanding of the structure and function of these proteins will be beneficial in a number of biotechnological applications, including the development of new pest control methods in agriculture. When searching for new pore formers, existing sequence homology-based methods fail to discover truly novel proteins with low sequence identity to known proteins. Search methodologies based on protein structures would help us move beyond this limitation. As the number of known structures for PFTs is very limited, it's quite challenging to identify new proteins having similar structures using computational approaches like deep learning. In this article, we therefore propose a sample-efficient graphical model, where a protein structure graph is first constructed according to consensus secondary structures. A semi-Markov conditional random fields model is then developed to perform protein sequence segmentation. We demonstrate that our method is able to distinguish structurally similar proteins even in the absence of sequence similarity (pairwise sequence identity < 0.4)-a feat not achievable by traditional approaches like HMMs. To extract proteins of interest from a genome-wide protein database for further study, we also develop an efficient framework for UniRef50 with 43 million proteins.

Efficient JPEG image steganography using pairwise conditional random field model

Recent research shows that the security performance of JPEG image steganography can be improved by developing an accurate statistical model to characterize the correlations between Discrete Cosine Transform (DCT) coefficients. In this paper, a novel JPEG steganographic scheme called J-CRF is proposed to characterize the correlations between DCT coefficients using a pairwise Conditional Random Field (CRF) model, in which the correlated DCT coefficient pairs are modeled as bivariate Gaussian random variables. Under the framework of pairwise CRF, the proposed scheme formulates the problem of minimizing the statistical detectability within the Gaussian image model as the minimization of the sum of a series of unary and pairwise potentials, both of which are utilized to measure the local detectability of embedding messages in the DCT coefficient pairs. All the potentials associated with statistical detectability are formalized as the KL divergence between the statistical distributions of the cover and the stego. Experimental results demonstrate that the proposed J-CRF can outperform other SOTA JPEG steganographic schemes, e.g., J-UNIWARD, J-MiPOD, and JEC-RL, in resisting the detection of various advanced steganalyzers. In addition, the performance of J-CRF could be further boosted by reformulating it as the weighted sum of unary and pairwise potentials, called J-CRFW.

Part-Of-Speech Tagging for Balochi Language: A Data driven application of Conditional Random Fields

Parts-of-Speech (POS) tagging involves the assignment of the correct part of speech or lexical category to individual words within a sentence in a natural language. This procedure holds significant in the field of Natural Language Processing (NLP) and find utility across a variety of NLP applications. Commonly, it constitutes the initial phase of natural language processing. Subsequent stages may encompass additional tasks such as chunking, parsing and more. Balochi stands as the predominant language in Balochistan,, ranking as the fourth most prevalent language in Pakistan. The field of natural language processing for Balochi is still in its nascent stages. In this research, we introduce an algorithm for Balochi part-of-speech tagging, leveraging machine learning techniques. The core of our approach relies on a Conditional Random Field model as the machine learning component. Careful consideration is given to selecting appropriate features for the CRF, taking into account the linguistic characteristics of Balochi. Balochi is currently considered a resource poor language, and thus, the available manually tagged data consists of only approximately 1500 sentences. The tagset used in this study created for research purpose, consisting of 16 different tags. The learning process incorporates tagged data. The algorithm demonstrates a high accuracy rate of 86.78% when applied to Balochi texts. The training corpus comprises 40000 words, while the test corpus contains 10000 words.

Application of BiLSTM-CRF model with different embeddings for product name extraction in unstructured Turkish text

Named entity recognition (NER) plays a pivotal role in Natural Language Processing by identifying and classifying entities within textual data. While NER methodologies have seen significant advancements, driven by pretrained word embeddings and deep neural networks, the majority of these studies have focused on text with well-defined grammar and structure. A significant research gap exists concerning NER in informal or unstructured text, where traditional grammar rules and sentence structure are absent. This research addresses this crucial gap by focusing on the detection of product names within unstructured Turkish text. To accomplish this, we propose a deep learning-based NER model which combines a Bidirectional Long Short-Term Memory (BiLSTM) architecture with a Conditional Random Field (CRF) layer, further enhanced by FastText embeddings. To comprehensively evaluate and compare our model’s performance, we explore different embedding approaches, including Word2Vec and Glove, in conjunction with the Bidirectional Long Short-Term Memory and Conditional Random Field (BiLSTM-CRF) model. Furthermore, we conduct comparisons against BERT to assess the efficacy of our approach. Our experimentation utilizes a Turkish e-commerce dataset gathered from the internet, where traditional grammatical and structural rules may not apply. The BiLSTM-CRF model with FastText embeddings achieved an F1 score value of 57.40%, a precision value of 55.78%, and a recall value of 59.12%. These results indicate promising performance in outperforming other baseline techniques. This research contributes to the field of NER by addressing the unique challenges posed by unstructured Turkish text and opens avenues for improved entity recognition in informal language settings, with potential applications across various domains.

Extracting Symptoms of Agitation in Dementia from Free-Text Nursing Notes Using Advanced Natural Language Processing.

Nursing staff record observations about older people under their care in free-text nursing notes. These notes contain older people's care needs, disease symptoms, frequency of symptom occurrence, nursing actions, etc. Therefore, it is vital to develop a technique to uncover important data from these notes. This study developed and evaluated a deep learning and transfer learning-based named entity recognition (NER) model for extracting symptoms of agitation in dementia from the nursing notes. We employed a Clinical BioBERT model for word embedding. Then we applied bidirectional long-short-term memory (BiLSTM) and conditional random field (CRF) models for NER on nursing notes from Australian residential aged care facilities. The proposed NER model achieves satisfactory performance in extracting symptoms of agitation in dementia with a 75% F1 score and 78% accuracy. We will further develop machine learning models to recommend the optimal nursing actions to manage agitation.

A Novel Constructive Unceasement Conditional Random Field and Dynamic Bayesian Network Model for Attack Prediction on Internet of Vehicle

A Novel Constructive Unceasement Conditional Random Field and Dynamic Bayesian Network Model for Attack Prediction on Internet of Vehicle

Data governance and Gensini score automatic calculation for coronary angiography with deep-learning-based natural language extraction.

With the widespread adoption of electronic health records, the amount of stored medical data has been increasing. Clinical data, often in the form of semi-structured or unstructured electronic medical records (EMRs), contains rich patient information. However, due to the use of natural language by physicians when composing these records, the effectiveness of traditional methods such as dictionaries, rule matching, and machine learning in the extraction of information from these unstructured texts falls short of clinical standards. In this paper, a novel deep-learning-based natural language extraction method is proposed to overcome current shortcomings in data governance and Gensini score automatic calculation in coronary angiography. A pre-trained model called bidirectional encoder representation from transformers (BERT) with strong text feature representation capabilities is employed as the feature representation layer. It is combined with bidirectional long short-term memory (BiLSTM) and conditional random field (CRF) models to extract both global and local features from the text. The study included an evaluation of the model on a dataset from a hospital in China and it was compared with another model to validate its practical advantages. Hence, the BiLSTM-CRF model was employed to automatically extract relevant coronary angiogram information from EMR texts. The achieved F1 score was 91.19, which is approximately 0.87 higher than the BERT-BiLSTM-CRF model.

3D conditional random fields simulation for rockfill compaction quality assessment with sparse EVD measurement

The compaction quality of rockfill materials plays a pivotal role in ensuring the stability and durability of runway structures. However, traditional assessment methods relying on limited random sampling may lack representativeness for the entire construction site and overlook potential areas of insufficient compaction. Addressing this challenge, this paper employs the dynamic deformation modulus (EVD), a rapid and nondestructive testing method, to assess the compaction quality of runway rockfill. Subsequently, this paper propose a 3D Conditional Random Field (CRF) model based on EVD to simulate the uncertain spatial distribution of rockfill compaction quality, allowing for the estimation of the overall construction area's compaction quality with a limited number of sample points. The validity of the model is demonstrated through a case study involving a runway expansion project. Additionally, three critical construction factors--namely, measurement points, compaction passes, and compaction materials--are analyzed throughout the construction process. This analysis provides managers with valuable insights and decision-making tools to promptly evaluate the quality of compaction areas, ensuring the stability and durability of the runway.

Segmentation of wheat scab fungus spores based on CRF_ResUNet++

Accurate segmentation of tiny spore pictures is a significant technique in computer-aided evaluation of wheat scab. A CRF_ResUNet++ spore segmentation model using UNet++ as the fundamental framework is proposed to address the issues of uneven brightness, low spore-background contrast, and spore adhesion in microscopically collected Fusarium spore images. ResNet and fully connected conditional random fields (CRF) are combined in this model to address these issues. First, a residual block ResNet is introduced into UNet++ to enhance the propagation of features and extract more spore detail information. Next, the spore image is initially segmented using UNet++ with an encoder-decoder structure. Finally, postprocessing is carried out using a fully connected conditional random field model to obtain more precise edges and complete spore regions. The laboratory collection dataset test results revealed that the F1-score and MIoU reached 0.943 and 0.925, respectively, with an average detection Precision improvement of 3.2 % over the original UNet++ model. In comparison to existing models, the suggested model can segment spore pictures in challenging circumstances, such as when they contain independent and inter-adherent spores, with excellent computer vision processing capabilities effects and objective evaluation metric segmentation effects. In conclusion, the method in this study can reliably detect and segment adherent wheat scab spores in complex backgrounds, which offers critical technical support for automatic detection of adherent wheat scab spores and early spore concentration prediction in complex environments in the field.

Deformation Gated Recurrent Network for Lane-Level Abnormal Driving Behavior Recognition

As a significant part of traffic accident prevention, abnormal driving behavior recognition has been receiving extensive attention. However, the granularity of existing abnormal driving behavior recognition is mostly at road-level, and these methods’ high complexity leads to high overhead on training and recognition. In this article, we propose a deformation gated recurrent network for lane-level abnormal driving behavior recognition. Firstly, we use conditional random field model to calculate the lane change necessity of the vehicle, which helps us to distinguish whether the lane-changing behavior is reasonable. Secondly, we propose deformation gated recurrent network (DF-GRN) and trajectory entropy to capture the implicit relationship between trajectories and shorten recognition time. Finally, we get classified results including aggressive, distracted and normal driving behavior from the network. Distracted and aggressive behavior will be marked as anomaly. The effectiveness and real-time nature of the network are verified by experiments on Hangzhou and Chengdu location datasets.

A Named Entity Recognition Approach for Electronic Medical Records Using BERT Semantic Enhancement and BiLSTM

Aiming at the problems of missing local context features, single word vector representation, and low entity recognition accuracy, a method for e-medical recording with named entity recognition, which is based on BERT and model fusion, is proposed. First, with the model of BERT for pre-training, the preceding and following contextual information is fused for the enhancement of word semantic representation and alleviation of the problem of polysemy; second, the network of bi-directional long-short term memory is for obtaining the sequence feature matrix, generation of optimal sequence in global sense achieved through the conditional random field model; finally, data enhancement is used to alleviate the class imbalance and improve the model ability in generalization. Results of the experiments find model proposal measured by F1 on CCKS21 data set reaches 0.8548, which is 0.51% and 0.08% higher than models with ID-CNNs-CRF and multi-task RNN. This demonstrates the excellent performance of the method proposed in this paper in improving named entity recognition.

A Convolutional Neural Network Model for Setswana Named Entity Recognition

Named entity recognition (NER) is a key component of the core task of natural language processing (NLP). In order to represent language, neural networks have been used starting in the 2000s, which enhanced entity recognition outcomes. The Setswana language, in contrast, has never been used with neural networks, in particular convolutional neural networks (CNN). Recently, problems with NLP have been addressed using CNNs, and the results have been quite interesting. CNNs are frequently used in NLP due to their ease of training and reputation as the best in sequence labelling. They depict the interdependence of all conceivable word combinations. Given the difficulties in identifying named entities for South African languages, including Setswana, and the inadequacy of resources, this research proposes the use of CNN model to identify named entities for Setswana. The results obtained are benchmarked with traditional methods such as Conditional random fields (CRF). The performance metrics such as F1-Score are explored in establishing the magnitude of trust and reliability of the proposed model. The model is evaluated using data from the South African Centre for Digital Language Resources' Setswana NER dataset. Compared to the present CRF model, which had an F-score performance of 78.0%, the testing results demonstrate that the model performs 94.0% better.

Multi-Meta Information Embedding Enhanced BERT for Chinese Mechanics Entity Recognition

The automatic extraction of key entities in mechanics problems is an important means to automatically solve mechanics problems. Nevertheless, for standard Chinese, compared with the open domain, mechanics problems have a large number of specialized terms and composite entities, which leads to a low recognition capability. Although recent research demonstrates that external information and pre-trained language models can improve the performance of Chinese Named Entity Recognition (CNER), few efforts have been made to combine the two to explore high-performance algorithms for extracting mechanics entities. Therefore, this article proposes a Multi-Meta Information Embedding Enhanced Bidirectional Encoder Representation from Transformers (MMIEE-BERT) for recognizing entities in mechanics problems. The proposed method integrates lexical information and radical information into BERT layers directly by employing an information adapter layer (IAL). Firstly, according to the characteristics of Chinese, a Multi-Meta Information Embedding (MMIE) including character embedding, lexical embedding, and radical embedding is proposed to enhance Chinese sentence representation. Secondly, an information adapter layer (IAL) is proposed to fuse the above three embeddings into the lower layers of the BERT. Thirdly, a Bidirectional Long Short-Term Memory (BiLSTM) network and a Conditional Random Field (CRF) model are applied to semantically encode the output of MMIEE-BERT and obtain each character’s label. Finally, extensive experiments were carried out on the dataset built by our team and widely used datasets. The results demonstrate that the proposed method has more advantages than the existing models in the entity recognition of mechanics problems, and the precision, recall, and F1 score were improved. The proposed method is expected to provide an automatic means for extracting key information from mechanics problems.

Evaluation of a prototype machine learning tool to semi-automate data extraction for systematic literature reviews

BackgroundEvidence-based medicine requires synthesis of research through rigorous and time-intensive systematic literature reviews (SLRs), with significant resource expenditure for data extraction from scientific publications. Machine learning may enable the timely completion of SLRs and reduce errors by automating data identification and extraction.MethodsWe evaluated the use of machine learning to extract data from publications related to SLRs in oncology (SLR 1) and Fabry disease (SLR 2). SLR 1 predominantly contained interventional studies and SLR 2 observational studies.Predefined key terms and data were manually annotated to train and test bidirectional encoder representations from transformers (BERT) and bidirectional long-short-term memory machine learning models. Using human annotation as a reference, we assessed the ability of the models to identify biomedical terms of interest (entities) and their relations. We also pretrained BERT on a corpus of 100,000 open access clinical publications and/or enhanced context-dependent entity classification with a conditional random field (CRF) model.Performance was measured using the F1 score, a metric that combines precision and recall. We defined successful matches as partial overlap of entities of the same type.ResultsFor entity recognition, the pretrained BERT+CRF model had the best performance, with an F1 score of 73% in SLR 1 and 70% in SLR 2. Entity types identified with the highest accuracy were metrics for progression-free survival (SLR 1, F1 score 88%) or for patient age (SLR 2, F1 score 82%). Treatment arm dosage was identified less successfully (F1 scores 60% [SLR 1] and 49% [SLR 2]). The best-performing model for relation extraction, pretrained BERT relation classification, exhibited F1 scores higher than 90% in cases with at least 80 relation examples for a pair of related entity types.ConclusionsThe performance of BERT is enhanced by pretraining with biomedical literature and by combining with a CRF model. With refinement, machine learning may assist with manual data extraction for SLRs.

Identification of mobile genetic elements with geNomad.

Identifying and characterizing mobile genetic elements in sequencing data is essential for understanding their diversity, ecology, biotechnological applications and impact on public health. Here we introduce geNomad, a classification and annotation framework that combines information from gene content and a deep neural network to identify sequences of plasmids and viruses. geNomad uses a dataset of more than 200,000 marker protein profiles to provide functional gene annotation and taxonomic assignment of viral genomes. Using a conditional random field model, geNomad also detects proviruses integrated into host genomes with high precision. In benchmarks, geNomad achieved high classification performance for diverse plasmids and viruses (Matthews correlation coefficient of 77.8% and 95.3%, respectively), substantially outperforming other tools. Leveraging geNomad's speed and scalability, we processed over 2.7 trillion base pairs of sequencing data, leading to the discovery of millions of viruses and plasmids that are available through the IMG/VR and IMG/PR databases. geNomad is available at https://portal.nersc.gov/genomad .

A management knowledge graph approach for critical infrastructure protection: Ontology design, information extraction and relation prediction

Critical Infrastructures (CI) underpin the basic functioning of society and the economy. Proper governance of CI security management remains a crucial challenge. This study aims to construct a knowledge graph for modeling CI protection. While the previous research has focused on threat intelligence modeling and open knowledge bases, they miss considering the defense side. Accordingly, we propose a knowledge graph for critical infrastructure protection, CIPKG, that extends the management ontology to include the defense side. It addresses the cross-industry and cross-time information gaps that occur in the process of CI protection management, making it more comprehensive in structure than the existing knowledge graph. We employ simplified Structured Threat Information Expression as attack ontology and design a new ontology for the defense side, which could combine with the existing threat ontology to form the CI protection knowledge graph. To dynamically extract information from emerging knowledge, we employ a Bi-directional Long Short-Term Memory and Conditional Random Field model with pre-trained cybersecurity domain-specific Bidirectional Encoder Representations from Transformers to recognize the named entities from CI regulations and standards. To associate the threat part with the management portion of the knowledge graph, we adopt the Knowledge Graph Bidirectional Encoder Representations from Transformer model to capture the semantic information and predict the relationship between threat and management. After information extraction and relation prediction, we build a knowledge graph with 529,360 nodes and about 3,335,000 edges.

Probabilistic multi-modal depth estimation based on camera–LiDAR sensor fusion

Multi-modal depth estimation is one of the key challenges for endowing autonomous machines with robust robotic perception capabilities. There have been outstanding advances in the development of uni-modal depth estimation techniques based on either monocular cameras, because of their rich resolution, or LiDAR sensors, due to the precise geometric data they provide. However, each of these suffers from some inherent drawbacks, such as high sensitivity to changes in illumination conditions in the case of cameras and limited resolution for the LiDARs. Sensor fusion can be used to combine the merits and compensate for the downsides of these two kinds of sensors. Nevertheless, current fusion methods work at a high level. They process the sensor data streams independently and combine the high-level estimates obtained for each sensor. In this paper, we tackle the problem at a low level, fusing the raw sensor streams, thus obtaining depth estimates which are both dense and precise, and can be used as a unified multi-modal data source for higher-level estimation problems. This work proposes a conditional random field model with multiple geometry and appearance potentials. It seamlessly represents the problem of estimating dense depth maps from camera and LiDAR data. The model can be optimized efficiently using the conjugate gradient squared algorithm. The proposed method was evaluated and compared with the state of the art using the commonly used KITTI benchmark dataset.