Range Of Evaluation Metrics Research Articles

IMotor-CNN: Identifying molecular functions of cytoskeleton motor proteins using 2D convolutional neural network via Chou's 5-step rule

Motor proteins are the driving force behind muscle contraction and are responsible for the active transportation of most proteins and vesicles in the cytoplasm. There are three superfamilies of cytoskeletal motor proteins with various molecular functions and structures: dynein, kinesin, and myosin. The functional loss of a specific motor protein molecular function has linked to a variety of human diseases, e.g., Charcot-Marie-Tooth disease, kidney disease. Therefore, creating a precise model to classify motor proteins is essential for helping biologists understand their molecular functions and design drug targets according to their impact on human diseases. Here we attempt to classify cytoskeleton motor proteins using deep learning, which has been increasingly and widely used to address numerous problems in a variety of fields resulting in state-of-the-art results. Our effective deep convolutional neural network is able to achieve an independent test accuracy of 97.5%, 96.4%, and 96.1% for each superfamily, respectively. Compared to other state-of-the-art methods, our approach showed a significant improvement in performance across a range of evaluation metrics. Through the proposed study, we provide an effective model for classifying motor proteins and a basis for further research that can enhance the performance of protein function classification using deep learning.

HLMCC: A Hybrid Learning Anomaly Detection Model for Unlabeled Data in Internet of Things

The Internet of Things (IoT) is a network of distributed devices or sensors connected through the internet to allow gathering and sharing of data. The data generated by these devices is affected by anomalies or abnormal behaviour due to attack issues, or breakdown in devices, as examples. However, most current anomaly detection systems rely on labelled data, while the class labels for IoT data are usually unavailable. Furthermore, the manual labelling task is expensive and time-consuming to perform due to the need for domain experts. More importantly, the volume of data in the IoT is growing rapidly, creating a need to predict the classification labels for future data. This study proposes a Hybrid Learning Model which uses both Clustering and Classification methods (HLMCC) to automate the labelling process and detect anomalies in IoT data. The model consists of two practical phases, automatic labelling and detecting anomalies. First, the HLMCC groups the data into normal and anomaly clusters by adopting Hierarchical Affinity Propagation (HAP) clustering. Second, the labelled data obtained from the clustering phase is used to train the Decision Trees (DTs) and to classify future unseen data. The results show that the HLMCC is able to automate the labelling of data, which is beneficial to minimize human involvement. Moreover, HLMCC outperforms the DTs on the originally labelled datasets and the state-of-the-art model over a wide range of evaluation metrics based on the average ranks. HLMCC produces the highest average ranks against other models in terms of False Positive Rate (FPR), recall, precision and the Area Under the Precision-Recall curve (AUCPR) with 1.8, 1.6, 1.8 and 1.8, respectively.

Graph Convolutional Network with Sequential Attention for Goal-Oriented Dialogue Systems

Domain-specific goal-oriented dialogue systems typically require modeling three types of inputs, namely, (i) the knowledge-base associated with the domain, (ii) the history of the conversation, which is a sequence of utterances, and (iii) the current utterance for which the response needs to be generated. While modeling these inputs, current state-of-the-art models such as Mem2Seq typically ignore the rich structure inherent in the knowledge graph and the sentences in the conversation context. Inspired by the recent success of structure-aware Graph Convolutional Networks (GCNs) for various NLP tasks such as machine translation, semantic role labeling, and document dating, we propose a memory-augmented GCN for goal-oriented dialogues. Our model exploits (i) the entity relation graph in a knowledge-base and (ii) the dependency graph associated with an utterance to compute richer representations for words and entities. Further, we take cognizance of the fact that in certain situations, such as when the conversation is in a code-mixed language, dependency parsers may not be available. We show that in such situations we could use the global word co-occurrence graph to enrich the representations of utterances. We experiment with four datasets: (i) the modified DSTC2 dataset, (ii) recently released code-mixed versions of DSTC2 dataset in four languages, (iii) Wizard-of-Oz style CAM676 dataset, and (iv) Wizard-of-Oz style MultiWOZ dataset. On all four datasets our method outperforms existing methods, on a wide range of evaluation metrics.

A breeding site model for regional, dynamical malaria simulations evaluated using in situ temporary ponds observations.

Daily observations of potential mosquito developmental habitats in a suburb of Kumasi in central Ghana reveal a strong variability in their water persistence times, which ranged between 11 and 81 days. The persistence of the ponds was strongly tied with rainfall, location and size of the puddles. A simple power-law relationship is found to fit the relationship between the average pond depth and area well. A prognostic water balance model is derived that describes the temporal evolution of the pond area and depth, incorporating the power-law geometrical relation. Pond area increases in response to rainfall, while evaporation and infiltration act as sink terms. Based on a range of evaluation metrics, the prognostic model is judged to provide a good representation of the pond coverage evolution at most sites. Finally, we demonstrate that the prognostic equation can be generalised and equally applied to a grid-cell to derive a fractional pond coverage, and thus can be implemented in spatially distributed models for relevant vector- borne diseases such as malaria.

An Appraisal Standard for a quick and reliable comparison between risk stratification methods for local healthcare systems

Introduction: European countries need to address the issues related to ageing populations and the consequent delivery of health care to multi-morbid and or frail patients. A potential solution is represented by the identification of those patients at greatest risk for costly and preventable service utilization by means of risk stratification tools. Currently, many stratification and predictive modelling software tools exist, with various degrees of (claimed) accuracy, internal/external validation, requirements on the input data, and target population. Because of this scenario, an Appraisal Standard (AS) model which evaluates different stratification techniques has been designed. The AS allows policy makers and health care managers to evaluate different stratification methods and to identify the most suitable for the health care environment under investigation. The work herewith described has been performed within the European Project ASSEHS (http://assehs.eu/). Methods – Literature review and Survey: First, a systematic literature review was conducted to extract data from validations of stratification tools in different populations that have been published in publically available reports or in peer-reviewed journal article resulting in a total of 216 publications or reports. A hand-selection of 26 publications was used to start designing the first version of the AS model. A second approach of data collection was considered to gather that information on either unpublished tools or aspects usually not included in scientific manuscripts (e.g. implementation 15th International Conference on Integrated Care, Edinburgh, UK, March 25-27, 2015 1 International Journal of Integrated Care – Volume 15, 27 May – URN:NBN:NL:UI:10-1-117019 – http://www.ijic.org/ and ease of use of the tool). A survey was developed to be sent out to appropriate professionals associated with one of the health care regions within and outside ASSEHS’ consortium. The surveys are meant to gather hands-on experiences that we could not retrieve from literature searches. The Appraisal Standard: The literature review and survey led to both the design of the data model underneath the AS and the collection of the information stored in the AS’ database. More specifically, the AS which comprises two main parts: an evaluation of the intrinsic properties of the stratification tools themselves and an evaluation of the study conditions that were used to develop and/or validate these tools. As a consequence, the core of the AS contains a table including a variety of properties of a stratification tool (e.g. model name, main technique, outcome type, etc.), and a table describing the different studies that evaluate those tools (e.g. authors, year, study type, measured risk, population, etc.). Multiple studies may have evaluated a particular stratification tool in different populations. Furthermore, one study may have compared a variety of stratification tools under the same conditions. There are also a wide range of evaluation metrics that can be used to determine the performance of stratification tools, such as area under the operator characteristic curve, mean squared prediction error etc. The Appraisal Standard therefore covers a wide range of evaluation metrics generally used in the scientific literature. These evaluation metrics were included in a table, using evaluation metric values as the connection between the tables describing stratification tool and the table describing study conditions. Finally, stratification tools generally make use of the same or similar (categories of) predictor variables (e.g. socio-demographic data, diagnostic data, pharmacy data, etc.). To account for this, the AS includes a table with such variables. The stratification model table can refer to this table. Discussion and conclusion: The work presented here is aligned with the priorities identified within several European health systems towards the implementation of proactive, anticipatory and integrated care. This approach is an answer to the strong urge of health care expenditures’ mitigation, enhancement of quality and continuity of care and health care sustainability. The AS will allow policy makers and health care managers to perform a critical comparison of different risk stratification techniques guiding them towards the selection of the best-in class stratification technique for their health care environment. The aim of this study is twofold: on one hand to support and guide those decision makers already aware of the efficacy of risk stratification techniques; on the other hand, to disseminate the knowledge on risk stratification tools via a comprehensive and unique source of information to reach those professionals and or organizations not yet informed about the deployment of risk stratification techniques. Limitations and future steps: As discussed in the section “Methods”, some health care organization developed their own stratification model but neither technical reports nor peerreviewed scientific articles have been published to describe their tool. For such scenario, we developed a survey to gather the information needed in order to include unpublished current or future tool in the database.

Fire weather simulation skill by the Weather Research and Forecasting (WRF) model over south-east Australia from 1985 to 2009

The fire weather of south-east Australia from 1985 to 2009 has been simulated using the Weather Research and Forecasting (WRF) model. The US National Oceanic and Atmospheric Administration Centers for Environmental Prediction and National Center for Atmospheric Research reanalysis supplied the lateral boundary conditions and initial conditions. The model simulated climate and the reanalysis were evaluated against station-based observations of the McArthur Forest Fire Danger Index (FFDI) using probability density function skill scores, annual cumulative FFDI and days per year with FFDI above 50. WRF simulated the main features of the FFDI distribution and its spatial variation, with an overall positive bias. Errors in average FFDI were caused mostly by errors in the ability of WRF to simulate relative humidity. In contrast, errors in extreme FFDI values were driven mainly by WRF errors in wind speed simulation. However, in both cases the quality of the observed data is difficult to ascertain. WRF run with 50-km grid spacing did not consistently improve upon the reanalysis statistics. Decreasing the grid spacing to 10km led to fire weather that was generally closer to observations than the reanalysis across the full range of evaluation metrics used here. This suggests it is a very useful tool for modelling fire weather over the entire landscape of south-east Australia.

Virtual reference services: Evaluation of online reference services

Virtual reference services: Evaluation of online reference services

Special issue on echo state networks and liquid state machines

Representation learning impacts the performance of Machine Learning (ML) models. Feature extraction-based methods such as Auto-Encoders (AEs) are used to find new, more accurate data representations from original ones. They perform efficiently on a specific task, in terms of: (1) high accuracy, (2) large short-term memory and (3) low execution time. The Echo-State Network (ESN) is a recent specific kind of a Recurrent Neural Networks (RNN), that presents very rich dynamics on account of its reservoir-based hidden layer. It is widely used in dealing with complex non-linear problems and has been shown to outperform classical approaches in a number of benchmark tasks. In this paper, the powerful dynamism and large memory provided by the ESN and complementary strengths of AEs in feature extraction are integrated, to develop a novel Echo-State Recurrent Autoencoder (ES-RA). In order to devise more robust alternatives to conventional reservoir-based networks, both single- (SL-ES-RA) and multi-layer (ML-ES-RA) models are formulated. The new features, once extracted from ESN’s hidden layer, are applied to various benchmark ML tasks including classification, time series prediction and regression. A range of evaluation metrics are shown to improve considerably compared to those obtained when applying original data features. An accuracy-based comparison is performed between our proposed recurrent AEs and two variants of ELM feed-forward AEs (Single and ML), for both noise free and noisy data. In summary, a comparative empirical study reveals the key contribution of exploiting recurrent connections in improving benchmark performance results.