Kinds Of Datasets Research Articles

On Characterization of Norm-Referenced Achievement Grading Schemes toward Explainability and Selectability

Grading is the process of interpreting learning competence to inform learners and instructors of the current learning ability levels and necessary improvement. For norm-referenced grading, the instructors use a conventionally statistical method, z score. It is difficult for such a method to achieve explainable grade discrimination to resolve dispute between learners and instructors. To solve such difficulty, this paper proposes a simple and efficient algorithm for explainable norm-referenced grading. Moreover, the rise of artificial intelligence nowadays makes machine learning techniques attractive to the norm-referenced grading in general. This paper also investigates two popular clustering methods, K-means and partitioning around medoids. The experiment relied on the data sets of various score distributions and a metric, namely, Davies–Bouldin index. The comparative evaluation reveals that our algorithm overall outperforms the other three methods and is appropriate for all kinds of data sets in almost all cases. Our findings however lead to a practically useful guideline for the selection of appropriate grading methods including both clustering methods and z score.

Machine learning for image-based detection of patients with obstructive sleep apnea: an exploratory study

PurposeIn 2-dimensional lateral cephalometric radiographs, patients with severe obstructive sleep apnea (OSA) exhibit a more crowded oropharynx in comparison with non-OSA. We tested the hypothesis that machine learning, an application of artificial intelligence (AI), could be used to detect patients with severe OSA based on 2-dimensional images.MethodsA deep convolutional neural network was developed (n = 1258; 90%) and tested (n = 131; 10%) using data from 1389 (100%) lateral cephalometric radiographs obtained from individuals diagnosed with severe OSA (n = 867; apnea hypopnea index > 30 events/h sleep) or non-OSA (n = 522; apnea hypopnea index < 5 events/h sleep) at a single center for sleep disorders. Three kinds of data sets were prepared by changing the area of interest using a single image: the original image without any modification (full image), an image containing a facial profile, upper airway, and craniofacial soft/hard tissues (main region), and an image containing part of the occipital region (head only). A radiologist also performed a conventional manual cephalometric analysis of the full image for comparison.ResultsThe sensitivity/specificity was 0.87/0.82 for full image, 0.88/0.75 for main region, 0.71/0.63 for head only, and 0.54/0.80 for the manual analysis. The area under the receiver-operating characteristic curve was the highest for main region 0.92, for full image 0.89, for head only 0.70, and for manual cephalometric analysis 0.75.ConclusionsA deep convolutional neural network identified individuals with severe OSA with high accuracy. Future research on this concept using AI and images can be further encouraged when discussing triage of OSA.

Estimating the Optimal Number of Clusters Via Internal Validity Index

Estimating the optimal number of clusters (NC) is pivotal in cluster analysis. From the viewpoint of sample geometry, a novel internal clustering validity index, which is termed the between-within cluster (BWC) index, is designed in this paper. Moreover, a method is proposed to estimate the optimal NC. The BWC index improves the well-known Silhouette index. BWC validates the clustering results from a certain clustering algorithm (e.g., affinity propagation or hierarchical) and estimates the optimal NC for many kinds of data sets, including synthetic data sets, benchmark data sets, UCI data sets, gene expression data sets, and images. Theoretical analysis and experimental studies demonstrate the effectiveness and high efficiency of the new index and method.

Review of Classification Methods on Unbalanced Data Sets

This paper studies the classification of unbalanced data sets. First, this kind of data sets is briefly introduced, and then the classification methods of unbalanced data sets are analyzed in detail from different perspectives such as data sampling method, algorithm level, feature level, cost-sensitive function, and deep learning. In addition, the data sampling methods are divided into different technologies for introduction: unbalanced data set classification method based on synthetic minority over-sampling technology (SMOTE), support vector machine (SVM) technology, and k-nearest neighbor (KNN) technology, etc. Then, the advantages and disadvantages of these methods are compared. Finally, the evaluation criteria of the unbalanced data set classifier are summarized, and the future work directions are prospected and summarized.

Long-term trends of fishery landings and target fish populations in the lower La Plata basin

Abstract The La Plata basin is the second largest basin of South America and has supported important river fisheries for more than a century. In this paper, we evaluate for the first time the historical trends of landings of 21 fish taxa and the recent population trends of 27 species of commercial fishes in the lower La Plata basin (Argentina). We compiled three kinds of data sets: Total fishery landings (between 1934 and 1986) and exports (1994‒2019), fisheries monitoring programs of Chaco and Santa Fe provinces in the Paraná River (2009‒2019), and surveys of fish populations in the Upper (Corrientes, 1993‒2020) and Middle (EBIPES, 2005‒2020) Paraná River. The analysis of the historical landings showed more species declining in the lower portion of the basin than in the upper basin. Regarding recent population trends, Pimelodus spp., Hoplias spp., Salminus brasiliensis, Luciopimelodus pati, and Ageneiosus spp. declined in more than one region, while Megaleporinus spp., Pterodoras granulosus, and Oxydoras kneri showed stable to positive trends, with the other species varying in their trends between regions. These tendencies could be associated to a combination of factors such as overfishing and environmental changes that would require an ecosystem approach for their adequate management.

Research on Human Travel Correlation for Urban Transport Planning Based on Multisource Data.

With the rapid development of positioning techniques, a large amount of human travel trajectory data is collected. These datasets have become an effective data resource for obtaining urban traffic patterns. However, many traffic analyses are only based on a single dataset. It is difficult to determine whether a single-dataset-based result can meet the requirement of urban transport planning. In response to this problem, we attempted to obtain traffic patterns and population distributions from the perspective of multisource traffic data using license plate recognition (LPR) data and cellular signaling (CS) data. Based on the two kinds of datasets, identification methods of residents’ travel stay point are proposed. For LPR data, it was identified based on different vehicle speed thresholds at different times. For CS data, a spatiotemporal clustering algorithm based on time allocation was proposed to recognize it. We then used the correlation coefficient r and the significance test p-values to analyze the correlations between the CS and LPR data in terms of the population distribution and traffic patterns. We studied two real-world datasets from five working days of human mobility data and found that they were significantly correlated for the stay and move population distributions. Then, the analysis scale was refined to hour level. We also found that they still maintain a significant correlation. Finally, the origin–destination (OD) matrices between traffic analysis zones (TAZs) were obtained. Except for a few TAZs with poor correlations due to the fewer LPR records, the correlations of the other TAZs remained high. It showed that the population distribution and traffic patterns computed by the two datasets were fairly similar. Our research provides a method to improve the analysis of complex travel patterns and behaviors and provides opportunities for travel demand modeling and urban transport planning. The findings can also help decision-makers understand urban human mobility and can serve as a guide for urban management and transport planning.

Data Validation and Mesoscale Assimilation of Himawari-8 Optimal Cloud Analysis Products

AbstractHimawari-8 optimal cloud analysis (OCA), which employs all 16 channels of the Advanced Himawari Imager, provides cloud properties such as cloud phase, top pressure, optical thickness, effective radius, and water path. By using OCA, the water vapor distribution can be inferred with high spatiotemporal resolution and with a wide coverage, including over the ocean, which can be useful for improving initial states for prediction of the torrential rainfalls that occur frequently in Japan. OCA products were first evaluated by comparing them with different kinds of datasets (surface, sonde, and ceilometer observations) and with model outputs, to determine their data characteristics. Overall, OCA data were consistent with observations of water clouds with moderate optical thicknesses at low to midlevels. Next, pseudorelative humidity data were derived from the OCA products, and utilized in assimilation experiments of a few heavy rainfall cases, conducted with the Japan Meteorological Agency’s nonhydrostatic model–based Variational Data Assimilation System. Assimilation of OCA pseudorelative humidities caused there to be significant differences in the initial conditions of water vapor fields compared to the control, especially where OCA clouds were detected, and their influence lasted relatively long in terms of forecast hours. Impacts of assimilation on other variables, such as wind speed, were also seen. When the OCA data successfully represented low-level inflows from over the ocean, they positively impacted precipitation forecasts at extended forecast times.

Neural-Network-Based Curve Fitting Using Totally Positive Rational Bases

This paper proposes a method for learning the process of curve fitting through a general class of totally positive rational bases. The approximation is achieved by finding suitable weights and control points to fit the given set of data points using a neural network and a training algorithm, called AdaMax algorithm, which is a first-order gradient-based stochastic optimization. The neural network presented in this paper is novel and based on a recent generalization of rational curves which inherit geometric properties and algorithms of the traditional rational Bézier curves. The neural network has been applied to different kinds of datasets and it has been compared with the traditional least-squares method to test its performance. The obtained results show that our method can generate a satisfactory approximation.

Interpoint Similarity-Based Uncertainty Measure for Robust Learning

Identifying reliable information from the information ocean is a natural talent of human being, which can be hardly formalized by machines. In this paper, we show how to measure the degree of uncertainty in terms of interpoint similarity. For applications under complex uncertainty, it is desirable that we provide a systematic way for users to identify reliable information. In our approach, the similarity uncertainty for different kinds of data sets is defined according to the Shannon entropy theory. Then, similarity constrained models are designed to guarantee superior learning performance under uncertainty. Experiments using both simulation data set and several public data sets, can clearly demonstrate significant improvements of the proposed method under large uncertainty.

SETNDS: A SET-Based Non-Dominated Sorting Algorithm for Multi-Objective Optimization Problems

Non-dominated sorting, used to find pareto solutions or assign solutions to different fronts, is a key but time-consuming process in multi-objective evolutionary algorithms (MOEAs). The best-case and worst-case time complexity of non-dominated sorting algorithms currently known are O(MNlogN) and O(MN2); M and N represent the number of objectives and the population size, respectively. In this paper, a more efficient SET-based non-dominated sorting algorithm, shorted to SETNDS, is proposed. The proposed algorithm can greatly reduce the number of comparisons on the promise of ensuring a shorter running time. In SETNDS, the rank of a solution to be sorted is determined by only comparing with the one with the highest rank degree in its dominant set. This algorithm is compared with six generally existing non-dominated sorting algorithms—fast non-dominated sorting, the arena’s principle sort, the deductive sort, the corner sort, the efficient non-dominated sort and the best order sort on several kinds of datasets. The compared results show that the proposed algorithm is feasible and effective and its computational efficiency outperforms other existing algorithms.

CDBH: A clustering and density-based hybrid approach for imbalanced data classification

The problem of imbalanced data set classification is prevalent in the studies of machine learning and data mining. In these kinds of data sets, the number of samples in classes is unequal so that one class has a lot more samples (the majority or negative class) than the other (the minority or positive class). The classical classifiers are ineffective in these conditions because they are biased toward the majority class and ignore the minority class, which is more important. Preprocessing the data distribution before training the classifier is one of the most effective methods to resolve this problem. These methods, balance the data distribution by decreasing the majority class size (under-sampling methods) or increasing the minority class size (over-sampling methods) or combining both of them (hybrid methods). In this paper, we propose an effective and simple hybrid approach based on the density concept and clustering, which is called Clustering and Density-Based Hybrid (CDBH). First, the minority class samples are clustered by the well-known k-means algorithm and their densities in each cluster are obtained. Then, the denser minority samples are selected with more likely to generate the new minority samples. To decrease the majority class size, the k-means algorithm is applied again on the majority class samples to cluster them and compute their densities, like the previous stage. Finally, the denser majority samples will have more chance to choose from the training set, and other samples are removed to balance the data samples distribution between classes. In the experiments, the Support Vector Machine (SVM) classifier is used as the classifier, and F-measure and AUC criteria are employed for evaluation. Also, preprocessing methods are compared in terms of the complexity of the classification model and the over-sampling rate. The results of comparing CDBH and other state of the art methods over 44 imbalanced data sets show the superiority of the proposed CDBH method based on the F-measure criterion.

Multi-scale supervised clustering-based feature selection for tumor classification and identification of biomarkers and targets on genomic data

BackgroundThe small number of samples and the curse of dimensionality hamper the better application of deep learning techniques for disease classification. Additionally, the performance of clustering-based feature selection algorithms is still far from being satisfactory due to their limitation in using unsupervised learning methods. To enhance interpretability and overcome this problem, we developed a novel feature selection algorithm. In the meantime, complex genomic data brought great challenges for the identification of biomarkers and therapeutic targets. The current some feature selection methods have the problem of low sensitivity and specificity in this field.ResultsIn this article, we designed a multi-scale clustering-based feature selection algorithm named MCBFS which simultaneously performs feature selection and model learning for genomic data analysis. The experimental results demonstrated that MCBFS is robust and effective by comparing it with seven benchmark and six state-of-the-art supervised methods on eight data sets. The visualization results and the statistical test showed that MCBFS can capture the informative genes and improve the interpretability and visualization of tumor gene expression and single-cell sequencing data. Additionally, we developed a general framework named McbfsNW using gene expression data and protein interaction data to identify robust biomarkers and therapeutic targets for diagnosis and therapy of diseases. The framework incorporates the MCBFS algorithm, network recognition ensemble algorithm and feature selection wrapper. McbfsNW has been applied to the lung adenocarcinoma (LUAD) data sets. The preliminary results demonstrated that higher prediction results can be attained by identified biomarkers on the independent LUAD data set, and we also structured a drug-target network which may be good for LUAD therapy.ConclusionsThe proposed novel feature selection method is robust and effective for gene selection, classification, and visualization. The framework McbfsNW is practical and helpful for the identification of biomarkers and targets on genomic data. It is believed that the same methods and principles are extensible and applicable to other different kinds of data sets.

Comparative Study of Clustering Approaches Applied to Spatial or Temporal Pattern Discovery

In the framework of ecological or environmental assessments and management, detection, characterization and forecasting of the dynamics of environmental states are of paramount importance. These states should reflect general patterns of change, recurrent or occasional events, long-lasting or short or extreme events which contribute to explain the structure and the function of the ecosystem. To identify such states, many scientific consortiums promote the implementation of Integrated Observing Systems which generate increasing amount of complex multivariate/multisource/multiscale datasets. Extracting the most relevant ecological information from such complex datasets requires the implementation of Machine Learning-based processing tools. In this context, we proposed a divisive spectral clustering architecture—the Multi-level Spectral Clustering (M-SC) which is, in this paper, extended with a no-cut criteria. This method is developed to perform detection events for data with a complex shape and high local connexity. While the M-SC method was firstly developed and implemented for a given specific case study, we proposed here to compare our new M-SC method with several existing direct and hierarchical clustering approaches. The clustering performance is assessed from different datasets with hard shapes to segment. Spectral methods are most efficient discovering all spatial patterns. For the segmentation of time series, hierarchical methods better isolated event patterns. The new M-SC algorithm, which combines hierarchical and spectral approaches, give promise results in the segmentation of both spatial UCI databases and marine time series compared to other approaches. The ability of our M-SC method to deal with many kinds of datasets allows a large comparability of results if applies within a broad Integrated Observing Systems. Beyond scientific knowledge improvements, this comparability is crucial for decision-making about environmental management.

Graph-based parallel large scale structure from motion

While Structure from Motion achieves great success in 3D reconstruction, it still meets challenges on large scale scenes. Incremental SfM approaches are robust to outliers, but are limited by low efficiency and easy suffer from drift problem. Though Global SfM methods are more efficient than incremental approaches, they are sensitive to outliers, and would also meet memory limitation and time bottleneck. In this work, large scale SfM is deemed as a graph problem, where graph are respectively constructed in image clustering step and local reconstructions merging step. By leveraging the graph structure, we are able to handle large scale dataset in divide-and-conquer manner. Firstly, images are modelled as graph nodes, with edges are retrieved from geometric information after feature matching. Then images are divided into independent clusters by a image clustering algorithm, and followed by a subgraph expansion step, the connection and completeness of scenes are enhanced by walking along a maximum spanning tree, which is utilized to construct overlapping images between clusters. Secondly, Image clusters are distributed into servers to execute SfM in parallel mode. Thirdly, after local reconstructions complete, we construct a minimum spanning tree to find accurate similarity transformations. Then the minimum spanning tree is transformed into a Minimum Height Tree to find a proper anchor node, and is further utilized to prevent error accumulation. We evaluate our approach on various kinds of datasets and our approach shows superiority over the state-of-the-art in accuracy and efficiency. Our algorithm is open-sourced in https://github.com/AIBluefisher/GraphSfM.

On the use of Archimedean copulas for insurance modelling

AbstractIn this paper, we explore the use of an extensive list of Archimedean copulas in general and life insurance modelling. We consider not only the usual choices like the Clayton, Gumbel–Hougaard, and Frank copulas but also several others which have not drawn much attention in previous applications. First, we apply different copula functions to two general insurance data sets, co-modelling losses and allocated loss adjustment expenses, and also losses to building and contents. Second, we adopt these copulas for modelling the mortality trends of two neighbouring countries and calculate the market price of a mortality bond. Our results clearly show that the diversity of Archimedean copula structures gives much flexibility for modelling different kinds of data sets and that the copula and tail dependence assumption can have a significant impact on pricing and valuation. Moreover, we conduct a large simulation exercise to investigate further the caveats in copula selection. Finally, we examine a number of other estimation methods which have not been tested in previous insurance applications.

Randomized multi-label subproblems concatenation via error correcting output codes

In this paper, we study the error correcting output codes for multi-label classification problems. Imbalance problem is very common in multi-label task and it has not been effectively solved yet. In previous works, base classifiers are learned from the same pool of training data which may lead to similar classifiers, especially when a large amount of labels exist. To tackle these challenges, we propose a novel model called Randomized Multi-Label Subproblems Concatenation via Error Correcting Output Codes (RMSC). In order to reduce the imbalance problem and enhance the diversity between base classifiers, in our model, the original multi-label problem is transformed into an ensemble of many multi-label sub-problems and the one-vs-all encoding strategy is conducted on these sub-problems and finally corresponding coding matrices are concatenated into a whole coding matrix. Moreover, we provide a proof of enhanced diversity in our model, together with some discussions about the parameter determination and time complexity. Extensive experimental results on various kinds of data sets well demonstrate that the proposed model consistently outperforms the state-of-the-art methods.

A hybrid approach for search and rescue using 3DCNN and PSO

Search and rescue are essential applications of disaster management in which people are evacuated from the disaster-prone area to a safer place. This overall process of search and rescue can be more efficient if an automated system can quickly locate the human or area where rescue is required. To provide a faster and accurate search of those places, this paper proposes a novel approach to search and rescue using automated drone surveillance. In this paper, a complex scene classification problem is solved using the proposed 3DCNN model. The proposed model uses spatial as well as temporal features of the video for the classification of the scene as help or non-help in the natural disaster. Due to the unavailability of such kind of dataset, it is impossible to train the model. Therefore, it is essential to develop a dataset for search and rescue. The proposed dataset is a first and unique dataset for scene classification using drone surveillance. The major contribution of this paper is (1) a novel 3DCNN powered model for scene classification in drone surveillance, (2) to develop the required dataset for the training of scene classification model, and (3) particular swarm optimization (PSO)-based hyper-parameter tuning for getting the best value of multiple parameters used for training the model. Our hybridization of parameter tuning with PSO helps for the convergence of parameter values of proposed 3DCNN model, and the proposed scene classification model (3DCNN+PSO) is applied to the dataset. The proposed model gives an impressive performance to help situation identification with 98% training and 99% validation accuracy.

Crop classification in cloudy and rainy areas based on the optical-synthetic aperture radar response mechanism

In the agricultural field, optical remote sensing technology plays an important role in crop monitoring or production estimation. However, the widespread distribution of clouds and rain limits the application of optical remote sensing. Synthetic aperture radar (SAR) has been widely used for studies of oceans, atmosphere, land, and space exploration, as well as by the military due to its all-weather nature, penetration to surface and cloud layers, and diversity of information carriers. However, it is difficult to classify ground objects with high accuracy based on SAR data. Considering the features of these two datasets, we proposed a framework to improve crop classifications in cloudy and rainy areas based on the optical-SAR response mechanism. Specifically, this method is designed to train a parametric analytic model in the area using both kinds of datasets and applied in the area with only SAR data to obtain the optical time-series features. Then crops from the second area were classified by the long-short-term memory network. As an example, the parametric analytic model in Lixian County was studied and was applied to Xifeng County to classify the crops with the OA of 61%, which had proved the robustness of the method.

Recognition of partial discharge signals in impaired datasets using cumulative energy signatures

The problem of impaired data sets refers to data sets containing a vast majority of unwanted signals than signals of interest. With increased interest in partial discharge (PD) testing with arbitrary waveforms and transients, these kind of data sets are becoming more and more common. Traditional clustering techniques cannot be applied due to big differences in spatial densities of the existing clusters in the data set. This paper contributes a simple yet efficient technique to recognize PD signals from noise and other disturbances. The signal recognition features are based on two specific areas extracted from the cumulative energy signal (CE) of each recorded waveform. These areas weigh up the extent to which the recorded signals have a pulse-like shape. A third feature, defined as a shape factor, extracts additional metrics from the CE signal that serves the purpose of accounting for the factors affecting the computation of the proposed recognition features and threshold for data size reduction. These three CE-based features are used to create a graph from which a real PD can be spotted in large impaired data sets. The performance of this technique is tested using PD measurements from superimposed impulse tests on a 150 kV cable system.

0594 Can a Deep Convolutional Neural Network Extract Diagnostic Information on Obstructive Sleep Apnea from Images?

Abstract Introduction Lateral cephalometric radiography is a simple way to provide craniofacial soft/hard tissue profiles specific for patients with obstructive sleep apnea (OSA) and may thus offer diagnostic information on the disease. We hypothesized that a machine learning technology, a deep convolutional neural network (DCNN), could make it possible to detect OSA based solely on lateral cephalometric radiographs without the need for either large amounts of subjective/laboratory data or skilled analyses. Methods In this diagnostic study, a DCNN was developed (n=1,258) and tested (n=131) using data from 1,389 lateral cephalometric radiographs obtained from individuals diagnosed with severe OSA (n=867; apnea hypopnea index &amp;gt;30/hour) or non-OSA (n=522; apnea hypopnea index &amp;lt; 5) at a single center for sleep disorders from March, 2006 to February, 2017. Three kinds of data sets were prepared by changing the area of interest using a single image; original image without any modification (Full Image), image containing a facial profile, upper airway, craniofacial soft/hard tissues, and image containing part of the occipital region (upper left corner of the image; Head Only). A radiologist and an orthodontist also performed a manual cephalometric analysis of the Full Image for comparison. Observers were blinded to the patient groupings. Data analysis was performed from April, 2018 to August, 2019. When the predictive score obtained from the DCNN analysis exceeded the threshold (0.50), the patient was judged to have OSA. The primary outcome was diagnostic accuracy in terms of area under the receiver-operating characteristic curve. Results The sensitivity/specificity was 0.87/0.82 for Full Image, 0.88/0.75 for Main Region, 0.71/0.63 for Head Only, and 0.54/0.80 for the manual analysis. The area under the curve was the highest for Main Region (0.92): 0.89 for Full Image, 0.70 for Head Only, and 0.75 for the manual analysis. Conclusion A DCNN identified individuals with OSA with high accuracy. This is a useful approach that does not require any laborious analyses in a primary care setting or in remote areas where an initial specialized OSA diagnosis is not feasible. Support This study was supported in part by the Japan Society for the Promotion of Science (grant numbers 17K11793, 19K10236).