Input Data Matrix Research Articles

Transformative crop insurance solution with big earth data: Implementation for potato in India

ContextCrop insurance has become an indispensable risk management tool in the agricultural sector because globally crops are being exposed to multiple hazards. The lack of reliable crop yield data has impacted the sustenance of area-yield crop insurance schemes. Index-based insurance, which links pay-outs to crop performance proxies rather than measured losses, is being explored to improve the effectiveness of crop insurance contracts. ObjectiveThis paper presents an innovative crop insurance scheme that has replaced the existing ‘area-yield’ approach using bias-prone crop yield estimates with the ‘area-crop performance approach’ using objectively measured satellite indices. MethodSatellite-based crop mapping, satellite and weather-based crop health indicators, field data collection and analysis, composite index generation, and insurance loss assessment are major tasks in the project. Data of Sentinel-1 and 2 satellites, weather datasets and mobile app-based field data from transplantation to harvesting of the crop constituted a huge repository of the database in this project. Metrics derived from established satellite indices, such as NDVI, LSWI and Backscatter, along with weather indices, were synthesized into a composite index of crop performance called Crop Health Factor (CHF). The input data matrix of the CHF model included eight input indicators. After data normalization, weights for these indicators were generated using the entropy technique, a proven method of information measurement that produces balanced relationships and unbiased weights. The CHF was first generated with data from the past years (2016–2019), and the resulting weights were then applied to the normalized data of the current year (2020). ResultsThe current crop insurance scheme, using CHF data instead of yield data, was implemented in the state of West Bengal, India, covering about 500,000 ha of potato across and one thousand insurance units in the 2020 crop season. The CHF and yield data from past years showed similar patterns in the majority of cases. The indemnity level was set at 70 % of the normal CHF, which was the average CHF of past years. Loss assessment and compensation payouts for the current year were determined by the extent of CHF reduction beyond the indemnity level. SignificanceThis new index-insurance scheme has many advantages over the conventional yield-based scheme in terms of transparency, objectivity and ease of implementation. There is scope for improving the composite index with additional features. Such technology-driven index-insurance schemes for field crops are expected to bring a paradigm shift in the crop insurance sector, giving rise to new business models and benefitting all the stakeholders.

The Auto-Correlation Function Aided Sparse Support Matrix Machine for EEG-Based Fatigue Detection

Due to support vector machine (SVM) methods generally vectorizing the input matrix and destroying the location dependence characteristic of EEG signals, the support matrix machine (SMM) methods have been presented to incorporate input signal with its location information into the input data matrix and to perform the classification algorithm. In this brief, we investigate an auto-correlation function based sparse support matrix machine (ACF-SSMM) algorithm to optimize and classify the EEG fatigue signals. The main contents include: 1) to input a matrix EEG signal with location information and compress the redundant features by using the sparse principle; 2) to extend the input matrix by adding data through an ACF transform, which takes into account information containing the previous/current instants, to accurately express the memory dynamics of the EEG signals; 3) to perform multiple similar classification algorithms on the EEG fatigue dataset (SEED-VIG dataset) to verify effectiveness of the presented ACF-SSMM algorithm in EEG-based fatigue detection. The experiments get better results, and show the effectiveness of the proposed algorithm.

Power-Efficient Deep Neural Network Accelerator Minimizing Global Buffer Access without Data Transfer between Neighboring Multiplier—Accumulator Units

This paper presents a novel method for minimizing the power consumption of weight data movements required by a convolutional operation performed on a two-dimensional multiplier–accumulator (MAC) array of a deep neural-network accelerator. The proposed technique employs a local register file (LRF) at each MAC unit in a manner such that once weight pixels are read from the global buffer into the LRF, they are reused from the LRF as many times as desired instead of being repeatedly fetched from the global buffer in each convolutional operation. One of the most evident merits of the proposed method is that the procedure is completely free from the burden of data transfer between neighboring MAC units. It was found from our simulations that the proposed method provides a power saving of approximately 83.33% and 97.62% compared with the power savings recorded by the conventional methods, respectively, when the dimensions of the input data matrix and weight matrix are 128 × 128 and 5 × 5, respectively. The power savings increase as the dimensions of the input data matrix or weight matrix increase.

Dynamic mode decomposition with core sketch

With the increase in collected data volumes, either from experimental measurements or high fidelity simulations, there is an ever-growing need to develop computationally efficient tools to process, analyze, and interpret these datasets. Modal analysis techniques have gained great interest due to their ability to identify patterns in the data and extract valuable information about the system being considered. Dynamic mode decomposition (DMD) relies on elements of the Koopman approximation theory to compute a set of modes, each associated with a fixed oscillation frequency and a decay/growth rate. Extracting these details from large datasets can be computationally expensive due to the need to implement singular value decomposition of the input data matrix. Sketching algorithms have become popular in numerical linear algebra where statistical theoretic approaches are utilized to reduce the cost of major operations. A sketch of a matrix is another matrix, which is significantly smaller, but still sufficiently approximates the original system. We put forth an efficient DMD framework, SketchyDMD, based on a core sketching algorithm that captures information about the range and corange (their mutual relationship) of input data. The proposed sketching-based framework can accelerate various portions of the DMD routines, compared to classical methods that operate directly on the raw input data. We conduct numerical experiments using the spherical shallow water equations as a prototypical model in the context of geophysical flows. We show that the proposed SketchyDMD is superior to existing randomized DMD methods that are based on capturing only the range of the input data.

Optimal number of clusters in explainable data analysis of agent-based simulation experiments

Dimension reduction and visualization of data generated from a complex simulation model are essential aspects for a better understanding of the behaviours of complex systems, and they are often required in many investigations besides computer simulation and modelling. Explainable data analysis or explainable data analytics is a new concept recently appearing in the literature within the machine learning community. The research on data explainability has become important in various fields where machine learning techniques are used. In this paper, we have extended this concept to the analysis of agent-based simulations. We refer to the explainability in the context of improving visualization and analysis of such type of simulation data, so that, a better understanding of nature of a complex system can be achieved. In this work, principal component analysis was first used as a feature extractor to reduce the dimension of the input data matrix to extract the feature vector, which consists of a set of low dimensional principal component scores. The K-means clustering method is then applied to cluster the principal component scores, so that, different treatments can be grouped to identify their similarity. Since it is crucial to determine the number of clusters in a clustering analysis, in order to improve the clustering results and meet our objective of reducing the number of treatments, two new algorithms are proposed to identify the optimal K value. The proposed methods are illustrated by applying them to data from an agent-based simulation model, and the obtained results demonstrate that the proposed methods are promising in determining the optimal number of clusters in K-means clustering and that they outperform the traditional methods.

Multiple partitions alignment via spectral rotation

Multi-view spectral clustering has drawn much attention due to the effectiveness of exploiting the similarity relationships among data points. These methods typically reveal the intrinsic structure using a predefined graph for each view. The predefined graphs are fused to a consensus one, on which the final clustering results are obtained. However, such common strategies may lead to information loss because of the inconsistency or noise among multiple views. In this paper, we propose to merge multi-view information in partition level instead of the raw feature space where the data points lie. The partition of each view is treated as a perturbation of the consensus clustering, and the multiple partitions are integrated by estimating a distinct rotation for each partition. The proposed model is formulated as a joint learning framework, i.e., with the input data matrix, our model directly outputs the final discrete clustering result. Hence it is an end-to-end single-stage learning model. An iterative updating algorithm is proposed to solve the learning problem, in which the involved variables can be optimized in a mutual reinforcement manner. Experimental results on real-world data sets illustrate the effectiveness of our model.

Learning Deep Models: Critical Points and Local Openness

With the increasing popularity of nonconvex deep models, developing a unifying theory for studying the optimization problems that arise from training these models becomes very significant. Toward this end, we present in this paper a unifying landscape analysis framework that can be used when the training objective function is the composite of simple functions. Using the local openness property of the underlying training models, we provide simple sufficient conditions under which any local optimum of the resulting optimization problem is globally optimal. We first completely characterize the local openness of the symmetric and nonsymmetric matrix multiplication mapping. Then we use our characterization to (1) provide a simple proof for the classical result of Burer-Monteiro and extend it to noncontinuous loss functions; (2) show that every local optimum of two-layer linear networks is globally optimal. Unlike many existing results in the literature, our result requires no assumption on the target data matrix [Formula: see text], and input data matrix [Formula: see text]; (3) develop a complete characterization of the local/global optima equivalence of multilayer linear neural networks (we provide various counterexamples to show the necessity of each of our assumptions); and (4) show global/local optima equivalence of overparameterized nonlinear deep models having a certain pyramidal structure. In contrast to existing works, our result requires no assumption on the differentiability of the activation functions and can go beyond “full-rank” cases.

Code design and latency analysis of distributed matrix multiplication with straggling servers in fading channels

This paper exploits coding to speed up computation offloading in a multi-server mobile edge computing (MEC) network with straggling servers and channel fading. The specific task we consider is to compute the product between a user-generated input data matrix and a large-scale model matrix that is stored distributively across the multiple edge nodes. The key idea of coding is to introduce computation redundancy to improve robustness against straggling servers and to create communication redundancy to improve reliability against channel fading. We utilize the hybrid design of maximum distance separable (MDS) coding and repetition coding. Based on the hybrid coding scheme, we conduct theoretical analysis on the average task uploading time, average edge computing time, and average output downloading time, respectively and then obtain the end-to-end task execution time. Numerical results demonstrate that when the task uploading phase or the edge computing phase is the performance bottleneck, the hybrid coding reduces to MDS coding; when the downlink transmission is the bottleneck, the hybrid coding reduces to repetition coding. The hybrid coding also outperforms the entangled polynomial coding that causes higher uplink and downlink communication loads.

Devising a new filtration method and proof of self-similarity of electromyograms

The main attention is paid to the analysis of electromyogram (EMG) signals using Poincaré plots (PP). It was established that the shapes of the plots are related to the diagnoses of patients. To study the fractal dimensionality of the PP, the method of counting the coverage figures was used. The PP filtration was carried out with the help of Haar wavelets. The self-similarity of Poincaré plots for the studied electromyograms was established, and the law of scaling was used in a fairly wide range of coverage figures. Thus, the entire Poincaré plot is statistically similar to its own parts. The fractal dimensionalities of the PP of the studied electromyograms belong to the range from 1.36 to 1.48. This, as well as the values of indicators of Hurst exponent of Poincaré plots for electromyograms that exceed the critical value of 0.5, indicate the relative stability of sequences. The algorithm of the filtration method proposed in this research involves only two simple stages: Conversion of the input data matrix for the PP using the Jacobi rotation. Decimation of both columns of the resulting matrix (the so-called "lazy wavelet-transformation", or double downsampling). The algorithm is simple to program and requires less machine time than existing filters for the PP. Filtered Poincaré plots have several advantages over unfiltered ones. They do not contain extra points, allow direct visualization of short-term and long-term variability of a signal. In addition, filtered PPs retain both the shape of their prototypes and their fractal dimensionality and variability descriptors. The detected features of electromyograms of healthy patients with characteristic low-frequency signal fluctuations can be used to make clinical decisions.

Kernel Partial Least Square Regression with High Resistance to Multiple Outliers and Bad Leverage Points on Near-Infrared Spectral Data Analysis

Multivariate statistical analysis such as partial least square regression (PLSR) is the common data processing technique used to handle high-dimensional data space on near-infrared (NIR) spectral datasets. The PLSR is useful to tackle the multicollinearity and heteroscedasticity problem that can be commonly found in such data space. With the problem of the nonlinear structure in the original input space, the use of the classical PLSR model might not be appropriate. In addition, the contamination of multiple outliers and high leverage points (HLPs) in the dataset could further damage the model. Generally, HLPs contain both good leverage points (GLPs) and bad leverage points (BLPs); therefore, in this case, removing the BLPs seems relevant since it has a significant impact on the parameter estimates and can slow down the convergence process. On the other hand, the GLPs provide a good efficiency in the model calibration process; thus, they should not be eliminated. In this study, robust alternatives to the existing kernel partial least square (KPLS) regression, which are called the kernel partial robust GM6-estimator (KPRGM6) regression and the kernel partial robust modified GM6-estimator (KPRMGM6) regression are introduced. The nonlinear solution on PLSR was handled through kernel-based learning by nonlinearly projecting the original input data matrix into a high-dimensional feature mapping that corresponded to the reproducing kernel Hilbert spaces (RKHS). To increase the robustness, the improvements on GM6 estimators are presented with the nonlinear PLSR. Based on the investigation using several artificial dataset scenarios from Monte Carlo simulations and two sets from the near-infrared (NIR) spectral dataset, the proposed robust KPRMGM6 is found to be superior to the robust KPRGM6 and non-robust KPLS.

An IoT based big data framework using equidistant heuristic and duplex deep neural network for diabetic disease prediction

A number of strategies empowered by Internet of Thing (IoT) have been utilized for the prediction of several dreadful diseases like diabetic for which ceaseless and real-time tracing system is an exceptionally predominant one. Wearable medical devices with sensor have consistently producing abundant data volume referred to as big data. With higher speed of data creation, it becomes cumbersome to accumulate, operate and analyze such abundant data volume during emergency. Deep Learning (DL) approaches have been largely made use of perceiving patterns, categorizing objects and the prediction of diabetic diseases at an early stage. Even so, DLs are fundamentally modest in evaluation precisely when the diabetic disease data size is abundant. To achieve the expectations of DLs for big data applications from an electronic device, the evaluation procedure for diabetic disease prediction must be speeded up, so that early analysis can be arrived at. In this work, a method called Equidistant Heuristic and Duplex Deep Neural Network (EH-DDNN) for early diabetic disease prediction is proposed. First, with the Big Data dataset as input, Equidistant Heuristic Pruning (EHP) algorithm is presented for feature selection. The EHP splits the input data matrix into rows and columns separately. By utilizing the notion of conditional non-alignment assessment and heuristics techniques, EHP, exploits neighbourhood evaluations into sub-division while reducing the communication time and overhead, thus enormously correlating computations. This in turn removes the irrelevant as well as redundant features, therefore resulting with fewer features easier for early prediction. Next, with the inherent features, a Duplex Deep Neural Network (DDNN) is designed for early prediction analyses a fusion of nonlinear processing features and linear response for stockpiling abundant data volume. Experiments are conducted and validation is performed on the benchmark datasets, Diabetes Data Set from UCI repository and Pima Indians Diabetes Disease dataset. Comparative analysis of diabetic disease prediction time, diabetic disease prediction overhead and ROC curve analysis are made.

Flood Forecasting Method of Small and Medium-sized Watershed Based on Convolutional Neural Network

A data-driven flood forecasting model based on a convolution neural network is proposed for the small and medium-sized watershed. Pearson correlation coefficient analysis was used to determine the time length of the input data. The historical rainfall and discharge were used to create the two-dimensional input data matrix. NAS was used to determine the structure of the model. The experiment results in Tunxi watershed in Anhui Province show that the accuracy is high under the 1˜3h forecast period, with the forecast period increased, accuracy would decrease.

Short-Term Traffic Flow Intensity Prediction Based on CHS-LSTM

Short-term traffic flow prediction is an important basis of intelligent transportation systems. Its accuracy directly affects the performance of traffic control and induction. To improve prediction accuracy, in this paper, a traffic flow prediction model is proposed by combining hierarchical agglomerative clustering (HAC), standardized Euclidean distance (SED), and a long short-term memory network (LSTM). The proposed model is called the CHS-LSTM model. In this model, HAC is used to carry out cluster analysis on the original traffic flow data sample set of target detection sections. Based on the results of cluster analysis, traffic flow data are divided into several categories. For different categories of traffic flow, SED is used to calculate the spatial correlation of the road network where the target sections are located, and the K sections most relevant to the target detection sections are used in the construction of an input data matrix for LSTM. The prediction result with the minimum root-mean-square error is regarded as the final prediction result. The electronic toll collection data of Taiwan are used as the fundamental data in this paper to verify the performance and effectiveness of the CHS-LSTM model. The experimental results indicate that the CHS-LSTM model can effectively improve the prediction accuracy of LSTM. Moreover, compared with well-known models generally used for predicting the intensity of traffic flow, our proposed model also shows its superiority.

Improved Subsampled Randomized Hadamard Transform for Linear SVM

Subsampled Randomized Hadamard Transform (SRHT), a popular random projection method that can efficiently project a d-dimensional data into r-dimensional space (r ≪ d) in O(dlog(d)) time, has been widely used to address the challenge of high-dimensionality in machine learning. SRHT works by rotating the input data matrix X ∈ ℝn × d by Randomized Walsh-Hadamard Transform followed with a subsequent uniform column sampling on the rotated matrix. Despite the advantages of SRHT, one limitation of SRHT is that it generates the new low-dimensional embedding without considering any specific properties of a given dataset. Therefore, this data-independent random projection method may result in inferior and unstable performance when used for a particular machine learning task, e.g., classification. To overcome this limitation, we analyze the effect of using SRHT for random projection in the context of linear SVM classification. Based on our analysis, we propose importance sampling and deterministic top-r sampling to produce effective low-dimensional embedding instead of uniform sampling SRHT. In addition, we also proposed a new supervised non-uniform sampling method. Our experimental results have demonstrated that our proposed methods can achieve higher classification accuracies than SRHT and other random projection methods on six real-life datasets.

Generalized maximum correntropy algorithm with affine projection for robust filtering under impulsive-noise environments

Combining affine projection (AP) with the generalized maximum correntropy (GMC) criterion, we propose a new family of AP-type filtering algorithms, called as APGMC, for system identification under impulsive-noise environments. By optimizing GMC of the a posterior error vector with a ℓ2-norm constraint on the filter weight vector, APGMC avoids the computation of the inversion of the input data matrix. Simulation results validate that APGMC achieves better filtering accuracy and faster convergence rate, compared to state-of-the-art algorithms.

Auto-weighted multi-view co-clustering via fast matrix factorization

Multi-view clustering is a hot research topic in machine learning and pattern recognition, however, it remains high computational complexity when clustering multi-view data sets. Although a number of approaches have been proposed to accelerate the computational efficiency, most of them do not consider the data duality between features and samples. In this paper, we propose a novel co-clustering approach termed as Fast Multi-view Bilateral K-means (FMVBKM), which can implement clustering task on row and column of the input data matrix, simultaneously. Specifically, FMVBKM applies the relaxed K-means clustering technique to multi-view data clustering. In addition, to decrease information loss in matrix factorization, we further introduce a new co-clustering method named as Fast Multi-view Matrix Tri-Factorization (FMVMTF). Extensive experimental results on six benchmark data sets show that the proposed two approaches not only have comparable clustering performance but also present the high computational efficiency, in comparison with state-of-the-art multi-view clustering methods.

Disambiguating the Context of the Concept Terms using Concept Hierarchies

Latent Semantic Analysis (LSA) makes the machine clearly conceptualize the terms of the document by learning the context in which these terms are written. However, LSA suffers from the limitation of input data matrix size in terms of number of terms and number of documents of the considered dataset. When the size of the dataset is huge, LSA becomes inefficient towards learning the correct context and thereby is unable to produce the intended concepts by the machine. To overcome this problem, Context Disambiguation (ConDis) ontology is engineered for a domain which has the capability of evolving itself based on automatic learning of concepts and relations from the ever scaling documents over the web. The concept hierarchies from general to specific concepts combined with corresponding object relations specify the particular context for a term. These object relations based concept hierarchies clearly help disambiguate the context of the concept terms in an effective manner.

A Task-Resource Mapping Algorithm for Large-Scale Batch-Mode Computational Marine Hydrodynamics Codes on Containerized Private Cloud

CPU time has long been a remaining problem for large-scale batch mode based scientific computing applications. To address this time-consuming problem, a container-based private cloud was employed, and a novel task-resource mapping algorithm was developed. Firstly, the execution features of typical batch mode codes were extracted and then computing jobs were formulated as a coarseness acyclic DAG. Secondly, to guarantee both job makespan and resource utilization, a novel task-resource mapping algorithm, along with container pre-planning and worst-case-first task placement phases, were developed. Finally, a typical Computational Marine Hydrodynamics software, Rotorysics, with a different scale of input data matrix was used as benchmark software. To manifest the effectiveness of the proposed method, a number of numerical examples were given via CloudSim and a small-medium containerized private cloud platform was adopted with three practical study cases. The computational results show that 1) compared with the traditional HPC workstation computing solution, container-based cloud solution shows significant savings in makespan by more than 6 times. 2) the new method is scalable to address bigger size batch computing problem up to a run matrix 108,.

Learning Parts-Based and Global Representation for Image Classification

Nonnegative matrix factorization (NMF), known as a famous matrix factorization technique, has been widely used in pattern recognition and computer vision. NMF represents the input data matrix as a product of two nonnegative factors. As NMF is based on the Euclidean distance, which is sensitive to noise or errors in the data, some robust NMF methods are proposed. Mainly focusing on parts-based representation, these robust NMF methods often neglect global representation of data. In fact, the global geometry information of data is more robust than the local information about the noisy data in terms of image classification. In order to effectively improve the robustness of NMF and learn part-based and global representation of the data, a novel method low-rank nonnegative factorization (LRNF) is proposed in this paper. First, we assume that the data are grossly corrupted, and the $L_{1} $ norm is used as a sparse constraint on the assumed noise matrix. Then, LRNF learns a low-rank matrix with the global representation ability. Finally, we make a nonnegative factorization of the learned low-rank matrix. We can obtain a base matrix, which preserves locality and globality properties of the data in the meantime. Extensive experiments have been conducted on nine real-world image databases to verify the performance of the proposed LRNF method by comparing with the state-of-the-art algorithms on robust dimensionality reduction.

Hyperspectral Unmixing Based on Incremental Kernel Nonnegative Matrix Factorization

Kernel nonnegative matrix factorization (KNMF) is an extension of NMF designed to capture nonlinear dependence features in data matrix through kernel functions. In KNMF, the size of the kernel matrices is closely associated with the input data matrix, of which the calculation consumes a large amount of memory and computing resource. When applied on large-scale hyperspectral data, KNMF often meets the bottleneck of memory and may cause the overflow of memory. And when dealing with dynamically acquired data, KNMF requires recomputation of the whole data set when newly acquired data arrived, which produces huge memory and computing resource requirements. To reduce the usage of memory and improve the computational efficiency when applying KNMF on large scale and dynamic hyperspectral data, we extend KNMF by introducing partition matrix theory and considering the relationships among dividing blocks. The decomposition results of hyperspectral data are derived from much smaller scale matrices containing the formerly achieved results and the newly data blocks incrementally. In this paper, we propose an incremental KNMF (IKNMF) to reduce the computing requirements for large-scale data in hyperspectral unmixing. An improved IKNMF (IIKNMF) is also proposed to further improve the abundance results of IKNMF. Experiments are conducted on both synthetic and real hyperspectral data sets. The experimental results demonstrate that the proposed methods can effectively save memory resources without degrading the unmixing performance and the proposed IIKNMF can achieve better abundance results than IKNMF and KNMF.