Scale Datasets Research Articles

HVAE: A deep generative model via hierarchical variational auto-encoder for multi-view document modeling

With the widespread development of the internet, multi-view text documents have become increasingly common, which has led to extensive research on multi-view text document modeling. As opposed to traditional single-view document modeling, which treats each document independently and learns each document as a single topic representation, the views of multi-view text documents have complicated correlation relationships that include both the global and local underlying topical information. In this study, we introduce a deep generative model for multi-view document modeling known as Hierarchical Variational Auto-Encoder (HVAE), which combines the advantages of the probability generative model for learning interpretable latent information and the deep neural network for efficient parameter inference. Specifically, a set of hierarchical topic representations is learned for each multi-view document to capture the document-level global topical information and view-level local topical information for each view. A two-level hierarchical topic inference network is investigated as the encoder network of HVAE, which is designed using an aligned variational auto-encoder, to learn the hierarchical topic representations. Subsequently, multi-view documents are generated through a two-layered generation network, considering both the view-level local and document-level global topic representations. Experiments on three real datasets of different scales for various tasks demonstrate the satisfactory results of the proposed method.

A deep learning-based fatigue crack growth rate measurement method using mobile phones

This paper proposes a fatigue crack growth rate (FCGR) measure method based on deep learning. Crack data sets of different scales are first collected by a camera, and are then used to train the faster region-based convolutional neural network (Faster R-CNN), respectively. A novel global and local dual-scale Faster R-CNN is further constructed and integrated into the mobile phone to predict the crack length during the entire loading cycle. The fatigue crack growth rate constants are finally obtained by integrating fracture mechanics. The proposed method can effectively measure the propagating small cracks during cyclic loading and extend to non-standard specimens.

TransCUNet: UNet cross fused transformer for medical image segmentation

Accurate segmentation of medical images is crucial for clinical diagnosis and evaluation. However, medical images have complex shapes, the structures of different objects are very different, and most medical datasets are small in scale, making it difficult to train effectively. These problems increase the difficulty of automatic segmentation. To further improve the segmentation performance of the model, we propose a multi-branch network model, called TransCUNet, for segmenting medical images of different modalities. The model contains three structures: cross residual fusion block (CRFB), pyramidal pooling module (PPM) and gated axial-attention, which achieve effective extraction of high-level and low-level features of images, while showing high robustness to different size segmentation objects and different scale datasets. In our experiments, we use four datasets to train, validate and test the models. The experimental results show that TransCUNet has better segmentation performance compared to the current mainstream segmentation methods, and the model has a smaller size and number of parameters, which has great potential for clinical applications.

Revisiting b-value for extended Kopili region of Northeast India and probabilistic estimation of seismic hazard attributes thereof

Kopili fault and its' surroundings of northeast India have remained witness to some devastating earthquakes. The complex tectonic feature of this region incurs a high seismicity rate relative to other regions. Very recently, an earthquake of magnitude 6.4 (MW) has jolted this region. Considering the persistently rising seismicity pattern of this region, the present study intends to integrate the frequency magnitude distribution of earthquakes with other pertinent seismic hazard parameters. This work presents the estimation of the return period and probability of non-exceedance of medium to large earthquakes for the Kopili fault and its' surroundings via the extreme value method. Two statistical methods, namely the Gutenberg-Richter(G-R) power law and Gumbel extreme value method (GEVM), are applied for the correct estimation of the b-value for the Kopili fault and its' surrounding region. The uniform and homogenous magnitude scale dataset consisting of 550 events observed within a limit of 350 km from the Kopili fault (1964 to 2022) are used in the study. The b-value computed using GR relation was found to be unsatisfactory (0.73) due to an inadequate dataset—thus Gumbel's extreme value method is used for the estimation of the b-value (0.97). Gumbel's extreme value method is used for the calculation of the probability of occurrence of earthquakes, return period, and the most probable earthquake magnitude for the study region. The most probable annual earthquake is found to be ∼ MW 4.99. The return period and probability of occurrence of design-based earthquakes are also calculated. The correlation between b-value, seismic moment, and fractal dimension for seismic hazard analysis is also investigated. The antithetical relationship between seismic moment and b-value is observed for the region while the deep-root character of the Kopili fault and the strong, unfractured character of the Shillong plateau is highlighted with the spatial distribution of fractal dimension in the study region.

Janus serumbank – jakten på biomarkører for kreft

Janus is a population-based prospective cancer biobank established in 1973. The biobank consists of residual blood serum samples from more than 300 000 men and women, who participated in different health surveys in Norway or as Red Cross blood donors during the period 1972–2004. It was established on the initiative of Professor of Pathology, Olav Torgersen (1907–78), referring to the longstanding maxim that ‘prevention is better than cure’. The purpose of the biobank was to build up a resource for studies of cancer aetiology and early detection of cancer, by measuring biochemical and immunological changes several years before the patient’s diagnosis. The Janus cohort, with comprehensive registry-based follow-up, enables longitudinal assessment of the preclinical stage in cancer patients or the latent period before the tumour has been established, and is ideal for the search of novel biomarkers of cancer. The Cancer Registry of Norway took over the cohort in 2004. In 2019 the biobank was moved into new semi-automated storage facilities and all samples were barcoded. The biobank is annually linked to the Cancer Registry for updates on new cancer cases and by the end of 2020 the number of incident cancer cases in Janus was 107 366. A continuous work in Janus has been on quality assurance of the biospecimens by investigating sample quality parameters like the effect of different pre-processing of the samples as well as storage time and temperature. We have investigated the stability of various hormones, proteins, metabolites, electrolytes and RNAs. This work has contributed to important knowledge in establishing Good Biobank Practice in Norway. In recent years we have also shown that the trace amounts of DNA in Janus is of sufficient quality for genotyping and methylation studies. Today Janus is used in a large number of national and international studies and is an active part in several international cancer consortia. The scientific output from the biobank contains a substantial proportion of high impact papers that have contributed to increased knowledge on cancer biomarkers for use in cancer control. Many of the projects have focused on investigating the association between infections and cancer, environmental exposures and cancer and early detection biomarkers. In recent years we have identified RNAs as early detection and potentially screening biomarkers of cancer. We have developed and optimized an RNA sequencing method for samples with low input RNA and produced RNA profiles of pre-clinical samples from 1631 cancer patients and 673 cancer-free controls. The sequencing data is combined with detailed cancer information from the Cancer Registry of Norway and information on environmental exposures from health surveys, in advanced biocomputational analysis. Results published on the healthy control group shows that RNA expression levels are significantly affected by age and smoking. For lung cancer the results showed dynamic changes in differentially expressed circulating RNAs specific to histology and stage. In the future we aim to utilize Janus in omics analyses and produce large scale datasets that can be shared and used in many research projects.

A Heuristic Video Recommendation Algorithm based on Similarity Computation for Multiple Features Analysis

Objective:: The short video applications have achieved great success in recent years. The number of videos being shot and uploaded to these platforms has increased greatly. In this way, mining and recommending videos for users based on their interests have become a very difficult problem in these video distribution platforms. Under this case, it becomes particularly important to design efficient video recommendation algorithms for these platforms. In order to solve the problem faced by high sparsity and large scale data sets in the field of media big data mining and recommendation, a heuristic video recommendation algorithm for multi-dimensional feature analysis and filtering is proposed. Methods:: Firstly, the video features are extracted from multiple dimensions such as user behavior and video tags. Then, the similarity analysis is carried out, and the video similarity degree is calculated by weighting, so as to obtain the similar video candidate set, and filter the similar video candidate set. After that, the videos with the highest scores are recommended to users by sorting. Finally, the video recommendation algorithm proposed in this paper is implemented by using the C language. Results:: Compared with the benchmark, the proposed video recommendation algorithm has improved the accuracy by 6.1%-136.4%, the recall rate by 19.3%-30.9%, the coverage rate by 55.6%-59.5%, the running time by 42.7%-60.4%, and the cache hit ratio by 10.9%-47.4%. Conclusion:: The proposed algorithm can effectively and greatly improve the accuracy, recall rate, coverage rate, running time and cache hit ratio.

Rookognise: Acoustic detection and identification of individual rooks in field recordings using multi-task neural networks

Individual-level monitoring is essential in many behavioural and bioacoustics studies. Collecting and annotating those data is costly in terms of human effort, but necessary prior to conducting analysis. In particular, many studies on bird vocalisations also involve manipulating the animals or human presence during observations, which may bias vocal production. Autonomous recording units can be used to collect large amounts of data without human supervision, largely removing those sources of bias. Deep learning can further facilitate the annotation of large amounts of data, for instance to detect vocalisations, identify the species, or recognise the vocalisation types in recordings. Acoustic individual identification, however, has so far largely remained limited to a single vocalisation type for a given species. This has limited the use of those techniques for automated data collection on raw recordings, where many individuals can produce vocalisations of varying complexity, potentially overlapping one another, with the additional presence of unknown and varying background noise. This paper aims at bridging this gap by developing a system to identify individual animals in those difficult conditions. Our system leverages a combination of multi-scale information integration, multi-channel audio and multi-task learning. The multi-task learning paradigm is based the overall task into four sub-tasks, three of which are auxiliary tasks: the detection and segmentation of vocalisations against other noises, the classification of individuals vocalising at any point during a sample, and the sexing of detected vocalisations. The fourth task is the overall identification of individuals. To test our approach, we recorded a captive group of rooks, a Eurasian social corvid with a diverse vocal repertoire. We used a multi-microphone array and collected a large scale dataset of time-stamped and identified vocalisations recorded, and found the system to work reliably for the defined tasks. To our knowledge, the system is the first to acoustically identify individuals regardless of the vocalisation produced. Our system can readily assist data collection and individual monitoring of groups of animals in both outdoor and indoor settings, even across long periods of time, and regardless of a species’ vocal complexity. All data and code used in this article is available online.

Digital Communication and Multimodal Features: Functioning of Emoji in Interpersonal Communication

Technical advances and digital means of communication have led to the development of digital semiotics which is characterised by its multimodality and abounds in paralinguistic elements such as emojis, emoticons, memes, etc. These extralinguistic elements serve as a compensatory mechanism in the new communication means. The increasing interest of users in various iconic signs and symbols generates the research interest in different fields of knowledge. The study aims to consider cognitive, semiotic and psycholinguistic features of emojis in interpersonal communication through analysing their functions in text messages and in social network messages. An attempt to reveal their persuasive mechanism is made. The research is based on a large scale dataset comprised of the private text messages as well as public posts on social networks which include verbal and nonverbal / iconic elements. The research data presents a multilingual bank of English, Russian and French sources. The research methods include context analysis, linguistic and pragmatic analysis and content analysis. The findings show that emojis in private interpersonal communication perform a number of functions, namely nonverbal, emotive, pragmatic, punctuation, substitutional, decorative and rhetorical functions. These iconic symbols incorporated in the interpersonal digital communication present a compensatory mechanism and the means of persuasion of a message addressee / recipient. The combination of verbal and iconic elements triggers a double focusing mechanism, and the perception is shaped by all cognitive mechanisms including rational and emotional, unconscious components.

Optimized lightweight CA-transformer: Using transformer for fine-grained visual categorization

As a rapidly developing research direction in computer vision (CV), related algorithms such as image classification and object detection have achieved inevitable research progress. Improving the accuracy and efficiency of algorithms for fine-grained identification of plant diseases and birds in agriculture is essential to the dynamic monitoring of agricultural environments. In this study, based on the computer vision detection and classification algorithm, combined with the architecture and ideas of the CNN model, the mainstream Transformer model was optimized, and then the CA-Transformer (Transformer Combined with Channel Attention) model was proposed to improve the ability to identify and classify critical areas. The main work is as follows: (1) The C-Attention mechanism is proposed to strengthen the feature information extraction within the patch and the communication between feature information so that the entire network can be fully attentive while reducing the computational overhead; (2) The weight-sharing method is proposed to transfer parameters between different layers, improve the reusability of model data, and at the same time increase the knowledge distillation link to reduce problems such as excessive parameters and overfitting; (3) Token Labeling is proposed to generate score labels according to the position of each Token, and the total loss function of this study is proposed according to the CA-Transformer model structure. The performance of the CA-Transformer model proposed in this study is compared with the current mainstream models on datasets of different scales, and ablation experiments are performed. The results show that the accuracy and mIoU of the CA-Transformer proposed in this study reach 82.89% and 53.17MS, respectively, and have good transfer learning ability, indicating that the model has good performance in fine-grained visual categorization tasks and can be used in ecological information. In the context of more diverse ecological information, this study can provide reference and inspiration for the practical application of information.

Network analysis approach to Likert-style surveys

Likert-style surveys are a widely used research instrument to assess respondents' preferences, beliefs, or experiences. In this paper, we propose and demonstrate how network analysis (NA) can be employed to model and evaluate the interconnectedness of items in Likert-style surveys. We explore the advantages of this approach by applying the methodology to the aspects of student experience scale datasets and compare the results to the principal component analysis. We successfully create a meaningful network based on survey item response similarity and use modular analysis of the network to identify larger themes built from the connections of particular aspects. The modular NA of the network of survey items identifies important themes that highlight differences in students' overall experiences. Our network analysis for Likert-style surveys methodology is widely applicable and provides a new way to investigate phenomena assessed by Likert-style surveys.

Kentsel Ekolojik Koridor Modeli: Malatya Örneği

Günümüzde kentsel alanlardaki genişleme ekosistem süreçleri üzerinde baskı oluşturmaktadır. Bu baskının saptanması ve kontrol edilmesi için ekoloji tabanlı modeller geliştirilmektedir. Bu noktadan hareketle çalışma, Malatya kenti örneğinde enerji akışını ve doğal tür akışını düzenleyecek, kentsel ısı adası etkisini azaltacak ve habitat bağlanabilirliğini artıracak arazi kullanım politikalarının geliştirilmesine katkı sağlayacak ve kentlere rehberlik edecek bir kentsel ekolojik koridor modeli geliştirmeyi amaçlamaktadır. Belirtilen amaca ulaşmak için farklı ölçek ve zamansallıktaki uydu görüntüleri, iklim verileri, topografik veriler ve demografik veri setleri farklı uzaktan algılama ve coğrafi bilgi sistemleri tekniklerini kullanılarak ilişkilendirilmiş ve kentsel ekolojik koridor modeli oluşturulmuştur.
 Malatya’da 1985-2021 yılları arasında kentsel büyümenin en çok etkilediği ekosistemin tarımsal ekosistem olduğu saptanmıştır. 2050 yılında kentsel nüfus 888.784 kişiye ulaşacağı öngörüsüne dayanarak kentsel alanların 173 km2’ye ulaşacağı tespit edilmiştir. 1985-2021 döneminde olduğu gibi 2021-2050 yılları arasında tarımsal ekosistemler üzerindeki baskı artarak devam ederken ekosistemler arasındaki bağlantı giderek zayıflayacaktır. 
 Ekolojik koridorlar, tarım ve bozkır ekosistemine ait ekolojik açıdan en duyarlı ve en büyük bütünleşik lekeler arasında kentsel yayılmayı yönlendirerek, enerji akışı, doğal tür akışı, kentsel ısı adası etkisinin azaltılması, yeraltı su rejiminin düzenlenmesi ve habitat bağlantılılığı gibi çeşitli ekolojik işlevler üstlenecektir. Malatya’da tarımsal ekosistemden 13 ve bozkır ekosisteminden 5 adet ekolojik açıdan duyarlı leke belirlenmiş ve ekosistem lekelerini birbirlerine bağlayan 694 km2 alana sahip ekolojik koridor önerilmiştir.

Computer Graphic and Photographic Image Classification Using Transfer Learning Approach

Advancement in information and communication technology has led to tremendous development in graphics techniques. Evolving multimedia tools are used to generate high quality Computer Graphics (CG) images. These images have wide applications in domains like video gaming, augmented reality, and virtual reality and many other. Computer graphic images are also used illegally in criminal activities. This article proposes an effective transfer learning approach to classify CG and Photographic (PG) images available in small scale dataset. Initially, pre-trained models such as AlexNet, GoogleNet, ResNet50, VGG-18 and SqueezeNet were modified and fine-tuned appropriately. Based on the validation accuracy, SqueezeNet was adapted as learning model for extracting deep features for classification. To evaluate the performance of squeezeNet, Columbia dataset and Photo realistic dataset were used. Finally, the performance of the proposed model was compared with state-of-the- art transfer learning approaches to prove its efficacy. Accuracy of 93.75% was attained using SqueezeNet for the folding ratio 80:20 when the input data is augmented.

Building a public service infrastructure: Lessons from the Northern Ireland Public Data Panel pilot.

There is an increasing awareness of the potential for creating benefit through the production, use and linkage of data sets to enhance evidence for public policymaking, service provision and the general public good. By linking large scale datasets across thematic areas, researchers are able to identify trends and areas of concern across a population, and answer questions not previously possible. These data are created in the delivery of national public services and their use in research is secondary. As each data point is an experience or event in the life of a person, it is vital to obtain the views and opinions of publics if we are to maintain a social license to access and use this data. Northern Ireland is currently the only of the three devolved jurisdictions of the UK (Scotland, Wales and Northern Ireland) that does not have a public panel to consider public data questions, including the secondary use of data, data research questions of public importance, data legislation, and other issues. While our data and research infrastructures develop, it is crucial that our public engagement infrastructure and processes do as well. Only with both of these in place will the data culture in Northern Ireland shift from a closed, conservative one to a landscape that acknowledges the power of evidence-led knowledge creation, understands the key importance that publics play in providing and maintaining acceptability of data use, and promotes the use of data in research and development, decision and policy-making, and public services. This paper will explore the rationale for and findings from a pilot to set up a public data panel unique in its approach, for the people of Northern Ireland, outlining success and challenges, and potential pathways forward.

Research on Side-Channel Analysis Based on Deep Learning with Different Sample Data

With the in-depth integration of deep learning and side-channel analysis (SCA) technology, the security threats faced by embedded devices based on the Internet of Things (IoT) have become increasingly prominent. By building a neural network model as a discriminator, the correlation between the side information leaked by the cryptographic device, the key of the cryptographic algorithm, and other sensitive data can be explored. Then, the security of cryptographic products can be evaluated and analyzed. For the AES-128 cryptographic algorithm, combined with the CW308T-STM32F3 demo board on the ChipWhisperer experimental platform, a Correlation Power Analysis (CPA) is performed using the four most common deep learning methods: the multilayer perceptron (MLP), the convolutional neural network (CNN), the recurrent neural network (RNN), and the long short-term memory network (LSTM) model. The performance of each model is analyzed in turn when the samples are small data sets, sufficient data sets, and data sets of different scales. Finally, each model is comprehensively evaluated by indicators such as classifier accuracy, network loss, training time, and rank of side-channel attacks. The experimental results show that the convolutional neural network CNN classifier has higher accuracy, lower loss, better robustness, stronger generalization ability, and shorter training time. The rank value is 2, that is, only two traces can recover the correct key byte information. The comprehensive performance effect is better.

A novel metric learning framework by exploiting global and local information

Distance metric learning plays a significant role in improving the generalization of algorithms related to distance metrics. In this paper, we first propose a generalized Mahalanobis metric learning framework (called GLML) to make use of the prior knowledge from the samples including global and local information, the main goal of which is to enlarge the distance of dissimilar sample pairs and shrink the distance of similar sample pairs simultaneously. Specifically, the metric learning problem is guided by a discriminative regularization through incorporating the pair-wise and class-wise information. Moreover, an effective alternating iterative algorithm is developed to optimize the proposed GLML, and the convergence of the algorithm is demonstrated theoretically. Then a boosting algorithm of GLML (BGLML) is proposed, where the low-rank basis learning is jointly optimized with the metric to better uncover the data structure and mitigate the computational burden. Following that, numerical experiments are carried out on binary classification and multi-classification problems. With different assessment criteria, experiment results from different scale datasets confirm that the proposed methods either improve generalization or have comparable results compared with the state-of-the-art algorithms.

COVID-19 chest X-ray detection through blending ensemble of CNN snapshots

The novel COVID-19 pandemic, has effectively turned out to be one of the deadliest events in modern history, with unprecedented loss of human life, major economic and financial setbacks and has set the entire world back quite a few decades. However, detection of the COVID-19 virus has become increasingly difficult due to the mutating nature of the virus, and the rise in asymptomatic cases. To counteract this and contribute to the research efforts for a more accurate screening of COVID-19, we have planned this work. Here, we have proposed an ensemble methodology for deep learning models to solve the task of COVID-19 detection from chest X-rays (CXRs) to assist Computer-Aided Detection (CADe) for medical practitioners. We leverage the strategy of transfer learning for Convolutional Neural Networks (CNNs), widely adopted in recent literature, and further propose an efficient ensemble network for their combination. The DenseNet-201 architecture has been trained only once to generate multiple snapshots, offering diverse information about the extracted features from CXRs. We follow the strategy of decision-level fusion to combine the decision scores using the blending algorithm through a Random Forest (RF) meta-learner. Experimental results confirm the efficacy of the proposed ensemble method, as shown through impressive results upon two open access COVID-19 CXR datasets — the largest COVID-X dataset, as well as a smaller scale dataset. On the large COVID-X dataset, the proposed model has achieved an accuracy score of 94.55% and on the smaller dataset by Chowdhury et al., the proposed model has achieved a 98.13% accuracy score.

Spatial extrapolation of stream thermal peaks using heterogeneous time series at a national scale

Abstract. Spatial reconstruction of stream temperature is relevant to water quality standards and fisheries management, yet large, regional scale datasets are rare because data are decentralized and inharmonious. This data discordance is a major limitation for understanding thermal regimes of riverine ecosystems. To overcome this barrier, we first aggregated one of the largest stream temperature databases on record with data from 1700 individual stations over 9 years from 2009–2017 (n = 45 000 000 hourly measurements) across France (area = 552 000 km2). For each station, we calculated a simple, ecologically relevant metric – the thermal peak – that captures the magnitude of summer thermal maximums. We then used three statistical models to extrapolate the thermal peak to nearly every stream reach in France and Corsica (n = 105 800) and compared relative model performances with an air temperature metric. In general, the hottest thermal peaks were found along major rivers, whereas the coldest thermal peaks were found along small rivers with forested riparian zones and strong groundwater inputs and were located in mountainous regions. Several key predictors of the thermal peak emerged, including drainage area, mean summer air temperature, minimum monthly specific discharge, and vegetation cover in the riparian zone. Despite differing predictor importance across model structures, we observed strong concordance among models in their spatial distributions of the thermal peak, suggesting its robustness as a useful metric at the regional scale. Finally, air temperature was found to be a poor proxy for the stream temperature thermal peak across nearly all stations and reaches, highlighting the growing need to measure and account for stream temperature in regional ecological studies.

Traffic Accident Detection via Self-Supervised Consistency Learning in Driving Scenarios

With the rapid progress of autonomous driving and advanced driver assistance systems, there are growing efforts to promote their safety in natural driving scenarios, especially for the detection of the traffic accidents. However, because of the dynamic camera motion and complex scene in driving situations, traffic accident detection is still challenging. In this work, we aim to give the ability of Traffic Accident Detection for driving systems by proposing a Self-Supervised Consistency learning framework, termed as SSC-TAD, that involves the appearance, motion, and context consistency learning. The key formulation is to find the inconsistency of video frames, object locations and the spatial relation structure of scene temporally between different frames captured by the dashcam videos. Within this field, different from the previous works which concentrate on predicting the future object locations or frames, we further focus on predicting the visual scene context in driving scenarios and detecting the traffic accident by considering the temporal frame consistency, temporal object location consistency, and the spatial-temporal relation consistency of road participants. In this work, this formulation is fulfilled by a collaborative multi-task consistency learning network and the visual scene context feature is represented by a graph convolution network. The superiority to the state-of-the-art is verified by exhaustive evaluations on two large scale datasets, i.e., the AnAn Accident Detection (A3D) dataset and DADA-2000 dataset collected recently.

Batch Active Learning with Graph Neural Networks via Multi-Agent Deep Reinforcement Learning

Graph neural networks (GNNs) have achieved tremendous success in many graph learning tasks such as node classification, graph classification and link prediction. For the classification task, GNNs' performance often highly depends on the number of labeled nodes and thus could be significantly hampered due to the expensive annotation cost. The sparse literature on active learning for GNNs has primarily focused on selecting only one sample each iteration, which becomes inefficient for large scale datasets. In this paper, we study the batch active learning setting for GNNs where the learning agent can acquire labels of multiple samples at each time. We formulate batch active learning as a cooperative multi-agent reinforcement learning problem and present a novel reinforced batch-mode active learning framework BiGeNe. To avoid the combinatorial explosion of the joint action space, we introduce a value decomposition method that factorizes the total Q-value into the average of individual Q-values. Moreover, we propose a novel multi-agent Q-network consisting of a graph convolutional network (GCN) component and a gated recurrent unit (GRU) component. The GCN component takes both the informativeness and inter-dependences between nodes into account and the GRU component enables the agent to consider interactions between selected nodes in the same batch. Experimental results on multiple public datasets demonstrate the effectiveness and efficiency of our proposed method.

Online Missing Value Imputation and Change Point Detection with the Gaussian Copula

Missing value imputation is crucial for real-world data science workflows. Imputation is harder in the online setting, as it requires the imputation method itself to be able to evolve over time. For practical applications, imputation algorithms should produce imputations that match the true data distribution, handle data of mixed types, including ordinal, boolean, and continuous variables, and scale to large datasets. In this work we develop a new online imputation algorithm for mixed data using the Gaussian copula. The online Gaussian copula model produces meets all the desiderata: its imputations match the data distribution even for mixed data, improve over its offline counterpart on the accuracy when the streaming data has a changing distribution, and on the speed (up to an order of magnitude) especially on large scale datasets. By fitting the copula model to online data, we also provide a new method to detect change points in the multivariate dependence structure for mixed data with missing values. Experimental results on synthetic and real world data validate the performance of the proposed methods.