Large-scale Clustering Research Articles

The role of scatter and satellites in shaping the large-scale clustering of X-ray AGN as a function of host galaxy stellar mass

ABSTRACT The co-evolution between central supermassive black holes (BHs), their host galaxies, and dark matter haloes is still a matter of intense debate. Present theoretical models suffer from large uncertainties and degeneracies, for example, between the fraction of accreting sources and their characteristic accretion rate. In recent work, we showed that active galactic nuclei (AGNs) clustering represents a powerful tool to break degeneracies when analysed in terms of mean BH mass, and that AGN bias at fixed stellar mass is largely independent of most of the input parameters, such as the AGN duty cycle and the mean scaling between BH mass and host galaxy stellar mass. In this paper, we take advantage of our improved semi-empirical methodology and recent clustering data derived from large AGN samples at z ∼ 1.2, demonstrate that the AGN bias as a function of host galaxy stellar mass is a crucial diagnostic of the BH–galaxy connection, and is highly dependent on the scatter around the BH mass–galaxy mass scaling relation and on the relative fraction of satellite and central active BHs. Current data at z ∼ 1.2 favour relatively high values of AGN in satellites, pointing to a major role of disc instabilities in triggering AGN, unless a high minimum host halo mass is assumed. The data are not decisive on the magnitude/covariance of the BH–galaxy scatter at z ∼ 1.2 and intermediate host masses $M_\mathrm{star} \lesssim 10^{11} \, \mathrm{M}_\mathrm{star}$. However, future surveys like Euclid/LSST will be pivotal in shedding light on the BH–galaxy co-evolution.

Large-scale clustering amongst Fermi blazars; evidence for axis alignments?

ABSTRACT We find evidence for large-scale clustering amongst Fermi-selected BL Lac objects but not amongst Fermi-selected flat spectrum radio quasars (FSRQs). Using two-point correlation functions, we have investigated the clustering properties of different classes of objects from the Fermi–LAT(Large Area Telescope) 4FGL catalogue. We wanted to test the idea based on optical polarization observations that there might be large volumes of space in which AGN axes are aligned. To do this, we needed a clean sample of blazars as these are objects with their jet axes pointing towards the observer and Fermi sources provide such a sample. We find that high latitude Fermi sources taken as a whole show a significant clustering signal on scales up to 30°. To investigate if all blazars behave in the same way we used he machine learning classifications from the literature, which are based only on gamma-ray information, to separate BL Lac-like objects from FSRQ-like objects. A possible explanation for the clustering signal we find amongst the BL Lac-like objects is that there are indeed large volumes of space in which AGN axes are aligned. This signal might be washed out in FSRQs since they occupy a much larger volume of space. Thus, our results support the idea that large scale polarization alignments could originate from coherent alignments of AGN axes. We speculate that these axis alignments may be related to the well-known intrinsic alignments of galaxy optical position angles.

Efficient Clustering of Short Messages into General Domains

The ever increasing activity in social networks is mainly manifested by a growing stream of status updating or microblogging. The massive stream of updates emphasizes the need for accurate and efficient clustering of short messages on a large scale. Applying traditional clustering techniques is both inaccurate and inefficient due to sparseness. This paper presents an accurate and efficient algorithm for clustering Twitter tweets. We break the clustering task into two distinctive tasks/stages: (1) batch clustering of user annotated data, and (2) online clustering of a stream of tweets. In the first stage we rely on the habit of `tagging', common in social media streams (e.g. hashtags), thus the algorithm can bootstrap on the tags for clustering of a large pool of hashtagged tweets. The stable clusters achieved in the first stage lend themselves for online clustering of a stream of (mostly) tagless messages.We evaluate our results against gold-standard classification and validate the results by employing multiple clusteringevaluation measures (information theoretic, paired, F and greedy). We compare our algorithm to a number of otherclustering algorithms and various types of feature sets. Results show that the algorithm presented is both accurate andefficient and can be easily used for large scale clustering of sparse messages as the heavy lifting is achieved ona sublinear number of documents.

Hierarchical coordinated anti-jamming channel access in clustering networks: a multi-leader multi-follower Stackelberg game approach

This paper mainly investigates the multi-user coordinated anti-jamming problem in clustering communication networks. In such kinds of networks, there exist multiple clusters and multiple users who communicate with their receivers simultaneously. Besides, a malicious jammer persistently attacks channels with wide-band and dynamic changing jamming signals. To cope with these challenges brought by the large-scale clustering network and the dynamic wide-band jamming, a hierarchical coordinated anti-jamming approach is proposed, and a multi-leader multi-follower Stackelberg game is introduced to model the anti-jamming problem. In detail, cluster heads act as leaders, and select available frequency bands to avoid jamming attacks, while users in each cluster act as followers and select corresponding channels distributedly and independently. Moreover, it is proved that there exist multiple Stackelberg equilibriums (SEs) in the proposed game. To obtain SEs, a hierarchical coordinated anti-jamming channel access (HCACA) algorithm is designed. Simulation results illustrate that the proposed approach is effective to cope with the dynamic wide-band jamming attacks. Furthermore, it is also depicted that the proposed approach outperforms the distributed anti-jamming comparative approach in terms of convergence speed.

Large-Scale Data Clustering Using Manifold-Regularized Ensemble of Posterior in GAN

Large-Scale Data Clustering Using Manifold-Regularized Ensemble of Posterior in GAN

Primordial non-Gaussianity from the completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey – I: Catalogue preparation and systematic mitigation

ABSTRACT We investigate the large-scale clustering of the final spectroscopic sample of quasars from the recently completed extended Baryon Oscillation Spectroscopic Survey (eBOSS). The sample contains 343 708 objects in the redshift range 0.8 < z < 2.2 and 72 667 objects with redshifts 2.2 < z < 3.5, covering an effective area of $4699\, {\rm deg}^{2}$. We develop a neural network-based approach to mitigate spurious fluctuations in the density field caused by spatial variations in the quality of the imaging data used to select targets for follow-up spectroscopy. Simulations are used with the same angular and radial distributions as the real data to estimate covariance matrices, perform error analyses, and assess residual systematic uncertainties. We measure the mean density contrast and cross-correlations of the eBOSS quasars against maps of potential sources of imaging systematics to address algorithm effectiveness, finding that the neural network-based approach outperforms standard linear regression. Stellar density is one of the most important sources of spurious fluctuations, and a new template constructed using data from the Gaia spacecraft provides the best match to the observed quasar clustering. The end-product from this work is a new value-added quasar catalogue with the improved weights to correct for non-linear imaging systematic effects, which will be made public. Our quasar catalogue is used to measure the local-type primordial non-Gaussianity in a companion paper.

Improvement of K-Means Algorithm for Accelerated Big Data Clustering

With the rapid development of the computer level, especially in recent years, “Internet +,” cloud platforms, etc. have been used in various industries, and various types of data have grown in large quantities. Behind these large amounts of data often contain very rich information, relying on traditional data retrieval and analysis methods, and data management models can no longer meet our needs for data acquisition and management. Therefore, data mining technology has become one of the solutions to how to quickly obtain useful information in today's society. Effectively processing large-scale data clustering is one of the important research directions in data mining. The k-means algorithm is the simplest and most basic method in processing large-scale data clustering. The k-means algorithm has the advantages of simple operation, fast speed, and good scalability in processing large data, but it also often exposes fatal defects in data processing. In view of some defects exposed by the traditional k-means algorithm, this paper mainly improves and analyzes from two aspects.

Reconstruction of reionization history through dispersion measurements of fast radio bursts

In this paper, we study the evolution of the ionization fraction x e(z) during the epoch of reionization by using the dispersion measurements (DMs) of fast radio bursts (FRBs). Different from the previous studies, here we turn to consider the large-scale clustering information of observed DMs of FRB catalog, which only needs the rough redshift distribution, instead of the exact redshift information of each FRB. Firstly, we consider the instantaneous “tanh” model for x e(z) and find that including the auto-correlation information of the mock catalog, about 104 FRBs with the intrinsic DM scatter of 100 pc/cm3 spanning 20% of all sky, could significantly improve the constraint on the width Δz of the model, when comparing with that from the CMB data alone. The evolution shape of the ionization fraction will be tightly narrowed, namely the duration of the epoch of reionization has been shrunk, z dur < 2.24 (95% C.L.). Furthermore, we also use another redshift-asymmetric reionization model and obtain that the FRB mock catalog could measure the ionization fraction at z = 6 precisely with the 1σ error Δ x e (z = 6) = 0.012, which means that the large-scale clustering information of observed DMs of FRB catalog is very sensitive to the ionization fraction of the end of reionization epoch. We conclude that the observation of high-redshift FRBs could be a complementary probe to study the reionization history in the future.

A flexible modelling of galaxy assembly bias

ABSTRACT We use the illustris TNG300 magneto-hydrodynamic simulation, the sage semi-analytical model, and the subhalo abundance matching technique (SHAM) to examine the diversity in predictions for galaxy assembly bias (i.e. the difference in the large-scale clustering of galaxies at a fixed halo mass due to correlations with the assembly history and other properties of host haloes). We consider samples of galaxies selected according to their stellar mass or star formation rate at various redshifts. We find that all models predict an assembly bias signal of different magnitude, redshift evolution, and dependence with selection criteria and number density. To model these non-trivial dependences, we propose an extension to the standard SHAM technique so it can include arbitrary amounts of assembly bias. We do this by preferentially selecting subhaloes with the same internal property but different individual large-scale bias. We find that with this model, we can successfully reproduce the galaxy assembly bias signal in either sage or the tng, for all redshifts and galaxy number densities. We anticipate that this model can be used to constrain the level of assembly bias in observations and aid in the creation of more realistic mock galaxy catalogues.

Strongly enhanced quadrupole deformation in a class of N≈ Z nuclei driven by large-scale clustering? * *Partly supported by the U.S. Department of Energy, Office of Science, Office of Nuclear Physics (DE-AC02-06CH11357)

The formation of large size clusters, and/or their relative motion as a possible excitation mode, are suggested to be closely related to the origin of deformation in specific cases, namely the case of two doubly-magic clusters or two clusters with nearby characterization. New lifetime data in N = Z 76Sr and 80Zr leading to large B(E2) values are reproduced consistently and well within this approach, along with data for a few neighboring N Z nuclei. These results are compared to previous studies of 32S and 20Ne and all of them support the ideas of the large-scale cluster approach.

From starburst to quiescence: post-starburst galaxies and their large-scale clustering over cosmic time

ABSTRACT We present the first study of the large-scale clustering of post-starburst (PSB) galaxies in the high-redshift Universe (0.5 &amp;lt; z &amp;lt; 3.0). We select ∼4000 PSB galaxies photometrically, the largest high-redshift sample of this kind, from two deep large-scale near-infrared surveys: the UKIDSS Ultra Deep Survey Data Release 11 and the Cosmic Evolution Survey. Using angular cross-correlation techniques, we estimate the halo masses for this large sample of PSB galaxies and compare them with quiescent and star-forming galaxies selected in the same fields. We find that low-mass, low-redshift (0.5 &amp;lt; z &amp;lt; 1.0) PSB galaxies preferentially reside in very high mass dark matter haloes (Mhalo &amp;gt; 1014 M⊙), suggesting that they are likely to be infalling satellite galaxies in cluster-like environments. High-mass PSB galaxies are more weakly clustered at low redshifts, but they reside in higher mass haloes with increasing look-back time, suggesting strong redshift-dependent halo downsizing. These key results are consistent with previous results, suggesting that two main channels are responsible for the rapid quenching of galaxies. While high-redshift (z &amp;gt; 1) galaxies appear to be quenched by secular feedback mechanisms, processes associated with dense environments are likely to be the key driver of rapid quenching in the low-redshift Universe (z &amp;lt; 1). Finally, we show that the clustering of photometrically selected PSBs is consistent with them being direct descendants of highly dust-enshrouded submillimetre galaxies, providing tantalizing evidence for the oft-speculated evolutionary pathway from starburst to quiescence.

A matheuristic for large-scale capacitated clustering

Clustering addresses the problem of assigning similar objects to groups. Since the size of the clusters is often constrained in practical clustering applications, various capacitated clustering problems have received increasing attention. We consider here the capacitated p-median problem (CPMP) in which p objects are selected as cluster centers (medians) such that the total distance from these medians to their assigned objects is minimized. Each object is associated with a weight, and the total weight in each cluster must not exceed a given capacity. Numerous exact and heuristic solution approaches have been proposed for the CPMP. The state-of-the-art approach performs well for instances with up to 5,000 objects but becomes computationally expensive for instances with a much larger number of objects. We propose a matheuristic with new problem decomposition strategies that can deal with instances comprising up to 500,000 objects. In a computational experiment, the proposed matheuristic consistently outperformed the state-of-the-art approach on medium- and large-scale instances while having similar performance for small-scale instances. As an extension, we show that our matheuristic can be applied to related capacitated clustering problems, such as the capacitated centered clustering problem (CCCP). For several test instances of the CCCP, our matheuristic found new best-known solutions.

Redondoviridae: High Prevalence and Possibly Chronic Shedding in Human Respiratory Tract, But No Zoonotic Transmission.

Redondoviridae is a recently discovered DNA virus family consisting of two species, vientovirus and brisavirus. Here we used PCR amplification and sequencing to characterize redondoviruses in nasal/throat swabs collected longitudinally from a cohort of 58 individuals working with animals in Vietnam. We additionally analyzed samples from animals to which redondovirus DNA-positive participants were exposed. Redondoviruses were detected in approximately 60% of study participants, including 33% (30/91) of samples collected during episodes of acute respiratory disease and in 50% (29/58) of baseline samples (with no respiratory symptoms). Vientovirus (73%; 24/33) was detected more frequently in samples than brisaviruses (27%; 9/33). In the 23 participants with at least 2 redondovirus-positive samples among their longitudinal samples, 10 (43.5%) had identical redondovirus replication-gene sequences detected (sampling duration: 35–132 days). We found no identical redondovirus replication genes in samples from different participants, and no redondoviruses were detected in 53 pooled nasal/throat swabs collected from domestic animals. Phylogenetic analysis described no large-scale geographical clustering between viruses from Vietnam, the US, Spain, and China, indicating that redondoviruses are highly genetically diverse and have a wide geographical distribution. Collectively, our study provides novel insights into the Redondoviridae family in humans, describing a high prevalence, potentially associated with chronic shedding in the respiratory tract with lack of evidence of zoonotic transmission from close animal contacts. The tropism and potential pathogenicity of this viral family remain to be determined.

Reconstruction of baryon fraction in intergalactic medium through dispersion measurements of fast radio bursts

ABSTRACT Fast radio bursts (FRBs) probe the total column density of free electrons in the intergalactic medium (IGM) along the path of propagation through the dispersion measures (DMs) that depend on the baryon mass fraction in the IGM, i.e. fIGM. In this letter, we investigate the large-scale clustering information of DMs to study the evolution of fIGM. When combining with the Planck 2018 measurements, we could give tight constraints on the evolution of fIGM(z) from about 104 FRBs with the intrinsic DM scatter of $30(1+z) \rm pc\, cm^{-3}$ spanning 80 per cent of the sky and redshift range z = 0–3. First, we consider the Taylor expansion of fIGM(z) up to second order, and find that the mean relative standard deviation σ(fIGM) ≡ 〈σ[fIGM(z)]/fIGM(z)〉 is about 6.7 per cent. In order to alleviate the dependence on fiducial model, we also adopt non-parametric methods in this work, the local principle component analysis. We obtain the consistent, but weaker constraints on the evolution of fIGM(z), namely the mean relative standard deviation σ(fIGM) is 21.4 per cent. With the forthcoming surveys, this could be a complimentary method to investigate the baryon mass fraction in the IGM.

A modified unconscious search algorithm for data clustering

Clustering is a widely used data mining technique with a diverse set of applications. Since clustering is an NP-hard problem, finding high-quality solutions for large-scale clustering problems can be an arduous and computationally expensive task. Therefore, many metaheuristics are utilized to solve these problems efficiently. In this paper, a modified unconscious search (US) and its k-means hybrid for data clustering are proposed with two main modifications: (1) generating initial population by combining solutions of k-means and random solutions, (2) replacing the usual local search step of the original US by an existing Heuristic Search method. Modified US is tested on the seven following well-known benchmarks from the UCI machine learning directory: Iris, Wine, Glass, Cancer, Vowel, CMC, and Ecoli. The results are then compared against metaheuristics, such as genetic algorithm, particle swarm optimization (PSO), black hole algorithm, hybrid of PSO k-means, and accelerated chaotic PSO. The results of experiments show that, on average, the quality of best solutions obtained by the proposed methods on all seven datasets is 0.176% better than the quality of the other six algorithms applied for experimentations.

A large-scale clustering and 3D trajectory optimization approach for UAV swarms

A large-scale clustering and 3D trajectory optimization approach for UAV swarms

Large-scale clustering of light small particles in developed turbulence

Small solid particles, droplets, and bubbles can form clusters in turbulent flows by the action of coherent vortices. This phenomenon, sometimes called the preferential concentration, was often thought to be most conspicuous when the velocity relaxation time τp of particles is comparable with the Kolmogorov time τη. However, since high-Reynolds number turbulence consists of coherent eddies with different timescales, particles can form clusters even when τp≫τη. We demonstrate, by direct numerical simulations, that light particles with different τp values form clusters around axes of coherent vortices with different sizes in developed turbulence.

A comparative study of many-objective optimizers on large-scale many-objective software clustering problems

Over the past 2 decades, several multi-objective optimizers (MOOs) have been proposed to address the different aspects of multi-objective optimization problems (MOPs). Unfortunately, it has been observed that many of MOOs experiences performance degradation when applied over MOPs having a large number of decision variables and objective functions. Specially, the performance of MOOs rapidly decreases when the number of decision variables and objective functions increases by more than a hundred and three, respectively. To address the challenges caused by such special case of MOPs, some large-scale multi-objective optimization optimizers (L-MuOOs) and large-scale many-objective optimization optimizers (L-MaOOs) have been developed in the literature. Even after vast development in the direction of L-MuOOs and L-MaOOs, the supremacy of these optimizers has not been tested on real-world optimization problems containing a large number of decision variables and objectives such as large-scale many-objective software clustering problems (L-MaSCPs). In this study, the performance of nine L-MuOOs and L-MaOOs (i.e., S3-CMA-ES, LMOSCO, LSMOF, LMEA, IDMOPSO, ADC-MaOO, NSGA-III, H-RVEA, and DREA) is evaluated and compared over five L-MaSCPs in terms of IGD, Hypervolume, and MQ metrics. The experimentation results show that the S3-CMA-ES and LMOSCO perform better compared to the LSMOF, LMEA, IDMOPSO, ADC-MaOO, NSGA-III, H-RVEA, and DREA in most of the cases. The LSMOF, LMEA, IDMOPSO, ADC-MaOO, NSGA-III, and DREA, are the average performer, and H-RVEA is the worst performer.

A Novel Unsupervised Classification Method for Sandy Land Using Fully Polarimetric SAR Data

Classification based on polarimetric synthetic aperture radar (PolSAR) images is an emerging technology, and recent years have seen the introduction of various classification methods that have been proven to be effective to identify typical features of many terrain types. Among the many regions of the study, the Hunshandake Sandy Land in Inner Mongolia, China stands out for its vast area of sandy land, variety of ground objects, and intricate structure, with more irregular characteristics than conventional land cover. Accounting for the particular surface features of the Hunshandake Sandy Land, an unsupervised classification method based on new decomposition and large-scale spectral clustering with superpixels (ND-LSC) is proposed in this study. Firstly, the polarization scattering parameters are extracted through a new decomposition, rather than other decomposition approaches, which gives rise to more accurate feature vector estimate. Secondly, a large-scale spectral clustering is applied as appropriate to meet the massive land and complex terrain. More specifically, this involves a beginning sub-step of superpixels generation via the Adaptive Simple Linear Iterative Clustering (ASLIC) algorithm when the feature vector combined with the spatial coordinate information are employed as input, and subsequently a sub-step of representative points selection as well as bipartite graph formation, followed by the spectral clustering algorithm to complete the classification task. Finally, testing and analysis are conducted on the RADARSAT-2 fully PolSAR dataset acquired over the Hunshandake Sandy Land in 2016. Both qualitative and quantitative experiments compared with several classification methods are conducted to show that proposed method can significantly improve performance on classification.

Mixed Membership Graph Clustering via Systematic Edge Query

This work considers clustering nodes of a largely incomplete graph. Under the problem setting, only a small amount of queries about the edges can be made, but the entire graph is not observable. This problem finds applications in large-scale data clustering using limited annotations, community detection under restricted survey resources, and graph topology inference under hidden/removed node interactions. Prior works tackled this problem from various perspectives, e.g., convex programming-based low-rank matrix completion and active query-based clique finding. Nonetheless, many existing methods are designed for estimating the single-cluster membership of the nodes, but nodes may often have mixed (i.e., multi-cluster) membership in practice. Some query and computational paradigms, e.g., the random query patterns and nuclear norm-based optimization advocated in the convex approaches, may give rise to scalability and implementation challenges. This work aims at learning mixed membership of nodes using queried edges. The proposed method is developed together with a systematic query principle that can be controlled and adjusted by the system designers to accommodate implementation challenges -- e.g., to avoid querying edges that are physically hard to acquire. Our framework also features a lightweight and scalable algorithm with membership learning guarantees. Real-data experiments on crowdclustering and community detection are used to showcase the effectiveness of our method.