Astronomical Data Processing Research Articles

Molecular spectral line data preprocessing container load grouping prediction algorithm based on EMD-LSTM

Unbalanced allocation of container resources in cluster environments is an urgent problem to be solved. Aiming at container load prediction and resource allocation strategy, this paper designs an astronomical data processing container load grouping prediction algorithm based on empirical mode decomposition-long short-term memory network, and proposes an adaptive recommendation value generation algorithm based on predicted load information, which can automatically allocate container computing resources according to the degree of load fluctuation. The load prediction accuracy is verified using simulated data and real astronomical observation data. The experimental results show that the algorithm proposed in this paper has higher prediction accuracy than the triple index method and the single long short-term memory network model. In the real-time preprocessing test of astronomical data, the recommendation value generation algorithm proposed in this paper can effectively improve the utilization efficiency of computing resources compared with the default strategy.

A novel edge computing approach to astronomical image data processing based on sCMOS camera using SoC

The ever-growing deluge of astronomical data challenges traditional server-based processing, hindering real-time analysis and scientific discovery. This paper proposes a novel approach: edge computing directly on an sCMOS camera using a System-on-Chip (SoC) architecture currently developed at Creotech Instruments. We present a custom-designed camera equipped with an FPGA-based SoC, enabling on-board preprocessing and feature extraction of astronomical images. This significantly reduces data transmission, minimizes latency, and empowers real-time decision-making for critical observations. We showcase the camera's capabilities through real-world scenarios, demonstrating its usability in astronomy.

Imputation of missing photometric data and photometric redshift estimation for CSST

ABSTRACT Accurate photometric redshift (photo-z) estimation requires support from multiband observational data. However, in the actual process of astronomical observations and data processing, some sources may have missing observational data in certain bands for various reasons. This could greatly affect the accuracy and reliability of photo-z estimation for these sources, and even render some estimation methods unusable. The same situation may exist for the upcoming Chinese Space Station Telescope (CSST). In this study, we employ a deep learning method called generative adversarial imputation networks (GAIN) to impute the missing photometric data in CSST, aiming to reduce the impact of data missing on photo-z estimation and improve estimation accuracy. Our results demonstrate that using the GAIN technique can effectively fill in the missing photometric data in CSST. Particularly, when the data missing rate is below 30 per cent, the imputation of photometric data exhibits high accuracy, with higher accuracy in the g, r, i, z, and y bands compared to the NUV and u bands. After filling in the missing values, the quality of photo-z estimation obtained by the widely used easy and accurate Zphot from Yale (eazy) software is notably enhanced. Evaluation metrics for assessing the quality of photo-z estimation, including the catastrophic outlier fraction (fout), the normalized median absolute deviation ($\rm {\sigma _{NMAD}}$), and the bias of photometric redshift (bias), all show some degree of improvement. Our research will help maximize the utilization of observational data and provide a new method for handling sample missing values for applications that require complete photometry data to produce results.

Resolution Enhancement of SOHO/MDI Magnetograms

Research on the solar magnetic field and its effects on solar dynamo mechanisms and space weather events has benefited from the continual improvements in instrument resolution and measurement frequency. The augmentation and assimilation of historical observational data timelines also play a significant role in understanding the patterns of solar magnetic field variation. Within the realm of astronomical data processing, super-resolution (SR) reconstruction refers to the process of using a substantial corpus of training data to learn the nonlinear mapping between low-resolution (LR) and high-resolution (HR) images, thereby achieving higher-resolution astronomical images. This paper is an application study in high-dimensional nonlinear regression. Deep learning models were employed to perform SR modeling on SOHO/MDI magnetograms and SDO/HMI magnetograms, thus reliably achieving resolution enhancement of full-disk SOHO/MDI magnetograms and enhancing the image resolution to obtain more detailed information. For this study, a data set comprising 9717 pairs of data from 2010 April to 2011 February was used as the training set, 1332 pairs from 2011 March were used as the validation set and 1034 pairs from 2011 April were used as the test set. After data preprocessing, we randomly cropped 128 × 128 sub-images as the LR cases from the full-disk MDI magnetograms, and the corresponding 512 × 512 sub-images as HR ones from the HMI full-disk magnetograms for model training. The tests conducted have shown that the study successfully produced reliable 4× SR reconstruction of full-disk MDI magnetograms. The MESR model’s results (0.911) were highly correlated with the target HMI magnetographs as indicated by the correlation coefficient values. Furthermore, the method achieved the best PSNR, SSIM, MAE and RMSE values, indicating that the MESR model can effectively reconstruct magnetograms.

Study on tiered storage algorithm based on heat correlation of astronomical data

With the surge in astronomical data volume, modern astronomical research faces significant challenges in data storage, processing, and access. The I/O bottleneck issue in astronomical data processing is particularly prominent, limiting the efficiency of data processing. To address this issue, this paper proposes a tiered storage algorithm based on the access characteristics of astronomical data. The C4.5 decision tree algorithm is employed as the foundation to implement an astronomical data access correlation algorithm. Additionally, a data copy migration strategy is designed based on tiered storage technology to achieve efficient data access. Preprocessing tests were conducted on 418GB NSRT (Nanshan Radio Telescope) formaldehyde spectral line data, showcasing that tiered storage can potentially reduce data processing time by up to 38.15%. Similarly, utilizing 802.2 GB data from FAST (Five-hundred-meter Aperture Spherical radio Telescope) observations for pulsar search data processing tests, the tiered storage approach demonstrated a maximum reduction of 29.00% in data processing time. In concurrent testing of data processing workflows, the proposed astronomical data heat correlation algorithm in this paper achieved an average reduction of 17.78% in data processing time compared to centralized storage. Furthermore, in comparison to traditional heat algorithms, it reduced data processing time by 5.15%. The effectiveness of the proposed algorithm is positively correlated with the associativity between the algorithm and the processed data. The tiered storage algorithm based on the characteristics of astronomical data proposed in this paper is poised to provide algorithmic references for large-scale data processing in the field of astronomy in the future.

Review of artificial intelligence applications in astronomical data processing

Review of artificial intelligence applications in astronomical data processing

RFI detection with spiking neural networks

Abstract Detecting and mitigating radio frequency interference (RFI) is critical for enabling and maximising the scientific output of radio telescopes. The emergence of machine learning (ML) methods capable of handling large datasets has led to their application in radio astronomy, particularly in RFI detection. Spiking neural networks (SNNs), inspired by biological systems, are well suited for processing spatio-temporal data. This study introduces the first exploratory application of SNNs to an astronomical data processing task, specifically RFI detection. We adapt the nearest latent neighbours (NLNs) algorithm and auto-encoder architecture proposed by previous authors to SNN execution by direct ANN2SNN conversion, enabling simplified downstream RFI detection by sampling the naturally varying latent space from the internal spiking neurons. Our subsequent evaluation aims to determine whether SNNs are viable for future RFI detection schemes. We evaluate detection performance with the simulated HERA telescope and hand-labelled LOFAR observation dataset the original authors provided. We additionally evaluate detection performance with a new MeerKAT-inspired simulation dataset that provides a technical challenge for machine-learnt RFI detection methods. This dataset focuses on satellite-based RFI, an increasingly important class of RFI and is an additional contribution. Our SNN approach remains competitive with the original NLN algorithm and AOFlagger in AUROC, AUPRC, and F1-scores for the HERA dataset but exhibits difficulty in the LOFAR and Tabascal datasets. However, our method maintains this accuracy while completely removing the compute and memory-intense latent sampling step found in NLN. This work demonstrates the viability of SNNs as a promising avenue for ML-based RFI detection in radio telescopes by establishing a minimal performance baseline on traditional and nascent satellite-based RFI sources and is the first work to our knowledge to apply SNNs in astronomy.

Ensemble Feature Selection Method for Single Pulse Classification

Affected by a large number of radio frequency interference signals, it has become an important task for astronomical data processing to quickly and accurately identify single pulse signals from massive observation data. Designing and extracting effective data features is the key issue for efficient identification of single pulse signals using machine learning. This paper proposes an ensemble feature selection method for single pulse signal classification. The method first mixed three types of features, including the parametric features, statistical features, and abstract features of single pulse signals, and then used five individual feature selection methods to select the corresponding optimal feature set, respectively. At last, the features selected by the five individual methods are mixed, and the greedy strategy was used to select the optimal ensemble feature set. The experimental results show that the ensemble feature set can improve F1-score by a value of 1.8% at most, and can obtain higher accuracy than the features selected by individual methods. Under the background of high-speed and large-scale sky survey, the ensemble feature selection method plays an important role in reducing the number of features, improving classification performance, and speeding up data processing.

DRAGONS—A Quick Overview

Data Reduction for Astronomy from Gemini Observatory North and South (DRAGONS) is a platform for the reduction and processing of astronomical data. The Python-based, open-source package includes infrastructure for automation and algorithms for the processing of imaging and spectroscopic data, up to the analysis-ready stage. DRAGONS currently focuses on the reduction of Gemini data, although it allows for support of data from other instruments and telescopes through third-party extensions. Its latest release (v3.1) enables automated reduction of all currently active Gemini imaging facility instruments, as well as optical longslit spectroscopic data, acquired with GMOS.

Statistical Data Retrieval Technique in Astronomy Computational Physics

Computational astronomy is a very important branch in today's era, where physicists or researchers can use computers to process statistics in astronomical physics. researchers can process abstract data from raw data and can convert data into data visualizations. Computational physics astronomy is a sophisticated and well-established method, this branch of science can provide and process data, solve complex problems, and is very helpful for statisticians and computer scientists. Astronomical physicists have many problems, among others; there is a problem that is hierarchical, and complex, so that this paper will provide a basis for methods for optimizing methods in processing statistical data on physics. The author's hope is that astronomical physicists can perform an important and effective processing of astronomical data optimally and effectively.

Estimating stellar parameters from LAMOST low-resolution spectra

ABSTRACTThe Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST) has acquired tens of millions of low-resolution spectra of stars. This paper investigates the parameter estimation problem for these spectra. To this end, we propose the deep learning model StarGRU network (StarGRUNet). This network is applied to estimate the stellar atmospheric physical parameters and 13 elemental abundances from LAMOST low-resolution spectra. On the spectra with signal-to-noise ratios greater than or equal to 5, the estimation precisions are 94 K and 0.16 dex on $T_\texttt {eff}$ and log g respectively, 0.07 to 0.10 dex on [C/H], [Mg/H], [Al/H], [Si/H], [Ca/H], [Ni/H] and [Fe/H], 0.10 to 0.16 dex on [O/H], [S/H], [K/H], [Ti/H] and [Mn/H], and 0.18 and 0.22 dex on [N/H] and [Cr/H]. The model shows advantages over other available models and high consistency with high-resolution surveys. We released the estimated catalogue computed from about 8.21 million low-resolution spectra in LAMOST DR8, code, trained model, and experimental data for astronomical science exploration and data processing algorithm research.

Decomposition and Classification of Carbon Star Spectra

Automatic classification of stellar spectra is an important research component of astronomical data processing and is the basis for studying stellar evolution and parameter measurements. As a rare kind of stellar spectra, carbon star spectra put forward more efficient and accurate requirements for classification methods. The traditional manual classification methods have the disadvantages of slow speed and low accuracy, which can no longer meet the practical needs of automatic classification of massive stellar spectra, especially low signal-to-noise ratio stellar spectra, and machine learning algorithms have been widely applied to stellar spectral classification. A distinctive feature of stellar spectra is high data dimensionality, and dimensionality reduction can not only realize feature extraction, but also reduce the computational effort, which is the first task of spectral classification. Traditional linear dimensionality reduction methods such as principal component analysis reduce the spectra only based on the variance, and different types of spectra will appear crossover after projection into the low-dimensional feature space, while streamwise learning can produce excellent classification boundaries, which will avoid overlap and facilitate subsequent classification. In view of the high dimensionality of spectral data, we investigate the distribution of spectral data in high-dimensional space and the principle of dimensionality reduction of high-dimensional linear data by stream shape learning, compare the effect of two-dimensionality reduction methods, t - SNE and principal component analysis, on spectral data, and finally analyze the experimental results and compare and validate them using various machine learning classifiers. The algorithm is implemented using Python language and Scikit - learn third-party library to perform experiments on 1000 low signal-to-noise carbon star spectra from LAMOST, and finally achieve high accuracy automatic processing and classification of the spectral data. The experimental results show that for the dimensionality reduction processing of spectral data, the t - SNE method based on stream shape learning can recover the low-dimensional stream shape structure in the high-dimensional spectral data, and after feature extraction, satisfactory classification accuracy can be achieved on the test dataset using a machine learning classifier.

Data mining techniques on astronomical spectra data – II. Classification analysis

ABSTRACT Classification is valuable and necessary in spectral analysis, especially for data-driven mining. Along with the rapid development of spectral surveys, a variety of classification techniques have been successfully applied to astronomical data processing. However, it is difficult to select an appropriate classification method in practical scenarios due to the different algorithmic ideas and data characteristics. Here, we present the second work in the data mining series – a review of spectral classification techniques. This work also consists of three parts: a systematic overview of current literature, experimental analyses of commonly used classification algorithms, and source codes used in this paper. First, we carefully investigate the current classification methods in astronomical literature and organize these methods into ten types based on their algorithmic ideas. For each type of algorithm, the analysis is organized from the following three perspectives. (1) their current applications and usage frequencies in spectral classification are summarized; (2) their basic ideas are introduced and preliminarily analysed; (3) the advantages and caveats of each type of algorithm are discussed. Secondly, the classification performance of different algorithms on the unified data sets is analysed. Experimental data are selected from the LAMOST survey and SDSS survey. Six groups of spectral data sets are designed from data characteristics, data qualities, and data volumes to examine the performance of these algorithms. Then the scores of nine basic algorithms are shown and discussed in the experimental analysis. Finally, nine basic algorithms source codes written in python and manuals for usage and improvement are provided.

A method for efficient radio astronomical data gridding on multi-core vector processor

Gridding is the performance-critical step in the data reduction pipeline for radio astronomy research, allowing astronomers to create the correct sky images for further analysis. Like the 2D stencil computation, gridding iteratively updates the output cells by convolution, where the value at each output cell in the space is computed as a weighted sum of neighboring point values. Existing state-of-the-art works have achieved performance improvement of gridding by using multi-core CPUs and GPUs in real-world applications, and their study proved that gridding is a type of scientific computation with high-density computing characteristics. However, low computational performance or high power consumption becomes the main limitation for their processing of large-scale astronomical data. The high-density computing feature of gridding provides opportunities to accelerate it on the multi-core vector processor with vector-SIMD architectures. However, existing works’ (such as those implemented on CPUs or GPUs) task parallelization and data transfer strategies are inefficient to perform gridding directly on the vector processor without any dedicated mapping algorithm.M-DSP is a multi-core vector processor with vector-SIMD architectures designed for the next-generation exascale supercomputer, delivering high performance with ultra-low power consumption. In this paper, we present, for the first time, a novel method to achieve efficient gridding on the M-DSP. Specifically, we propose a gridding workflow designed for the vector-SIMD architectures and present a vectorized version of the gridding convolution algorithm to fully exploit the computational power of the M-DSP. In addition, centering on the processor architectures, we propose task-based parallelization strategies for block and line computing as well as different data loading strategies to achieve high parallel performance and high data transfer efficiency. Experimental results show that our work on M-DSP exhibits very competitive performance compared to other methods running on CPUs or GPUs. This demonstrates the efficiency of our method and the fact that the vector-SIMD architecture is beneficial for scientific computing with ”high density” characteristics, which can exploit its wide vector core and achieve higher performance than its competitors.

Automatic Detection and Classification of Radio Galaxy Images by Deep Learning

Surveys conducted by radio astronomy observatories, such as SKA, MeerKAT, Very Large Array, and ASKAP, have generated massive astronomical images containing radio galaxies (RGs). This generation of massive RG images has imposed strict requirements on the detection and classification of RGs and makes manual classification and detection increasingly difficult, even impossible. Rapid classification and detection of images of different types of RGs help astronomers make full use of the observed astronomical image data for further processing and analysis. The classification of FRI and FRII is relatively easy, and there are more studies and literature on them at present, but FR0 and FRI are similar, so it is difficult to distinguish them. It poses a greater challenge to image processing. At present, deep learning has made breakthrough progress in the field of image analysis and processing and has preliminary applications in astronomical data processing. Compared with classification algorithms that can only classify galaxies, object detection algorithms that can locate and classify RGs simultaneously are preferred. In target detection algorithms, YOLOv5 has outstanding advantages in the classification and positioning of small targets. Therefore, we propose a deep-learning method based on an improved YOLOv5 object detection model that makes full use of multisource data, combining FIRST radio with SDSS optical image data, and realizes the automatic detection of FR0, FRI, and FRII RGs. The innovation of our work is that on the basis of the original YOLOv5 object detection model, we introduce the SE Net attention mechanism, increase the number of preset anchors, adjust the network structure of the feature pyramid, and modify the network structure, thereby allowing our model to demonstrate galaxy classification and position detection effects. Our improved model produces satisfactory results, as evidenced by experiments. Overall, the mean average precision (mAP@0.5) of our improved model on the test set reaches 89.4%, which can determine the position (R.A. and decl.) and automatically detect and classify FR0s, FRIs, and FRIIs. Our work contributes to astronomy because it allows astronomers to locate FR0, FRI, and FRII galaxies in a relatively short time and can be further combined with other astronomically generated data to study the properties of these galaxies. The target detection model can also help astronomers find FR0s, FRIs, and FRIIs in future surveys and build a large-scale star RG catalog. Moreover, our work is also useful for the detection of other types of galaxies.

Why Is the Mean Anomaly at Epoch Not Used in Tests of Non-Newtonian Gravity?

The mean anomaly at epoch η is one of the standard six Keplerian orbital elements in terms of which the motion of the two-body problem is parameterized. Along with the argument of pericenter ω, η experiences long-term rates of change induced, among other things, by general relativity and several modified models of gravity. Thus, in principle, it may be fruitfully adopted together with ω in several tests of post-Newtonian gravity performed with astronomical and astrophysical binary systems. This would allow us to enhance the gravitational signature we are interested in and to disentangle some competing disturbing effects acting as sources of systematic bias. Nonetheless, for some reasons unknown to the present author, η has never been used so far by astronomers in actual data reductions. This note aims to raise interest in the community about the possible practical use of such an orbital element or, at least, to induce experts in astronomical data processing to explicitly make clear if it is not possible to use η for testing gravitational models and, if this is the case, why.

Design and Evaluation of 2-bit-Input Single-Flux-Quantum Autocorrelator System for Astronomical Data Analysis

To improve the accuracy of astronomical data analysis, an increase in the number of output bits from the A/D converter is effective. We investigated a single-flux-quantum (SFQ) autocorrelator and an integrator that supports multiple-bit signal output from an SFQ A/D converter for astronomical data processing. The SFQ binary counter was used as the integrator that decimates the output from the SFQ autocorrelator for communication with equipment at room temperature. We designed the test circuits composed of 2-bit inputs, an autocorrelator, and an integrator that contains 5631 Josephson junctions using the 10 kA/cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> Nb process. We confirmed the correct operation of the test circuit with a normalized bias voltage margin of 102%-114% by the low-speed test. We obtained its correct high-speed operation up to a 55.1 GHz clock frequency. In terms of circuit design, scaling up the test circuit is easy. These results indicate the effectiveness of a high-precision and large-bandwidth radio astronomical observation using SFQ circuit technology.

Efficiency Near the Edge: Increasing the Energy Efficiency of FFTs on GPUs for Real-Time Edge Computing

The Square Kilometre Array (SKA) is an international initiative for developing the world's largest radio telescope with a total collecting area of over a million square meters. The scale of the operation, combined with the remote location of the telescope, requires the use of energy-efficient computational algorithms. This, along with the extreme data rates that will be produced by the SKA and the requirement for a real-time observing capability, necessitates in-situ data processing in an edge style computing solution. More generally, energy efficiency in the modern computing landscape is becoming of paramount concern. Whether it be the power budget that can limit some of the world's largest supercomputers, or the limited power available to the smallest Internet-of-Things devices. In this paper, we study the impact of hardware frequency scaling on the energy consumption and execution time of the Fast Fourier Transform (FFT) on NVIDIA GPUs using the cuFFT library. The FFT is used in many areas of science and it is one of the key algorithms used in radio astronomy data processing pipelines. Through the use of frequency scaling, we show that we can lower the power consumption of the NVIDIA V100 GPU when computing the FFT by up to 60% compared to the boost clock frequency, with less than a 10% increase in the execution time. Furthermore, using one common core clock frequency for all tested FFT lengths, we show on average a 50% reduction in power consumption compared to the boost core clock frequency with an increase in the execution time still below 10%. We demonstrate how these results can be used to lower the power consumption of existing data processing pipelines. These savings, when considered over years of operation, can yield significant financial savings, but can also lead to a significant reduction of greenhouse gas emissions.

Real-time abnormal light curve detection based on a Gated Recurrent Unit network

Targeting the problem of high real-time requirements in astronomical data processing, this paper proposes a real-time early warning model for light curves based on a Gated Recurrent Unit (GRU) network. Using the memory function of the GRU network, a prediction model of the light curve is established, and the model is trained using the collected light curve data, so that the model can predict a star magnitude value for the next moment based on historical star magnitude data. In this paper,we calculate the difference between the model prediction value and the actual observation value and set a threshold. If the difference exceeds the set threshold, the observation value at the next moment is considered to be an abnormal value, and a warning is given. Astronomers can carry out further certification based on the early warning and in combinationwith other means of observation. Themethod proposed in this paper can be applied to real-time observations in time domain astronomy.

Research and Application of Recurrent Neural Network in Solar Radio Interference Filtering

The various interfering signals present in the space result in the inability to obtain a clean and clear solar radio dynamic spectrum, which affects the effective observation of solar radio burst events. At the data processing level, we propose a method for predicting and processing radio interference signals in solar radio burst events using recurrent neural network in deep learning. Firstly, the radio signal that satisfies the condition is selected by the amount of solar radio flux of all frequency channels at some moment, and then the position of the initial time of the burst event is located by using the variation curve of the solar radio flux with time under the single frequency channel. After that, the constructed recurrent neural network is used to predict the signal value of the radio in the burst area. Finally, according to the linear additivity of the signal, the value of the clean pure burst event is obtained by subtracting the predicted radio value from the original value of the burst area. The experimental results show that the proposed method can effectively remove the interference in the solar radio dynamic spectrum and preserve the effective information of the burst event to the greatest extent. This provides a new idea and research direction for deep learning in the anti-interference processing of astronomical big data.