Real-time Data Reduction Research Articles

The Umbrella software suite for automated asteroid detection

We present the Umbrella software suite for asteroid detection, validation, identification and reporting. The current core of Umbrella is an open-source modular library, called Umbrella2, that includes algorithms and interfaces for all steps of the processing pipeline, including a novel detection algorithm for faint trails. Building on the library, we have also implemented a detection pipeline accessible both as a desktop program (ViaNearby) and via a web server (Webrella), which we have successfully used in near real-time data reduction of a few asteroid surveys on the Wide Field Camera of the Isaac Newton Telescope. In this paper we describe the library, focusing on the interfaces and algorithms available, and we present the results obtained with the desktop version on a set of well-curated fields used by the EURONEAR project as an asteroid detection benchmark.

AI-enabled real-time dual-comb molecular fingerprint imaging.

Hyperspectral imaging provides spatially resolved spectral information. Utilizing dual-frequency combs as active illumination sources, hyperspectral imaging with ultra-high spectral resolution can be implemented in a scan-free manner when a detector array is used for heterodyne detection. Here, we show that dual-comb hyperspectral imaging can be performed with an uncooled near-to-mid-infrared detector by exploiting the detector array's high frame rate, achieving 10 Hz acquisition in 30 spectral channels across 16,384 pixels. Artificial intelligence (AI) enables real-time data reduction and imaging of gas concentration based on characteristic molecular absorption signatures. Owing to the detector array's sensitivity from 1 to 5 µm wavelength, this demonstration lays the foundation for real-time versatile imaging of molecular fingerprint signatures across the infrared wavelength regime with high temporal resolution.

Introduction to APOSOS project: 15 cm aperture electro-optical telescopes to track space objects

Asia-Pacific ground-based Optical Space object Observation System (APOSOS) is an international cooperation project, which is initiated and supported by Asia-Pacific Space Cooperation Organization (APSCO). Currently, APOSOS consists of three Observation Nodes in Pakistan, Peru and Iran, and a Data and Operation Management Center in National Astronomical Observatories, Chinese Academy of Sciences. Each Observation Node is equipped with a dedicated 15 cm aperture refracting telescope, which is capable of real-time astrometric data reduction. APOSOS aims to develop a tracking network of space object to acquire accurate astrometric measurements of space objects owned by APSCO member states. In this paper, we present an overview of APOSOS’s design and development. The APOSOS’s detecting capability of tracking a space object is darker than 12m in LEO or 16m in GEO, the size of which is equivalent to a diameter of 10-cm class in LEO or meter class in GEO respectively. A parametric estimation model is setup to profile APOSOS’s capability in different tracking missions, which is verified by experimental observations. Preliminary results of tracking observations of SLR satellites showed that RMS of APOSOS astrometric measurements in RA/DEC is better than 5″.

The Performance of Belle II High Level Trigger in the First Physics Run

The Belle II experiment is a new generation B-factory experiment at KEK in Japan aiming at the search for New Physics in a huge sample of B-meson decays. The commissioning of the accelerator and the detector for the first physics run has started from March this year. The Belle II High Level Trigger (HLT) is fully working in the beam run. The HLT is now operated with 1600 cores clusterized in 5 units, which is 1/4 of the full configuration. The software trigger is performed using the same offline reconstruction code, and events are classified into a set of physics categories. Only the events in the categories of interest are finally sent out to the storage. Live data quality monitoring is also performed on HLT. For the selected events, the reconstructed tracks are extrapolated to the surface of the pixel detector (PXD) and quickly fed back to the readout electronics for the real time data reduction by sending only the associated hits. The maximum trigger rate in the first physics run was 3.5kHz, and the Belle II data acquisition system was stably operated. There were several problems in the HLT operation, but they have successfully been fixed during the data taking period. The HLT reduction factor is measured to be 8 which is still higher than the design because of the high background environment.

A method of real-time data reduction of asteroid optical observation image based on GPU

A method of real-time data reduction of asteroid optical observation image based on GPU

An adaptive framework for real-time data reduction in AMI

In existing Advanced Metering Infrastructure (AMI), data collection intervals for each smart meter (SM) typically vary from 15 to 60 min. If we have 1 million SMs that transmit data every 15 min, these SMs will export 4 million records per hour. This leads to dramatically increasing bandwidth usage, energy consumption, traffic cost and I/O congestion. In this work, we present an adaptive framework for minimizing the amount of data transfer from SMs. The reduction in the framework is forecasting-based; when an SM reading is close to the forecasted value, the SM does not transmit the reading. In order for the framework to be adaptive to the ever-changing pattern of SM data, it is provided with a pool of forecasting methods. A supervised-learning scheme is employed to switch in real-time to the forecasting method most suitable to the current data pattern. The experimental results demonstrate that the proposed framework achieves data reduction rates up to 98% with accuracy 96%, depending on the operational parameters of the framework and consumer behavior (statistical features of SM data).

Real-time diffraction computed tomography data reduction.

Diffraction imaging is an X-ray imaging method which uses the crystallinity information (cell parameter, orientation) as a signal to create an image pixel by pixel: a pencil beam is raster-scanned onto a sample and the (powder) diffraction signal is recorded by a large area detector. With the flux provided by third-generation synchrotrons and the speed of hybrid pixel detectors, the acquisition speed of these experiments is now limited by the transfer rate to the local storage as the data reduction can hardly be performed in real time. This contribution presents the benchmarking of a typical data analysis pipeline for a diffraction imaging experiment like the ones performed at ESRF ID15a and proposes some disruptive techniques to decode CIF binary format images using the computational power of graphics cards to be able to perform data reduction in real time.

Automated data processing architecture for the Gemini Planet Imager Exoplanet Survey

The Gemini Planet Imager Exoplanet Survey (GPIES) is a multi-year direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow-up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our data reduction pipelines. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real-time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.

APRS: adaptive real-time payload data reduction scheme for IoT/WSN sensor board with multivariate sensors

In this paper, a new simple mechanism called as adaptive real-time payload data reduction scheme (APRS) for energy efficiency purpose is proposed. APRS aims to reduce the transmitted packet size for each sensed payload, moreover it prevents any transmissions if no significant change is reported by the payload sensing block. Performance of APRS was evaluated through simulation by utilising various real-time datasets. In addition, APRS was successfully implemented in testbed. In conclusion, APRS has managed to show its simplicity and flexibility for the real-time IoT/WSN application when it is compared with the other algorithms and its reduction ratio during a transmission is within acceptable range of 81-94%. The average of the total percentage of energy saved by applied APRS in all nodes is 95%. Overall, APRS has high performance in the reduction ratio of data and efficiency in energy consumption when it is compared with other recent multivariate data reduction methods.

APRS: adaptive real-time payload data reduction scheme for IoT/WSN sensor board with multivariate sensors

In this paper, a new simple mechanism called as adaptive real-time payload data reduction scheme (APRS) for energy efficiency purpose is proposed. APRS aims to reduce the transmitted packet size for each sensed payload, moreover it prevents any transmissions if no significant change is reported by the payload sensing block. Performance of APRS was evaluated through simulation by utilising various real-time datasets. In addition, APRS was successfully implemented in testbed. In conclusion, APRS has managed to show its simplicity and flexibility for the real-time IoT/WSN application when it is compared with the other algorithms and its reduction ratio during a transmission is within acceptable range of 81-94%. The average of the total percentage of energy saved by applied APRS in all nodes is 95%. Overall, APRS has high performance in the reduction ratio of data and efficiency in energy consumption when it is compared with other recent multivariate data reduction methods.

Intermediate Palomar Transient Factory: Realtime Image Subtraction Pipeline

A fast-turnaround pipeline for realtime data reduction plays an essential role in discovering and permitting follow-up observations to young supernovae and fast-evolving transients in modern time-domain surveys. In this paper, we present the realtime image subtraction pipeline in the intermediate Palomar Transient Factory. By using high-performance computing, efficient databases, and machine-learning algorithms, this pipeline manages to reliably deliver transient candidates within 10 minutes of images being taken. Our experience in using high-performance computing resources to process big data in astronomy serves as a trailblazer to dealing with data from large-scale time-domain facilities in the near future.

A Method for Correcting Telescope Pointing Error in Optical Space Debris Surveys

In the data processing of optical space debris observations, the background stars may be mismatched or unable to get matched due to the influence of the telescope pointing error, then it is difficult to calculate the plate model and to determine accurately the object position. Based on the characteristics of debris CCD images, a multiple neighborhood matching method based on feature stars is proposed. The experimental results show that the influence of the telescope pointing error can be removed, and that the efficiency and correctness of background star map matching are improved significantly. Meanwhile, the time cost of this method is low, so it is beneficial to the real-time data reduction.

The RoboPol pipeline and control system

We describe the data reduction pipeline and control system for the RoboPol project. The RoboPol project is monitoring the optical $R$-band magnitude and linear polarization of a large sample of active galactic nuclei that is dominated by blazars. The pipeline calibrates and reduces each exposure frame, producing a measurement of the magnitude and linear polarization of every source in the $13'\times 13'$ field of view. The control system combines a dynamic scheduler, real-time data reduction, and telescope automation to allow high-efficiency unassisted observations.

The Belle II Pixel Detector Data Acquisition and Reduction System

The upcoming Belle II experiment is designed to work at a luminosity of 8×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">35</sup> cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-2</sup> s <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> , 40 times higher than its predecessor. The pixel detector of Belle II with its ~ 8 million channels will deliver ten times more data than all other sub-detectors together. A data rate of 22 Gbytes/s is expected for a trigger rate of 30 kHz and an estimated pixel detector occupancy of 3%, which is by far exceeding the specifications of the Belle II event builder system. Therefore a realtime data reduction of a factor > 30 is needed. A hardware platform capable of processing this amount of data is the ATCA based Compute Node (CN). Each CN consists of an xTCA carrier board and four AMC/xTCA daughter boards. The carrier board supplies the high bandwidth connectivity to the other CNs via Rocket-IO links. In the current prototype design, each AMC board is equipped with a Xilinx Virtex-5 FX70T FPGA, 4 GB of memory, Gbit Ethernet and two bi-directional optical links allowing for a bandwidth of up to 12.5 Gbits/s. IPMI control of mother and daughter board is foreseen. One ATCA shelf containing 10 mother boards/40 daughter boards is sufficient to process the data from the 40 front end boards. The data reduction on the CN is done in two steps. First, the event data delivered by the front end electronics via optical links is stored in memory until the high level trigger (HLT) decision has been made. The HLT rejects >2/3 of these events. In a second step, pixel data of positively triggered events is reduced with the help of regions of interest (ROI), calculated by the HLT from projecting trajectories back to the pixel detector plane. The design allows additional ROI inputs computed from hit cluster properties or tracklets from the surrounding silicon strip detector. The final data reduction is achieved by sending only data within these ROIs to the main event builder.

A new source extraction algorithm for optical space debris observation

Specific challenges arise in the task of real-time automatic data reduction of optical space debris observations. Here we present an automatic technique that optimally detects and measures the sources from images of optical space debris observations. We show that highly reliable and accurate results can be obtained on most images produced by our specific sensors, and due to optimizations, the whole pipeline works fast and efficiently. Tests demonstrate that the technique performs better than SExtractor from the point of view of fast and accurate detection, therefore it is well suited for data reduction of optical space debris observations.

KMTNet 실시간 자료처리 파이프라인 개발

Real-time data reduction pipeline for the Korea Microlensing Telescope Network (KMTNet) was developed by Korea Astronomy and Space Science Institute (KASI). The main goal of the data reduction pipeline is to find variable objects and to record their light variation from the large amount of observation data of about 200 GB per night per site. To achieve the goal we adopt three strategic implementations: precision pointing of telescope using the cross correlation correction for target fields, realtime data transferring using kernel-level file handling and high speed network, and segment data processing architecture using the Sun-Grid engine. We tested performance of the pipeline using simulated data which represent the similar circumstance to CTIO (Cerro Tololo Inter-American Observatory), and we have found that it takes about eight hours for whole processing of one-night data. Therefore we conclude that the pipeline works without problem in real-time if the network speed is high enough, e.g., as high as in CTIO.

The FastTracker Real Time Processor and Its Impact on Muon Isolation, Tau and b-Jet Online Selections at ATLAS

As the LHC luminosity is ramped up to 3 <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$\,\times 10^{34}~{\hbox {cm}}^{2}~s^{1}$</tex></formula> and beyond, the high rates, multiplicities, and energies of particles seen by the detectors will pose a unique challenge. Only a tiny fraction of the produced collisions can be stored offline and immense real-time data reduction is needed. An effective trigger system must maintain high trigger efficiencies for the physics we are most interested in while suppressing the enormous QCD backgrounds. This requires massive computing power to minimize the online execution time of complex algorithms. A multi-level trigger is an effective solution to meet this challenge. The Fast Tracker (FTK) is an upgrade to the current ATLAS trigger system that will operate at full Level-1 output rates and provide high-quality tracks reconstructed over the entire inner detector by the start of processing in the Level-2 Trigger. FTK solves the combinatorial challenge inherent to tracking by exploiting the massive parallelism of associative memories that can compare inner detector hits to millions of pre-calculated patterns simultaneously. The tracking problem within matched patterns is further simplified by using pre-computed linearized fitting constants and relying on fast DSPs in modern commercial FPGAs. Overall, FTK is able to compute the helix parameters for all tracks in an event and apply quality cuts in less than 100 <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\mu{\hbox {s}}$</tex> </formula> . The system design is defined and the performance presented with respect to high transverse momentum (high- <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$p_{\rm T}$</tex></formula> ) Level-2 objects: b jets, tau jets, and isolated leptons. We test FTK algorithms using the full ATLAS simulation with WH events up to <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$3\times 10^{34}~{\hbox {cm}}^{2}{\hbox {s}}^{1}$</tex></formula> luminosity and compare the FTK results with the offline tracking capability. We present the architecture and the reconstruction performance for the mentioned high- <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$p_{\rm T}$</tex></formula> Level-2 objects.

FPGA-based architecture for real-time data reduction of ultrasound signals

This paper describes a novel method for on-line real-time data reduction of radiofrequency (RF) ultrasound signals. The approach is based on a field programmable gate array (FPGA) system intended mainly for steel thickness measurements. Ultrasound data reduction is desirable when: (1) direct measurements performed by an operator are not accessible; (2) it is required to store a considerable amount of data; (3) the application requires measuring at very high speeds; and (4) the physical space for the embedded hardware is limited. All the aforementioned scenarios can be present in applications such as pipeline inspection where data reduction is traditionally performed on-line using pipeline inspection gauges (PIG). The method proposed in this work consists of identifying and storing in real-time only the time of occurrence (TOO) and the maximum amplitude of each echo present in a given RF ultrasound signal. The method is tested with a dedicated immersion system where a significant data reduction with an average of 96.5% is achieved.

The application of the instantaneous states reduction to the orbital monitoring of pivotal arcs of the Chang’E-1 satellite

In the Chinese lunar exploration project, the Chang’E-1 (CE-1) satellite was jointly monitored by the United S-band range and Doppler and the VLBI technique. A real-time reduction of the tracking data is realized to deduce the time series of the instantaneous state vectors (ISV) (position and velocity vectors) of the CE-1 satellite, and is applied to the orbital monitoring of pivotal arcs. This paper introduces this real-time data reduction method and its application to the orbital monitoring of pivotal arcs of the CE-1 satellite in order to serve as a source of criticism and reference.

Toward Online Data Reduction for Portable Electroencephalography Systems in Epilepsy

Portable EEG units are key tools in epilepsy diagnosis. Current systems could be made physically smaller and longer lasting by the inclusion of online data reduction methods to reduce the power required for storage or transmission of the EEG data. This paper presents a real-time data reduction algorithm based upon the discontinuous recording of the EEG: noninteresting background sections of EEG are discarded online, with only potentially diagnostically interesting sections being saved. MATLAB simulations of the algorithm on an EEG dataset containing 982 expert marked events in 4 days of data show that 90% of events can be correctly recorded while achieving a 50% data reduction. The described algorithm is formulated to have a direct, low power, hardware implementation and similar data reduction strategies could be employed in a range of body-area-network-type applications.