Data Processing Chain Research Articles

Results from the First Phase of the Seafloor Backscatter Processing Software Inter-Comparison Project

Seafloor backscatter mosaics are now routinely produced from multibeam echosounder data and used in a wide range of marine applications. However, large differences (>5 dB) can often be observed between the mosaics produced by different software packages processing the same dataset. Without transparency of the processing pipeline and the lack of consistency between software packages raises concerns about the validity of the final results. To recognize the source(s) of inconsistency between software, it is necessary to understand at which stage(s) of the data processing chain the differences become substantial. To this end, willing commercial and academic software developers were invited to generate intermediate processed backscatter results from a common dataset, for cross-comparison. The first phase of the study requested intermediate processed results consisting of two stages of the processing sequence: the one-value-per-beam level obtained after reading the raw data and the level obtained after radiometric corrections but before compensation of the angular dependence. Both of these intermediate results showed large differences between software solutions. This study explores the possible reasons for these differences and highlights the need for collaborative efforts between software developers and their users to improve the consistency and transparency of the backscatter data processing sequence.

Airborne Oblique Imaging: Towards the Hybrid Era

Abstract If the use of oblique aerial camera systems is steadily growing for 3D capture of urban areas, their combination with a LiDAR unit seems to have all the potential to lead the airborne mapping sector a step forward. To fully exploit the complementary sensor behaviour, a new perspective should be adopted that looks beyond the traditional data processing chains and extends them towards an hybrid data processing concept. Assisted tie point matching, integrated sensor orientation and augmented 3D reconstruction are the keystones of a rigorous hybrid workflow for hybrid sensors. They should all rely on a deep understanding of the different properties of active and passive 3D imaging, and of the uncertainty components in their measurements. The paper will focus on the most recent answers to these issues, that open new opportunities for boosting the quality of the geospatial products w.r.t completeness, geometric quality, object detection and processing efficiency

Benefits of the Open-Loop Tracking Command (OLTC): Extending conventional nadir altimetry to inland waters monitoring

Abstract In the past 25 years, radar altimeters have been designed and operated primarily for ocean observation. However, over the past decade there has been a growing interest for altimetry measurements over inland waters. Studying lakes, reservoirs and rivers water level is of prime importance for the hydrology community to assess the Earth’s global resources of fresh water. Satellite altimetry is key to providing such global and continuous datasets of water surface height. Indeed, much progress has been made in altimeters capability to acquire quality measurements over inland waters. In this paper we present an overview of major technical evolutions of the tracking function of altimeters, from Jason-2/POS3 and SARAL/AltiKa to Jason-3/POS3B and Sentinel-3/SRAL and the improvements brought by the Open-Loop Tracking Command (OLTC) to extend the altimeter observation domain to inland waters. The OLTC is an on-board feature used to set the altimeter waveforms reception window, containing a priori elevation information built from a dedicated database of hydrological targets. This paper focuses on the use of OLTC for acquiring reliable water signal rather than discussing the accuracy of the retrieved height, which constitutes a further step in the altimeter data processing chain. We show how Jason-3 and Sentinel-3 altimetry missions are currently able to observe and monitor about 70,000 hydrological targets over the globe and how it is contributing in building a global dataset of inland waters level, in preparation for future missions such as Sentinel-6 and the Surface Water Ocean Topography mission (SWOT).

Calibration, Conversion, and Quantitative Multi-Layer Inversion of Multi-Coil Rigid-Boom Electromagnetic Induction Data.

Multi-coil electromagnetic induction (EMI) systems induce magnetic fields below and above the subsurface. The resulting magnetic field is measured at multiple coils increasingly separated from the transmitter in a rigid boom. This field relates to the subsurface apparent electrical conductivity (σa), and σa represents an average value for the depth range investigated with a specific coil separation and orientation. Multi-coil EMI data can be inverted to obtain layered bulk electrical conductivity models. However, above-ground stationary influences alter the signal and the inversion results can be unreliable. This study proposes an improved data processing chain, including EMI data calibration, conversion, and inversion. For the calibration of σa, three direct current resistivity techniques are compared: Electrical resistivity tomography with Dipole-Dipole and Schlumberger electrode arrays and vertical electrical soundings. All three methods obtained robust calibration results. The Dipole-Dipole-based calibration proved stable upon testing on different soil types. To further improve accuracy, we propose a non-linear exact EMI conversion to convert the magnetic field to σa. The complete processing workflow provides accurate and quantitative EMI data and the inversions reliable estimates of the intrinsic electrical conductivities. This improves the ability to combine EMI with, e.g., remote sensing, and the use of EMI for monitoring purposes.

Comparison of MongoDB and Cassandra Databases for Spectrum Monitoring As-a-Service

Due to the growing number of devices accessing the Internet through wireless networks, the radio spectrum has become a highly contended resource. The availability of low cost radio spectrum monitoring sensors enables a geographically distributed, real-time observation of the spectrum to spot inefficiencies and to develop new strategies for its utilization. The potentially large number of sensors to be deployed and the intrinsic nature of data make this task a Big Data problem. In this work we design, implement, and validate a hardware and software architecture for wideband radio spectrum monitoring inspired to the Lambda architecture. This system offers Spectrum Sensing as a Service to let end users easily access and process radio spectrum data. To minimize the latency of services offered by the platform, we fine tune the data processing chain. From the analysis of sensor data characteristics, we design the data models for MongoDB and Cassandra, two popular NoSQL databases. A MapReduce job for spectrum visualization has been developed to show the potential of our approach and to identify the challenges in processing spectrum sensor data. We experimentally evaluate and compare the performance of the two databases in terms of application processing time for different types of queries applied on data streams with heterogeneous generation rate. Our experiments show that Cassandra outperforms MongoDB in most cases, with some exceptions depending on data stream rate.

Wearable-Based Affect Recognition-A Review.

Affect recognition is an interdisciplinary research field bringing together researchers from natural and social sciences. Affect recognition research aims to detect the affective state of a person based on observables, with the goal to, for example, provide reasoning for the person’s decision making or to support mental wellbeing (e.g., stress monitoring). Recently, beside of approaches based on audio, visual or text information, solutions relying on wearable sensors as observables, recording mainly physiological and inertial parameters, have received increasing attention. Wearable systems enable an ideal platform for long-term affect recognition applications due to their rich functionality and form factor, while providing valuable insights during everyday life through integrated sensors. However, existing literature surveys lack a comprehensive overview of state-of-the-art research in wearable-based affect recognition. Therefore, the aim of this paper is to provide a broad overview and in-depth understanding of the theoretical background, methods and best practices of wearable affect and stress recognition. Following a summary of different psychological models, we detail the influence of affective states on the human physiology and the sensors commonly employed to measure physiological changes. Then, we outline lab protocols eliciting affective states and provide guidelines for ground truth generation in field studies. We also describe the standard data processing chain and review common approaches related to the preprocessing, feature extraction and classification steps. By providing a comprehensive summary of the state-of-the-art and guidelines to various aspects, we would like to enable other researchers in the field to conduct and evaluate user studies and develop wearable systems.

The offline data quality monitoring of the BESIII end cap TOF system

The end-cap time-of-flight (TOF) at Beijing Spectrometer was upgraded with multi-gap resistive plate chamber technology in order to improve the particle identification capability in 2015. The offline data quality monitoring (ODQM) is a critical aspect of the data processing chain aiming at providing data with good quality for physics analyses. An ODQM tool for upgraded end-cap TOF has been developed to provide feedback about the functioning and performance of detector hardware and data processing chain. Detector information and reconstructed time-of-flight characteristics of charged tracks are filled into plots using full Bhabha events’ reconstruction results, and then, these plots are used to assess the operational conditions of the detector and the quality of the data by the experts. This paper describes the design and the content of performance of the ODQM of end-cap TOF and the data quality performance achieved during the last 2-year physical data taking.

SIFSpec: Measuring Solar-Induced Chlorophyll Fluorescence Observations for Remote Sensing of Photosynthesis.

Solar-induced chlorophyll fluorescence (SIF) is regarded as a proxy for photosynthesis in terrestrial vegetation. Tower-based long-term observations of SIF are very important for gaining further insight into the ecosystem-specific seasonal dynamics of photosynthetic activity, including gross primary production (GPP). Here, we present the design and operation of the tower-based automated SIF measurement (SIFSpec) system. This system was developed with the aim of obtaining synchronous SIF observations and flux measurements across different terrestrial ecosystems, as well as to validate the increasing number of satellite SIF products using in situ measurements. Details of the system components, instrument installation, calibration, data collection, and processing are introduced. Atmospheric correction is also included in the data processing chain, which is important, but usually ignored for tower-based SIF measurements. Continuous measurements made across two growing cycles over maize at a Daman (DM) flux site (in Gansu province, China) demonstrate the reliable performance of SIF as an indicator for tracking the diurnal variations in photosynthetically active radiation (PAR) and seasonal variations in GPP. For the O2–A band in particular, a high correlation coefficient value of 0.81 is found between the SIF and seasonal variations of GPP. It is thus concluded that, in coordination with continuous eddy covariance (EC) flux measurements, automated and continuous SIF observations can provide a reliable approach for understanding the photosynthetic activity of the terrestrial ecosystem, and are also able to bridge the link between ground-based optical measurements and airborne or satellite remote sensing data.

Super-resolution radar imaging for breast cancer detection with microwaves: the integrated information selection criteria.

This paper focuses on the data preprocessing scheme, as well as on the frequency selection and spatial filtering modules integrated with a Time-Reversal Multiple Signal Classification (TR-MUSIC) algorithm, for microwave breast imaging. This algorithm is part of the data processing chain of the Wavelia Microwave Breast Imaging (MBI) system prototype, which has been recently installed at the University Hospital of Galway, Ireland, for a first-in-human clinical trial. Indicative results from application of the algorithm on an experimental phantom dataset, and on a first patient dataset, are presented in this paper. Good correspondence between the two datasets is demonstrated, confirming the validity of the experimental setup used so far for the on-site acceptance of the Wavelia system, after installation at the hospital for clinical testing.

The Gaia ultracool dwarf sample – II. Structure at the end of the main sequence

ABSTRACT We identify and investigate known late M, L, and T dwarfs in the Gaia second data release. This sample is being used as a training set in the Gaia data processing chain of the ultracool dwarfs work package. We find 695 objects in the optical spectral range M8–T6 with accurate Gaia coordinates, proper motions, and parallaxes which we combine with published spectral types and photometry from large area optical and infrared sky surveys. We find that 100 objects are in 47 multiple systems, of which 27 systems are published and 20 are new. These will be useful benchmark systems and we discuss the requirements to produce a complete catalogue of multiple systems with an ultracool dwarf component. We examine the magnitudes in the Gaia passbands and find that the GBP magnitudes are unreliable and should not be used for these objects. We examine progressively redder colour–magnitude diagrams and see a notable increase in the main-sequence scatter and a bivariate main sequence for old and young objects. We provide an absolute magnitude – spectral subtype calibration for G and GRP passbands along with linear fits over the range M8–L8 for other passbands.

Regularised patient-specific stopping power calibration for proton therapy planning based on proton radiographic images

Proton transmission imaging has been proposed and investigated as imaging modality complementary to x-ray based techniques in proton beam therapy. In particular, it addresses the issue of range uncertainties due to the conversion of an x-ray patient computed tomography (CT) image expressed in Hounsfield Units (HU) to relative stopping power (RSP) needed as input to the treatment planning system. One approach to exploit a single proton radiographic projection is to perform a patient-specific calibration of the CT to RSP conversion curve by optimising the match between a measured and a numerically integrated proton radiography.In this work, we develop the mathematical tools needed to perform such an optimisation in an efficient and robust way. Our main focus lies on set-ups which combine pencil beam scanning with a range telescope detector, although most of our methods can be employed in combination with other set-ups as well. Proton radiographies are simulated in Monte Carlo using an idealised detector and applying the same data processing chain used with experimental data. This approach allows us to have a ground truth CT-RSP curve to compare the optimisation results with.Our results show that the parameters of the CT-RSP curve are strongly correlated when using a pencil beam based set-up, which leads to unrealistic variation in the optimised CT-RSP curves. To address this issue, we introduce a regularisation procedure which guarantees a plausible degree of smoothness in the optimised CT-RSP curves. We investigate three different methods to perform the numerical projection operation needed to generate a proton digitally reconstructed radiography. We find that the approximate and computationally faster method performs as well as the more accurate but more demanding method. We perform a Monte Carlo experiment based on a head and neck patient to evaluate the range accuracy achievable with the optimised CT-RSP curves and find an agreement with the ground truth expectation of better than . Our results further indicate that the region in the patient in which the proton radiography is acquired does not necessarily have to correspond to the treatment volume to achieve this accuracy. This is important as the imaged region could be freely chosen, e.g. in order to spare organs at risk.

Toward a multimodal fusion of layered cultural object images: complementarity of optical coherence tomography and terahertz time-domain imaging in the heritage field.

Terahertz time-domain imaging (THz-TDI) and spectral-domain optical coherence tomography (SD-OCT) are two techniques capable of providing 3D datasets from which depth profiles and cross-sectional images of an object can be derived. They are novel photonics technologies of particular relevance to the field of heritage science, for which the comprehension of the stratigraphic structure of a cultural heritage object may help in the understanding of its artistic technology and state of preservation. The differences in imaging depth, field of view, and axial/lateral resolutions of the two imaging techniques provide different but complementary information of the same scene. Through the use of multimodal image fusion, the user will benefit from access to a detailed comparison of the information content of the two different datasets and images. To carry this out, we have developed a first data processing chain with the aim of representing THz-TDI and SD-OCT cross-sectional images in a unique image grid and space, in which the different datasets are dimensionally consistent.

Evolution of the ALICE Software Framework for Run 3

ALICE is one of the four major LHC experiments at CERN. When the accelerator enters the Run 3 data-taking period, starting in 2021, ALICE expects almost 100 times more Pb-Pb central collisions than now, resulting in a large increase of data throughput. In order to cope with this new challenge, the collaboration had to extensively rethink the whole data processing chain, with a tighter integration between Online and Offline computing worlds. Such a system, code-named ALICE O2, is being developed in collaboration with the FAIR experiments at GSI. It is based on the ALFA framework which provides a generalized implementation of the ALICE High Level Trigger approach, designed around distributed software entities coordinating and communicating via message passing. We will highlight our efforts to integrate ALFA within the ALICE O2 environment. We analyze the challenges arising from the different running environments for production and development, and conclude on requirements for a flexible and modular software framework. In particular we will present the ALICE O2 Data Processing Layer which deals with ALICE specific requirements in terms of Data Model. The main goal is to reduce the complexity of development of algorithms and managing a distributed system, and by that leading to a significant simplification for the large majority of the ALICE users.

3D convolutional GAN for fast simulation

Machine Learning techniques have been used in different applications by the HEP community: in this talk, we discuss the case of detector simulation. The need for simulated events, expected in the future for LHC experiments and their High Luminosity upgrades, is increasing dramatically and requires new fast simulation solutions. We describe an R&D activity aimed at providing a configurable tool capable of training a neural network to reproduce the detector response and speed-up standard Monte Carlo simulation. This represents a generic approach in the sense that such a network could be designed and trained to simulate any kind of detector and, eventually, the whole data processing chain in order to get, directly in one step, the final reconstructed quantities, in just a small fraction of time. We present the first application of three-dimensional convolutional Generative Adversarial Networks to the simulation of high granularity electromagnetic calorimeters. We describe detailed validation studies comparing our results to Geant4 Monte Carlo simulation. Finally we show how this tool could be generalized to describe a whole class of calorimeters, opening the way to a generic machine learning based fast simulation approach.

Porting the LHCb Stack from x86 (Intel) to aarch64 (ARM) and ppc64le (PowerPC)

LHCb is undergoing major changes in its data selection and processing chain for the upcoming LHC Run 3 starting in 2021. With this in sight several initiatives have been launched to optimise the software stack. This contribution discusses porting the LHCb Stack from x86_64 architecture to both architectures aarch64 and ppc64le with the goal to evaluate the performance and the cost of the computing infrastructure for the High Level Trigger (HLT). This requires porting a stack with more than five million lines of code and finding working versions of external libraries provided by LCG. Across all software packages the biggest challenge is the growing use of vectorisation - as many vectorisation libraries are specialised on x86 architecture and do not have any support for other architectures. In spite of these challenges we have successfully ported the LHCb High Level Trigger code to aarch64 and ppc64le. This contribution discusses the status and plans for the porting of the software as well as the LHCb approach for tackling code vectorisation in a platform independent way.

RDataFrame: Easy Parallel ROOT Analysis at 100 Threads

The Physics programmes of LHC Run III and HL-LHC challenge the HEP community. The volume of data to be handled is unprecedented at every step of the data processing chain: analysis is no exception. Physicists must be provided with first-class analysis tools which are easy to use, exploit bleeding edge hardware technologies and allow to seamlessly express parallelism. This document discusses the declarative analysis engine of ROOT, RDataFrame, and gives details about how it allows to profitably exploit commodity hardware as well as high-end servers and manycore accelerators thanks to the synergy with the existing parallelised ROOT components. Real-life analyses of LHC experiments’ data expressed in terms of RDataFrame are presented, highlighting the programming model provided to express them in a concise and powerful way. The recent developments which make RDataFrame a lightweight data processing framework are described, such as callbacks and I/O capabilities. Finally, the flexibility of RDataFrame and its ability to read data formats other than ROOT’s are characterised, as an example it is discussed how RDataFrame can directly read and analyse LHCb’s raw data format MDF.

Advanced Ultraviolet Radiation and Ozone Retrieval for Applications (AURORA): A Project Overview

With the launch of the Sentinel-5 Precursor (S-5P, lifted-off on 13 October 2017), Sentinel-4 (S-4) and Sentinel-5 (S-5)(from 2021 and 2023 onwards, respectively) operational missions of the ESA/EU Copernicus program, a massive amount of atmospheric composition data with unprecedented quality will become available from geostationary (GEO) and low Earth orbit (LEO) observations. Enhanced observational capabilities are expected to foster deeper insight than ever before on key issues relevant for air quality, stratospheric ozone, solar radiation, and climate. A major potential strength of the Sentinel observations lies in the exploitation of complementary information that originates from simultaneous and independent satellite measurements of the same air mass. The core purpose of the AURORA (Advanced Ultraviolet Radiation and Ozone Retrieval for Applications) project is to investigate this exploitation from a novel approach for merging data acquired in different spectral regions from on board the GEO and LEO platforms. A data processing chain is implemented and tested on synthetic observations. A new data algorithm combines the ultraviolet, visible and thermal infrared ozone products into S-4 and S-5(P) fused profiles. These fused products are then ingested into state-of-the-art data assimilation systems to obtain a unique ozone profile in analyses and forecasts mode. A comparative evaluation and validation of fused products assimilation versus the assimilation of the operational products will seek to demonstrate the improvements achieved by the proposed approach. This contribution provides a first general overview of the project, and discusses both the challenges of developing a technological infrastructure for implementing the AURORA concept, and the potential for applications of AURORA derived products, such as tropospheric ozone and UV surface radiation, in sectors such as air quality monitoring and health.

Pattern Recognition Scheme for Large-Scale Cloud Detection Over Landmarks

Landmark recognition and matching is a critical step in many image navigation and registration models for geostationary satellite services, as well as to maintain the geometric quality assessment in the instrument data processing chain of earth observation satellites. Matching the landmark accurately is of paramount relevance, and the process can be strongly impacted by the cloud contamination of a given landmark. This paper introduces a complete pattern recognition methodology able to detect the presence of clouds over landmarks using Meteosat second generation (MSG) data. The methodology is based on the ensemble combination of dedicated support vector machines dependent on the particular landmark and illumination conditions. This divide-and-conquer strategy is motivated by the data complexity and follows a physically based strategy that considers variability both in seasonality and illumination conditions along the day to split observations. In addition, it allows training the classification scheme with millions of samples at an affordable computational costs. The image archive was composed of 200 landmark test sites with near 7 million multispectral images that correspond to MSG acquisitions during 2010. Results are analyzed in terms of cloud detection accuracy and computational cost. We provide illustrative source code and a portion of the huge training data to the community.

Generative models for fast simulation

Machine Learning techniques have been used in different applications by the HEP community: in this talk, we discuss the case of detector simulation. The need for simulated events, expected in the future for LHC experiments and their High Luminosity upgrades, is increasing dramatically and requires new fast simulation solutions. We will present results of several studies on the application of computer vision techniques to the simulation of detectors, such as calorimeters. We will also describe a new R&D activity, within the GeantV project, aimed at providing a configurable tool capable of training a neural network to reproduce the detector response and replace standard Monte Carlo simulation. This represents a generic approach in the sense that such a network could be designed and trained to simulate any kind of detector and, eventually, the whole data processing chain in order to get, directly in one step, the final reconstructed quantities, in just a small fraction of time. We will present the first three-dimensional images of energy showers in a high granularity calorimeter, obtained using Generative Adversarial Networks.

Multi-sensors measuring system for geodetic monitoring of elevator guide rails

The elevator guide rails geometry, which covers such features as rectilinearity, verticality, parallelism and span, is of key importance as uninterrupted, failure-free operation of elevators and safety of elevators’ users are prerequisites. The geometric parameters have to measured and controlled periodically according to elevators industry standards. The paper discusses the construction and principles of operation of developed Multi-Sensors Measuring System (MSMS) in the control of geometric parameters of elevator guide rails. The system is made of laser rangefinders and two laser emitters with a collimation system. To detect and register the location of laser spots the system uses CCD/CMOS detectors. The elements of the system are deployed in the elevator shaft. This solution allows you to control the position of the laser beams in the external reference system and to determine the torsion angles of the guide rails in the elevator shaft. Algorithm and data processing chain are also provided in the paper. The experimental results demonstrate that the system ensure elevator check according to current industry standard.