Scientific Data Management and Application in High Energy Physics

Abstract

High energy physics experiments have been producing a large amount of data at PB or EB level, and there will be ambitious experimental programs in the coming decades. The efficiency of data-intensive researches is closely related to how fast data can be accessed and how many computational resources can be used. Changes in computing technology and large increases in data volume require new computing models. This paper will give an overall introduction to scientific data management technologies and applications in high energy physics. The current data management framework and workflow will be investigated at first. These include data acquisition, data transfer, data storage, data processing, data sharing and data preservation. Then some ongoing research and development on data organization, management and access will be introduced. Finally the EventDB, an event-based big scientific data management system will be introduced. The test on more than ten billion physics events shows the query speed is greatly improved than traditional file-base data management system.