Managing and analyzing large data sets

Abstract

Data is power, assuming you know how to use it. The organizations that will be most successful over the next decade are those that recognize this simple fact. The advantage provided by data that has been transformed into useful information is undeniable: information derived from measurement data can be used to drive differentiation, optimization, and innovation. The future's biggest data challenge lies in effectively managing acquired data to give context to what is otherwise just a series of numbers across thousands of files. Instrumentation hardware vendors have accelerated data collection rates so quickly and enabled engineers and scientists to break through data rate and resolution-barriers so rapidly that today, data is coming in at a truly blistering pace. Unfortunately, the same can't be said for the rate of education on proper data storage, management and post-processing techniques. As a result, many organizations have been forced to devise their own schemes for data storage and management since they've effectively been left to their own accord. As a further consequence, many groups within the same organizational entity have adopted conflicting data strategies; this fractured outcome makes it nearly impossible to yield efficiency gains from data collaboration and knowledge sharing. After a brief introduction to the significance of data in the world today and the challenges that engineers and scientists currently face, this presentation will educate the audience on several technologies and techniques that provide a more modular, scalable approach to data management and yield the following benefits: Acquired data will gain context: descriptive information stored throughout the data acquisition and processing life cycle makes data traceable, trackable, usable, and logical to store. Acquired data will be searchable: data mining makes it easy to locate the data that you need, when you need it. Acquired data will be easier to process: from analysis to reporting, data management reduces the many roadblocks in data processing, particularly as data sets grow in size.