Chapter 2 - Deep learning in big data and data mining

Abstract

The growth of the digital age has led to a colossal leap in data generated by the average user. This growing data has several applications: businesses can use it to give a more personalized touch to their services, governments can use it to better allocate their funds, and companies can utilize it to select the best candidates for a job. While these applications may seem extremely enticing, there are a couple of problems that must be solved first, namely, data collection and extraction of useful patterns from the data. The disciplines of data mining and big data deal with these problems, respectively. But, as we have already discussed, the amount of data is so vast that any manual approach is extremely time intensive and costly. Thus this limits the potential outcomes from this data. This problem has been solved by the application of deep learning. Deep learning has allowed us to automate processes that were not only time intensive but also mentally arduous. It has achieved better than human accuracy in several types of discriminative and recognition tasks making it a viable alternative to inefficient human labor. Deep learning plays a vital role in this analysis and has enabled several businesses to comprehend customer needs and accordingly improve their own services, thus giving them the opportunity to outdo their competitors. Similarly, deep learning has also been instrumental in analyzing the trends and associations of securities in the financial market. It has even helped to create fraud detection and loan underwriting applications, which have contributed to making financial institutions more transparent and efficient. Apart from directly improving the efficiency in these fields, deep learning has also been instrumental in improving the fields of data mining and big data. Machine learning algorithms can actually utilize the existing data to predict the unknowns, including future trends in data. Due to its potential applications the field of machine learning is deeply interconnected with data mining. Nevertheless, machine learning algorithms are often heavily dependent on the availability of huge datasets to ensure useful accuracy. Deep learning algorithms have allowed the different components of data (i.e., multimedia data) in the data mining process itself to be identified. Similarly, semantic indexing and tagging algorithms have allowed the processes of big data to speed up.