Airline Datasets Research Articles

Sentiment analysis for airline twitter based of machine learning and deep learning

The study of public opinion can be beneficial for obtaining certain knowledge. Thus, the sentiment analysis of the social networks, for example, the Twitter or Facebook has grown into an effective way of understanding users’ opinion and has many uses. Nevertheless, the efficiency and accuracy of sentiment analysis seem to be hampered by the problems rising from natural language processing processes that are inherent with the text. Currently the airline sector is considered the significant field of the market. To sustain that sector and constantly update it, mind mining becomes inevitable. This paper, proposed a model for sentiment analysis based on extracting two different features. Term frequency-inverse document frequency and Word2vec. These feature introduced separately to different classifiers to classify the sentences as positive, negative, or neutral. Twitter US Airline dataset used to evaluate the performance of the proposed model. Bi-directional Long short Term memory outperformed others methods with recall, Precision, and F-Score reached to 0.97, 0.98,and 0.97 respectively when using Word2vec feature.

GenCo: A Generative Learning Model for Heterogeneous Text Classification Based on Collaborative Partial Classifications

The use of generative learning models in natural language processing (NLP) has significantly contributed to the advancement of natural language applications, such as sentimental analysis, topic modeling, text classification, chatbots, and spam filtering. With a large amount of text generated each day from different sources, such as web-pages, blogs, emails, social media, and articles, one of the most common tasks in NLP is the classification of a text corpus. This is important in many institutions for planning, decision-making, and creating archives of their projects. Many algorithms exist to automate text classification tasks but the most intriguing of them is that which also learns these tasks automatically. In this study, we present a new model to infer and learn from data using probabilistic logic and apply it to text classification. This model, called GenCo, is a multi-input single-output (MISO) learning model that uses a collaboration of partial classifications to generate the desired output. It provides a heterogeneity measure to explain its classification results and enables a reduction in the curse of dimensionality in text classification. Experiments with the model were carried out on the Twitter US Airline dataset, the Conference Paper dataset, and the SMS Spam dataset, outperforming baseline models with 98.40%, 89.90%, and 99.26% accuracy, respectively.

Departure Delay Prediction using Machine Learning

Abstract: The project aims to investigate forecasting strategies for forecasting weather-related planes delays. In order to plan ahead and reduce the effects of disruptions, it is essential for airlines and travelers to be able predict such delays with accuracy. The research focuses on developing an ensemble model-based machine learning flight delay prediction system. The Airline dataset is subjected to the use of three distinct machine learning methods: Random Forest Clasifier, k-nearest-neighbor (KNN), and Support Vector Machine (SVM). The suggested approach is then assessed for efficiency and outcomes using a comparative analysis

On the asymptotic properties of a bagging estimator with a massive dataset

Bagging is a useful method for large‐scale statistical analysis, especially when the computing resources are very limited. We study here the asymptotic properties of bagging estimators for ‐estimation problems but with massive datasets. We theoretically prove that the resulting estimator is consistent and asymptotically normal under appropriate conditions. The results show that the bagging estimator can achieve the optimal statistical efficiency, provided that the bagging subsample size and the number of subsamples are sufficiently large. Moreover, we derive a variance estimator for valid asymptotic inference. All theoretical findings are further verified by extensive simulation studies. Finally, we apply the bagging method to the US Airline Dataset to demonstrate its practical usefulness.

Using Hadoop, Analyze Airport Authority

A modern airport has access to a vast quantity of information, including the number of flights, arrival and departure times, flight routes, the number of airports that operate in each nation, a list of the current active airlines in each nation, etc. DBMS has so far been utilized to store and analyze the airport database. An airport database management system uses SQL to apply queries to all of the records. B-tree or distributed hash tables are used by My SQL to analyze the database. It is unable to analyze a large volume of data. Data is stored in a parallel database. This system's disadvantage is that it can't handle an increase in massive amounts of data to a significant degree. Therefore, the Hadoop database file system is employed to solve this issue. Hadoop uses HDFS and the Map-Reduce processing architecture to store vast amounts of data and provide the data in parallel. We made an effort to examine in-depth research of airline data sets, such as a list of airports in India, a list of airlines with no stops, a list of codeshare airlines by country with the most airports, and a list of active airlines in the United States. Here, we concentrated on processing large data sets in a distributed setting utilizing the hive component of the Hadoop ecosystem. The developers and business analysts will be helped by this effort when it comes to obtaining and handling their user requests

Adaptive windowing based recurrent neural network for drift adaption in non-stationary environment.

In today’s digital era, many applications generate massive data streams that must be sequenced and processed immediately. Therefore, storing large amounts of data for analysis is impractical. Now, this infinite amount of evolving data confronts concept drifts in data stream classification. Concept drift is a phenomenon in which the distribution of input data or the relationship between input data and target label changes over time. If the drifts are not addressed, the learning model’s performance suffers. Non-stationary data streams must be processed as they arrive, and neural networks’ built-in capabilities aid in the processing of huge non-stationary data streams. We proposed an adaptive windowing approach based on a gated recurrent unit, a variant of the recurrent neural network incrementally trained on incoming data (for the real-world airline and synthetic Streaming Ensemble Algorithm (SEA) datasets), and employed elastic weight consolidation with the Fisher information matrix to prevent forgetting. Unlike the traditional fixed window methodology, the proposed model dynamically increases the window size if the prediction is correct and reduces it if drifts occur. As a result, an adaptive recurrent neural network model can adapt to changes in the non-stationary data stream and provide consistent performance. Moreover, the findings revealed that on the airline and the SEA dataset, the proposed model outperforms state-of-the-art methods by achieving 67.74% and 91.70% accuracy, respectively. Further, the results demonstrated that the proposed model has a better accuracy of 3.6% and 1.6% for the SEA and the airline dataset, respectively.

Sentiment Prediction of Textual Data Using Hybrid ConvBidirectional-LSTM Model

With the emergence of social media platforms, most people have changed their way of interacting. Perhaps, sharing day-to-day lifestyle updates is a trend substantially influenced by microblogging sites, specifically Twitter, Facebook, Instagram, and many more. Moreover, text and messages are the most preferred way for such interactions. Twitter is one of the most commonly used microblogging tools that enable people to express their thoughts, opinions, emotions, happiness, sadness, excitement, ideas, mental stress, and so on. Hence, the sentiment prediction furnished by such textual data becomes a complex and challenging task. In this research, the authors proposed a hybridization of the convolutional neural network and bi-directional long short-term memory model (named ConvBidirectional-LSTM), which aims to better the categorization of sentiments of text data. Then, this proposed hybrid ConvBidirectional-LSTM model is compared with the existing state-of-the-art models, GloVe-based CNN-LSTM and Hierarchical Bi-LSTM (HeBiLSTM) models. Furthermore, the performance of the proposed hybrid ConvBidirectional-LSTM model is evaluated on the US airline dataset using various performance parameters like accuracy, precision, recall, and f 1 score. The proposed model outperformed the existing state-of-the-art models with an accuracy rate of 93.25% in sentiment prediction.

Feature Screening for Massive Data Analysis by Subsampling

Modern statistical analysis often encounters massive datasets with ultrahigh-dimensional features. In this work, we develop a subsampling approach for feature screening with massive datasets. The approach is implemented by repeated subsampling of massive data and can be used for analyzing tasks with memory constraints. To conduct the procedure, we first calculate an R-squared screening measure (and related sample moments) based on subsamples. Second, we consider three methods to combine the local statistics. In addition to the simple average method, we design a jackknife debiased screening measure and an aggregated moment screening measure. Both approaches reduce the bias of the subsampling screening measure and therefore increase the accuracy of the feature screening. Last, we consider a novel sequential sampling method, that is more computationally efficient than the traditional random sampling method. The theoretical properties of the three screening measures under both sampling schemes are rigorously discussed. Finally, we illustrate the usefulness of the proposed method with an airline dataset containing 32.7 million records.

Enhanced sentiment extraction architecture for social media content analysis using capsule networks.

Recent research has produced efficient algorithms based on deep learning for text-based analytics. Such architectures could be readily applied to text-based social media content analysis. The deep learning techniques, which require comparatively fewer resources for language modeling, can be effectively used to process social media content data that change regularly. Convolutional Neural networks and recurrent neural networks based approaches have reported prominent performance in this domain, yet their limitations make them sub-optimal. Capsule networks sufficiently warrant their applicability in language modelling tasks as a promising technique beyond their initial usage of image classification. This study proposes an approach based on capsule networks for social media content analysis, especially for Twitter. We empirically show that our approach is optimal even without the use of any linguistic resources. The proposed architectures produced an accuracy of 86.87% for the Twitter Sentiment Gold dataset and an accuracy of 82.04% for the CrowdFlower US Airline dataset, indicating state-of-the-art performance. Hence, the research findings indicate noteworthy accuracy enhancement for text processing within social media content analysis.

Least-Square Approximation for a Distributed System

In this work, we develop a distributed least-square approximation (DLSA) method that is able to solve a large family of regression problems (e.g., linear regression, logistic regression, and Cox’s model) on a distributed system. By approximating the local objective function using a local quadratic form, we are able to obtain a combined estimator by taking a weighted average of local estimators. The resulting estimator is proved to be statistically as efficient as the global estimator. Moreover, it requires only one round of communication. We further conduct a shrinkage estimation based on the DLSA estimation using an adaptive Lasso approach. The solution can be easily obtained by using the LARS algorithm on the master node. It is theoretically shown that the resulting estimator possesses the oracle property and is selection consistent by using a newly designed distributed Bayesian information criterion. The finite sample performance and computational efficiency are further illustrated by an extensive numerical study and an airline dataset. The airline dataset is 52 GB in size. The entire methodology has been implemented in Python for a de-facto standard Spark system. The proposed DLSA algorithm on the Spark system takes 26 min to obtain a logistic regression estimator, which is more efficient and memory friendly than conventional methods. Supplementary materials for this article are available online.

Particle swarm optimization for airlines fleet assignment

Airline fleet assignment is the process of assigning aircraft types to scheduled flight legs in order to minimize operating cost and achieve maximize revenue, while satisfying a set of constraints. This paper formulate the fleet assignment problem for airlines that optimization goal is to minimize the total assignment cost. Particle swarm optimization proposed to solve this model. The model successfully applied to Egyptair airline dataset using the particle swarm optimization and mixed integer programming. The proposed method compared with mixed integer programming and current Egyptair assignment methodology. The results showed that the particle swarm optimization is the best method for the Egyptair fleet assignment process. The solution quality is better than mixed integer programming and Egyptair assignment methodology where we saw a daily cost reduction with a percentage of 14.6% and 19.3% respectively.

Competitive revenue management models with loyal and fully flexible customers

Developing practical models for capturing competitive effects in revenue management and pricing systems has been a significant challenge for airlines and other industries. The prevalent mechanisms of accounting for competitive effects rely on changing the price structure and making manual adjustments to respond to dynamically evolving competitive scenarios. Furthermore, micro-economic models have also not become popular in practice primarily because of the simplistic mechanisms proposed for modeling consumer behavior in a competitive setting. In particular, many of these models assume that the customers always seek the lowest price in the market, that is they are fully flexible. In practice, customers may display some degree of affinity or loyalty to an airline and may pay a premium for their preferred choice. On the other hand, almost all early revenue management models did not explicitly consider competitive effects and assumed that an airline’s demand only depends on their prices i.e., demand is fully dedicated to an airline (loyal). This paper develops a model to capture more realistic competitive dynamics by including both these types of customer behavior. We also develop a Bayesian machine learning based demand forecasting methodology for such models with explicit competitive considerations and show the benefit of this approach over traditional models on a real airline data set.

An Automatic Convolutional Neural Network Optimization Using a Diversity-Guided Genetic Algorithm

Hyperparameters and architecture greatly influence the performance of convolutional neural networks (CNNs); therefore, their optimization is important to obtain the desired results. One of the state-of-the-art methods to achieve this is the use of neuroevolution that utilizes a genetic algorithm (GA) to optimize a CNN. However, the GA is often trapped into a local optimum resulting in premature convergence. In this study, we propose an approach called the “diversity-guided genetic algorithm-convolutional neural network (DGGA-CNN)” that uses adaptive parameter control and random injection to facilitate the search process by exploration and exploitation while preserving the population diversity. The alternation between exploration and exploitation is guided by using an average pairwise Hamming distance. Moreover, the DGGA fully handles the architecture of the CNN by using a novel finite state machine (FSM) combined with three novel mutation mechanisms that are specifically created for architecture chromosomes. Tests conducted on suggestion mining and twitter airline datasets reveal that the DGGA-CNN performs well with valid architectures and a comparison with other methods demonstrates its capability and efficiency.

Discovering predictive ensembles for transfer learning and meta-learning

Recent meta-learning approaches are oriented towards algorithm selection, optimization or recommendation of existing algorithms. In this article we show how data-tailored algorithms can be constructed from building blocks on small data sub-samples. Building blocks, typically weak learners, are optimized and evolved into data-tailored hierarchical ensembles. Good-performing algorithms discovered by evolutionary algorithm can be reused on data sets of comparable complexity. Furthermore, these algorithms can be scaled up to model large data sets. We demonstrate how one particular template (simple ensemble of fast sigmoidal regression models) outperforms state-of-the-art approaches on the Airline data set. Evolved hierarchical ensembles can therefore be beneficial as algorithmic building blocks in meta-learning, including meta-learning at scale.

Predicting Fuel Consumption and Flight Delays for Low-Cost Airlines

Low-cost airlines (LCAs) represent a new category of airlines that provides low-fare flights. The rise and growth of LCAs has intensified the price competition among airlines, and LCAs require continuous efforts to reduce their operating costs to lower flight prices; however, LCA passengers still demand high-quality services. A common measure of airline service quality is on-time departure performance. Because LCAs apply efficient aircraft utilization and the time between flights is likely to be small, additional effort is required to avoid flight delays and improve their service quality. In this paper, we apply state-of-the-art predictive modeling approaches to real airline datasets and investigate the feasibility of machine learning methods for cost reduction and service quality improvement in LCAs. We address two prediction problems: fuel consumption prediction and flight delay prediction. We train predictive models using flight and passenger information, and our experiment results show that our regression model predicts the amount of fuel consumption more accurately than flight dispatchers, and our binary classifier achieves an area under the ROC curve (AUC) of 0.75 for predicting a delay of a specific flight route.

Impact of national culture on airline operations

Organizations absorb their nations’ culture, norms, and beliefs; and therefore culture has unbeknownst influence on the process of service design and delivery. In this paper, we test the role of a nation’s culture on airline service quality. We use a non-perceptual global, airline dataset where cultural differences are measured by Hofstede’s and the GLOBE project’s national characteristics. As indicators of service quality, we use assessments of an independent evaluation agency. We find that cultures that place high value on future orientation provide higher quality airline services. On the other hand, cultures that are high in individualism and uncertainty avoidance tend to perform poorly. High individualism might impair true concern for passenger welfare, and high uncertainty avoidance means that employees are apprehensive towards the constant changes and fine-tuning necessary in airline services. We also determine that the implications of national culture may vary by passenger segments identified by cabin types.

Frequent Item Set Mining Using INC_MINE in Massive Online Analysis Frame Work

Frequent Pattern Mining is one of the major data mining techniques, which is exhaustively studied in the past decade. The technological advancements have resulted in huge data generation, having increased rate of data distribution. The generated data is called as a ‘data stream’. Data streams can be mined only by using sophisticated techniques. The paper aims at carrying out frequent pattern mining on data streams. Stream mining has great challenges due to high memory usage and computational costs. Massive online analysis frame work is a software environment used to perform frequent pattern mining using INC_MINE algorithm. The algorithm uses the method of closed frequent mining. The data sets used in the analysis are Electricity data set and Airline data set. The authors also generated their own data set, OUR-GENERATOR for the purpose of analysis and the results are found interesting. In the experiments five samples of instance sizes (10000, 15000, 25000, 35000, 50000) are used with varying minimum support and window sizes for determining frequent closed itemsets and semi frequent closed itemsets respectively. The present work establishes that association rule mining could be performed even in the case of data stream mining by INC_MINE algorithm by generating closed frequent itemsets which is first of its kind in the literature.

Impact of national culture on the quality of information delivery in services

A nation’s culture has a significant influence on an employee’s behavior and communication. Culture-specific characteristics cause information delivery biases, especially in services. This paper studies how a nation’s culture affects the quality of information delivery using a global airline dataset. Cultural differences are measured by two commonly used datasets, Hofstede and Globe projects. Our results show that cultural characteristics have statistically significant effects on the website quality, inflight media options, and inflight communications. We suggest firms in each culture become aware of these findings and step forward to remedy their shortcomings or to bolster their performance.

Customized Policies for Handling Partial Information in Relational Databases

Most real-world databases have at least some missing data. Today, users of such databases are “on their own” in terms of how they manage this incompleteness. In this paper, we propose the general concept of partial information policy (PIP) operator to handle incompleteness in relational databases. PIP operators build upon preference frameworks for incomplete information, but accommodate different types of incomplete data (e.g., a value exists but is not known; a value does not exist; a value may or may not exist). Different users in the real world have different ways in which they want to handle incompleteness-PIP operators allow them to specify a policy that matches their attitude to risk and their knowledge of the application and how the data was collected. We propose index structures for efficiently evaluating PIP operators and experimentally assess their effectiveness on a real-world airline data set. We also study how relational algebra operators and PIP operators interact with one another.

Evolving forecast combination structures for airline revenue management

Forecasting is at the heart of every revenue management system, providing necessary input to capacity control, pricing and overbooking functionalities. For airlines, the key to efficient capacity control is determining the time of when to restrict bookings in a lower-fare class to leave space for later booking high-fare customers. This work presents findings of a collaboration project between Bournemouth University and Lufthansa Systems AG, a company providing revenue management software for airline carriers. The main aim is to increase net booking forecast accuracy by modifying one of its components, the cancellation forecast. Complementing an available set of three traditional individual algorithms, an additional method is presented and added to the method pool. Furthermore, diversification of model parameters and level of learning is discussed to increase the number of individual forecasts even further. Finally, the evolution of forecast combination structures is investigated and shown to be beneficial on an airline data set.