Big Data Optimization Research Articles

Quasi-singular methods for determining the limiting properties of the spacecraft entry trajectories into the planet’s atmosphere

The problem of the limiting dynamic properties determining of the spacecraft (SC) trajectories with non-zero aerodynamic quality entered the atmosphere of a planet (Earth, Venus, Mars etc.) often has singularities. First of all, the singularity of the optimal control and of the SC’ dynamic characteristics is appeared after the interplanetary flight or after the SC maneuver in deep space. The target landing point is either the regular landing site on the Earth, or is the selected region with the specified geological characteristics on the surface of the planet (for example, Venus). The bulk and very laborious numerical experiments have shown that the optimal in terms of maximum coverage of the planet’s surface by virtual trajectories of the descending spacecraft often presented as the trajectory with non-monotonic altitude changes (up to secondary spacecraft departure from the atmosphere and secondary entrance) and with the compact atmospheric regions of the spacecraft velocity break points. As it turned out, at the same time, the reachability areas on the surface of the planet obtained by optimization methods, have non-trivial geometric details. One of the most interesting nuances is the non-smooth beake-like border of the reverse side of the reachable area. In the certain sense, we can talk about generalized SC ricochets on the planet’s atmosphere, which allow us to expand the reachability of the SC (both in terms of range and lateral range). The paper presents a phenomenological semi-analytical model for the phase beams constructing of the quasi-singular SC descent in the planet’s atmosphere, which allows us to synthesize the main nuances of the reachability areas, instead the long-time big data optimization. As the result, the time of the preliminary mission ballistic analysis can be significantly reduced

A Review on Big Data Optimization Techniques

Abstract Analysis of representative tools for SQL query processing on Hadoop (SQL-on-Hadoop systems), such as Hive, Impala, Presto, Shark, show that they are not still sufficiently efficient for complex analytical queries and interactive query processing. Existing SQL-on-Hadoop systems have many benefits from the application of modern query processing techniques that have been studied extensively for many years in the database community. It is expected that with the application of advanced techniques, the performance of SQL-on-Hadoop systems can be improved. The main idea of this paper is to give a review of big data concepts and technologies, and summarize big data optimization techniques that can be used for improving performance when processing big data.

Solutions of fractional-stochastic Bao’s system

Systems of high-dimensional nonlinear ordinary differential equations play a significant role in Physics and applied sciences including big-data optimization, financial models, epidemic disease models. In this paper, we are concerned with numerical solutions of Bao’s system that is a 4-dimensional hyperchaotic system introduced by Bo-Cheng and Zhong (2008). We solve the Bao’s system with both the Crank-Nicolson and power series methods. Crank-Nicolson method is eventually evolved into a new system whose solution is presented in a quite neat algorithmic manner. By adding standard Brownian motion to each term in the model, we express the Bao’s system as a system of stochastic differential equations. We solve the stochastic system with an Euler-type approximate solution method. By adding noise and expressing time derivatives with Caputo-type fractional derivative, we study on synchronization and parameter estimation of the models. To the best of our knowledge, Bao’s system has not been numerically solved with the methods employed in this paper previously, and this paper considers fractional and stochastic Bao’s system for the first time in the history of research. Techniques employed by us in this paper may serve as a framework for solutions of many other systems of ordinary differential equations including Lorenz types and epidemic models.

Randomized Block Proximal Methods for Distributed Stochastic Big-Data Optimization

In this paper we introduce a class of novel distributed algorithms for solving stochastic big-data convex optimization problems over directed graphs. In the addressed set-up, the dimension of the decision variable can be extremely high and the objective function can be nonsmooth. The general algorithm consists of two main steps: a consensus step and an update on a single block of the optimization variable, which is then broadcast to neighbors. Three special instances of the proposed method, involving particular problem structures, are then presented. In the general case, the convergence of a dynamic consensus algorithm over random row stochastic matrices is shown. Then, the convergence of the proposed algorithm to the optimal cost is proven in expected value. Exact convergence is achieved when using diminishing (local) stepsizes, while approximate convergence is attained when constant stepsizes are employed. The convergence rate is shown to be sublinear and an explicit rate is provided in the case of constant stepsizes. Finally, the algorithm is tested on a distributed classification problem, first on synthetic data and, then, on a real, high-dimensional, text dataset.

Bandwidth optimization of information application system under fine integral method of fuzzy fractional order ordinary differential equations

In order to solve the problems that the network bandwidth of previous information application systems can’t guarantee the quality of big data transmission, resulting in low transmission efficiency and slow data processing, etc., an improved information application system is proposed by optimizing the fine integration method of fuzzy fractional ordinary differential equations and combining with software-defined networking (SDN). Firstly, the fractional-order fuzzy differential equation fine integration method of Pade approximation is derived. Then, based on the error analysis theory, the optimization formula for the adaptive selection of weighted parameter N and expanded item number q is obtained. Combined with SDN, an improved information application system is designed, and the improved algorithm and system are detected by example simulation and performance test. The results show that the numerical accuracy and the computational efficiency of the improved algorithm are higher than that of the improved algorithm. The improved port data merge rate and task completion efficiency under different bandwidth are also significantly higher than the original system. It shows that the information application system proposed in this research can better solve the problems of insufficient bandwidth and low communication speed of traditional systems, and provide a new perspective for bandwidth optimization of big data.

The concept of the formation and development of a digital intellectual ecosystem of blended university learning

The article proposes the concept of the formation and development of an adaptive ecosystem of university learning. The concept can allow not only to eliminate the shortcomings inherent in the distance education system, but also to create the basis for building a full-fledged educational technology. The basis for constructing such an ecosystem, in addition to purely didactic developments, can be modern achievements in the field of systems theory, digitalization and artificial intelligence. The education market is seriously affected by advances in artificial intelligence and the rapid development of Industry 4.0. It is also necessary to consider rather unpredictable natural disasters and pandemics. Under these conditions, the only way to maintain and strengthen their positions in the education market, which will rapidly change in the coming decades, is the transformation of processes within the framework of new technological trends and integrated network cluster ecosystems. Decentralized training and outsourcing can become two key functions for the successful application of artificial intelligence in education. Modeling, optimization and analytics of big data make it possible to form a complete set of technologies for creating an outsourcing network and digital educational chains, which allows us to identify the state model of all processes in real time. At each moment in time, the digital twin displays the status of outsourcing processes and educational chains with actual data on planning, preparing the necessary equipment, directly preparing educational programs, loading teachers, accounting and monitoring learning outcomes, etc. The digital twin can be used both for making decisions in real-time, and for forecasting and planning outsourcing. In fact, the university and the companies providing outsourcing services within the framework of this approach are integrated into a single mechanism for solving tasks of flexible individual training. Within the framework of the proposed approach, it is possible to build an educational university environment integrated with real objects of the economy of the territory, which is a component of the educational ecosystem. The concept under consideration allows predicting and planning the training of required specialists, since the model of its work is closely connected with enterprises in the real sector. This becomes possible due to the fact that training takes place according to flexible programs that reflect the ever-changing requirements of enterprises to the competencies of their employees. In fact, a university or a group of universities is becoming an essential component of territorial industrial clusters, which makes it possible to increase the efficiency and quality of specialist training and to quickly develop new curricula and courses that will quickly develop competencies demanded by the real sector of the economy. The use of artificial intelligence technology in combination with the capabilities of the Internet of things and digitalization of the main business processes provides, in fact, the functioning and development of the university’s ecosystem by analogy with the ecosystems of large sectoral system-forming enterprises.

Large-scale Ising emulation with four body interaction and all-to-all connections

Optical Ising machines with two-body interactions have shown potential in solving combinatorial optimization problems which are extremely hard to solve with digital computers. Yet, some physical systems cannot be properly described by only two-body interactions. Here, we propose and demonstrate a nonlinear optics approach to emulate Ising machines containing many spins (up to a million in the absence of optical imperfections) and with tailored all-to-all two and four-body interactions. Our approach employs a spatial light modulator to encode and control the spins in the form of the binary-phase values, and emulates the high-order interaction with frequency conversion in a nonlinear crystal. By implementing adaptive feedback, the system can be evolved into effective spin configurations that well-approximate the ground-states of Ising Hamiltonians with all-to-all connected many-body interactions. Our technique could serve as a tool to probe complex, many-body physics and give rise to exciting applications in big-data optimization, computing, and analytics.

An Artificial Bee Colony-Guided Approach for Electro-Encephalography Signal Decomposition-Based Big Data Optimization

The digital age has added a new term to the literature of information and computer sciences called as the big data in recent years. Because of the individual properties of the newly introduced term, the definitions of the data-intensive problems including optimization problems have been substantially changed and investigations about the solving capabilities of the existing techniques and then developing their specialized variants for big data optimizations have become important research topic. Artificial Bee Colony (ABC) algorithm inspired by the clever foraging characteristics of the real honey bees is one of the most successful swarm intelligence-based metaheuristics. In this study, a new ABC algorithm-based technique that is named source-linked ABC (slinkABC) was proposed by considering the properties of the optimization problems related with the big data. The slinkABC algorithm was tested on the big data optimization problems presented at the Congress on Evolutionary Computation (CEC) 2015 Big Data Optimization Competition. The results obtained from the experimental studies were compared with the different variants of the ABC algorithm including gbest-guided ABC (GABC), ABC/best/1, ABC/best/2, crossover ABC (CABC), converge-onlookers ABC (COABC), quick ABC (qABC) and modified gbest-guided ABC (MGABC) algorithms. In addition to these, the results of the proposed ABC algorithm were also compared with the results of the Differential Evolution (DE) algorithm, Genetic algorithm (GA), Firefly algorithm (FA), Phase-Based Optimization (PBO) algorithm and Particle Swarm Optimization (PSO) algorithm-based approaches. From the experimental studies, it was understood that the ABC algorithm modified by considering the unique properties of the big data optimization problems as in the slinkABC produces better solutions for most of the tested instances compared to the mentioned optimization techniques.

A comparative study between artificial bee colony (ABC) algorithm and its variants on big data optimization

The big data term and its formal definition have changed the properties of some of the computational problems. One of the problems for which the fundamental properties change with the existence of the big data is the optimization problems. Artificial bee colony (ABC) algorithm inspired by the intelligent source search, consumption and communication characteristics of the real honey bees has proven its efficiency on solving different numerical and combinatorial optimization problems. In this study, the standard ABC algorithm and its well-known variants including the gbest-guided ABC algorithm, the differential evolution based ABC/best/1 and ABC/best/2 algorithms, crossover ABC algorithm, converge-onlookers ABC algorithm and quick ABC algorithm were assessed using the electroencephalographic signal decomposition based optimization problems introduced at the 2015 Congress on Evolutionary Computing Big Data Competition. The experimental studies on solving big data optimization problems showed that the phase-divided structure of the standard ABC algorithm still protects its advantageous sides when the candidate food sources or solutions are generated by referencing the global best solution in the onlooker bee phase.

A genetic Artificial Bee Colony algorithm for signal reconstruction based big data optimization

In recent years, the researchers have witnessed the changes or transformations driven by the existence of the big data on the definitions, complexities and future directions of the real world optimization problems. Analyzing the capabilities of the previously introduced techniques, determining possible drawbacks of them and developing new methods by taking into consideration of the unique properties related with the big data are nowadays in urgent demands. Artificial Bee Colony (ABC) algorithm inspired by the clever foraging behaviors of the real honey bees is one of the most successful swarm intelligence based optimization algorithms. In this study, a novel ABC algorithm based big data optimization technique was proposed. For exploring the solving abilities of the proposed technique, a set of experimental studies has been carried out by using different signal decomposition based big data optimization problems presented at the Congress on Evolutionary Computation (CEC) 2015 Big Data Optimization Competition. The results obtained from the experimental studies first were compared with the well-known variants of the standard ABC algorithm named gbest-guided ABC (GABC), ABC/best/1, ABC/best/2, crossover ABC (CABC), converge-onlookers ABC (COABC) and quick ABC (qABC). The results of the proposed ABC algorithm were also compared with the Differential Evolution (DE) algorithm, Genetic algorithm (GA), Firefly algorithm (FA), Fireworks algorithm (FW), Phase Base Optimization (PBO) algorithm, Particle Swarm Optimization (PSO) algorithm and Dragonfly algorithm (DA) based big data optimization techniques. From the experimental studies, it was understood that the newly introduced ABC algorithm based technique is capable of producing better or at least promising results compared to the mentioned big data optimization techniques for all of the benchmark instances.

A real-time Decision Support System for Big Data Analytic: A case of Dynamic Vehicle Routing Problems

Recently, the explosion of large amounts of traffic data has guided data scientists to create models with big data for a better decision-making. Big Data applications process and analyze this huge amounts of data (collected from a variety of heterogeneous data sources) that cannot be processed with traditional technologies. In this paper, Big Data frameworks are used for solving an optimization problem known as Dynamic Vehicle Routing Problem (DVRP). Hence, due to the NP-Hardness of the problem and to deal with a large size of data, we develop a parallel Spark Genetic Algorithm named (S-GA). This parallelism aims to take the advantage of Spark’s in-memory computing ability (as a master-slave distribution computing) and GA’s iterations operations. Parallel operations were used for fitness evaluation and genetic operations. Based on the parallel S-GA a decision support system is developed for the DVRP in order to generate the best routes. The experiments show that our proposed architecture is improved due to its capacity when coping with Big Data optimization problems by interconnecting components and deploying on different nodes of a cluster.

Hybrid multi-objective evolutionary algorithm based on Search Manager framework for big data optimization problems

Big Data optimization (Big-Opt) refers to optimization problems which require to manage the properties of big data analytics. In the present paper, the Search Manager (SM), a recently proposed framework for hybridizing metaheuristics to improve the performance of optimization algorithms, is extended for multi-objective problems (MOSM), and then five configurations of it by combination of different search strategies are proposed to solve the EEG signal analysis problem which is a member of the big data optimization problems class. Experimental results demonstrate that the proposed configurations of MOSM are efficient in this kind of problems. The configurations are also compared with NSGA-III with uniform crossover and adaptive mutation operators (NSGA-III UCAM), which is a recently proposed method for Big-Opt problems.

A Novel Real-Time, Lightweight Chaotic-Encryption Scheme for Next-Generation Audio-Visual Hearing Aids

Next-generation audio-visual (AV) hearing aids stand as a major enabler to realize more intelligible audio. However, high data rate, low latency, low computational complexity, and privacy are some of the major bottlenecks to the successful deployment of such advanced hearing aids. To address these challenges, we propose an integration of 5G Cloud-Radio Access Network (C-RAN), Internet of Things (IoT), and strong privacy algorithms to fully benefit from the possibilities these technologies have to offer. Existing audio-only hearing aids are known to perform poorly in noisy situations where overwhelming noise is present. Current devices make the signal more audible but remain deficient in restoring intelligibility. Thus, there is a need for hearing aids that can selectively amplify the attended talker or filter out acoustic clutter. The proposed 5G IoT-enabled AV hearing-aid framework transmits the encrypted compressed AV information and receives encrypted enhanced reconstructed speech in real time to address cybersecurity attacks such as location privacy and eavesdropping. For security implementation, a real-time lightweight AV encryption is proposed, based on a piece-wise linear chaotic map (PWLSM), Chebyshev map, and a secure hash and S-Box algorithm. For speech enhancement, the received secure AV (including lip-reading) information in the cloud is used to filter noisy audio using both deep learning and analytical acoustic modelling. To offload the computational complexity and real-time optimization issues, the framework runs deep learning and big data optimization processes in the background, on the cloud. The effectiveness and security of the proposed 5G-IoT-enabled AV hearing-aid framework are extensively evaluated using widely known security metrics. Our newly reported, deep learning-driven lip-reading approach for speech enhancement is evaluated under four different dynamic real-world scenarios (cafe, street, public transport, pedestrian area) using benchmark Grid and ChiME3 corpora. Comparative critical analysis in terms of both speech enhancement and AV encryption demonstrates the potential of the envisioned technology to deliver high-quality speech reconstruction and secure mobile AV hearing aid communication. We believe our proposed 5G IoT enabled AV hearing aid framework is an effective and feasible solution and represents a step change in the development of next-generation multimodal digital hearing aids. The ongoing and future work includes more extensive evaluation and comparison with benchmark lightweight encryption algorithms and hardware prototype implementation.

Multiobjective big data optimization based on a hybrid salp swarm algorithm and differential evolution

This paper developed a multiobjective Big Data optimization approach based on a hybrid salp swarm algorithm and the differential evolution algorithm. The role of the differential evolution algorithm is to enhance the capability of the feature exploitation of the salp swarm algorithm because the operators of the differential evolution algorithm are used as local search operators. In general, the proposed method contains three stages. In the first stage, the population is generated, and the archive is initialized. The second stage updates the solutions using the hybrid salp swarm algorithm and the differential evolution algorithm, and the final stage determines the nondominated solutions and updates the archive. To assess the performance of the proposed approach, a series of experiments were performed. A set of single-objective and multiobjective problems from the 2015 Big Data optimization competition were tested; the dataset contained data with and without noise. The results of our experiments illustrated that the proposed approach outperformed other approaches, including the baseline nondominated sorting genetic algorithm, on all test problems. Moreover, for single-objective problems, the score value of the proposed method was better than that of the traditional multiobjective salp swarm algorithm. When compared with both algorithms, that is, the adaptive DE algorithm with external archive and the hybrid multiobjective firefly algorithm, its score was the largest. In contrast, for the multiobjective functions, the scores of the proposed algorithm were higher than that of the fireworks algorithm framework.

Optimization of Big Data Scheduling in Social Networks

In social network big data scheduling, it is easy for target data to conflict in the same data node. Of the different kinds of entropy measures, this paper focuses on the optimization of target entropy. Therefore, this paper presents an optimized method for the scheduling of big data in social networks and also takes into account each task’s amount of data communication during target data transmission to construct a big data scheduling model. Firstly, the task scheduling model is constructed to solve the problem of conflicting target data in the same data node. Next, the necessary conditions for the scheduling of tasks are analyzed. Then, the a periodic task distribution function is calculated. Finally, tasks are scheduled based on the minimum product of the corresponding resource level and the minimum execution time of each task is calculated. Experimental results show that our optimized scheduling model quickly optimizes the scheduling of social network data and solves the problem of strong data collision.

Hybrid Efficient Genetic Algorithm for Big Data Feature Selection Problems

Due to the huge amount of data being generating from different sources, the analyzing and extracting of useful information from these data becomes a very complex task. The difficulty of dealing with big data optimization problems comes from many factors such as the high number of features, and the existing of lost data. The feature selection process becomes an important step in many data mining and machine learning algorithms to reduce the dimensionality of the optimization problems and increase the performance of the classification or clustering algorithms. In this paper, a set of hybrid and efficient genetic algorithms are proposed to solve feature selection problem, when the handled data has a large feature size. The proposed algorithms use a new gene-weighted mechanism that can adaptively classify the features into strong relative features, weak or redundant features, and unstable features during the evolution of the algorithm. Based on this classification, the proposed algorithm gives the strong features high priority and the weak features less priority when generating new candidate solutions. In the same time, the proposed algorithm tries to more concentrate on unstable features that sometimes appear and sometimes disappear from the best solutions of the population. The performance of proposed algorithms is investigated by using different datasets and feature selection algorithms. The results show that our proposed algorithms can outperform the other feature selection algorithms and effectively enhance the classification performance over the tested datasets.

Distributed Partitioned Big-Data Optimization via Asynchronous Dual Decomposition

In this paper, we consider a novel partitioned framework for distributed optimization in peer-to-peer networks. In several important applications, the agents of a network have to solve an optimization problem with two key features: i) the dimension of the decision variable depends on the network size, and ii) cost function and constraints have a sparsity structure related to the communication graph. For this class of problems, a straightforward application of existing consensus methods would show two inefficiencies: poor scalability and redundancy of shared information. We propose an asynchronous distributed algorithm, based on dual decomposition and coordinate methods, to solve partitioned optimization problems. We show that by exploiting the problem structure, the solution can be partitioned among the nodes, so that each node just stores a local copy of a portion of the decision variable (rather than a copy of the entire decision vector) and solves a small-scale local problem.

RETRACTED ARTICLE: Multimedia based intelligent network big data optimization model

The data of a city space is very large. In the city, there are tens of the thousands of sensors of video, audio and image at the same time. Real-time communication and reception of relevant information are carried out. Real-time monitoring of key areas and key persons and real-time warning information, of course, these operations are in the protection of the privacy and the security monitoring object under the premise of intelligent application analysis. To solve this challenge, this paper proposes the multimedia based intelligent network big data optimization model. To make the game decision between service behavior and ethnic behavior and to achieve a comparison between the specific behaviors in the action domain, two behavior comparison criteria need to be defined. The data level includes the data layer memory module and the forwarding module. The data layer memory module is used to store the content of the service, and the forward module is used to forward the data. Forward when the data flows through the module, the data layer memory modules can be according to the requirements of component control level and store the corresponding service content, and further described the service identification and service information notices to component level control. The intelligent multimedia information processing technology used in multimedia sensor networks should take into account two factors: one is the complexity of the processing, the computing power of the multimedia sensor nodes is limited, and the overly complicated processing technology is not suitable; and the second is the multimedia sensor network features and application requirements. The proposed method dealt with the challenges well, the validation proves the robustness.

Categories for (Big) Data models and optimization

This paper proposes a theoretical foundation for Big Data. More precisely, it explains how “functors”, a concept coming from Category Theory, can serve to model the various data structures commonly used to represent (large) data sets, and how “natural transformations” can formalize relations between these structures. Algorithms, such as querying a precise information, mainly depend on the data structure considered, and thus natural transformations can serve to optimize these algorithms and get a result in a shorter time. The paper details four functors modeling tabular data, graph structures (e.g. triple stores), cached and split data. Next, the paper explains how, by considering a functional programming language, the concepts can be implemented without effort to propose new tools (e.g. efficient information servers and query languages). And, as a complement to the mathematical models proposed, the paper also presents a optimized data server and a specific query language (based on “unification” to facilitates the search of information). Finally, the paper gives a comparison study and shows that this tool is more efficient than most of the standards available in the market: the functional server appears to be 10+ times faster than relational or document oriented databases (Mysql and MongoDB), and 100+ times faster than a graph database (Neo4j).

An improved NSGA-III algorithm with adaptive mutation operator for Big Data optimization problems

One of the major challenges of solving Big Data optimization problems via traditional multi-objective evolutionary algorithms (MOEAs) is their high computational costs. This issue has been efficiently tackled by non-dominated sorting genetic algorithm, the third version, (NSGA-III). On the other hand, a concern about the NSGA-III algorithm is that it uses a fixed rate for mutation operator. To cope with this issue, this study introduces an adaptive mutation operator to enhance the performance of the standard NSGA-III algorithm. The proposed adaptive mutation operator strategy is evaluated using three crossover operators of NSGA-III including simulated binary crossover (SBX), uniform crossover (UC) and single point crossover (SI). Subsequently, three improved NSGA-III algorithms (NSGA-III SBXAM, NSGA-III SIAM, and NSGA-III UCAM) are developed. These enhanced algorithms are then implemented to solve a number of Big Data optimization problems. Experimental results indicate that NSGA-III with UC and adaptive mutation operator outperforms the other NSGA-III algorithms.