Machine-generated Data Research Articles

Hybrid Electro-Optical Intra-Data Center Networks Tailored for Different Traffic Classes

This work presents a novel approach in intradata center network design and performance evaluation, based on a tailored, long-range dependence data traffic generation model for different application classes. We examine how an intra-data center network can be efficiently evaluated and optimized by applying accurate models for machine-generated data and using assumptions on current and future commercially available data center hardware. Moreover, we show that by migrating such a Fat-Tree network to its hybrid, electro-optical counterpart by employing optical circuit switching for selected, intense traffic connections, significant capital and operational expenditure cost savings can be obtained.

Data Science Approach for Simulating Educational Data: Towards the Development of Teaching Outcome Model (TOM)

The increasing availability of educational data provides the educational researcher with numerous opportunities to use analytics to extract useful knowledge to enhance teaching and learning. While learning analytics focuses on the collection and analysis of data about students and their learning contexts, teaching analytics focuses on the analysis of the design of the teaching environment and the quality of learning activities provided to students. In this article, we propose a data science approach that incorporates the analysis and delivery of data-driven solution to explore the role of teaching analytics, without compromising issues of privacy, by creating pseudocode that simulates data to help develop test cases of teaching activities. The outcome of this approach is intended to inform the development of a teaching outcome model (TOM), that can be used to inspire and inspect quality of teaching. The simulated approach reported in the research was accomplished through Splunk. Splunk is a Big Data platform designed to collect and analyse high volumes of machine-generated data and render results on a dashboard in real-time. We present the results as a series of visual dashboards illustrating patterns, trends and results in teaching performance. Our research aims to contribute to the development of an educational data science approach to support the culture of data-informed decision making in higher education.

Application of data analytics for product design: Sentiment analysis of online product reviews

Advanced data analytics is one of the most revolutionary technological developments in the 21st century, which enables the discovery of underlining trends via sophisticated computational methods On various e-commerce and social platforms, millions of online product reviews are published by customers, which can potentially provide designers with invaluable insights into product design. This paper presents a design framework to analyze online product reviews. The objective is to use this machine-generated data to identify a series of customer needs. The framework aims to distill large volumes of qualitative data into quantitative insights on product features, so that designers can make more informed decisions. The framework combines the elements of online product reviews, design theory and methodology, and data analytics to reveal new insights. The effectiveness of the proposal framework is validated through a case study on product reviews from the e-commerce website, Amazon. The framework demonstrates a statistical approach for analyzing online product reviews. The framework acts as an interface between quantitative outputs and the qualitative and creative process of design. Further analysis of results identifies many of incorporating logical, computational methods into the highly subjective and creative process of design.

Application and Usefulness of Internet of Things in Information Technology

The Internet of Things (IoT) is a system of interrelated computing devices, mechanical and digital machines, objects, animals or people that are provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or human-tocomputer interaction. It is an ambiguous term, but it is fast becoming a tangible technology that can be applied in data centers to collect information on just about anything that IT wants to control. IoT has evolved from the convergence of wireless technologies, micro-electromechanical systems (MEMS), microservices and the internet. The convergence has helped tear down the silo walls between operational technology (OT) and information technology (IT), allowing unstructured machine-generated data to be analyzed for insights that will drive improvements. The Internet of Things (IoT) is essentially a system of machines or objects outfitted with data-collecting technologies so that those objects can communicate with one another. The machineto-machine (M2M) data that is generated has a wide range of uses, but is commonly seen as a way to determine the health and status of things -- inanimate or living.

Written Evidence to HM Treasury’s Digital Competition Expert Panel

Building upon my previous research on ‘The Rise of Big Data’, I can highlight the existence of a two-tier system of micro- and large-scale (big) data analytics. All companies that dominate local or neighbouring markets collect data at a micro-scale level for the purpose of predictive analytics, such as linear, including decision trees, vector, and cluster modelling. They use geo-demographic variables, such as income, age, and so on, and behavioural data to predict the target’s willingness to buy a particular product. Major retailers attempt to test their customers’ willingness to make certain purchases. Regularly, credit-rating companies use sample modelling to test the probability of fraud; insurance companies for the probability of claims; life insurance companies to estimate life expectancy; banks for the probability of a mortgage’s voluntary foreclosure; and so on. However, given the limited size of the sample, such predictive analytics may not prove accurate. In contrast, large-scale corporations that possess or harvest a large amount of big data may inter alia use raw data from the unstructured content of emails or the web for data mining purposes; machine-generated data; statistical software packages, such as IBM, Stata, Rapid Minder, Google’s open source software, Apache Hadoop, Revolution, and so on; and automated data that is a mix of data-driven and expert-derived rules to analyse big data. Software packages act as intelligent agents that allow for quick automation and processing of big data analytics. With the help of the quantitative and statistical analysis of big data, it is, however, possible to accurately measure the consumers’ willingness to pay for particular products, determine the elasticity of demand in response to price changes, observe trends in the life cycle of a product, identify under-performing products, and categorise customers. While the micro-scale behavioural modelling of data serves for the analysis and prediction of the risks associated with the use of targeted advertising and promotional campaigns, when the same modelling is being applied at a large-scale level to forecast customers’ demand, to predict product trends, and to make strategic pricing recommendations, the latter inevitably becomes part of a wider social experiment of intensive platform monitoring and data sharing with data analytics companies. Due to the size of the sample of participants due to be observed, the latter forecasts tend to be even more accurate and to reliably inform producers of estimated demand and future pricing options. I would argue that no marketing research harms consumers as long as the sample of the targeted consumers remains meaningful, but limited for a specific purpose. Otherwise, big data analytics is a perfect substitute for direct or indirect exchanges of strategic information regarding actual or future pricing methods; estimated demand; consumers’ preferences, location, investment; and so much more. However, larger companies or corporations are in a stronger position to extract strategic data that can later be exploited tactically, i.e., through targeted advertising, and strategically, by informing the price setting mechanism. Instead of a business-to-business exchange of information (B2B: ‘hub and spoke’ conspiracy), this large-scale marketing experiment moves on to the prospective consumers (B2C: track-and-monitor conspiracy). In my opinion, this phenomenon, which I have previously identified as a track-and-monitor conspiracy on the basis of consumers’ geographical location; socio-economic demographics, i.e., income status; and behavioural data, i.e., preferences and interests, allows for a pricing conspiracy to be implemented with the help of consumers rather than competitors. For example, consumers identified as living in remote areas, i.e., the Highlands or small islands, usually have less choice and can therefore be charged more for other terms and conditions, such as transportation costs. Knowing consumers’ category of income, businesses can more accurately predict their reservation price in terms of bargaining. It is similar to a meeting of minds between the buyer and the seller, where the latter knows how much the former is able to potentially spend.

Antitrust Risks and Big Data

The explosive growth in companies’ exploitation of big data is drawing intense scrutiny from European antitrust authorities. EU Competition Commissioner Margrethe Vestager has promised to “keep a close eye on how companies use data” and a number of European antitrust authorities have conducted full-blown studies on big data issues, including an especially comprehensive May 2016 Franco/German study on “Competition Law and data”. Antitrust authorities fear that big data can create barriers to entry and market power, especially where companies hold unique datasets that cannot be replicated by competitors. However, a closer look at big data-related theories of harm suggests that the focus on uniqueness may be misplaced. Many companies collect voluminous datasets that are unique and non-replicable, as well as, being valuable and competitively significant, without raising exclusionary concerns of the types highlighted by antitrust authorities. The perception that limited access to big data may create barriers to entry and stifle the growth of the digital economy risks provoking an overbroad legislative reaction. In January 2017, the EU Commission consulted on a wide range of data-related issues, proposing among other things that companies be required to share non-personal, machine-generated data with third parties, including competitors, whether or not the data-holder holds a dominant position or engages in abusive conduct. As such, it would go far beyond any access remedy recognized in the antitrust context. This article discusses European antitrust authorities’ concerns about the foreclosure risks of big data from the perspective of the different types and uses of big data. This approach reveals that exclusionary concerns arise in a relatively small segment of big data uses, and those situations can be assessed using traditional antitrust tools. Some important big data issues not addressed in this article include the mirror image of these exclusionary concerns – i.e., that ubiquitous big data can make markets so transparent that competition may be impaired by competitors using pricing algorithms – the role of big data in merger review, and the relationship between antitrust enforcement and data protection. Big data are a big topic, and a complex one.

Autonomous Landmark Calibration Method for Indoor Localization.

Machine-generated data expansion is a global phenomenon in recent Internet services. The proliferation of mobile communication and smart devices has increased the utilization of machine-generated data significantly. One of the most promising applications of machine-generated data is the estimation of the location of smart devices. The motion sensors integrated into smart devices generate continuous data that can be used to estimate the location of pedestrians in an indoor environment. We focus on the estimation of the accurate location of smart devices by determining the landmarks appropriately for location error calibration. In the motion sensor-based location estimation, the proposed threshold control method determines valid landmarks in real time to avoid the accumulation of errors. A statistical method analyzes the acquired motion sensor data and proposes a valid landmark for every movement of the smart devices. Motion sensor data used in the testbed are collected from the actual measurements taken throughout a commercial building to demonstrate the practical usefulness of the proposed method.

Competing with Big Data

This paper studies competition in data-driven markets, that is, markets where the cost of quality production is decreasing in the amount of machine-generated data about user preferences or characteristics, which is an inseparable byproduct of using services offered in such markets. This gives rise to data-driven indirect network effects. We construct a dynamic model of R&D competition, where duopolists repeatedly determine their innovation investments, and show that such markets tip under very mild conditions, moving towards monopoly. In a tipped market, innovation incentives both for the dominant firm and for competitors are small. We also show under which conditions a dominant firm in one market can leverage its position to a connected market, thereby initiating a domino effect. We show that market tipping can be avoided if competitors share their user information.

Big data analytics for exploratory social network analysis

If an organisation desires to retrieve productive insights, big data analytics plays a vital role in analysing the unstructured, semi-structured and structured data. Big data assumes human-sourced information (social network analysis), machine-generated data and process-mediated data. Big data as a product of social networks comes from human experiences in works of art or in books, video, photographs, etc. A small piece of information that might have begun with a suggestion of purchasing a smart phone during group chats amongst a circle of friends might end up on the desk of a smart phone company manager as an aid to decision making. This paper aims to address big data analytics for exploratory social network and proposes an experimental study with results. Experimentation has been carried out on SocNetV Version 1.9 using Pajek and different metrics of SNA are evaluated and analysed to strengthen decision making.

An empirical study of on-line models for relational data streams

To date, Inductive Logic Programming (ILP) systems have largely assumed that all data needed for learning have been provided at the onset of model construction. Increasingly, for application areas like telecommunications, astronomy, text processing, financial markets and biology, machine-generated data are being generated continuously and on a vast scale. We see at least four kinds of problems that this presents for ILP: (1) it may not be possible to store all of the data, even in secondary memory; (2) even if it were possible to store the data, it may be impractical to construct an acceptable model using partitioning techniques that repeatedly perform expensive coverage or subsumption-tests on the data; (3) models constructed at some point may become less effective, or even invalid, as more data become available (exemplified by the problem when identifying concepts); and (4) the representation of the data instances may need to change as more data become available (a kind of language drift problem). In this paper, we investigate the adoption of a stream-based on-line learning approach to relational data. Specifically, we examine the representation of relational data in both an infinite-attribute setting, and in the usual fixed-attribute setting, and develop implementations that use ILP engines in combination with on-line model-constructors. The behaviour of each program is investigated using a set of controlled experiments, and performance in practical settings is demonstrated by constructing complete theories for some of the largest biochemical datasets examined by ILP systems to date, including one with a million examples; to the best of our knowledge, the first time this has been empirically demonstrated with ILP on a real-world data set.

The development of Korean online dictionaries: a case study of Naver dictionary services

The advances in information technology have been affecting people’s consultation behaviors and thus shaping the direction of development of Korean online dictionaries. This paper aims (1) to describe the development of Korean online dictionaries by analyzing a case study of the dictionary services of Naver, South Korea’s biggest web portal site; (2) to identify the characteristics of the online dictionary evolution; and (3) to provide primary data for further research. In order to provide background information on the evolution of Naver dictionary services, it introduces the history of Korean online dictionaries and identifies the changes in the sizes and characteristics of the audiences, which were affected by the shift from the personal computer to mobile devices. Then, the development of Naver dictionary services is examined in terms of the development phases, entry data acquisition methods, and the development of tool features. This study reveals three findings: (1) the history could be divided into three periods: the formation period (1999–2004), the growth period (2005–2009), and the expansion period (2010–now). The expansion period is characterized by the popularity of smartphones, the enlargement of dictionary services, and the emergence of mechanical translation; (2) the four main sources for entries in the dictionary services are learner dictionaries, terminological–technical dictionaries, machine-generated data such as web-crawled data and parallel texts collected from the internet, and user-participated entries; (3) most of the tool features appeared during the growth period, and they are now evolving in the direction towards appification and individualization. These findings may allow predictions about the future of Korean online dictionaries.

Evaluating Gas Turbine Performance Using Machine-Generated Data: Quantifying Degradation and Impacts of Compressor Washing

Gas turbine (GT) operators are often met with the challenge of utilizing and making meaning of the vast measurement data collected from machine sensors during operation. This can easily be about 576 × 106 data points of gas path measurements for one machine in a base load operation in a year, if the width of the data is 20 columns of measured and calculated parameters. This study focuses on the utilization of large data in the context of quantifying the degradation that is mostly related to compressor fouling, in addition to investigations on the impact of offline and online compressor washing. To achieve this, four GT engines operating for about 3.5 years with 51 offline washes and 1184 occasions of online washes were examined. This investigation includes different wash frequencies, liquid concentrations, and one engine operation without online washing (only offline). This study has involved correcting measurement data not only just with compressor inlet temperatures (CITs) and pressures but also with relative humidity (RH). turbomatch, an in-house GT performance simulation software has been implemented to obtain nondimensional factors for the corrections. All of the data visualization and analysis have been conducted using tableau analytics software, which facilitates the investigation of global and local events within an operation. The concept of using of handles and filters is proposed in this study, and it demonstrates the level of insight to the data and forms the basis of the outcomes obtained. This work shows that during operation, the engine performance is mostly deteriorating, though to varying degrees. Online washing also showed an influence on this, reducing the average degradation rate each hour by half, when compared to the engine operating only with offline washing. Hourly marginal improvements were also observed with an increased average wash frequency of nine hours and a similar outcome obtained when the washing solution is 2.3 times more concentrated. Clear benefits of offline washes are also presented, alongside the typically obtainable values of increased power output after a wash, also in relation to the number of operating hours before a wash.

A REVIEW PAPER ON BIG DATA ANALYTICS

The basis of this dissertation is the rise of “big data” and the use of analytics to mine that data. Big data is basically used to analyze the large amount of data. Companies have been storing and analyzing large volumes of data. In the recent times, data warehousing was mostly used for this purpose which was invented in the early 1990s. Big Data warehouses means terabytes that is data stored in the warehouses was terabytes in storage. But now it is petabytes, and the data is growing at a high speed. The growth of data is because the companies and organizations store and analyze the greater levels of transaction details, as well as Web data and machine-generated data, to achieve a better understanding of customer behavior and market behavior. Big data helps to achieve this goal. A large number of companies are using Big Data. It seems in coming years the growth of big data concept must be higher.

AN ANALYTICAL APPROACH TO ANALYZE WEB DATA

The basis of this dissertation is the rise of “big data” and the use of analytics to mine that data. Big data is basically used to analyze the large amount of data. Companies have been storing and analyzing large volumes of data. In the recent times, data warehousing was mostly used for this purpose which was invented in the early 1990s. Big Data warehouses means terabytes that is data stored in the warehouses was terabytes in storage. But now it is petabytes, and the data is growing at a high speed. The growth of data is because the companies and organizations store and analyze the greater levels of transaction details, as well as Web data and machine-generated data, to achieve a better understanding of customer behavior and market behavior. Big data helps to achieve this goal. A large number of companies are using Big Data. It seems in coming years the growth of big data concept must be higher.

Rise of the Machines: Machine-Generated Data and the Confrontation Clause

Forensic machines and other such devices can create powerfully incriminating evidence, and some do so with little or no human aid. In a world where machines increasingly perform that task, what happens to the right to cross-examine the human witnesses who would have testified but for the machines? Through errors, bias, and even fabrication, machine operators, calibrators, and others involved in the forensic process still have the power to cause these tools to incriminate the wrong person. There is no shortage of evidence that this occurs. Many courts have concluded, however, that there is no right to cross-examine the operators of these machines. This Article analyzes that result. The Article begins by discussing Confrontation Clause jurisprudence from across the nation, including machine-generated testimony cases and areas of Confrontation Clause jurisprudence that might be instructive in addressing machine-generated data, such as cases involving photographs, videos, interpreters, and dog handler testimony. The article then considers the strengths and weaknesses of several potential approaches to machine-generated testimony and concludes that multiple approaches are defensible, but the approach that best adheres to the purposes of the Confrontation Clause focuses on the degree and nature of the operator’s control over the machine. Under that model, there is no right to cross-examine the operators of many modern machines. Though that result is troubling, there are alternative models that might extend the lifespan of the Confrontation Clause. However, as machines become increasingly automated, the right to cross-examine their human assistants and progenitors will approach extinction.

An Architecture for Big Data Analytics

Big Data is the new experience curve in the new economy driven by data with high volume, velocity, variety, and veracity. They come from various sources that include the Internet, mobile devices, social media, geospatial devices, sensors, and other machine-generated data. Unlocking the value of Big Data allows business to better sense and respond to the environment, and is becoming a key to creating competitive advantages in a complex and rapidly changing market. Government is also taking notice of the Big Data phenomenon and has created initiatives to exploit Big Data in many areas such as science and engineering, healthcare and national security. Traditional data processing and analysis of structured data using RDBMS and data warehousing no longer satisfy the challenges of Big Data. Technology trends for Big Data embrace open source software, commodity servers, and massively parallel-distributed processing platforms. Analytics is at the core of exploiting values from Big Data to create consumable insights for business and government. This paper presents architecture for Big Data Analytics and explores Big Data technologies that include NoSQL databases, Hadoop Distributed File System and MapReduce.

Agile BI � The Future of BI

In rapidly changing economy, Business Intelligence solutions have to become more agile. This paper attempts to discuss some questions which help in creating agile BI solution such as: What is Agile? Why agile is so well suited for BI? Which are key elements that promote agile BI solution? Also, paper briefly looks at technologies that can be used for enabling agile BI solution.Keywords: Agile Business Intelligence, Agile Business Analytics, Agile Development Methodologies, In-Memory Bi Approaches, Data Virtualization Server1 IntroductionBusiness Intelligence (BI) was defined in different ways. The Data-Warehousing Institute has defined Business Intelligence as the tools, technologies and processes required to turn data into information and information into knowledge and plans that optimize business actions [6], Turban has defined BI as a broad category of applications and techniques for gathering, storing, analyzing and providing access to data to help enterprise user make better business and strategic decisions. [23], The range of capabilities that can be defined as business intelligence is very broad. The spectrum of BI technologies is presented in [16].Most enterprises have hundreds of internal and external data sources such as: databases, e-mail archives, file systems, spreadsheets, digital images, audio files and more. 80% of organizational data are unstructured and semi-structured data.Traditional Business Intelligence systems use small fraction of all data available. Also, traditional BI systems use only structured The core components of traditional BI architecture are: ETL tools, enterprise data warehouse with metadata repository and business analytics (Figure 1).Traditional BI systems use ETL tools for extracting data from multiple sources and temporarily storing those datasets at staging area. Organizations use data warehouses to aggregate cleaned and structured Business analytics/BI tools include enterprise reporting tools, ad hoc query tools, statistical analysis tools, OLAP tools, spatial-OLAP analysis tools, dashboards, scorecards and advanced analytics. Advanced analytics typically refer to: data mining tools, text mining tools, predictive analytics, artificial intelligence, and natural language processing. But architecture is unable to get adapted to change. A study by Aberdeen Group [24] showed that this style ofBI is predominantly controlled, driven and delivered by corporate IT. Often, only static views of data are available and any changes or enhancements must be made by IT organization. These characteristics are in contradiction with frequently changing business and big data. Big data typically refers to following types of data: semi-structured data (XML and similar standards), unstructured data, Web data (social data, Web logs) and real-time data (event data, spatial data, machine-generated data).Table 1 briefly summarizes main disadvantages of traditional BI systems.How to eliminate these problems? By building agile BI solution. A study by TDWI Research [22] showed that many traditional business intelligence systems are not agile:* 33% of organizations needed more than three months to add new data source to existing business intelligence system* developing complex report or dashboard with about 20 dimensions, 12 measures and 6 user access rides took on average 7 weeks in 2011The next section presents briefly concept of agile BI and key elements that together promote agile BI solution.2 Agile BIAgile means ability to be adaptable. Agile BI was defined in different ways. The Forrester Research defines agile BI as an approach that combines processes, methodologies, organizational structure, tools and technologies that enable strategic, tactical and operational decision -makers to be more flexible and more responsive to fast pace of changes to business and regulatory requirements [9], According to Data Warehousing Institute agile BI addresses broad need to enable flexibility by accelerating time it takes to deliver value with BI projects. …

6–10 × pyrosequencing is a practical approach for whole prokaryote genome studies

Next generation 454 pyrosequencing technology for whole bacterial genome sequencing involves a deep sequencing strategy with at least 15–20× in depth proposed by official protocols but usually done with over 20× in practices. In this study, we carried out a comprehensive evaluation of quality of the de novo assemblies based on realistic pyrosequencing simulated data from 1480 prokaryote genomes and 7 runs of machine-generated data. Our results demonstrated that for most of the prokaryote genomes, 6–10× sequencing in qualified runs with 400bp reads could produce high quality draft assembly (>98% genome coverage, <100 contigs with N50 size >100kb, single base accuracy >99.99, indel error rate <0.01%, false gene loss/duplication rate <0.5%). Our study proves the power of low depth pyrosequencing strategy, which provides a cost-effective way for sequencing whole prokaryote genomes in a short time and enables further studies in microbial population diversity and comparative genomics.

Generation of Realistic Social Network Datasets For Testing of Analysis and Simulation Tools

Abstract : Testing large-scale dynamic network simulation packages, such as NetWatch, requires a large quantity of test data to be available for each of the experiments. The test data includes initial topologies of agents' social networks and specification of knowledge networks for each of the agents to fit an empirically derived distribution of knowledge. Testing the software on machine-generated data, as opposed to empirical data only, allows the user to conduct repeatable tests that stress certain aspects of the software and help in debugging and optimization of software performance.