Empirical Performance Evaluation Research Articles

Anomaly Detection in Railway Sensor Data Environments: State-of-the-Art Methods and Empirical Performance Evaluation.

To date, significant progress has been made in the field of railway anomaly detection using technologies such as real-time data analytics, the Internet of Things, and machine learning. As technology continues to evolve, the ability to detect and respond to anomalies in railway systems is once again in the spotlight. However, railway anomaly detection faces challenges related to the vast infrastructure, dynamic conditions, aging infrastructure, and adverse environmental conditions on the one hand, and the scale, complexity, and critical safety implications of railway systems on the other. Our study is underpinned by the three objectives. Specifically, we aim to identify time series anomaly detection methods applied to railway sensor device data, recognize the advantages and disadvantages of these methods, and evaluate their effectiveness. To address the research objectives, the first part of the study involved a systematic literature review and a series of controlled experiments. In the case of the former, we adopted well-established guidelines to structure and visualize the review. In the second part, we investigated the effectiveness of selected machine learning methods. To evaluate the predictive performance of each method, a five-fold cross-validation approach was applied to ensure the highest accuracy and generality. Based on the calculated accuracy, the results show that the top three methods are CatBoost (96%), Random Forest (91%), and XGBoost (90%), whereas the lowest accuracy is observed for One-Class Support Vector Machines (48%), Local Outlier Factor (53%), and Isolation Forest (55%). As the industry moves toward a zero-defect paradigm on a global scale, ongoing research efforts are focused on improving existing methods and developing new ones that contribute to the safety and quality of rail transportation. In this sense, there are at least four avenues for future research worth considering: testing richer data sets, hyperparameter optimization, and implementing other methods not included in the current study.

Role of Transformational Leadership and Social Entrepreneurship in Achieving Enterprise Performance: Taking Creativity as a Mediator

Purpose: In the recent business context, the pivotal drivers of organizational success are increasingly identified as transformational leadership and social entrepreneurship. These dynamic forces have emerged as crucial predictors, shaping the trajectory of enterprise performance. The present study proposed to explore the role of transformational leadership and social entrepreneurship in achieving enterprise performance through creativity among employees of small and medium-sized enterprises (SMEs) in Egypt. Design/Methodology/ Approach: We applied quantitative methods to detect this problem by collecting data from the employees of Small and Medium-sized Enterprises (SMEs) in Egypt. Finally, we utilized 294 samples to get the results. Findings: Applying the analysis of moment structures (AMOS) version 26.0, the results show a positive and significant impact of transformational leadership, social entrepreneurship and creativity on enterprise performance. The mediating path leads to a positive mediating impact of creativity in developing the connection between transformational leadership and enterprise performance. On the other hand, creativity did not mediate the relationship between social entrepreneurship and enterprise performance. Research, practical and social implications: The research outcomes serve as a strategic compass for policymakers and planners, offering actionable insights to elevate enterprise performance through the strategic integration of transformational leadership and social entrepreneurship. Furthermore, the study provides valuable guidance for organizations aspiring to cultivate a culture of innovation and creativity, fostering the essential elements for sustained success in today's dynamic business environment. Originality/ value: This study addresses the gaps in the empirical evaluation of enterprise performance through transformational leadership, social entrepreneurship and creativity among Egyptian employees of Egyptian SMEs.

Entrepreneurial Leadership and Enterprise Performance: A Way Out of Sustainability

Purpose: Developing enterprise performance and sustainability is essential for firms’ survival. This study explores the effect of accountability, responsibility and analytical thinking on enterprise among Egyptian firms. Design/Methodology/ Approach: We applied quantitative methods to detect this problem by collecting data from entrepreneurs working in different firms in Egypt. The study concludes the results based on 304 valid samples. Findings: Employing path analysis through analysis of moment structures (AMOS) version 26.0, the findings suggest a positive and significant effect of responsibility and analytical thinking on enterprise performance. On the other hand, accountability negatively affects enterprise performance. Finally, the effect of enterprise performance on enterprise sustainability is positive and significant. Research, practical and social implications: The findings of the study may be helpful for investors to Invest in training for accountability, responsibility, and analytical thinking to optimize venture success. They may foster a corporate culture balancing accountability, risk-taking, and innovation for enduring success. Finally, they Align performance metrics with sustainability goals and integrate ethical practices for resilient business outcomes. Originality/ value: This study addresses the lacunae in the empirical evaluation of enterprise performance and sustainability among Egyptian entrepreneurs by emphasizing accountability, responsibility, and analytical thinking. Doing so contributes to a more comprehensive understanding of the factors influencing business success and long-term viability in the Egyptian context.

Owner-free Distributed Symmetric Searchable Encryption Supporting Conjunctive Queries

Symmetric Searchable Encryption (SSE), as an ideal primitive, can ensure data privacy while supporting retrieval over encrypted data. However, existing multi-user SSE schemes require the data owner to share the secret key with all query users or always be online to generate search tokens. While there are some solutions to this problem, they have at least one weakness, such as non-supporting conjunctive query, result decryption assistance of the data owner, and unauthorized access. To solve the above issues, we propose an O wner-free Di stributed S ymmetric searchable encryption supporting C onjunctive query (ODiSC). Specifically, we first evaluate the Learning-Parity-with-Noise weak Pseudorandom Function (LPN-wPRF) in dual-cloud architecture to generate search tokens with the data owner free from sharing key and being online. Then, we provide fine-grained conjunctive query in the distributed architecture using additive secret sharing and symmetric-key hidden vector encryption. Finally, formal security analysis and empirical performance evaluation demonstrate that ODiSC is adaptively simulation-secure and efficient.

Nearby connections strategies: Features, usage, and empirical performance evaluation

The Device-to-Device (D2D) communications involve a direct, peer-to-peer link between devices that operates independently of fixed network infrastructures. It can either complement existing network infrastructures or be used as a standalone network to provide services, such as distributing content, supporting emergency services during natural disasters, and enabling smartphone applications like AirDrop (iOS) and Nearby Share (Android). However, the behavior of D2D communication is not always predictable, and there are many challenges to implementing and deploying D2D communication systems in real-world situations. In particular, it is hard to know what throughput one can obtain, as nominal numbers provided by the documentation are seldom observed in practice. This paper focuses on Nearby Connections Application Programming Interface (API) provided by Google. This API offers distinct strategies in function of the network’s topology, and choosing the right one for an application is a challenging task as they perform differently. In this paper, we contribute to the research community in two ways. Firstly, we investigate the real-time performance of throughput of two strategies (STAR and POINT-TO-POINT, as they are the only ones to use Bluetooth and Wi-Fi Direct to transfer data). Our study shows that the POINT-TO-POINT strategy generally achieves high throughput performance and is suitable for use cases involving high network traffic. In contrast, the STAR performs poorly in throughput, which can result in link instability and slow transfers in some cases. Secondly, we disclose a tool to help network designers evaluate their networks and fine-tune their protocols and algorithms according to their specificities.

E-trikes for urban delivery: An empirical mixed-fleet simulation approach to assess city logistics sustainability

Increasing urbanization has resulted in changes in land use, environment, demographics, and consumer behavior. These changes have magnified the challenges of city logistics. In particular, last-mile delivery freight has become a growing challenge for sustainable cities. Despite numerous studies on last-mile delivery solutions, little attention has been given to assessing the performance of e-cargo in mixed fleet systems. This study's contribution lies in its empirical monitoring and evaluation of e-trike performance for two months and parameterization of a citywide simulation. We demonstrated how GPS data coupled with publicly available data can be used to develop various performance indicators and provided comparison between real-world data and the Google Distance Matrix API. Furthermore, we estimated the costs if the company used vans (all or a mixed fleet) for the same e-trike deliveries. The costs associated with an e-trike-based delivery system are not dramatically higher than those associated with traditional methods. Labor costs were found to be the primary factor influencing cost trade-offs. However, emissions costs are reduced to a large extent supporting sustainability initiatives of the system. This proposed approach can be generalized to evaluate alternative vehicle types that simultaneously change vehicle form factor and performance and fuel types.

Computing Phylo- k-Mers.

Finding the correct position of new sequences within an established phylogenetic tree is an increasingly relevant problem in evolutionary bioinformatics and metagenomics. Recently, alignment-free approaches for this task have been proposed. One such approach is based on the concept of phylogenetically-informative k-mers or phylo- k-mers for short. In practice, phylo- k-mers are inferred from a set of related reference sequences and are equipped with scores expressing the probability of their appearance in different locations within the input reference phylogeny. Computing phylo- k-mers, however, represents a computational bottleneck to their applicability in real-world problems such as the phylogenetic analysis of metabarcoding reads and the detection of novel recombinant viruses. Here we consider the problem of phylo- k-mer computation: how can we efficiently find all k-mers whose probability lies above a given threshold for a given tree node? We describe and analyze algorithms for this problem, relying on branch-and-bound and divide-and-conquer techniques. We exploit the redundancy of adjacent windows of the alignment to save on computation. Besides computational complexity analyses, we provide an empirical evaluation of the relative performance of their implementations on simulated and real-world data. The divide-and-conquer algorithms are found to surpass the branch-and-bound approach, especially when many phylo- k-mers are found.

Addressing Binary Classification over Class Imbalanced Clinical Datasets Using Computationally Intelligent Techniques.

Nowadays, healthcare is the prime need of every human being in the world, and clinical datasets play an important role in developing an intelligent healthcare system for monitoring the health of people. Mostly, the real-world datasets are inherently class imbalanced, clinical datasets also suffer from this imbalance problem, and the imbalanced class distributions pose several issues in the training of classifiers. Consequently, classifiers suffer from low accuracy, precision, recall, and a high degree of misclassification, etc. We performed a brief literature review on the class imbalanced learning scenario. This study carries the empirical performance evaluation of six classifiers, namely Decision Tree, k-Nearest Neighbor, Logistic regression, Artificial Neural Network, Support Vector Machine, and Gaussian Naïve Bayes, over five imbalanced clinical datasets, Breast Cancer Disease, Coronary Heart Disease, Indian Liver Patient, Pima Indians Diabetes Database, and Coronary Kidney Disease, with respect to seven different class balancing techniques, namely Undersampling, Random oversampling, SMOTE, ADASYN, SVM-SMOTE, SMOTEEN, and SMOTETOMEK. In addition to this, the appropriate explanations for the superiority of the classifiers as well as data-balancing techniques are also explored. Furthermore, we discuss the possible recommendations on how to tackle the class imbalanced datasets while training the different supervised machine learning methods. Result analysis demonstrates that SMOTEEN balancing method often performed better over all the other six data-balancing techniques with all six classifiers and for all five clinical datasets. Except for SMOTEEN, all other six balancing techniques almost had equal performance but moderately lesser performance than SMOTEEN.

Performance Evaluation of Different Raspberry Pi Models as MQTT Servers and Clients

Performance analysis for devices in Internet of Things (IoT) environments is an important consideration, especially with their increasing integration in technological solutions, worldwide. The Single Board Computers (SBCs) of the Raspberry Pi Foundation have been widely accepted by the community, and hence, they have been incorporated in numerous IoT projects. To ease their integration, it is essential to assess their network performance. In this paper, we made an empirical performance evaluation of one of the most popular network protocols for IoT environments, named the Message Queuing Telemetry Transport (MQTT) protocol, on Raspberry Pi. To do so, we set up two different testbeds scenarios and assessed the performance with benchmarks. At the software level, we focused on Mosquitto, a popular open-source MQTT broker implementation and client library. Our principal metric is the transmission time, but we also investigated the throughput. In our experiments, we varied several parameters, such as the size of the payload of the published messages, the WiFi bandwidth, the QoS level, the security level (MQTT vs. MQTT with TLS), and the hardware for the clients and broker. We focus mainly on packet sizes ranging from 100 to 25,000 bytes. We also investigate how these low-cost devices handle a TCP SYN flood attack. In the research work presented within this paper, we aim to guide developers, researchers, network administrators, and hobbyists who plan to use these low-cost devices in an MQTT or IoT network by showing the performance that they should expect according to different Raspberry Pi options.

Performance Evaluation of Public Management in View of Fuzzy DEA Model

The research on management performance evaluation and governance countermeasures of public utilities, as an important entry point, focus, and flash point of the combination of management science, engineering research, and public management research, not only highlights the implementation and implementation of the scientific outlook on development and correct outlook on political performance, so that the topic has distinct theoretical frontier, practical pertinence, and practical guidance. It also highlights the intersection and integration of multiple disciplines, combines normative research with empirical research, qualitative research with quantitative research, macro research with micro research, and method and technology research with concept innovation and institutional arrangement research, and strives to realize the integration and complementarity of research methods of management science and engineering and public management. The research of public management broadens new horizons, expands new fields, and provides new paradigms, methods, and means. Based on the weight of public utilities management performance evaluation indicators and the selection of empirical samples, this paper studies the basic principles and comparative advantages of fuzzy comprehensive evaluation model and DEA model, makes an empirical evaluation of public utilities management performance, draws evaluation conclusions, and analyzes the key factors affecting and restricting the level of public utilities management performance. From the research level of evaluation technology and methods to the level of institutional arrangement, it is the only way to realize the scientific, standardized, and institutionalized performance evaluation of local government public utilities management. The basic procedures of performance evaluation and the performance evaluation mechanism are introduced, and the establishment and improvement of the information fidelity system and result application system of public utilities performance evaluation are studied. Based on the evaluation conclusion and empirical analysis of public utilities management performance, aiming at the key factors affecting the level of public utilities management performance, this paper puts forward the governance countermeasures of public utilities from the aspects of establishing scientific public utilities management performance concept, innovating public utilities management system, establishing and perfecting public utilities investment management mechanism, and so on.

Municipal fiber in the United States: A financial assessment

Despite growing interest in broadband provided by municipally owned and operated fiber-to-the-home networks, the academic literature has yet to undertake a systematic assessment of these projects' financial performance. To fill this gap, we utilize municipalities' official reports to offer an empirical evaluation of the financial performance of every municipal fiber project in the U.S. operating in 2010 through 2019. An analysis of the actual performance of the resulting fifteen-project panel dataset reveals that none of the projects generated sufficient nominal cash flow in the short run to maintain solvency without infusions of additional cash from outside sources or debt relief. Similarly, 87% have not actually generated sufficient nominal cash flow to put them on track to achieve long-run solvency. In addition, 73% generated negative nominal cash flow over the past three fiscal years, leaving them poorly positioned to make up their deficits and causing them to fall farther into debt. An assessment based on the net present value of these projects' operating cash flow indicates that 53% of projects would not be on track to breakeven even assuming the theoretical best-case performance in terms of capital expenditures and debt service. Close analysis of these projects’ performance reveals that revenue generation likely plays a more important role in generating cash flow than efficiency in construction costs or operating efficiency.

Recurrent Stroke Prediction using Machine Learning Algorithms with Clinical Public Datasets: An Empirical Performance Evaluation

Recurrent strokes can be devastating, often resulting in severe disability or death. However, nearly 90% of the causes of recurrent stroke are modifiable, which means recurrent strokes can be averted by controlling risk factors, which are mainly behavioral and metabolic in nature. Thus, it shows that from the previous works that recurrent stroke prediction model could help in minimizing the possibility of getting recurrent stroke. Previous works have shown promising results in predicting first-time stroke cases with machine learning approaches. However, there are limited works on recurrent stroke prediction using machine learning methods. Hence, this work is proposed to perform an empirical analysis and to investigate machine learning algorithms implementation in the recurrent stroke prediction models. This research aims to investigate and compare the performance of machine learning algorithms using recurrent stroke clinical public datasets. In this study, Artificial Neural Network (ANN), Support Vector Machine (SVM) and Bayesian Rule List (BRL) are used and compared their performance in the domain of recurrent stroke prediction model. The result of the empirical experiments shows that ANN scores the highest accuracy at 80.00%, follows by BRL with 75.91% and SVM with 60.45%.

Hazelcast jet

Jet is an open source, high performance, distributed stream processor built at Hazelcast during the last five years. Jet was engineered with millisecond latency on the 99.99th percentile as its primary design goal. Originally Jet's purpose was to be an execution engine that performs complex business logic on top of streams generated by Hazelcast's In-memory Data Grid (IMDG): a set of in-memory, partitioned and replicated data structures. With time, Jet evolved into a full-fledged, scale-out stream processor that can handle out-of-order streams and provide exactly-once processing guarantees. Jet's end-to-end latency lies in the order of milliseconds, and its throughput in the order of millions of events per CPU-core. This paper presents the main design decisions we made in order to maximize the performance per CPU-core, alongside lessons learned, and an empirical performance evaluation.

Empirical Performance Evaluation of Communication Libraries for Multi-GPU based Distributed Deep Learning in a Container Environment

Recently, most cloud services use Docker container environment to provide their services. However, there are no researches to evaluate the performance of communication libraries for multi-GPU based distributed deep learning in a Docker container environment. In this paper, we propose an efficient communication architecture for multi-GPU based deep learning in a Docker container environment by evaluating the performances of various communication libraries. We compare the performances of the parameter server architecture and the All-reduce architecture, which are typical distributed deep learning architectures. Further, we analyze the performances of two separate multi-GPU resource allocation policies ― allocating a single GPU to each Docker container and allocating multiple GPUs to each Docker container. We also experiment with the scalability of collective communication by increasing the number of GPUs from one to four. Through experiments, we compare OpenMPI and MPICH, which are representative open source MPI libraries, and NCCL, which is NVIDIA’s collective communication library for the multi-GPU setting. In the parameter server architecture, we show that using CUDA-aware OpenMPI with multi-GPU per Docker container environment reduces communication latency by up to 75%. Also, we show that using NCCL in All-reduce architecture reduces communication latency by up to 93% compared to other libraries.

Empirical Performance and Energy Consumption Evaluation of Container Solutions on Resource Constrained IoT Gateways.

Containers virtually package a piece of software and share the host Operating System (OS) upon deployment. This makes them notably light weight and suitable for dynamic service deployment at the network edge and Internet of Things (IoT) devices for reduced latency and energy consumption. Data collection, computation, and now intelligence is included in variety of IoT devices which have very tight latency and energy consumption conditions. Recent studies satisfy latency condition through containerized services deployment on IoT devices and gateways. They fail to account for the limited energy and computing resources of these devices which limit the scalability and concurrent services deployment. This paper aims to establish guidelines and identify critical factors for containerized services deployment on resource constrained IoT devices. For this purpose, two container orchestration tools (i.e., Docker Swarm and Kubernetes) are tested and compared on a baseline IoT gateways testbed. Experiments use Deep Learning driven data analytics and Intrusion Detection System services, and evaluate the time it takes to prepare and deploy a container (creation time), Central Processing Unit (CPU) utilization for concurrent containers deployment, memory usage under different traffic loads, and energy consumption. The results indicate that container creation time and memory usage are decisive factors for containerized micro service architecture.

The effect of benchmark data characteristics during empirical strip packing heuristic performance evaluation

In this paper, we consider the two-dimensional strip packing problem in which a fixed set of items has to be packed into a single object of unlimited height with the aim of minimising the packing height. A new, systematic way of comparing the relative performances of packing algorithms is proposed. A large set of strip packing benchmark instances from various repositories in the literature is clustered into different classes of test problems based on their underlying features. We compare a representative sample of existing strip packing heuristics in terms of solution quality in respect of the clustered data. The effectiveness of the designs of the different algorithms is contrasted for each of the data categories. It is found that the aspects of the test problems affect the solution qualities and relative rankings achieved by the various packing algorithms, but that no single heuristic is superior in respect of all the data categories.

An empirical evaluation of machine learning performance in corporate sales growth prediction

An empirical evaluation of machine learning performance in corporate sales growth prediction

International reserves and economic growth

The channels through which reserves affect economic performance have been studied extensively; however, an empirical evaluation of the actual performance of reserve holders is missing from the literature. This paper’s goal is to fill this gap. We investigate the relationship between the level of international reserves and economic growth for a sample of 120 developed and developing nations for the period 1981–2010. Using dynamic panel data techniques and controlling for a wide range of covariates, we find that international reserves have a positive effect on growth. Furthermore, the effect weakens as the opportunity cost of holding reserves increases.

A survey of empirical performance evaluation of permissioned blockchain platforms: Challenges and opportunities

Blockchain-based platforms, particularly those based on permissioned blockchain, are increasingly popular in a broad range of settings. In addition to security and privacy concerns, organizations seeking to implement such platforms also need to consider performance, especially in latency- or delay-sensitive applications. Performance is generally less studied in comparison to security and privacy, and therefore in this paper we survey existing empirical performance evaluations of different permissioned blockchain platforms published between 2015 and 2019, using a comparative framework. The framework comprises ten criteria. We then conclude the paper with a number of potential future research directions.

An Energy Aware Task Scheduling Model Using Ant-Mating Optimization in Fog Computing Environment

Fog computing has become a platform of choice for executing emerging applications with low latency requirements. Since the devices in fog computing tend to be resource constraint and highly distributed, how fog computing resources can be effectively utilized for executing delay-sensitive tasks is a fundamental challenge. To address this problem, we propose and evaluate a new task scheduling algorithm with the aim of reducing the total system makespan and energy consumption for fog computing platform. The proposed approach consists of two key components: 1) a new bio-inspired optimization approach called Ant Mating Optimization (AMO) and 2) optimized distribution of a set of tasks among the fog nodes within proximity. The objective is to find an optimal trade-off between the system makespan and the consumed energy required by the fog computing services, established by end-user devices. Our empirical performance evaluation results demonstrate that the proposed approach outperforms the bee life algorithm, traditional particle swarm optimization and genetic algorithm in terms of makespan and consumed energy.