Benchmarking Methodology Research Articles

БЕНЧМАРКИНГ IT-КОМПАНИИ: СУЩНОСТЬ И ОСОБЕННОСТИ ПРИМЕНЕНИЯ

The essence of benchmarking is revealed, its features are highlighted and the advantages of using the benchmarking methodology in the activities of an IT company are substantiated

Comparison of Cheap Imported Stainless Steel Samples with Indian-made Samples and a Crystalline Phase Based Methodology for Bench-marking them

In the emerging South Asian markets, from the commercial as well as engineering point of view, performance and quality of any local engineering product like steel structures vis a vis similar but imported, cheap materials is always an issue. In this paper, as a representative case, we discuss the composition, crystalline phase and microstructure of two most common stainless steel grades manufactured locally by the major government of India sponsored (GOIS) Steel maker and compare them with similar cheap imported items that some local private businesses prefer due to slightly lower costs but having similar materials usage parameters. We have also used wet etch based micrographic analysis to compare surface morphology and have also done composition analysis in our results. It is shown that even if ASTM standards of composition are followed, since their crystalline phase components are not the same, their performance and properties are never comparable. More stringent quality bench-marking of such alloy materials for checking repeatibility of batches and processes using X-ray diffraction and Raman spectral data analysis is thereby proposed.

Benchmarking City Layouts—A Methodological Approach and an Accessibility Comparison between a Real City and the Garden City

This article presents a comparative accessibility study between a real city and its redraft as a Garden City. The benchmarking methodology involves defining and evaluating a location-based accessibility indicator in a GIS environment for the city of Coimbra, Portugal, and for the same city laid out as a Garden City, with the same number of inhabitants, jobs, and similar number of urban facilities. The results are derived as maps and weighted average distances per inhabitant to the facilities and jobs, and show that, for the Garden City, average distances drop to around 500 m for urban facilities and 1500 m for the combination of facilities and jobs, making much of the city accessible by walking and practically the whole of it accessible by cycling, with positive impact on transport sustainability and accessibility equity. The methodology can be extended to other benchmarking indicators and city layouts, and the quantitative results it yields make a valuable contribution to the debate on the ideal layout of cities. Moreover, it gives directions on how to improve real cities to address current and future sustainability concerns.

Development of methods for identifying an appropriate benchmarking peer to establish information security policy

Benchmarking methodology provides organizations with appropriate information security policy. However, selecting an appropriate organization as a benchmarking peer can be a challenge due to firms’ heterogeneous implementation and usage of information systems. Our goal is to develop and propose methods to appropriately identify a benchmarking peer organization by incorporating machine learning methods and mathematics set theory. We incorporate vague soft set, entropy, dynamic time warping, and Gaussian process. We use log data from information security management systems in multiple companies to validate our methods. Our experimental results indicate that the combined use of Gaussian process, vague soft set, and dynamic time warping can be more effective in identifying an appropriate benchmarking peer than conventional machine learning methods.

Cybertwin-driven resource allocation using deep reinforcement learning in 6G-enabled edge environment

The recent emergence of sixth-generation (6G) enabled wireless communication technology has resulted in the rapid proliferation of a wide range of real-time applications. These applications are highly data-computation intensive and generate huge data traffic. Cybertwin-driven edge computing emerges as a promising solution to satisfy massive user demand, but it also introduces new challenges. One of the most difficult challenges in edge networks is efficiently offloading tasks while managing computation, communication, and cache resources. Traditional statistical optimization methods are incapable of addressing the offloading problem in a dynamic edge computing environment. In this work, we propose a joint resource allocation and computation offloading scheme by integrating deep reinforcement learning in Cybertwin enabled 6G wireless networks. The proposed system uses the potential of the MATD3 algorithm to provide QoS to end-users by minimizing the overall latency and energy consumption with better management of cache resources. As these edge resources are deployed in inaccessible locations, therefore, we employ secure authentication mechanism for Cybertwins. The proposed system is implemented in a simulated environment, and the results are calculated for different performance metrics with previous benchmark methodologies such as RRA, GRA, and MADDPG. The comparative analysis reveals that the proposed MATD3 reduces end-to-end latency and energy consumption by 13.8% and 12.5% respectively over MADDPG with a 4% increase in successful task completion.

Extracting network based attack narratives through use of the cyber kill chain: A replication study

Abstract The defense of a computer network requires defenders to both understand when an attack is taking place and understand the larger strategic goals of their attackers. In this paper we explore this topic through the replication of a prior study “Extracting Attack Narratives from Traffic Datasets” by Mireles et al. [Athanasiades, N., et al., Intrusion detection testing and benchmarking methodologies, in First IEEE International Workshop on Information Assurance. 2003, IEEE: Darmstadt, Germany]. In their original research Mireles et al. proposed a framework linking a particular cyber-attack model (the Mandiant Life Cycle Model) and identification of individual attack signatures into a process as to provide a higher-level insight of an attacker in what they termed as attack narratives. In our study we both replicate the original authors work while also moving the research forward by integrating many of the suggestions Mireles et al. provided that would have improved their study. Through our analysis, we confirm the concept that attack narratives can provide additional insight beyond the review of individual cyber-attacks. We also built upon one of their suggested areas by exploring their framework through the lens of Lockheed Martin Cyber Kill Chain. While we found the concept to be novel and potentially useful, we found challenges replicating the clarity Mireles et al. described. In our research we identify the need for additional research into describing additional components of an attack narrative including the nonlinear nature of cyber-attacks and issues of identity and attribution.

Benchmarking In-Memory Computing Architectures

In-memory computing (IMC) architectures have emerged as a compelling platform to implement energy efficient machine learning (ML) systems. However, today, the energy efficiency gains provided by IMC designs seem to be leveling off and it is not clear what the limiting factors are. The conceptual complexity of IMCs combined with the absence of a rigorous benchmarking methodology makes it difficult to gauge progress and identify bottlenecks in this exciting field. This paper presents a benchmarking methodology for IMCs comprising: 1) a compositional view of IMCs that enables one to parse an IMC design into its canonical components; 2) a set of benchmarking metrics to quantify the performance, efficiency, and accuracy of IMCs; and 3) a strategy for analyzing the reported IMC data and metrics. We apply the proposed benchmarking methodology on an extensive database of IMC metrics extracted from >70 IC designs published since 2018, in order to infer and comprehend trends in this area. Our benchmarking effort indicates: 1) SRAM-based IMCs show a clear win in terms of energy efficiency and compute density over digital accelerators at the bank-level but the energy efficiency gap reduces dramatically when comparing at the processor level; 2) eNVM-based IMCs lag behind SRAM-based IMCs in terms of both energy efficiency and compute density, and surprisingly lag digital accelerators in terms of compute density; 3) the compute (bank-level) accuracy of IMCs, though a critical metric, is pervasively neglected in publications as is the energy vs. accuracy trade-off inherent to IMCs.

ОЦЕНКА СООТВЕТСТВИЯ ПРИОРИТЕТОВ СТРАТЕГИЧЕСКОГО РАЗВИТИЯ РЕГИОНОВ ИХ ОТРАСЛЕВОЙ СПЕЦИАЛИЗАЦИИ НА ОСНОВЕ TEXT MINING

The task of determining the correctness of self-positioning of regions in terms of verifying the compliance of texts of regional development strategies with their industry specialization was solved in the course of the research presented in the article. Using the "smart" benchmarking methodology, as well as the Text Mining tools, long-term development strategies of 11 regions with a total text corpus of 415,780 words were analyzed. The main sections of the all-Russian classifier of economic activities that characterize the sectoral priorities of regional development were selected as keywords. The extraction of key concepts from strategy texts, as well as their quantitative analysis, was carried out using the high-level Python programming language. The obtained quantitative results of comparing the named entities of the development strategies of the subjects of the Russian Federation proved that the insufficiency of unique goal-setting in terms of identifying promising specializations in regional development strategies distorts the system of priority development directions. This is objectively one of the reasons why the territories do not achieve the planned indicators. The paper uses methods of text mining, mathematical statistics, grouping and generalization, as well as techniques for visualizing the analyzed data. The author's method of conducting intellectual analysis of texts is universal for any field of science. The developed algorithms for extracting named entities from the text and algorithms for quantitative analysis of the text open up wide horizons for further research in the field of strategy analysis, as public documents addressed to interested subjects.

Inferential Statistics and Machine Learning Models for Short-Term Wind Power Forecasting

The inherent randomness, intermittence and volatility of wind power generation compromise the quality of the wind power system, resulting in uncertainty in the system's optimal scheduling. As a result, it's critical to improve power quality and assure real-time power grid scheduling and grid-connected wind farm operation. Inferred statistics are utilized in this research to infer general features based on the selected information, confirming that there are differences between two forecasting categories: Forecast Category 1 (0–11 h ahead) and Forecast Category 2 (12–23 h ahead). In z-tests, the null hypothesis provides the corresponding quantitative findings. To verify the final performance of the prediction findings, five benchmark methodologies are used: Persistence model, LMNN (Multilayer Perceptron with LM learning methods), NARX (Nonlinear autoregressive exogenous neural network model), LMRNN (RNNs with LM training methods) and LSTM (Long short-term memory neural network). Experiments using a real dataset show that the LSTM network has the highest forecasting accuracy when compared to other benchmark approaches including persistence model, LMNN, NARX network, and LMRNN, and the 23-steps forecasting accuracy has improved by 19.61%.

A Comprehensive Survey on the Most Important IPv4aaS IPv6 Transition Technologies, their Implementations and Performance Analysis

As the central public IPv4 address pool has already been exhausted, the deployment of IPv6 has become inevitable. However, the users still require IPv4 Internet access due to some IPv4-only applications. The IPv4aaS (IPv4-as-a-Service) IPv6 transition technologies facilitate that ISPs provide IPv4 service to their customers while using only IPv6 in their access and core networks. This paper discusses the widely used IPv4aaS IPv6 transition technologies in ISP/enterprise networks; we explain their operations, advantages, properties and consider their performances. There are currently many IPv6 transition technologies, nevertheless, in this paper, the five most prominent IPv4aaS IPv6 transition technologies are discussed, namely 464XLAT, Dual-Stack Lite, Lightweight 4over6, MAP-E, and MAP-T. Moreover, the deployment and implementations of these technologies are being analysed and inspected. This paper also overviews the benchmarking methodology for IPv6 transition technologies and surveys several papers that investigated metrics and tools utilized in analysing the performance of different IPv6 transition technologies.

Assessment of the PLC Code generated with the GEMMA-GRAFCET Methodology

Industry 4.0 has fostered the digital transformation of enterprises through the retrofitting or replacement of traditional manufacturing processes with Cyber Physical Systems (CPS). Even if interoperability is paramount to correctly operate CPS, it still remains a challenge due to lack of universal standards for the management of the operational modes of CPS through the PLC code. The GEMMA-GRAFCET Methodology (GG-Methodology) was introduced as a standard approach and vocabulary to face this issue. However, the PLC code obtained from its application has not been analyzed yet. Thus, the novelty of this research work consists in the assessment of the PLC code obtained from the application of the GG-Methodology using metrics resulting from static and dynamic analysis. In comparison to a benchmark methodology, the GG-Methodology results in PLC code more complex and with larger response time, but more maintainable and with similar execution time. Given the large number of methodologies for the development of PLC code presented in the literature, authors hope that the approach adopted within this work may be utilized for generating a quantitative and objective classification that may guide the control practitioner in the methodology selection process.

Towards Implementing a Software Tester for Benchmarking MAP-T Devices

Several IPv6 transition technologies have been designed and developed over the past few years to accelerate the full adoption of the IPv6 address pool. To make things more organized, the Benchmarking Working Group of IETF has standardized a comprehensive benchmarking methodology for these technologies in its RFC 8219. The Mapping of Address and Port using Translation (MAP-T) is one of the most important transition technologies that belong to the double translation category in RFC 8219. This paper aims at presenting our progress towards implementing the world’s first RFC 8219 compliant Tester for the MAP-T devices, more specifically, the MAP-T Customer Edge (CE) and the MAP-T Border Relay (BR). As part of the work of this paper, we presented a typical design for the Tester, followed by a discussion about the operational requirements, the scope of measurements, and some design considerations. Then, we installed a testbed for one of the MAP-T implementations, called Jool, and showed the results of the testbed. And finally, we ended up with a brief description of the MAP-T test program and its configuration parameters in case of testing the BR device.

Robust relation between public procurement for innovation and economic development

We found a solid and robust relationship between the share of public procurement for innovation (PPI) in public procurement and GDP per capita for 30 European countries. The share of PPI is highly associated with determinants from “demand pull” as well as “supply push”. These findings open new opportunities for the study of the drivers of public procurement for innovation. The study also provides a new methodology for benchmarking.

Fast and versatile analysis of liposome encapsulation efficiency by nanoParticle exclusion chromatography

Liposomes are an attractive drug delivery platform for a wide variety of pharmaceutical molecules. Encapsulation efficiency, which refers to the amount of drug contained inside liposomes compared with the total amount of drug, is a critical quality attribute of liposome products, as the free drug in a liposomal formulation may cause toxicity or undesired biodistribution. The determination of encapsulation efficiency requires the measurement of at least two of the three drug populations: total drug, encapsulated drug and free drug. However, direct measurement of the encapsulated drug and free drug remains a challenging analytical task. Nanoparticle exclusion chromatography (nPEC), an emerging high-performance liquid chromatography (HPLC) technique, has shown great potential in separating and quantifying the free drug in liposomal formulations. In this study, nPEC was systematically evaluated for two representative liposomal formulations containing either hydrophilic or hydrophobic small molecule drugs. It is reported for the first time that the insoluble free drug suspended in the aqueous formulation can be directly measured by nPEC. This free drug in the suspension sample was quantified with excellent accuracy and precision. On the other hand, the total drug measurement from dissociated liposomes was confirmed by the benchmark methodology of reversed phase liquid chromatography (RPLC). The facile quantitation of free and total drug in the liposome formulation enables the fast and accurate determination of the encapsulation efficiency, which can be used to guide the formulation development and characterize the product quality.

Capacity Building for Internationalization at a Technical University in Kazakhstan

This article discusses the internationalization capacity building project at the Karaganda Technical University (‘KTU’) in Kazakhstan. The project, entitled “Capacity building for the internationalization of a technical university by means of digital learning technologies,” was supported by the fund “Scientific foundations “Mangilik el” (education of the 21st century, fundamental and applied research in the humanities)” for scientists 2020–2022. The project’s primary purpose was to explicate a sustainable strategy for internationalizing a Kazakh technical university, considering the national and international contexts, evaluating internationalization capacity needs, formulating and implementing a capacity-building response for internationalization, and enhancing the qualifications and abilities of students and teachers to an internationally comparable level. In addition, the project delivers a set of strategies for internationalization and a benchmarking methodology for evaluating the effectiveness of internationalization processes. The benchmarking analysis of the internationalization process of a university has been conducted through a classification method, comparative analysis, multi-factor SWOT analysis, and correlation analysis.

Análisis del Diseño de la Estrategia RIS3 de Andalucía a través del Benchmarking Regional

Research and innovation strategies for smart specialisation (RIS3), conceived as a tool to improve the economic performance of regions through innovation, has played a key role in the European Union regional policy in the 2014-2020 programing period.
 One of the main points of RIS3 strategies is the selection of few specialisation priorities in which regional economies have a certain competitive advantage. Regional benchmarking exercises are encouraged by the European Commission, to provide a better vision of the position of the region and detect its strengths and weaknesses.
 In our work we focus our attention in Andalusia, a region in which European regional policy has a very relevant historical importance. We intend to reinforce the SWOT analysis included in the RIS3 strategy of Andalusia 2014-2020, performing a detailed comparative analysis of the region in relation to a selection of reference regions, using the benchmarking methodology proposed by the European Commission. This exercise allows us to enrich the SWOT analysis included in the Andalusian RIS3 strategy 2014-2020 and, therefore, to design new proposals for action that could have been included and to propose a cualitative methodology that could be useful in the design of RIS3 strategies in the period 2021-2027.

MiRPreM and tiRPreM: Improved methodologies for the prediction of miRNAs and tRNA-induced small non-coding RNAs for model and non-model organisms

In recent years, microRNAs (miRNAs) and tRNA-derived RNA fragments (tRFs) have been reported extensively following different approaches of identification and analysis. Comprehensively analyzing the present approaches to overcome the existing variations, we developed a benchmarking methodology each for the identification of miRNAs and tRFs, termed as miRNA Prediction Methodology (miRPreM) and tRNA-induced small non-coding RNA Prediction Methodology (tiRPreM), respectively. We emphasized the use of respective genome of organism under study for mapping reads, sample data with at least two biological replicates, normalized read count support and novel miRNA prediction by two standard tools with multiple runs. The performance of these methodologies was evaluated by using Oryza coarctata, a wild rice species as a case study for model and non-model organisms. With organism-specific reference genome approach, 98 miRNAs and 60 tRFs were exclusively found. We observed high accuracy (13 out of 15) when tested these genome-specific miRNAs in support of analyzing the data with respective organism. Such a strong impact of miRPreM, we have predicted more than double number of miRNAs (186) as compared with the traditional approaches (79) and with tiRPreM, we have predicted all known classes of tRFs within the same small RNA data. Moreover, the methodologies presented here are in standard form in order to extend its applicability to different organisms rather than restricting to plants. Hence, miRPreM and tiRPreM can fulfill the need of a comprehensive methodology for miRNA prediction and tRF identification, respectively, for model and non-model organisms.

Early Benchmarking Total Hip Arthroplasty Implants Using Data from the Michigan Arthroplasty Registry Collaborative Quality Initiative (MARCQI).

BackgroundBenchmarking arthroplasty implant revision risk is an informative way to address implant performance. National benchmarking efforts exist in the United Kingdom, Netherlands, and Australia. Recently, the International Prosthesis Benchmarking Working Group, including representatives from industry, academia, and national registries, produced a guideline describing arthroplasty benchmarking methodology. The proposal was applied to data from the Michigan Arthroplasty Registry Collaborative Quality Initiative (MARCQI) to assess its feasibility for benchmarking implants in the United States.MethodsPrimary elective total hip arthroplasty procedures performed for osteoarthritis between 2/15/2012 and 12/31/2018 and their associated revisions were identified in the MARCQI registry. The guidelines recommend that all prostheses combinations receive an early benchmark if they have at least 250 procedures at risk and the revision rate does not exceed the pre-determined standard of 2% at 2 years and 3% at 5 years.ResultsA total of 72,949 primary cases met the inclusion criteria. Of these, 1369 had revisions. Twenty-nine and six stem/cup combinations satisfied the minimum case requirement at 2 and 5 years, respectively. Three implant combinations would not receive a benchmark at 2 years: Secur-Fit/Trident, Anthology/Reflection 3, Taperloc 133/G7.ConclusionThe guideline can be implemented in the United States by a regional registry. Moreover, not all hip implants currently in use would receive an early benchmark. This raises concern as these implant combinations represent a significant number of cases in Michigan, some with increasing utilization.

Robust Evaluation of Ultraviolet-C Sensitivity for SARS-CoV-2 and Surrogate Coronaviruses.

ABSTRACTUV light, more specifically UV-C light at a wavelength of 254 nm, is often used to disinfect surfaces, air, and liquids. In early 2020, at the cusp of the COVID-19 pandemic, UV light was identified as an efficient means of eliminating coronaviruses; however, the variability in published sensitivity data is evidence of the need for experimental rigor to accurately quantify the effectiveness of this technique. In the current study, reliable and reproducible UV techniques have been adopted, including accurate measurement of light intensity, consideration of fluid UV absorbance, and confirmation of uniform dose delivery, including dose verification using an established biological target (T1UV bacteriophage) and a resistant recombinant virus (baculovirus). The experimental results establish the UV sensitivity of SARS-CoV-2, HCoV-229E, HCoV-OC43, and mouse hepatitis virus (MHV) and highlight the potential for surrogate viruses for disinfection studies. All four coronaviruses were found to be easily inactivated by 254 nm irradiation, with UV sensitivities of 1.7, 1.8, 1.7, and 1.2 mJ/cm2/log10 reduction for SARS-CoV-2, HCoV-229E, HCoV-OC43, and MHV, respectively. Similar UV sensitivities for these species demonstrate the capacity for HCoV-OC43, HCoV-229E, and MHV to be considered surrogates for SARS-CoV-2 in UV-inactivation studies, greatly reducing hazards and simplifying procedures for future experimental studies.IMPORTANCE Disinfection of SARS-CoV-2 is of particular importance due to the global COVID-19 pandemic. UV-C irradiation is a compelling disinfection technique because it can be applied to surfaces, air, and water and is commonly used in drinking water and wastewater treatment facilities. UV inactivation depends on the dose received by an organism, regardless of the intensity of the light source or the optical properties of the medium in which it is suspended. The 254 nm irradiation sensitivity was accurately determined using benchmark methodology and a collimated beam apparatus for four coronaviruses (SARS-CoV-2, HCoV-229E, HCoV-OC43, and MHV), a surrogate indicator organism (T1UV), and a resistant recombinant virus (baculovirus vector). Considering the light distribution across the sample surface, the attenuation of light intensity with fluid depth, the optical absorbance of the fluid, and the sample uniformity due to mixing enable accurate measurement of the fundamental inactivation kinetics and UV sensitivity.

Monolithic Ontological Methodology (MOM): An Effective Software Project Management Approach

Due to rapid changes in software applications, especially incorporating the demands of self-regulating technologies becomes a major challenge in software projects. This research focuses on technological, managerial, and procedural challenges, which are believed to the most significant factors contributing to projects failure. To address these issues, this study proposes Monolithic Ontological Methodology (MOM) which addresses the weakness in the existing benchmark methodologies including PRINCE2, Extreme Programming, and Scrum in terms of project management, quality control, and stakeholder involvement. The MOM consists of seven phases and each phase has the required number of iterations until it is approved by management. The updated information is recorded and shared with the respective teams. The standard documentation with control language is structured by Descriptive Logic (DL) that reduces ambiguity and technical debate. Furthermore, the illustration of the MOM includes figures, logical expressions, and descriptions. To demonstrate the effectiveness of the proposed methodology, an Analytic Hierarchy Process (AHP) was performed. The findings indicate the validity of MOM concerning considered performance metrics. Although the applicability of the proposed methodology involves relatively more documentation and formalities. The adaptive nature of MOM makes it suitable for the standard organization and brings sustainability to the organization by implementing distributed project management.