Methodology For Evaluation Of Systems Research Articles

Performance evaluation of underwater vision systems

The article describes a methodology for performance evaluation of vision systems for remotely operated underwater vehicles. The methodology is based on a system approach and uses mathematical models of the aqueous medium where an optical signal propagates, the underwater object image registration system, and the mathematical model of the human visual system. The detection and recognition probabilities of underwater object image at a given registration range are used as performance evaluation indicators of underwater vision systems. The mathematical model of the aqueous medium developed by the authors allows quantitative evaluation of the influence of backscattering interference arising during objects illumination on the underwater vision system performance. The results of numerical experiments presented in the paper illustrate the possibility of using the proposed technique to optimize the underwater object image registration system parameters in order to achieve the required values of detection or recognition probabilities at the given ranges.

Decision Structuring and Robustness Analysis in Selecting Urban Transit Alternatives

Public policy-makers in many cities worldwide have recognized the need to seek urban passenger transport solutions in the domain of urban transit systems. However, with the availability of many transit technologies and systems, decision-makers need support in identifying the advantages and disadvantages of each system, and in evaluating their suitability for the specific urban context being considered.In this article, the Analytical Hierarchy Process (AHP) is proposed as a decision-support methodology for evaluation of urban transit systems. A hierarchy is proposed for the evaluation of system choice, and an illustrative example is presented. The hierarchy reflects overall objectives of transit systems, which include achieving betterment in the urban transport picture and mitigating possible implementation impediments. On the betterment side, the transit system objectives considered in the hierarchy include appropriate level of service and performance, congestion reduction potential, support of economic development, and flexibility. On the other hand, transit system impediment factors comprise system cost, environmental impacts, and implementation and operation barriers. The article demonstrates the value of the proposed approach in decision structuring for selecting among urban transit alternatives and in providing the facility of analyzing the robustness of the decision with respect to various judgments that are made in the selection process.

Performance of PV systems in Slovenia with the help of typical daily profiles and automatic detection of orientation and inclination angles

Operators of photovoltaic (PV) systems, especially the small ones, monitor only the produced energy output, since they are not equipped with a meteorological station, or there is none in the neighbourhood. When evaluating a large number of PV systems across a region or country, access to the data due to privacy restrictions is usually very limited. To resolve both aspects, we report on a methodology to evaluate the performance of photovoltaic systems where only produced energy data and rated power of the PV system are available and all other data, including orientation and inclination angles, and the exact location are missing. We developed a novel algorithm for automatic detection of orientation/inclination angles and used the recently developed Typical Daily Profiles (TDP) methodology for a fast and accurate evaluation of PV systems. TDPs are representative hourly profiles of the observed parameter (e.g., power, irradiance and temperature) on a monthly basis. For the PV system degradation rate extraction, year-to-year TDPs are used. The case study of 957 PV systems in Slovenia in the period 2015–2019 reveals an average PV system performance ratio exceeding 85% and an average PV system rated power degradation rate of −0.7% per year.

A methodology for techno-economic evaluation of asymmetric energy storage systems: A nuclear energy case study

This paper discusses a novel methodology for the analysis of energy storage system. This methodology combines engineering and economic modelling to assess the relative economic performance of plant designs within a simulated UK electricity market. It presents a case study that explores whether a nuclear power plant can be combined with a liquid air energy storage plant to allow the resultant hybrid plant to provide a load-following electricity supply. The approach adopted is distinct from much of the work on conventional load following nuclear power plant operations, as in our case the underlying nuclear energy production does not vary. The methodology expands the previous literature by performing market-led system optimisation to best design the output profile of the hybrid plant to improve economic performance in the UK electricity grid. Whilst this combined modelling approach has been demonstrated in the context of a specific plant design, this methodology might be applied to any asymmetric energy storage system to assess its economic viability. This work is both contrasting and complementary to the levelized cost of storage approach. Rather than providing the price at which energy must be sold to make a storage system economically viable, this approach considers how volatile the electricity spot market would have to become to make such a system profitable. The ability to explore different market conditions is a major omission from the existing literature on energy storage systems.

РЕЗУЛЬТАТЫ ПРИМЕНЕНИЯ МЕТОДИКИ ЧИСЛЕННОЙ ОЦЕНКИ СИСТЕМ МЕНЕДЖМЕНТА ИННОВАЦИЙ

In order to confirm the adequacy of the developed methodology for the numerical evaluation of innovation management systems, with the participation of the Russian Register Certification Association, an audit of the innovation management system in a technology company was organized and conducted. The paper presents the results of applying the above methodology when assessing compliance with the requirements of CEN/TS 16555/GOST R 56273.

Environmental Value Engineering (EVE) Assessment: A Case Study of Comparing Curtain Wall System Alternatives

The need to explore environmentally friendly materials to avoid the depletion of the non-renewable resources is very significant. Humans have been heavily dependent on non-renewable resources since the industrial revolution. This has led to environmental impacts such as increase in CO2 in the atmosphere and global warming, which is arguably one of the major problems we are facing today. Faced with the possible shortages of natural resources, pollution, population overgrowth, and concern for protecting the environment, human beings are coming to realize that new concepts are needed to analyze the interdependent parts of the built environment. Considering the present need for optimal use of our resources, as opposed to maximizing, there is an increasing shift of focus to system evaluation methodologies that can be used to evaluate the environmental impact of a product or system. It is with this ever-increasing need for products that are not only functional and cost-effective, but also environmental friendly, that Environmental Value Engineering (EVE) assessment is gaining popularity. This paper explores the application of the EVE methodology through a case study of comparing the environmental impact of the C.T.W. Series 3 Wood Hybrid Curtain Wall system and the Kawneer 1600 Wall System. The EVE methodology, which is an environmental life cycle assessment method, was used to compare the inputs of the environment, fuel energy, goods, and services in terms of emergy for both systems.

Modeling for evaluation of safety instrumented systems with heterogeneous components

A novel methodology for evaluation of safety instrumented systems (SISs) with heterogeneous components based on multi-stage dynamic Bayesian networks (MDBNs) is proposed. The unified evaluation model for M-out-of-N architectures based on MDBNs is constructed. An automatic modeling method of conditional probability tables for time-slice and inter-slice transitions is established. Average probability of failure on demand (PFDavg) and probability of failing safety (PFS) of the structure can be calculated by the established model. A method for calculating the life-cycle cost of SISs is presented by using the results of system evaluation and specific cost details. A high-integrity pressure protection system (HIPPS) is adopted to demonstrate the application of the proposed approach. A MDBNs-based evaluation and cost analysis software of heterogeneous redundant structure is developed by using MATLAB graphical user interface. The analysis of system evaluation results and life-cycle cost can help the designer determine the configuration of the heterogeneous redundant structure.

Reliability evaluation of bulk power systems with wind generation using small signal stability analysis

This paper presents a computationally efficient methodology for reliability evaluation of bulk power systems with wind power generation using small-signal stability analysis. The methodology is based on the application of Monte Carlo simulation to calculate reliability indices, where the sampled system states are evaluated by means of their small-signal stability condition. Therefore, the system small-signal stability is evaluated for scenarios that consider the failure of generation and transmission assets and the unavailability and power variations of wind farms. Unlike previous works, the proposed methodology does not consider the probability of small variations in system parameters, but instead the probability of occurrence of the scenarios themselves including the large and nonlinear variations resulting from network failures. This allows comparing different alternatives for the planning of large power systems, such as different PSS settings or system reinforcement solutions, in a more realistic approach. Additionally, the methodology uses a non-conventional partial eigenvalue solution focusing on a user-defined subset of poles of interest, efficient and practical for the small-signal analysis of large-scale power systems.

Overcoming barriers confronting application of protein therapeutics in bone fracture healing.

Bone fracture is a major contributor to debilitation and death among patients with bone diseases. Thus, osteogenic protein therapeutics and their delivery to bone have been extensively researched as strategies to accelerate fracture healing. To prevent morbidity and mortality of fractures, which occur frequently in the aging population, there is a critical need for development of first-line therapeutics. Bone morphogenic protein-2 (BMP-2) has been at the forefront of bone regeneration research for its potent osteoinduction, despite safety concerns and biophysiological obstacles of delivery to bone. However, continued pursuit of osteoinductive proteins as a therapeutic option is largely aided by drug delivery systems, playing an imperative role in enhancing safety and efficacy. In this work, we highlighted several types of drug delivery platforms and their biomaterials, to evaluate the suitability in overcoming challenges of therapeutic protein delivery for bone regeneration. To showcase the clinical considerations for each type of platform, we have assessed the most common route of administration strategies for bone regeneration, classifying the platforms as implantable or injectable. Additionally, we have analyzed the commonly utilized models and methodology for safety and efficacy evaluation of these osteogenic protein-loaded systems, to present clinical opinions for future directions of research in this field. It is hoped that this review will promote research and development of clinically translatable osteogenic protein therapeutics, while targeting first-line treatment status for achieving desired outcomes of fracture healing. Graphical abstract.

Mapping the spatial distribution of emissions from domestic shipping in Izmir Bay

The purpose of this study is to determine the risky areas in terms of air pollution from domestic shipping in Izmir Bay with the integration of Geographic Information System (GIS) and emission evaluation methodology. To do this, initially, the emissions as a result of 435 trips of 19 ships engaged in domestic transportation in Izmir Bay were evaluated by the bottom-up method. The spatial distribution map was then compiled using the point density analysis through MapInfo v.16 software as a GIS instrument. The study revealed that annual emissions of PM10,CO2,NOx, SO2, and HC from domestic shipping were 601.5 t, 925.9 t, 62612.1 t, 22.0 t, and 61.2 t, respectively. In addition, the highest emissions as ton per daily trip were recorded in the cruising mode due to longer operation periods compared to the other modes. It was found that the lines between Pasaport, Alsancak and Karsiyaka piers are the most polluted routes due to dense ship traffic. Furthermore, Bostanli and Karsiyaka piers were determined as the points with the lowest air quality due to heavy traffic. We concluded that the ship emissions do not spread homogeneously throughout the region. In order to reduce the emissions in the area, the number of daily voyages and ship size should be optimized for passenger and vehicle density. The study also recommends installing emission monitoring stations at the piers to control air pollution.

A machine learning methodology for reliability evaluation of complex chemical production systems.

System reliability evaluation is very important for safe operation and sustainable development of complex chemical production systems. This paper proposes a hybrid model for the reliability evaluation of chemical production systems. First, the influential factors in system reliability are categorized into five aspects: Man, Machine, Material, Management and Environment (4M1E), each of which represents a component subsystem of a complex chemical production process. Second, the Support Vector Machine (SVM) algorithm is used to develop machine learning models for the reliability evaluation of each subsystem, during which Particle Swarm Optimization (PSO) is applied for model parameter optimization. Third, the Random Forest (RF) algorithm is employed to correlate the reliability of the five subsystems with the reliability of the corresponding whole chemical production system. Then, Markov Chain Residual error Correction (MCRC) is adopted to improve the predictive accuracy of the machine learning model. The efficacy of the proposed hybrid model is tested on a case study, and the results indicate that the proposed model is capable of delivering satisfactory prediction results.

External therapies of traditional Chinese medicine combined with sodium hyaluronate injected in articular cavity therapy on knee osteoarthritis: Meta-analysis

To systematically evaluate the efficacy and safety of external therapies of traditional Chinese medicine(TCM) combined with sodium hyaluronate(SH) injected in articular cavity therapy on knee osteoarthritis(KOA). The following databases such as CNKI, WanFang, VIP, CBM, PubMed and Medline were researched to collect the randomized controlled trails on external therapies of TCM combined with sodium hyaluronate injected in articular cavity therapy on KOA. The selection of studies, assessment of methodological quality and data extraction were performed independently by two researchers. The methodological quality was assessed by using the Cochrane system evaluation methodology and Meta-analysis were performed by using Cochrane Collaboration's the RevMan 5.3 software. Forteen studies involving 1 449 patients were included. All of the trails were not adequate enough in methodological quality. Meta-analysis indicated that compared with control group, external therapies of TCM combined with sodium hyaluronate injected in articular cavity could raise effectiv rate(P<0.000 01) and cure-rate(P<0.000 01), improve Lysholm score(P=0.003) and reduce VAS score(P<0.000 1). But two groups have no difference in Womac score (P=0.13).Compared with the treatment with sodium hyaluronate injected in articular cavity, external therapies of TCM combined with sodium hyaluronate injected in articular cavity, a promising treatment options, can be complementary advantages, improve the clinical curativ effect. But it still needs low risk and high quality clinical trials to verify.

Comparisons of Bitcoin Cryptosystem with Other Common Internet Transaction Systems by AHP Technique

This paper describes proposed methodology for evaluation of critical systems and prioritization of critical risks and assets identified in highly secured information systems. For different types of information assets or security environments it is necessary to apply different techniques and methods for their prioritization and evaluation. In this article, VECTOR matrix method for prioritization of critical assets and critical risks is explained and integrated into AHP (Analytic Hierarchy Process) technique as a set of fixed criteria for evaluation of defined alternatives. Bitcoin cryptocurrency was compared and evaluated along with other common Internet transaction systems by information security professionals according to defined VECTOR criteria. Also, the newly proposed hybrid AHP model is presented with potential case studies for future research. This article tries to discover security posture of Bitcoin cryptocurrency in the context of information security risks related to the existing most common online payment systems like e-banking, m-banking, and e-commerce

A Blueprint for Causal Inference in Implementation Systems

Background: Following a decade of significant progress in implementation science, research efforts are increasingly focused on the investigation of implementation mechanisms and the adoption of multiple EBPs in service delivery systems. This calls for the development of a systems approach to learn about causal mechanisms in implementation systems and a formal methodology for evaluation of systems of care. Methods: We develop a multilevel decision juncture approach to study implementation systems based on advances in implementation research and methods developed in the literature on causal inference and program evaluation. Implementation phases are linked through a sequence of decisions made by relevant agents in the systems at various levels. Formulating the decision juncture as a system of structural equations, each describing a causal relationship between variables, causal mechanisms can be identified throughout the phases of implementation. We use a hypothetical case study of a parenting program in the child welfare sector to illustrate the approach following a staged process analysis model taken from the econometric literature. Following the research on causal inference in recursive structural models, we also demonstrate how graphical models can be used to identify treatment effects at different levels of the system. Results: The models presented make contributions to implementation research in two areas. First, structural systems focus on the causal mechanisms of implementation. This allows efficient selection of covariates and optimal choice of implementation strategies based on identified leverage points in the system. Second, structural models provide a rigorous approach to the evaluation of implementation success. Taking a system approach, implementation, systems and effectiveness outcomes can be investigated as causally linked parts of an implementation system. Conclusions: As implementation science shifts its focus towards large dynamic systems of care, structural causal models connecting actors’ choices and implementation phases across levels of the system provide a rigorous scientific approach to investigate implementation and intervention effectiveness as causally linked parts of the system. Furthermore, using this approach it is straight forward to extend static models of implementation to dynamic systems that can account for sustainment and provide an avenue to integrate continuous quality improvement cycles into implementation research.

A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms

In this paper a review is presented of the research on eye gaze estimation techniques and applications, that has progressed in diverse ways over the past two decades. Several generic eye gaze use-cases are identified: desktop, TV, head-mounted, automotive and handheld devices. Analysis of the literature leads to the identification of several platform specific factors that influence gaze tracking accuracy. A key outcome from this review is the realization of a need to develop standardized methodologies for performance evaluation of gaze tracking systems and achieve consistency in their specification and comparative evaluation. To address this need, the concept of a methodological framework for practical evaluation of different gaze tracking systems is proposed.

Approaching information system evaluation methodology and techniques: a comprehensive review

This paper advances our knowledge of information systems (IS) by drawing ideas and insights from IT/IS evaluation. The emergence and increasing pressure to invest in new IS architecture and infrastructures has become a high priority issue within organisations, and largely influenced by the need to deliver better value products and service through robust and responsive supply chains. With this in mind, business managers are seeking to use appropriate methods and techniques to appraise and justify the financial contribution of IS at strategic, operational and tactical levels. Thus, comprehensive but understandable methodologies are needed to solve the complicated project justification problems arising from the complexity of new technologies. Notwithstanding, a wide body of literature has emerged that sheds light on various methods for evaluating the cost and benefits of IT/IS investment, ranging from simple to very complex techniques. This paper sets out to provide a review of this literature in order to lend insights into our understanding of the plethora of reported research in the literature that has propagated investment justification and benefits management models and frameworks. We show that the IT/IS evaluation literature contributes significantly to scholarship on the IS.

Approaching information system evaluation methodology and techniques: a comprehensive review

This paper advances our knowledge of information systems IS by drawing ideas and insights from IT/IS evaluation. The emergence and increasing pressure to invest in new IS architecture and infrastructures has become a high priority issue within organisations, and largely influenced by the need to deliver better value products and service through robust and responsive supply chains. With this in mind, business managers are seeking to use appropriate methods and techniques to appraise and justify the financial contribution of IS at strategic, operational and tactical levels. Thus, comprehensive but understandable methodologies are needed to solve the complicated project justification problems arising from the complexity of new technologies. Notwithstanding, a wide body of literature has emerged that sheds light on various methods for evaluating the cost and benefits of IT/IS investment, ranging from simple to very complex techniques. This paper sets out to provide a review of this literature in order to lend insights into our understanding of the plethora of reported research in the literature that has propagated investment justification and benefits management models and frameworks. We show that the IT/IS evaluation literature contributes significantly to scholarship on the IS.

A conceptual approach for evaluating the multiple benefits of urban flood management practices

This paper focuses on the spatial distribution of the dominant and relevant benefits of urban flood management based on context‐ and location‐specific evaluations. We explore the conceptual rationale and describe a detailed methodology for assessing the benefit profile, benefit intensity, and comment on benefit dependencies arising from urban flood management practices that utilise green infrastructure. A case study is described which demonstrates the application of the concepts in Portland, Oregon, USA. A Geographic Information System approach is developed to evaluate some of the multiple benefits of the East Lents Floodplain Restoration Project. Results are presented in the form of a comparative benefit profile, and a spatially distributed benefit intensity. The paper concludes with the implications of the methodology for future multiple benefit evaluation of urban drainage and flood management systems.

Apparent performance ratio of photovoltaic systems—A methodology for evaluation of photovoltaic systems across a region

Most of the photovoltaic (PV) solar systems track none or only basic monitoring data, thus the investor has no or limited information about the performance, quality, and reliability of a PV system. In this paper, we report on a methodology to evaluate a large number of PV systems on a regional or even countrywide level. The methodology is based on an apparent performance ratio, a factor close to the temperature compensated performance ratio, although based on solar irradiation data from the closest meteorological station. The methodology is compared to a more detailed evaluation procedure and a comprehensive evaluation of a PV system where all the required—both electrical and meteorological—data are monitored. The results confirm that the apparent performance ratio is very similar to the temperature compensated performance ratio of a PV system.

Session details: Special Issue on Challenges in Software Performance

Software systems have become increasingly complex. Such complexity makes it extremely challenging to deliver systems that meet their performance requirements. In recent years, there have been several high profile instances of projects that failed due to performance-related issues, e.g., the rollout of the healthcare.gov Web site. This motivates the need for new approaches for software developers to address performance problems. This special issue presents 6 papers that address this theme. A brief summary of these papers follows. Gesvindr and Buhnova study the challenges of building Platform as a Service (PaaS) cloud applications that meet their performance requirements. Using two case studies involving the Microsoft Azure cloud platform, the authors show that the use of a cloud platform in itself does not guarantee good performance and scalability. The authors then identify several best practices for ensuring the performance and scalability of PaaS applications. The work of Heinrich considers the problem of managing\ the performance of software systems composed of thirdparty cloud services. The lack of control over such thirdparty services can cause deviations from performance objectives that can be hard to foresee and manage. The author outlines a model-based framework called iObserve that can help a system adapt and evolve so as to address such problems. Li, O'Brien, and Kihl address the lack of standard methodologies for experimental performance evaluation of software systems. The paper outlines a method for defining domainspecific methodologies for performance evaluation in a way that is consistent across domains. The authors demonstrate the utility of their approach using a cloud services case study. The paper by Grottke et al. considers the problem of handling cyber attacks on networked computer systems andconnected industrial control systems supporting critical infrastructuresuch as energy, water, and gas. Using analytic models, the authors study the performance of various attack mitigation approaches. In particular, the model shows how parameters such as malware scanning rates, network topology, and types of infections impact the effective processing capacity of a system. Rehmann et al. describe the performance monitoring in SAP HANA's continuous integration process. They describe how performance regression tests have been integrated in the delivery process and what services are available to detect and report performance anomalies. Being an extended version of a previous workshop paper, this paper now also presents the reporting of performance test results and anomaly detection. Finally, Nambiar et al. examine issues and experience with software performance, and consider the shortcomings and potential benefits of modeling from the perspective of a potential industrial user. In particular, it considers why modeling is little used, and what conditions it would have to satisfy to fulfill its potential. We sincerely thank all authors for their work and contribution to this special issue. Additionally, we thank our reviewers Davide Arcelli, Danilo Ardagna, Alberto Avritzer, Steffen Becker, Ian Gorton, Andre van Hoorn, Raffaela Mirandola, Juan F. Perez, and Murray Woodside for their timely and constructive reviews as well as editor-in-chief Nidhi Hedge for her support.