Multi-criteria Performance Evaluation Research Articles

Decision support in the budget management of a public institution: a constructivist multicriteria model

The need for having controls that provide support for decision-making and the improvement of management techniques encourages managers to seek greater efficiency in the use of public resources. Utilizing tools that allow for a more in-depth evaluation and analysis, using clearly defined and transparent criteria encompassing the public budget, assists in decision support and enhancing the budgeting process. By defining criteria for each context, those responsible gain the means to support their decisions, backed by consistent and measurable information. In this way, this study aimed to construct a multicriteria performance evaluation model to support the budget management of a public institution, based on the needs and perceptions of its budget manager. To do this, the MCDA-C methodology was adopted as the methodological procedure. This instrument allowed for the identification of 34 descriptors considered relevant by the decision-maker, an understanding of the impact of their decisions, and the recommendation of improvement actions. Furthermore, the study compared the criteria identified in the literature, directed toward performance evaluation and public budget, with the descriptors highlighted in the model. It was found that some concerns of the decision-maker, listed in the model, are not present in the criteria identified in the literature, resulting from a constructivist bias. As a contribution, the model provides a new tool to support the manager in making decisions regarding the use of public resources, as well as serving as a reference for other institutions and in the development of further research.

Uni- and multivariate bias adjustment of climate model simulations in Nordic catchments: Effects on hydrological signatures relevant for water resources management in a changing climate

Hydrological climate-change-impact studies depend on climatic variables simulated by climate models. Due to parametrization and numerous simplifications, however, climate-model outputs come with systematic biases compared to the observations. In the past decade, several methods of different complexity and dimensionality for adjustment of such biases were introduced, but their benefits for impact studies and accurate streamflow projections are still debated. In this paper, we evaluated the ability of two state-of-the-art, advanced multivariate bias-adjustment methods to accurately reproduce 16 hydrological signatures, and compared their performance against two parsimonious univariate bias-adjustment methods based on a multi-criteria performance evaluation. The results indicated that all bias-adjustment methods considerably reduced biases and increased the consistency of simulated hydrological signatures. The added value of multivariate methods in maintaining dependence structures between precipitation and temperature was not systematically reflected in the resulting hydrological signatures, as they were generally outperformed by univariate methods. The benefits of multivariate methods only emerged for low-flow signatures in snowmelt-driven catchments. Based on these findings, we identified the most suitable bias-adjustment methods for water-resources management in Nordic regions under a changing climate, and provided practical guidelines for the selection of bias-adjustment methods given specific research targets and hydroclimatic regimes.

Multi-Criteria Performance Evaluation System in the Industrial Heritage Reuse Plan

Industrial heritage is unique in terms of its patterns, reuse characteristics, economic relevance and social operations. Under the government’s “Industrial Heritage Regeneration Project”, a menu of outcome/performance assessments have been in practice and have evolved. This paper combines expert methods, the Analytic Hierarchy Process, and the Multi-criteria Evaluation Method, along with qualitative and quantitative data, to clarify the hierarchical structure and weight of factors that influence outcome/performances. The research purpose is to establish an integrated multi-criteria performance evaluation model for the reuse of industrial heritage. The findings identify a long list of factors by crossing the four major factors concerning the reuse of industrial heritage, namely regeneration creativity, operational management, implementation effectiveness and sustainable developments. Regeneration creativity is considered as the most important element, and the presentation of thematic plans and characteristics is the most important influencing factor. It is suggested that clarifications should be made concerning the factors that affect different facets in the execution of reuse initiatives. The research findings can serve as a reference for decision-making in reuse and development by reflecting the culture and value for the reuse and implementations of industrial heritage.

Multi-Criteria Performance Evaluation of Gridded Precipitation and Temperature Products in Data-Sparse Regions

Inadequate climate data stations often make hydrological modelling a rather challenging task in data-sparse regions. Gridded climate data can be used as an alternative; however, their accuracy in replicating the climatology of the region of interest with low levels of uncertainty is important to water resource planning. This study utilised several performance metrics and multi-criteria decision making to assess the performance of the widely used gridded precipitation and temperature data against quality-controlled observed station records in the Lake Chad basin. The study’s findings reveal that the products differ in their quality across the selected performance metrics, although they are especially promising with regards to temperature. However, there are some inherent weaknesses in replicating the observed station data. Princeton University Global Meteorological Forcing precipitation showed the worst performance, with Kling–Gupta efficiency of 0.13–0.50, a mean modified index of agreement of 0.68, and a similarity coefficient SU = 0.365, relative to other products with satisfactory performance across all stations. There were varying degrees of mismatch in unidirectional precipitation and temperature trends, although they were satisfactory in replicating the hydro-climatic information with a low level of uncertainty. Assessment based on multi-criteria decision making revealed that the Climate Research Unit, Global Precipitation Climatology Centre, and Climate Prediction Centre precipitation data and the Climate Research Unit and Princeton University Global Meteorological Forcing temperature data exhibit better performance in terms of similarity, and are recommended for application in hydrological impact studies—especially in the quantification of projected climate hazards and vulnerabilities for better water policy decision making in the Lake Chad basin.

Life-cycle assessment and multi-criteria performance evaluation of pervious concrete pavement with fly ash

Pervious concrete pavement has great potential in reducing surface water runoff and improving water quality. This study aims to conduct life cycle assessment (LCA) and multi-criteria assessment of pervious concrete pavement with the use of fly ash. The research significance lies in the integration of hydraulic and structure designs for developing functional unit of pervious concrete pavement and combining environmental impacts with engineering and economic indicators. A full LCA of pervious concrete pavement system was conducted including the stages of material, construction, transportation, use, maintenance, and end-of-life stage. The impact assessment included global warming potential, energy consumption, and eutrophication potential due to runoff purification. The pervious concrete pavement structures were designed based on mechanical properties of mixtures, hydrological requirement of reservoir layer, and subgrade soil type to meet equivalent performance on parking lot and highway shoulder. Multi-criteria analysis of pervious concrete mixes and pavement systems were presented based on the normalization of environmental impacts, engineering performances, and economic costs. The LCA results showed that the environmental impacts of pervious pavement system were mainly generated from material stage in terms of 81% to 92% GHG emissions and 70% to 83% energy consumption. The performance rankings of pervious concrete mixes were different from those of pervious pavement structures. The pervious concrete pavement with fly ash can cause greater or less environmental impact, depending on mechanic properties of pervious concrete and the required surface layer thickness to achieve equivalent structural performance.

Virtual reality supported interactive tower crane layout planning for high-rise modular integrated construction

Tower crane layout planning (TCLP) is critical to achieving safe and efficient module installation for high-rise modular integrated construction (MiC). Nevertheless, the existing TCLP methods are inefficient in addressing the multiple heavy lifts in high-rise MiC. Therefore, this paper aims to develop an innovative virtual reality (VR) tool to support main contractors in selecting the optimal tower crane layout plan for high-rise MiC. The paper adopts the design science methodology for tool development and evaluation. The tool is designed with three functional components, i.e., crane layout generation with real-time feasibility checking, layout selection using multi-criteria performance evaluation, and interactive crane-lift simulation. Using a real-life high-rise MiC project, the tool is proved to be innovative in interactive human-in-the-loop planning, effective in determining the safe and efficient crane layout, and efficient in use. The work described integrates human-in-the-loop simulation and multi-criteria decision-making via VR technologies.

Multicriteria performance evaluation of fiber-reinforced cement composites: An environmental perspective

Cement composites have the intrinsic drawback of low flexural strength and crack endurance. Many studies have been conducted to improve the mechanical performance of cement composites by employing commercially available fibers, but less attention has been paid to the environmental impacts associated with these fibers. Therefore, this study aims to evaluate and compare the performance of different fiber reinforced cement composites (FRCCs) based on their mechanical strength, environmental impact potential, and cost. For this purpose, the overall performance of six different fiber (such as Recycled carbon fiber (rCF), Basalt Fiber (BF), Glass fiber (GF), Polypropylene fiber (PPF), Polyvinyl alcohol fiber (PVAF), and Virgin carbon fiber (vCF)) reinforced cement composites were evaluated. Results revealed that the flexural strength of recycled carbon fiber reinforced cement composite (M-rCF) was highest among all other FRCCs. Moreover, depending on the environmental impact category, the lowest impacts were employed by M-rCF and M-PPF. At the same time, the minimum production cost was associated with M-PPF. Therefore, a multicriteria decision technique was employed to evaluate the overall performance of all the FRCCs. Based on the performance score, the FRCCs can be ranked as: M-rCF > M-PPF > M-BF > M-GF > M-PVAF > M-vCF.

A Dynamic Evaluation of Distress Predication Models Using Malmquist-DEA

So far, the dominant comparative studies of competing distress prediction models (DPMs) have been restricted to the application of static evaluation frameworks and as such overlooked their performance over time. This study fills this gap by proposing a multi-period, multi-criteria performance evaluation framework; namely, Malmquist DEA, which by design provides multi-criteria rankings of competing DPMs over multiple periods and thus highlights the performance dynamics over time and reveals sources of inefficiencies of these models, and discuss its implementation decisions. In addition, we use the proposed framework to address a variety of research questions. Using data on UK listed companies, this study provides a dynamic comprehensive comparison between the most cited static and dynamic DPMs. Our empirical findings suggest that increasing the length of the training sample does not necessarily lead to an improvement in the performance of models; in fact, the type of classifier and information fed into the models have critical roles in improving the performance of DPMs during both regular and high distress rate periods.

MARCOS technique under intuitionistic fuzzy environment for determining the COVID-19 pandemic performance of insurance companies in terms of healthcare services

Assessing and ranking private health insurance companies provides insurance agencies, insurance customers, and authorities with a reliable instrument for the insurance decision-making process. Moreover, because the world’s insurance sector suffers from a gap of evaluation of private health insurance companies during the COVID-19 outbreak, the need for a reliable, useful, and comprehensive decision tool is obvious. Accordingly, this article aims to identify insurance companies’ priority ranking in terms of healthcare services in Turkey during the COVID-19 outbreak through a multi-criteria performance evaluation methodology. Herein, alternatives are evaluated and then ranked as per 7 criteria and assessments of 5 experts. Experts’ judgments and assessments are full of uncertainties. We propose a Measurement of Alternatives and Ranking according to the Compromise Solution (MARCOS) technique under an intuitionistic fuzzy environment to rank insurance companies. The outcomes yielded ten insurance companies ranking in terms of healthcare services in the era of COVID-19. The payback period, premium price, and network are determined as the most crucial factors. Finally, a comprehensive sensitivity analysis is performed to verify the proposed methodology’s stability and effectiveness. The introduced approach met the insurance assessment problem during the COVID-19 pandemic very satisfactory manner based on sensitivity analysis findings.

Fuzzy logic decision support system for hospital employee performance evaluation with maple implementation

In any organization, the capability, knowledge and skills play a significant role in success of an employee. For instant, in hospital management, the performance evaluation of an employee is mainly based on qualitative in nature. Evaluation of certain factors such as uncertainty, vagueness and imprecision is based on the decision making ability of the evaluator. The performance evaluation for promotion, incentives, bonuses, growth and development of any employee should be recognized in effective and efficiency manner with appropriate ratings. Therefore, this paper aims to design and implement a multi criteria performance evaluation for hospital employees to get promotion, incentives, bonus, growth and development. In this paper, we propose an Interval Valued Fuzzy Weighted Distance Algorithm (IVFWDA) for performance evaluation. The expected interval and the actual interval of the work achievement, management skill, personal quality, care, safety and risk management features are extracted by Interval Valued Fuzzy Soft Matrix (IVFSM) and it is effectively done using the proposed algorithm. An example is presented as a case study to illustrate the proposed algorithm. Implementation of the proposed algorithm in Maple is also discussed with sample computations.

A simulation-based dynamic scheduling and dispatching system with multi-criteria performance evaluation for Industry 3.5 and an empirical study for sustainable TFT-LCD array manufacturing

As the existing manufacturing systems may not be ready to support flexible decisions for smart production with increasing product mix and shortening product life cycle, it is crucial to rapidly respond to dynamic needs to improve bottleneck productivity and ensure the throughput of the whole manufacturing system. Limitations of the existing approaches can be traced in part to the lack of a framework within which different decisions in real settings can be integrated and aligned in light of the changes of manufacturing contexts. To fill the gaps, this study aims to develop a dynamic scheduling and dispatching system with the constructed discrete event simulation model to optimise the scheduling for bottleneck and associated dispatching rules for remaining processes, while considering the manufacturing system as a whole to empower smart manufacturing and reduce waste for sustainable production. A hybrid genetic algorithm is developed to minimise photolithography capacity loss, while considering the waiting time constraints and determining the optimal dispatching rules for non-bottleneck workstations by the design of experiments and multi-criteria decision analysis to integrate related decisions for the manufacturing system as a whole. An empirical study was conducted for validation. The results have shown its practical viability.

Multi-criteria performance evaluation of a glass panel system using full-scale experimental data

In order to better evaluate the lifetime performance of glass panel facade systems, this study firstly proposes to conduct a full-scale wind resistance, air/water infiltration, and seismic resistance tests. Two specimens were designed, constructed, and tested for this purpose. A fatigue process is also applied to one of the manufactured systems in addition to a standard test procedure. Secondly, data acquired from the experimental part of this work have been used in a multi-criteria performance evaluation process to determine the performance levels of each of the specimens. In the evaluation process, a total of 10 different criteria and 2 alternatives are identified for decision making. Performance indicator data obtained from the detailed calculations using the AHP (Analytic Hierarchy Process), COPRAS (Complex Proportional Assessment) and TOPSIS (The Technique for Order of Preference by Similarity to Ideal Solution) methods are then utilized during the numerical assessment process. The ‘criticality level’ is defined within the evaluation process and it indicates the percent (%) values of the obtained performance indicators. As a result of the performance evaluations, the AHP and TOPSIS methods gave similar results and could be selected as the preferred methods for the evaluation of such experimental data. Results obtained from the AHP and TOPSIS methods are consistent and easy to interpret when compared to the COPRAS method. Furthermore, in this work, the most critical criterion is found to be the frontal deflections.

A standard-based approach for multi-criteria performance evaluation of engineered systems

It is a common practice to periodically monitor the key performance indicators of an engineered system (ES) and quantitatively evaluate the system's ability to perform its functions. For a specific ES, the technical standards for managing its operations and maintenance are usually available and deal with multiple aspects. Therefore, the performance evaluation of ESs should use a standard-based multi-criteria decision analysis (MCDA) approach. It is different from the generic MCDA methods where the performance evaluation for alternatives is based on the decision matrices of alternatives. The literature on performance evaluation of ESs is vast but the literature on standard-based performance evaluation is very limited. This paper aims to fill this gap by proposing a standard-based MCDA approach for evaluating operational performance of ESs. The proposed approach derives the weights of the performance parameters and the parameters of their normalization functions based on the information specified in the technical standards. The robustness and weak compensation degree of the reverse quadratic mean model are identified and it can be used as an aggregation model. A real-world case study is included to illustrate the proposed approach and its usefulness.

A novel multi-criteria analysis model for the performance evaluation of bank regions: an application to Turkish agricultural banking

The banks serve in a highly dynamic and competitive environment and need to systematically evaluate their performance to improve their competitiveness. Performance evaluation is an important and complex process that requires flexible and analytic methods while handling the multidimensionality of the problem. This study presents a hybrid multi-criteria performance evaluation model for banking sector which combines two multi-criteria decision making methods that are simulation-integrated hesitant fuzzy linguistic term sets-based analytic hierarchy process method to determine the importance level of each criterion according to the decision makers’ subjective judgements and grey relational analysis method to rank bank regions according to their performance values. The presented model is based on both probability theory and fuzzy sets theory and thus better represents all the dimensions of the uncertainty inherent in decision making process. A real-life application of the proposed performance evaluation model for a private bank operating in agricultural banking sector in Turkey is also given to illustrate the effectiveness and the applicability of the model.

Comparative analysis of company performance evaluation methods

The evaluation of a company's performance is an integral part of the management of every company, which allows determining the impact of business management decisions on the performance results, as well as the direction of the results and the decisions that need to be made to improve them. Traditionally, a company's performance is evaluated on the basis of an analysis of financial performance indicators, but it is noted that in today's dynamic business environment the timely control of the performance is crucial, thus, there is an increasing attention on non-financial performance indicators. The aim of the article is to investigate and compare the modern methods for company performance evaluation that are based on the analysis of financial and non-financial performance indicators. The research is carried out using systematic analysis of scientific literature, comparison, and aggregation methods. First of all, the article unfolds the essence of the concept of the company performance evaluation and highlights the differences between traditional and modern performance evaluation systems. Moreover, the advantages of non-financial performance indicators are compared with financial indicators. Furthermore, a classification scheme for performance evaluation methods is provided; by identifying their main advantages and disadvantages the most popular and widely used modern performance evaluation methods are overviewed: economic value added method, balanced scorecard, performance prism, performance pyramid, six-sigma model and multi-criteria company performance evaluation method. Finally, according to certain criteria, a comparative analysis of the latter methods is performed. The comparative analysis confirms that non-financial performance indicators complement financial indicators; therefore, in the process of a company's performance evaluation, modern performance evaluation methods, combining financial and non-financial performance indicators and allowing the performance to be evaluated both quantitatively and qualitatively, should be used. However, there is no single method that would be appropriate for all companies, so further work can be targeted at establishing a multi-criteria performance evaluation method that would satisfy the needs of a particular business' activity and targets for evaluation.

A study on pipe-embedded wall integrated with ground source-coupled heat exchanger for enhanced building energy efficiency in diverse climate regions

The study proposes a new low-grade energy utilization system, i.e. a pipe-embedded wall integrated with ground source-coupled heat exchanger, and investigates its energy saving potentials in five typical climate regions of China. The cool/hot water produced by the ground source-coupled heat exchanger is supplied to the pipe-embedded wall to intercept the heat gain/loss through the wall. A real-scale building energy simulation platform has been developed for estimating the energy saving potentials. The study results show that the energy saving potential of the proposed system varies with the climate regions. With the same sized pipe-embedded wall system, the building in warm climate regions achieves the largest energy saving percentage. It is followed by the buildings in the hot summer and cold winter climate region, the severe cold climate region, the cold climate region, and the hot summer and warm winter climate region in the declining sequence of energy saving. Meanwhile, a multi-criteria performance evaluation has been conducted to search for the optimal size of the low-grade-energy system in each climate region. The proposed system can be utilized in practice for energy saving, and the study results are helpful in guiding the practical applications of the pipe-embedded wall system in different climate regions.

Multi-criteria performance evaluation of an airline maintenance programme – A case study

It has always been a concern to most airline operators why their aircraft are on the ground despite all the resources being incurred on maintenance. It is the main goal of this study to evaluate the performance of an airline maintenance programme. Primary data were sourced from the maintenance department of a local airline operator while the secondary data were sourced from aviation journals and statutory regulatory provisions respectively. The information obtained was quantitatively analysed according to maintenance practice factor (MPF), maintenance practice contribution (MPC) and analytical hierarchy process (AHP). Maintenance practices analysed were; visual inspection, servicing, lubrication, cleaning and checking. Cleaning had the highest maintenance practice with a mean MPF of 95.3%. Scheduled maintenance method had the highest contribution of use with a mean MPC of 100%, showing how effective it is. The AHP revealed that the Logistics and Planning elements of the maintenance programme contribute more to the success of the maintenance programme with a contribution of value for money of 26.1%. The consistency ratios indicate consistency in the judgments obtained from experts in the maintenance department. To have aircraft that are ready to fly, emphasis should be placed on the effectiveness and efficiency of the Logistics and Planning activities of the airline’s maintenance department.

Modified Grey Relational Analysis Integrated with Grey Dematel Approach for the Performance Evaluation of Retail Stores

Performance evaluation is one of the most important problems for retail chains and may have effect on tactical and strategic decisions. This paper proposes a grey-based multi-criteria performance evaluation model for retail sector. This model integrates Decision-Making Trial and Evaluation Laboratory (DEMATEL) and modified Grey Relational Analysis (GRA) methods. First, the grey-based DEMATEL method is used for determining the importance of performance indicators to be used in GRA based on the experts’ assessments. Then, the proposed modified GRA method is used for the performance evaluation and ranking of retail stores with respect to the predetermined performance indicators. Finally, the effectiveness and the applicability of the developed approach are illustrated with a case study with the actual data taken from a retail chain in Turkey.

The performance profile: A multi–criteria performance evaluation method for test–based problems

Abstract In test-based problems, solutions produced by search algorithms are typically assessed using average outcomes of interactions with multiple tests. This aggregation leads to information loss, which can render different solutions apparently indifferent and hinder comparison of search algorithms. In this paper we introduce the performance profile, a generic, domain-independent, multi-criteria performance evaluation method that mitigates this problem by characterizing the performance of a solution by a vector of outcomes of interactions with tests of various difficulty. To demonstrate the usefulness of this gauge, we employ it to analyze the behavior of Othello and Iterated Prisoner’s Dilemma players produced by five (co)evolutionary algorithms as well as players known from previous publications. Performance profiles reveal interesting differences between the players, which escape the attention of the scalar performance measure of the expected utility. In particular, they allow us to observe that evolution with random sampling produces players coping well against the mediocre opponents, while the coevolutionary and temporal difference learning strategies play better against the high-grade opponents. We postulate that performance profiles improve our understanding of characteristics of search algorithms applied to arbitrary test-based problems, and can prospectively help design better methods for interactive domains.

University Management with Focus on Multicriteria Performance Evaluation: Illustration in the Brazilian Context

Public universities have been facing situations that impel them to improvements in its management towards excellence in reaching, with quality, the objectives for which they were constituted, i.e., to offer teaching, research and extension. In this context, it is up to the managers of the universities to act proactively. The literature points out that the internal control systems are managerial tools capable of providing information for the manager’s actions in his/her activity; however we notice that the academic literature devoted to this demand is still scarce, especially as regards support for decision-making considering organizational specificities. Given the importance of the manager's activity and the lack of literature about the subject, it becomes important to develop tools to support the decision of those managers. For this purpose, we see Performance Evaluation, with a multicriteria focus, as a tool capable of filling this gap. Thus, this study proposes the construction of a performance evaluation model, through the Methodology Multicriteria Decision Aid – Constructivist (MCDA-C), to support the management and decision making of managers in a Brazilian public university. To illustrate the proposal for a university management we developed a case study in the Federal Technological University in southern Brazil. Among the main findings of this exploratory research we can mention: 129 aspects (indicators) were identified, grouped in the areas of Operational Support, Budget Management, Administrative Management and Transparency/Accountability, which account for the performance evaluation of DIRPLAD/UTFPR; the performance evaluation was proceeded and resulted in a score of 21 points on a scale from 0 to 100; it was shown how the built model serves as a management tool demonstrating how the manager can establish new action strategies to improve the performance of DIRPLAD /UTFPR.