Consistent Ranking Research Articles

Comparison of metrics for the evaluation of medical segmentations using prostate MRI dataset

Nine previously proposed segmentation evaluation metrics, targeting medical relevance, accounting for holes, and added regions or differentiating over- and under-segmentation, were compared with 24 traditional metrics to identify those which better capture the requirements for clinical segmentation evaluation. Evaluation was first performed using 2D synthetic shapes to highlight features and pitfalls of the metrics with known ground truths (GTs) and machine segmentations (MSs). Clinical evaluation was then performed using publicly-available prostate images of 20 subjects with MSs generated by 3 different deep learning networks (DenseVNet, HighRes3DNet, and ScaleNet) and GTs drawn by 2 readers. The same readers also performed the 2D visual assessment of the MSs using a dual negative-positive grading of −5 to 5 to reflect over- and under-estimation. Nine metrics that correlated well with visual assessment were selected for further evaluation using 3 different network ranking methods - based on a single metric, normalizing the metric using 2 GTs, and ranking the network based on a metric then averaging, including leave-one-out evaluation. These metrics yielded consistent ranking with HighRes3DNet ranked first then DenseVNet and ScaleNet using all ranking methods. Relative volume difference yielded the best positivity-agreement and correlation with dual visual assessment, and thus is better for providing over- and under-estimation. Interclass Correlation yielded the strongest correlation with the absolute visual assessment (0–5). Symmetric-boundary dice consistently yielded good discrimination of the networks for all three ranking methods with relatively small variations within network. Good rank discrimination may be an additional metric feature required for better network performance evaluation.

Assessment of Regression Models for Predicting Rice Yield and Protein Content Using Unmanned Aerial Vehicle-Based Multispectral Imagery

Unmanned aerial vehicle-based multispectral imagery including five spectral bands (blue, green, red, red-edge, and near-infrared) for a rice field in the ripening stage was used to develop regression models for predicting the rice yield and protein content and to select the most suitable regression analysis method for the year-invariant model: partial least squares regression, ridge regression, and artificial neural network (ANN). The regression models developed with six vegetation indices (green normalization difference vegetation index (GNDVI), normalization difference red-edge index (NDRE), chlorophyll index red edge (CIrededge), difference NIR/Green green difference vegetation index (GDVI), green-red NDVI (GRNDVI), and medium resolution imaging spectrometer terrestrial chlorophyll index (MTCI)), calculated from the spectral bands, were applied to single years (2018, 2019, and 2020) and multiple years (2018 + 2019, 2018 + 2020, 2019 + 2020, and all years). The regression models were cross-validated through mutual prediction against the vegetation indices in nonoverlapping years, and the prediction errors were evaluated via root mean squared error of prediction (RMSEP). The ANN model was reproducible, with low and sustained prediction errors of 24.2 kg/1000 m2 ≤ RMSEP ≤ 59.1 kg/1000 m2 in rice yield and 0.14% ≤ RMSEP ≤ 0.28% in rice-protein content in all single-year and multiple-year analyses. When the importance of each vegetation index of the regression models was evaluated, only the ANN model showed the same ranking in the vegetation index of the first (MTCI in both rice yield and protein content) and second importance (CIrededge in rice yield and GRNDVI in rice-protein content). Overall, this means that the ANN model has the highest potential for developing a year-invariant model with stable RMSEP and consistent variable ranking.

Diversity and Discrepancies in Water‐related University Rankings: Is There a Need for More Consistency or Is There Value in Breadth?

Diversity and Discrepancies in Water‐related University Rankings: Is There a Need for More Consistency or Is There Value in Breadth?

SENSITIVITY ANALYSIS OF SUPPLY CHAIN MANAGEMENT PROBLEM BASED ON HIERARCHICAL AND NETWORK DECISION SUPPORT MODELS

Background. Predicting future sales is necessary to control the flow of goods in supply chains (SC), which is why firms perform forecasting of consumer demand. To study the reliability of the solution, the sensitivity analysis (SA) of the results to changes in the original data is performed. Network decision support (DS) models usually contain a very large number of elements and connections between them, which makes it difficult to perform SA. Objective. The aim of the paper is to assess the sensitivity of the solution, given in the form of the element ranking of the network DS model, to inaccuracies and contradictions in elements of expert pairwise comparison matrices (PCMs), as well as to changes in individual elements of the supermatrix of the network DS model. To assess the priority of different types of information in the SC management system and to assess the sensitivity of the decision to obtain a more accurate forecast of consumer demand. Methods. Evaluation of DS models to prioritize information needs in the SC management system is carried out based on a developed method of network analysis. The method for assessing the sensitivity of the decision, proposed for the hierarchical DS model, includes finding stable elements of each level of the hierarchy and assessing the degree of sensitivity of global ranking of elements. The sensitivity of a solution based on a network DS model includes an assessment of the stability of local rankings, finding the matrix elements that most affect the change in consistency and change in the local ranking, as well as the sensitivity to changes in individual elements of the model supermatrix. Results. The method of complex assessment of sensitivity has been further developed, the stages of assessing the stability of local ranking of elements of the network DS model and the resilience of the pairwise comparison matrix elements to changes in the permissible inconsistency have been improved. A method for SA of results based on the network DS model to changes in individual elements of the supermatrix using machine learning tools is generalized. Conclusions. In the SC management problem, the elements of the expert PCM are found, which to the greatest extent affect the change in consistency and change in the rankings of decision alternatives. The stable elements of the matrix and elements for revision by an expert were calculated in order to increase the reliability of the solution to the SC management problem based on the hierarchical and network DS models.

The underlying drivers of economy-wide energy efficiency and asymmetric energy price responses

A good understanding of energy efficiency trend and its potential underlying drivers is required when considering policies for energy conservation and environmental sustainability. In light of this, we estimate energy efficiency trends using index decomposition analysis, data envelopment analysis and stochastic frontier analysis as robust methods. The driving factors (including asymmetric price responses) are examined using a dynamic panel model estimated by Arellano and Bond (1991) GMM estimator. The application for 32 OECD countries found that none of the three methods leads to a consistent ranking between energy efficiency estimates and energy intensity – corroborating the criticism that energy intensity is not a good proxy for energy efficiency. The panel-data regressions using the energy efficiency estimates from the three methods show similarities in the impacts of the drivers (including energy price, foreign direct inflows, trade openness, population growth, temperature etc.) on energy efficiency. Although the results of asymmetric price responses of energy efficiency estimates vary slightly, we found insignificant evidence of asymmetric effects of total energy price but there are asymmetric responses with energy-specific prices. Thus, using energy-specific prices as well as allowing for asymmetric price effects in analysing drivers of energy efficiency is apt and informative in formulating energy policies.

Aggregating Multidimensional Criteria in Audit Decision Making

Measuring total performance of prospective alternatives in decision making (DM) needs aggregation of their noncomparable properties which often have different scales of measure. In order to address this, scale transformations and normalization are applied, but problems of validation of such additive models and interpretation of the physical sense of the result arise. In this paper, a multiplicative performance aggregation of properties' values is suggested for DM. Application of the approach enables consistent and unambiguous ranking of the alternatives allowing values of all their properties of interest. A realistic example of DM related to audit activity planning is considered where several competing control areas should be selected into an audit plan. Beyond audit planning, the approach is generalizable to other DM problems in corporate management and governance where one needs to rank alternatives compared in multiple dimensions.

Consistent Poplar Clone Ranking Based on Leaf Phenology and Temperature Along a Latitudinal and Climatic Gradient in Northern Europe

In Northern Europe, poplars (Populus) can provide biomass for energy and material use, but most available clones were developed for lower latitudes and are unlikely to be well adapted to higher latitudes, even under warmer climates. We thus need to understand how clones respond to climatic conditions and photoperiod, and how these responses can be predicted. We answer these questions exploiting leaf phenological data of Populus clones, grown in six sites across the Baltic region, in Northern Europe, for 2 years with contrasting climatic conditions. Regarding the effects of climatic conditions and photoperiod, within each site, higher temperatures advanced the timing and enhanced the speed of spring and autumn phenology, but reduced the effective growing season length. Across sites, latitude affected the timing of spring and autumn phenology, the speed of spring phenology, and the effective growing season length; clone affected only the timing of phenology. Regarding the predictability of clone response to growing conditions, the growing degree day (GDD) model could not predict spring phenology, because the growing degree day threshold for a specific phenological stage was not only clone-, but also latitude- and year-specific. Yet, this GDD threshold allowed a robust ranking of clones across sites and years, thus providing a tool to determine the relative differences across clones, independently of latitude and temperature. A similar, but not as strong, pattern was observed in the timing of spring and autumn phenological stages. Hence, while prediction of spring phenology remains elusive, the ranking of clones based on observations of their phenology in a single location can provide useful indications on the clones’ relative performance under different latitudes and climates.

The Impact of Correlated Metrics on the Interpretation of Defect Models

Defect models are analytical models for building empirical theories related to software quality. Prior studies often derive knowledge from such models using interpretation techniques, e.g., ANOVA Type-I. Recent work raises concerns that correlated metrics may impact the interpretation of defect models. Yet, the impact of correlated metrics in such models has not been investigated. In this paper, we investigate the impact of correlated metrics on the interpretation of defect models and the improvement of the interpretation of defect models when removing correlated metrics. Through a case study of 14 publicly- available defect datasets, we find that (1) correlated metrics have the largest impact on the consistency, the level of discrepancy, and the direction of the ranking of metrics, especially for ANOVA techniques. On the other hand, we find that removing all correlated metrics (2) improves the consistency of the produced rankings regardless of the ordering of metrics (except for ANOVA Type-I); (3) improves the consistency of ranking of metrics among the studied interpretation techniques; (4) impacts the model performance by less than 5 percentage points. Thus, when one wishes to derive sound interpretation from defect models, one must (1) mitigate correlated metrics especially for ANOVA analyses; and (2) avoid using ANOVA Type-I even if all correlated metrics are removed.

A computationally efficient metric for identification of critical links in large transportation networks

Maintaining safety and resilience in transportation system requires identification of critical components. The existing transportation link criticality metrics typically require multiple traffic assignments to perform a full scan of the network. Link removals create computational burden and can cause network disconnectivity which makes it problematic to run traffic assignment. The authors previously introduced Link Criticality Index (LCI) that identifies the criticality ranking within a single User Equilibrium (UE) traffic assignment using Frank-Wolfe (FW) algorithm. While LCI is shown to provide balanced rankings with respect to connectivity and flow conditions, its computational efficiency diminishes for larger networks due to the need for path enumeration. This paper formulates an adjusted LCI to make it computationally efficient for larger networks. Adjusted LCI utilizes 1) viable routes instead of the complete list of enumerated paths; and, 2) a path-based traffic assignment algorithm (Gradient Projection) to replace FW. The consistency of rankings between original/adjusted LCIs, and other measures from the literature are analyzed and compared for three networks, and through an experimental setup. The proposed method is also applied on a large-scale transportation network, i.e., Chicago network. The results show that both modifications are effective and yield faster results without compromising LCI's advantages.

Linguistic scale consistency issues in multi-granularity decision making contexts

The symbolic model based on the linguistic scale has been widely used to represent linguistic knowledge to deal with various linguistic decision problems. However, linguistic scales with different granularity may yield inconsistent decision outcomes in the linguistic decision making. Thus, this paper systematically studies the linguistic scale consistency issues in multi-granularity decision making contexts. We first define the concepts of the consistent multi-granularity representation, consistent multi-granularity aggregation and consistent multi-granularity ranking. After that, we analytically present a necessary and sufficient condition to guarantee the consistent multi-granularity representation and a sufficient condition to characterize the intrinsic mechanism of the consistent multi-granularity aggregation. Then, an attitude-based linguistic representation method (ALRM) is proposed to improve the consistent multi-granularity ranking. Finally, a detailed numerical analysis and simulation experiments are presented to show the advantages of the ALRM over the traditional linguistic approach. These results will provide new insights into the use of linguistic scales in the linguistic decision making.

Decision Supporting System For Malang City Baseball and Softball Athletes Selection Using The Vikor Method

Decision support system is a computer-based system that is used to solve problems by semi-structured and unstructured conditions. In this case, the best solution can be found from certain criteria and provided alternatives and is easy to use by users. In this study, researchers designed and built a website-based decision support system to select athletes in PERBASASI Malang with the provided test criteria, namely the hit test, catch test, throw test, and run test. The purpose of this study was to provide recommendations for participants who passed the selection based on test scores that have been processed using the VlseKriterijumska Optimizacija I Kompromisno Resenje (VIKOR) method. From the results of calculations, the use of the VIKOR method for this case study produced an accuracy value of 90.90%, a precision value of 93.33%, and a recall value of 93.33%. In addition, the VIKOR sensitivity test showed a consistent ranking of the calculation of the value with veto (value of v less than 0.5), by consensus (value of v is 0.5), and voting by majority rule (value of v more than 0.5). User testing that was applied by using each level of the existing account, i.e. administrator account level, selection account level, and members account level conducted on this information system concluded that the system that was built was running smoothly and was easy to use.

Evaluating negative emissions technologies using neutrosophic data envelopment analysis

Growing energy demand due to population growth and economic progress has led to the need for intensive climate change mitigation. Negative emissions technologies (NETs) will play an important role in limiting global warming by reducing greenhouse gas concentration in the atmosphere. Available options for NETs can be integrated into energy and climate change policies. However, this strategy requires an assessment tool to select a suitable set of NETs to be integrated. A mathematical framework considering multiple environmental and economic factors under uncertainty is needed for the assessment tool. This paper has two main contributions: (1) development of a novel multi-criteria decision analysis tool based on data envelopment analysis method under neutrosophic environment, and (2) application of this tool in the evaluation of NETs considering the risks associated with the uncertainties in these technologies. The model incorporates uncertainties that result in risks when higher performance levels are assumed. A novel neutrosophic data envelopment analysis (NDEA) model is developed for evaluating NETs; this approach uses neutrosophic set theory to account for membership, nonmembership and indeterminacy in uncertain parameter values. A case study is presented to illustrated this technique. Then, a representative case study considering four alternative options for bioenergy with carbon capture and storage (BECCS) is used to demonstrate the tool. The performance assessment of candidate NETs based on literature data is made. Results show different characteristics of NETs in terms of their sensitivity to expert’s risk attitude and perception towards the different dimensions of neutrosophic decision-making environment. For instance, the preference of underground storage over ocean storage for BECCS is evident when improving the technology is assumed to be challenging. Various NETs are evaluated; technologies such as BECCS, and afforestation and reforestation show consistent ranking in different decision environments. The capability of the model to account for uncertainties can allow useful insights to be drawn by policy-makers, and thus, accelerate the deployment of NETs.

New similarity measures for single-valued neutrosophic sets with applications in pattern recognition and medical diagnosis problems

The single-valued neutrosophic set (SVNS) is a well-known model for handling uncertain and indeterminate information. Information measures such as distance measures, similarity measures and entropy measures are very useful tools to be used in many applications such as multi-criteria decision making (MCDM), medical diagnosis, pattern recognition and clustering problems. A lot of such information measures have been proposed for the SVNS model. However, many of these measures have inherent problems that prevent them from producing reasonable or consistent results to the decision makers. In this paper, we propose several new distance and similarity measures for the SVNS model. The proposed measures have been verified and proven to comply with the axiomatic definition of the distance and similarity measure for the SVNS model. A detailed and comprehensive comparative analysis between the proposed similarity measures and other well-known existing similarity measures has been done. Based on the comparison results, it is clearly proven that the proposed similarity measures are able to overcome the shortcomings that are inherent in existing similarity measures. Finally, an extensive set of numerical examples, related to pattern recognition and medical diagnosis, is given to demonstrate the practical applicability of the proposed similarity measures. In all numerical examples, it is proven that the proposed similarity measures are able to produce accurate and reasonable results. To further verify the superiority of the suggested similarity measures, the Spearman’s rank correlation coefficient test is performed on the ranking results that were obtained from the numerical examples, and it was again proven that the proposed similarity measures produced the most consistent ranking results compared to other existing similarity measures.

Rank consistent ordinal regression for neural networks with application to age estimation

In many real-world prediction tasks, class labels include information about the relative ordering between labels, which is not captured by commonly-used loss functions such as multi-category cross-entropy. Recently, the deep learning community adopted ordinal regression frameworks to take such ordering information into account. Neural networks were equipped with ordinal regression capabilities by transforming ordinal targets into binary classification subtasks. However, this method suffers from inconsistencies among the different binary classifiers. To resolve these inconsistencies, we propose the COnsistent RAnk Logits (CORAL) framework with strong theoretical guarantees for rank-monotonicity and consistent confidence scores. Moreover, the proposed method is architecture-agnostic and can extend arbitrary state-of-the-art deep neural network classifiers for ordinal regression tasks. The empirical evaluation of the proposed rank-consistent method on a range of face-image datasets for age prediction shows a substantial reduction of the prediction error compared to the reference ordinal regression network.

Quantifying the structure of strong gravitational lens potentials with uncertainty-aware deep neural networks

ABSTRACTGravitational lensing is a powerful tool for constraining substructure in the mass distribution of galaxies, be it from the presence of dark matter sub-haloes or due to physical mechanisms affecting the baryons throughout galaxy evolution. Such substructure is hard to model and is either ignored by traditional, smooth modelling, approaches, or treated as well-localized massive perturbers. In this work, we propose a deep learning approach to quantify the statistical properties of such perturbations directly from images, where only the extended lensed source features within a mask are considered, without the need of any lens modelling. Our training data consist of mock lensed images assuming perturbing Gaussian Random Fields permeating the smooth overall lens potential, and, for the first time, using images of real galaxies as the lensed source. We employ a novel deep neural network that can handle arbitrary uncertainty intervals associated with the training data set labels as input, provides probability distributions as output, and adopts a composite loss function. The method succeeds not only in accurately estimating the actual parameter values, but also reduces the predicted confidence intervals by 10 per cent in an unsupervised manner, i.e. without having access to the actual ground truth values. Our results are invariant to the inherent degeneracy between mass perturbations in the lens and complex brightness profiles for the source. Hence, we can quantitatively and robustly quantify the smoothness of the mass density of thousands of lenses, including confidence intervals, and provide a consistent ranking for follow-up science.

An Alternate Unsupervised Technique Based on Distance Correlation and Shannon Entropy to Estimate λ0-Fuzzy Measure

λ0-measure is a special type of fuzzy measure. In the context of multi-attribute decision making (MADM), the measure can be used together with Choquet integral to model the interdependencies that usually present between the decision attributes. Unfortunately, the range of techniques available to estimate λ0-measure values is too limited i.e., only four techniques are available to this date. Besides, the review on literature shows that each of these existing techniques either requires some initial data from the decision-makers or misrepresents the actual interdependencies held by the attributes. Thus, an alternate unsupervised technique is needed for the estimation of λ0-measure values. This study has developed such a technique by integrating the idea of distance correlation and Shannon entropy. In this technique, the two inputs required to estimate λ0-measure values, namely, the interdependence degrees and fuzzy densities are determined by utilizing the distance correlation measures and entropy weights, respectively. An evaluation to rank the websites owned by five different hospitals located in Sabah, Malaysia, was conducted to illustrate the usage of the technique. A similar evaluation was also performed with a few selected MADM techniques for comparison purposes, where the proposed technique is found to have produced the most consistent ranking. From the literature perspective, this study has contributed an alternate unsupervised technique that can estimate λ0-measure values without necessitating any additional data from the decision-makers, and at the same time can better capture the interdependencies held by the attributes.

Efficacy of intravesical therapies on the prevention of recurrence and progression of non‐muscle‐invasive bladder cancer: A systematic review and network meta‐analysis

Intravesical instillation therapy is the mainstay of prophylaxis of tumor recurrence and progression in non‐muscle‐invasive bladder cancer. However, there is no study evaluating the superiority of monotherapy. The aim of this study is to compare the efficacy of preventing recurrence and progression of intravesical monotherapies via network meta‐analysis (NMA) of randomized controlled trials. Database searches were conducted on Embase, Ovid Medline, Web of Science, ScienceDirect, Cochrane Library, and ClinicalTrials.com from the time of establishment to February 6, 2020. The monotherapies included Bacille Calmette‐Guérin (BCG), mitomycin C (MMC), interferon (IFN), adriamycin, epirubicin, gemcitabine (GEM), and thiotepa (THP). A Bayesian consistency network model was generated under a random‐effects model. The superiority of therapy was identified based on the surface under the cumulative ranking curve (SUCRA). Fifty‐seven studies with 12462 patients are included. NMA shows that GEM (SUCRA = 0.92), BCG (SUCRA = 0.82), and IFN (SUCRA = 0.78) are the top three effective drugs to reduce recurrence. GEM (SUCRA = 0.87) is the most effective therapy to prevent progress, followed by BCG, MMC, THP, and IFN with similar efficacy. Subgroup analysis of pairwise meta‐analysis and NMA was performed on publication year, trial initiation year, study origin, center involvement, sample size, drug schedule, tumor characteristics, and trial quality to address confounding factors, which suggests the robustness of the results with stable effect sizes. Network meta‐regression also indicates consistent rank by analyzing year, sample size, and quality. Compared with BCG, GEM is also a promising therapy with favorable efficacy to reduce tumor recurrence and progression. IFN and MMC could be alternative therapies for BCG with slightly inferior efficacy in recurrence prevention and similar efficacy in progression prevention. However, the results of this study should be treated with caution since most of the included studies are of moderate to high risk of bias.

A GIS-based multi-criteria model for offshore wind energy power plants site selection in both sides of the Aegean Sea.

Location selection for offshore wind farms is a major challenge for renewable energy policy, marine spatial planning, and environmental conservation. This selection constitutes a multi-criteria decision-making problem, through which parameters like wind velocity, water depth, shorelines, fishing areas, shipping routes, environmental protection areas, transportation, and military zones should be jointly investigated. The aim of the present study was thus to develop an integrated methodology for assessing the siting of bottom-fixed offshore wind farms in two different countries (with different legal, political, and socio/economic characteristics). Our methodology combined multi-criteria decision-making methods and geographical information systems and was implemented in Cyclades (Greece) and in the sea area of İzmir region (Turkey). Experts used fuzzy sets and linguistic terms to achieve more consistent and independent rankings and results. In the Turkish region, the results showed that 519 km2 (10.23%) of the study area is suitable for offshore wind farms, while in the Greek region, only 289 km2 (3.22%) of the study area was found to be suitable. This spatial suitability analysis may contribute to provide some useful recommendations for the spatial marine planning at the regional scale, as well as for the preliminary assessment of new offshore wind farms in both countries.

Combining Experimental and Modeling Approaches to Understand Genotype x Sowing Date x Environment Interaction Effects on Emergence Rates and Grain Yield of Soybean.

Soybean emergence and yield may be affected by many factors. A better understanding of the cultivar x sowing date x environment interactions could shed light into the competitiveness of soybean with other crops, notably, to help manage major biotic and abiotic factors that limit soybean production. We conducted a 2-year field experiments to measure emergence dynamics and final rates of three soybean cultivars from different maturity groups, with early and conventional sowing dates and across three locations. We also measured germination parameter values of the three soybean cultivars from different maturity groups under controlled experiments to parametrize the SIMPLE crop emergence model. This allowed us to assess the prediction quality of the model for emergence rates and to perform simulations. Final emergence rates under field conditions ranged from 62% to 92% and from 51% to 94% for early and conventional sowing, respectively. The model finely predicted emergence courses and final rates (root mean square error of prediction (RMSEP), efficiency (EF), and mean deviation (MD) ranging between 2% to 18%, 0.46% to 0.99%, and −10% to 15%, respectively) across a wide range of the sowing conditions tested. Differences in the final emergence rates were found, not only among cultivars but also among locations for the same cultivar, although no clear trend or consistent ranking was found in this regard. Modeling suggests that seedling mortality rates were dependent on the soil type with up to 35% and 14% of mortality in the silty loam soil, due to a soil surface crust and soil aggregates, respectively. Non-germination was the least important cause of seedling mortality in all soil types (up to 3% of emergence losses), while no seedling mortality due to drought was observed. The average grain yield ranged from 3.1 to 4.0 t ha−1, and it was significantly affected by the irrigation regime (p < 0.001) and year (p < 0.001) but not by locations, sowing date or cultivars. We conclude that early sowing is unlikely to affect soybean emergence in South-West of France and therefore may represent an important agronomic lever to escape summer drought that markedly limit soybean yield in this region.

Early Health Technology Assessment during Nonalcoholic Steatohepatitis Drug Development: A Two-Round, Cross-Country, Multicriteria Decision Analysis.

Background. The assessment of value along the clinical development of new biopharmaceutical compounds is a challenging task. Complex and uncertain evidence has to be analyzed, considering a multitude of value preferences from different stakeholders. Objective. To investigate the use of multicriteria decision analysis (MCDA) to support decision making during drug development while considering payer and health technology assessment (HTA) value concerns, by applying the Advance Value Framework in nonalcoholic steatohepatitis (NASH) and testing for the consistency of the results. Design. A multiattribute value theory methodology was applied and 2 rounds of decision conferences (DCs) were organized in 3 countries (England, France, and Germany), with the participation of national key experts and stakeholders using the MACBETH questioning protocol and algorithm. A total of 51 health care professionals, patient advocates, and methodologists, including (ex-) committee members or assessors from national HTA bodies, participated in 6 DCs in the study countries. Target Population. NASH patients in fibrosis stages F2 to 3 were considered. Interventions. The value of a hypothetical product profile was assessed against 3 compounds under development using their phase 2 results. Outcome Measures. DC participants’ value preferences were elicited involving criteria selection, options scoring, and criteria weighting. Results. Highly consistent valuation rankings were observed in all DCs, always favoring the same compound. Highly consistent rankings of criteria clusters were observed, favoring therapeutic benefit criteria, followed by safety profile and innovation level criteria. Limitations. There was a lack of comparative treatment effects, early evidence on surrogate endpoints was used, and stakeholder representativeness was limited in some DCs. Conclusions. The use of MCDA is promising in supporting early HTA, illustrating high consistency in results across countries and between study rounds.