Mitigating subjectivity and bias in AI development indices: A robust approach to redefining country rankings

Abstract

Artificial Intelligence (AI) indices have emerged to assess countries’ progress in AI development, aiding decision-making on investments and policy choices. Typically, these indices are derived from a linear weighted sum of various criteria, employing deterministic weights. However, this approach fails to capture interactions among criteria, and the use of deterministic weights is susceptible to debate. To mitigate these issues, we conduct a methodological analysis based on Choquet integral (CI) and Stochastic Multicriteria Acceptability Analysis 2 (SMAA-2). We assess correlations between different AI dimensions and employ CI to model them. Additionally, we apply SMAA-2 to conduct a sensitivity analysis using both weighted sum and CI in order to evaluate the stability of the indices with regard to the weights. Finally, we introduce a ranking methodology based on SMAA-2, which considers several sets of weights to derive the ranking of countries. In the computational analysis, we evaluate our approach using the dataset employed in The Global AI Index, as proposed by the British news website Tortoise. The results reveal that our approach effectively mitigates bias. Furthermore, we scrutinize changes in the ranking resulting from weight adjustments and demonstrate that our proposed rankings closely align with those derived from variations in weights, indicating robustness.