Suitability Of Metrics Research Articles

Offense specialization among outlaw motorcycle gang members: Comparing specialization metrics

PurposeWe examine the extent of offense specialization among outlaw motorcycle gang affiliates across Australia using a multi-metric approach, and by doing so compare diverse measures of specialization. MethodsThe sample consisted of 19,633 offenses, committed by 2283 affiliates of outlaw motor cycle gangs (OMCGs), disaggregated into four offense categories: ongoing criminal enterprise, short-term instrumental, violence and intimidation, and other offenses against person. Six specialization metrics were employed to measure specialization among the full sample of affiliates and relevant subgroups. ResultsOf the six metrics employed to measure specialization, five suggested a high degree of versatility in offending among OMCG affiliates. Results were inconsistent across metrics when examining levels of specialization among subgroups of OMCG affiliates, and across offense categories. ConclusionsOMCGs are highly versatile in their offending, demonstrating the importance of interventions targeting organized, violent and lower-level volume crime. However, different measures of specialization yielded inconsistent results and appeared to be measuring different constructs. The suitability of metrics is dependent on the nature of the data and research question of interest. Future specialization research should employ a multi-metric approach to thoroughly explore specialization and improve the ability for cross study comparisons.

Bias audit laws: how effective are they at preventing bias in automated employment decision tools?

ABSTRACT Automated employment decision tools use machine learning, artificial intelligence, predictive analytics, and other data-driven approaches to enhance candidate experiences and streamline employment related decision-making, allowing human resources to be concentrated where they are needed most. However, the use of these tools without appropriate safeguards has resulted in a number of high-profile scandals in recent years, particularly in regard to bias. Accordingly, lawmakers have started to propose laws that require bias audits of automated employment decision tools to examine their outputs for subgroup differences. The first of its kind was New York City Local Law 144, but other US states have since followed suit. In this paper, we examine the concerns about the effectiveness of this and other similar laws, including the suitability of metrics, the scope of the law, and low levels of compliance. We conclude that despite the law being a good initial first step towards greater transparency around automated employment decision tools and reducing bias, examining outcomes alone is not sufficient to prevent bias elsewhere in the tool. Moreover, effective bias prevention will require a multidisciplinary approach that combines expertise in IO psychology, law, and computer science to develop appropriate metrics and maximize the enforceability of such laws.

Robustness Metrics: How Are They Calculated, When Should They Be Used and Why Do They Give Different Results?

AbstractRobustness is being used increasingly for decision analysis in relation to deep uncertainty and many metrics have been proposed for its quantification. Recent studies have shown that the application of different robustness metrics can result in different rankings of decision alternatives, but there has been little discussion of what potential causes for this might be. To shed some light on this issue, we present a unifying framework for the calculation of robustness metrics, which assists with understanding how robustness metrics work, when they should be used, and why they sometimes disagree. The framework categorizes the suitability of metrics to a decision‐maker based on (1) the decision‐context (i.e., the suitability of using absolute performance or regret), (2) the decision‐maker's preferred level of risk aversion, and (3) the decision‐maker's preference toward maximizing performance, minimizing variance, or some higher‐order moment. This article also introduces a conceptual framework describing when relative robustness values of decision alternatives obtained using different metrics are likely to agree and disagree. This is used as a measure of how “stable” the ranking of decision alternatives is when determined using different robustness metrics. The framework is tested on three case studies, including water supply augmentation in Adelaide, Australia, the operation of a multipurpose regulated lake in Italy, and flood protection for a hypothetical river based on a reach of the river Rhine in the Netherlands. The proposed conceptual framework is confirmed by the case study results, providing insight into the reasons for disagreements between rankings obtained using different robustness metrics.