There is widespread consensus in the comparative welfare state literature that the welfare state can be best conceptualized in terms of social rights of citizenship. The Social Citizenship Indicator Program (SCIP) and the Comparative Welfare Entitlements Dataset (CWED), which rely on operational definitions of quantified legislated social rights, constitute centrepieces of this thriving research. As leading state-of-the-art tools for capturing welfare stateness, these two datasets are being widely used. Scholars in general have also been treating them as interchangeable measurement tools. Upon closer inspection, however, we discover that the two datasets point to contrasting images of welfare state change for certain countries and time periods. This article aims to contribute to the scholarly exchange on the validity problem in measuring welfare state generosity. The exchange has hitherto been confined to problems of dataset choice with respect to only replacement rates, a set of key indicators included in both datasets. However, there are 11 key non-replacement rate indicators SCIP and CWED have in common, whose convergent validity has yet to be questioned. We thus explore the convergent validity of these non-replacement rate indicators across the two datasets. We then replicate the two leading composite indexes (Decommodification Index (DI) and Benefit Generosity Index (BGI)) constructed on the basis of these indicators. We identify problems of invalidity manifested in discrepancies in non-replacement rate indicator scores and index values for DI and BGI. We show how these discrepancies could lead to contrasting assessments of welfare state change. We then identify a set of potential sources for these discrepancies, most of which are related to different operationalizations of similar concepts. We conclude by calling for more dialogue among developers of SCIP and CWED to further clarify their conceptual and operational points of departure.