Profiling As Inferred Data. Amplifier Effects and Positive Feedback Loops

Abstract

Extracting profiles and other hidden knowledge from large amounts of data via techniques like data mining and machine learning is often regarded as an input-output process in which knowledge (i.e., profiles) are extracted from raw data. In this provocation, a different perspective is taken, in which profiles are not regarded as knowledge, but rather as (new) data, namely as inferred data. Using this perspective, it is shown that profiles are not only an end result or an end product, but can also be reused as ingredients for further data analytics. However, in this way, profiling processes may function as amplifiers, amplifying bias and inaccuracies via positive feedback loops, that further entrench consequences for data subjects. Effects of small disturbances (like incorrect or incomplete data, or flaws in the data analysis) may lead to an increase of the magnitude of perturbations. Obviously, this may have some serious consequences for data subjects. Profiling based on datasets from data brokers that contain large amounts of inferred data, may propagate any existing biased patterns, leading, for instance, to disparate impact. The reuse of inferred data may also lead to self-fulfilling prophecies – a phenomenon well-known in profiling. In case of inferred data, however, the effect might be much stronger: because of the self-reinforcing effect, patterns may be amplified and become much more entrenched. These effects may amplify inequality, undermine democracy and further push people into categories that are hard to break out. From a legal perspective, under the EU General Data Protection Regulation (GDPR) inferred data may or may not be personal data. If so, people have a right to access the inferred data and to receive meaningful information about the logic involved in the data analytics. However, since data subjects have no right to access the algorithms and data of other data subject used in the analyses, it is impossible for them to check whether data is inferred correctly.