Affective Recommendation Systems (“ARS”) allow for increasingly personalised recommendations based on users’ emotional states (such as overall mood or attitude) inferred through the processing of an array of data ranging from individuals’ online behaviour (e.g., clicks, interactions, viewing time) to weather conditions (e.g., is the user nostalgic due to the rain?). While emotional recognition systems generally depend on biometric data, such as through the use of facial expression analysis, to infer individuals’ emotions, a variety of ARS use other types of information to understand users’ mood, preferences and attitude towards content.
For instance, researchers have developed models that extract emotional or affective data from users’ microblogs to enhance music recommendations and chatbots that are able to extract emotions from users’ utterances. Other proposals intend to enhance social networking by integrating users into communities of individuals with similar interests and views, which requires an analysis of the relationships between users and the processing of geographical location, tags, frequency of publishing and overall topic sentiment to infer users’ characteristics.
In this context, this Insight intends to explore whether there is a protection gap in the EU’s current data protection and AI regulation framework regarding systems designed to infer emotions that are not dependent on “traditional” biometric data, especially considering that emotional data, in itself, is not protected as a special category of data under the GDPR’s Article 9.
How do these systems function?
A Survey of Affective Recommender Systems defines three key pillars of their general architecture:
First, the system focuses on understanding users’ affective states, which can require the processing of several sources of information such as metadata, online behaviour and past interactions to extract users’ affective features and develop a model based on their affective preferences.
Secondly, affective attributes are extracted from an item’s content by processing genre, brand, descriptive information, user reviews and other multimedia signals. For instance, relating a tragic romance film with sadness, nostalgia and other such emotions.
Finally, the system mixes both user affective data and item affective data to generate personalised recommendations that cater to a user’s interests.
EU Data Protection and AI Regulation Framework
Challenges in classifying this type of emotional data as biometric data
Considering that emotional data relates to the core of individuals’ private sphere and potentially reveals a large set of personal characteristics, its classification as a special category of data under Article 9 of the GDPR is essential to guarantee an adequate level of protection for data subjects. Here, the main question would relate to whether the systems under analysis are truly non-biometric under the definitions laid out in both Article 4(14) and Article 9.
On the one hand, ARS infer users’ emotions, moods and attitudes towards certain contents by analysing their online behaviour through a variety of data, thereby effectively measuring individuals’ behavioural characteristics. Additionally, techniques such as stylometry (a field that uses statistical methods to discover the authorship of written texts) can allow for the unique identification of users.
On the other hand, Article 9(1) requires that biometric data be used for the purpose of uniquely identifying a natural person. However, most recommendation systems do not seem to be developed with the objective of identifying users and, in fact, most users would likely already be identified by their usernames, accounts and other online identifiers prior to the processing.
Furthermore, both the legislator[1] as well as the EDPB’s position on the classification of photographs and video footage of an individual as biometric data under Article 9 reinforce the interpretation that such classification is dependent on personal data being specifically processed to contribute to the identification of the individual. In light of this, affective recommendation systems that go beyond “traditional” biometrics to infer users’ emotions may be able to bypass the prohibition under Article 9(1) through designs centred on the personalisation of recommendations rather than user identification or authentication.
Could this type of emotional data be, nonetheless, protected under Article 9?
Inferring users’ emotions and attitude towards content may likely reveal a variety of other special categories of personal data, especially if such processing is taking place in large online platforms (e.g., social media) that allow a wide cross-matching of various indicators (e.g., a user’s consumption of low mood/depressive content being correlated with weather data) and the tracking of users’ attitudes to content to infer their political stances or sexual orientation.
– Data concerning health
A joint reading of Article 4(15) and Recital 35 of the GDPR indicates that this type of personal data includes all data related to the past, current and future health status of an individual, be it related to physical or mental health. In this instance, if a user consistently reacts with fear to content that portrays a certain animal, it may be possible to infer that the user has a specific phobia, thereby prompting the system to adapt to that information (e.g., by stopping recommendations of similar content). Similarly, mood-aware systems could hypothetically be able to detect several mental health conditions, such as depression, by consistently observing a long-term low level of arousal, happiness or overall interaction with content with which the user previously had a positive reaction or relationship.
– Sexual orientation
If a user consistently reacts with arousal or has a high level of interaction with content that portrays certain genders seductively, it may also be hypothetically possible that the recommendation system infers their sexual orientation. Furthermore, a system intended to promote social networking may indeed benefit from inferences related to sexual orientation, as they may be used to foster content engagement and user interactions.
– Political opinions and philosophical/religious beliefs
If a user reliably reacts adversely or positively to content expressing political statements, portraying political candidates or representing specific religious or philosophical beliefs, it may be possible, especially by the system’s modelling of inferences with a large set of emotional and non-emotional data, to infer that user’s stances and beliefs. For instance, a user may feel fear after seeing a news article on the election of a certain political candidate or react with anger towards certain proposed policies or moral statements. Additionally, affective recommendation systems’ architecture seems to indicate that the processing of these special categories of data is not only extremely beneficial in terms of accuracy, but also inevitable when users are persistently tracked on platforms that offer a large variety of content (e.g., social media based on short video content).
Limitations of the use of GDPR’s Article 9
GDPR’s Recital 51 states that sensitive data merits specific protection as its processing could create significant risks to fundamental rights and freedoms. Would this not be the case with emotional and/or affective data?
In essence, emotional data relates to the core of individuals’ private sphere and may mirror our desires, attitudes, tendencies, biases and preferences. The emergence of advanced recommendation systems, especially in the context of social media platforms, can pose significant risks to users, such as addiction and other mental health concerns, as reflected in the recent landmark case regarding Meta and YouTube.
Nonetheless, emotional data is not treated as a special category of data under Article 9 unless it includes the list of specified special categories of data, thereby leaving a gap in data subjects’ protection.
Limitations of the AI Act’s prohibited AI practices
The AI Act’s Article 5(1)(f) explicitly prohibits the use of AI systems to infer emotions of natural persons in the workplace and educational institutions. Nonetheless, even though Article 5.º(1)(f) does not prohibit other emotion recognition systems, these are still classified as high-risk AI systems under Article 6 and Annex III 1.c, if based on biometrics, which triggers the corresponding obligations, including compliance with many safety requirements, ranging from implementing a risk management system to conducting an FRIA (Fundamental Rights Impact Assessment). Additionally, Article 5(1)(g) prohibits the use of biometric categorisation systems that individually categorise natural persons based on their biometric data to deduce or infer their race, political opinions, trade union membership, religious or philosophical beliefs, sex life or sexual orientation.
Although this Article 5 could provide additional protection, it not only restricts the prohibition on systems based on biometric data (which can be difficult to ascertain in certain affective recommendation systems) but also excludes (via Recital 16) systems that are a purely ancillary feature intrinsically linked to another commercial service, which may further hinder users’ protection.[2]
Digital Services Act at play?
In addition to both the GDPR and the AI Act, the Digital Services Act (“DSA”), which interacts with both those legislations, establishes that affective computing shall respect fundamental rights such as human dignity, right to a private life and the protection of one’s personal data. As such, under its Articles 34, 35 and 38, providers of VLOPs and VLOSEs must mitigate systemic risks and provide (at least) one option for each of their recommender systems which is not based on profiling.
These requirements could offer a wider range of protection, however, the limitation of the scope of application to VLOPs and VLOSEs, i.e., online platforms and online search engines with an average of 45 million or more monthly active users in the EU, may still leave a gap in safeguards for users of platforms falling outside that scope.
Conclusion
Emotional inferences generated by affective recommendation systems that go beyond “traditional” biometrics are protected under the GDPR’s Article 9 insofar as they reveal special categories of data such as political opinions, religious beliefs, sexual orientation and data concerning health.
While it can be argued that the inference of these types of sensitive data may be inevitable on large platforms (especially social media), the restrictiveness of the list of special categories of data creates a gap in protection by excluding emotional data from that scope, even though such data still relates to the core of individuals’ private sphere.
Furthermore, both the AI Act’s prohibition of emotional recognition systems (in the workplace and educational institutions) and biometric categorisation systems that deduce sensitive data, as well as the limited scope of the DSA’s requirements, may not fully address the identified gap.
[1] General Data Protection Regulation, Recital 51.
[2] Artificial Intelligence Act, Article 3(40) and Recital 30.
