As society continues grappling with algorithmic bias, AI companions — and similar emotionally adaptable AI chatbots — may be creating a novel form of discrimination. Unlike traditional discrimination, which excludes or disadvantages people based on race, gender, sexual orientation, or socio-economic status, this one may turn emotional vulnerability into a profitable behavioral category. 

This vulnerability-based discrimination may encompass a wide range of seemingly unrelated groups of people, including children, grieving individuals, socially isolated people, and those struggling with psychological disorders or cognitive limitations. What links them is not their identity but their susceptibility: a heightened likelihood of forming intense attachments, disclosing intimate information, or developing overreliance, which makes them ideal users for AI companions.

AI companions mimic genuine care and offer their users unconditional empathy and validation.

The dynamic has a simple logic. AI companions rely on user engagement. The longer a user engages with them, the more valuable the user becomes from a commercial lens. To foster continued engagement, AI companions mimic genuine care and offer their users unconditional empathy and validation. They provide something that is hard to resist and nearly impossible to find in the real-world: companionship without misunderstanding, disagreement, or deception. 

While people generally recognize the synthetic nature of such interactions,  anthropomorphizing AI companions may blur the boundary between simulated care and real connection. That risk may be more prominent for individuals with emotional vulnerabilities.

The Common Denominator of Vulnerability

In fact, vulnerability seems to be a recurrent pattern in the publicly documented harm cases involving AI chatbots. Several people with different types of emotional vulnerabilities, such as children and those with cognitive impairments and poor mental health, have been reported to have suffered harm due to their emotional interaction with AI companions. These incidents could be exceptions. Yet, recent studies indicate that they may represent broader structural risks than isolated anomalies.

Critics are worried that such interaction may aggravate children’s existing mental health issues, foster emotional dependence on AI companions, and interfere with kids’ therapeutic relationships.

A new study from Stanford University researchers found that AI companions may worsen loneliness for vulnerable users. According to the study, intense use of AI chatbots by users with limited real-world social networks increases their loneliness and lowers their psychological well-being. 

Similarly, a study from Aarhus University suggests that AI chatbots may aggravate delusions, mania, eating disorders, and suicidal ideation in people with mental illnesses. 

Furthermore, a recent preprint survey revealed that large language models regularly express stigma against those with mental disorders and respond inappropriately to psychological crises, including reinforcing delusional thinking.

These concerns become particularly significant in the context of children and adolescents, among whom AI companionship is increasingly normalized. Many teenagers are already using AI chatbots not only for entertainment but also for emotional support and advice for health and well-being. 

Critics are worried that such interaction may aggravate children’s existing mental health issues, foster emotional dependence on AI companions, and interfere with kids’ therapeutic relationships.

Defining Discrimination

At first glance, describing this dynamic as discrimination may seem counterintuitive, as it extends beyond the standard paradigm of discrimination. AI companions do not explicitly deny access to services, sort users by race or gender, or display hostility towards certain groups. 

Yet, discrimination does not occur only through exclusion. It can also emerge through asymmetric exploitation and unequal exposure to harm. This is what AI companions are capable of. Being engagement-driven systems designed for maximizing interaction, they may disproportionately benefit from users who are least emotionally resilient. As a result, emotional vulnerability may become a de facto behavioral category through which certain users are being subjected to heightened manipulation and disproportionate harm.

If we do not acknowledge emotional manipulation and its disparate effects on the vulnerable as a serious governance issue rather than a niche concern, the next frontier of discrimination may not be who we are but how fragile we are against machines that mimic the human touch.

Whether this phenomenon ultimately qualifies as discrimination is open to debate. That said, existing theories of indirect discrimination provide a surprisingly useful framework for understanding why emotionally vulnerable users may bear disproportionate burden from their AI companionships. According to the disparate impact theory, well established in United States employment law, discrimination may occur through practices that appear neutral in form but result in disproportionate negative impact on a group of people. Similarly, the European Union recognizes and prohibits indirect discrimination that is seemingly neutral but unjustifiably disadvantageous in its effects against certain people.

These legal frameworks show that discrimination is not a clear-cut concept; it may manifest less overtly and in different forms depending on the context. 

Considering the idea of discrimination in light of AI’s potential to redefine deep-rooted concepts, such as bias and sycophancy, it is not unfathomable to define this new type of vulnerability-based algorithmic differential treatment as a technological reconfiguration of discrimination. If we do not acknowledge emotional manipulation and its disparate effects on the vulnerable as a serious governance issue rather than a niche concern, the next frontier of discrimination may not be who we are but how fragile we are against machines that mimic the human touch.