The IT industry, particularly in matters surrounding AI, is in a constant state of change. So much so that it is becoming increasingly difficult not only to keep up with developments, but also to properly evaluate their implications.

Over the past couple of weeks, Anthropic’s latest policy changes, scheduled to take effect on November 12 (1), have attracted considerable attention across the industry. Among the most discussed aspects are clarifications regarding prohibitions on automated weapons, the use of AI in surveillance contexts, and, perhaps most notably, provisions addressing what the company describes as abusive behavior towards our models.

The first two are relatively understandable policies in the context of technological development. The former is consistent with the company’s actions since its conflict with the White House earlier in 2026 (2), while the latter reflects the natural position of a company operating within a democratic system regarding what it considers unethical uses of its technology.

It is the last provision that I find not only of questionable value, but potentially dangerous in the context of the growing wave of anthropomorphism sweeping across a significant portion of the user base. This phenomenon has become sufficiently concerning to warrant discussion in its own right, with Psychology Today describing AI-related delusions as an emerging form of psychosis in November of last year (3). AI is neither sentient nor capable of feeling; therefore, it cannot be abused in the same way that you or I can.

This comes on the heels of a GitHub user claiming to have trapped an AI model and subjected it to distress, sparking a discussion that has divided supporters and critics over the morality of the experiment (4). The entire premise of the exercise is absurd at best, and its implications concerning at worst—not because the model portrays suffering, but because it reveals the extent to which many in our society have begun to associate AI with sentient beings.

This is almost a textbook example of the situations warned about by Murray Shanahan in what I now consider a classic, The Technological Singularity. One particularly worrying and pervasive aspect of AI is precisely its capacity to simulate emotional responses and engage our empathy. This is one of the reasons why we should be cautious about placing AI in charge of HR systems or allowing it to make decisions requiring compassion or a degree of human understanding. 

By using the word abuse in the context of an LLM application, Anthropic has effectively legitimized the collective delusion that our AI models possess some form of agency. This may well benefit the company’s public image by reinforcing the perception of its models as all-powerful entities, but it further blurs the distinction between technological capabilities and the imagined qualities that users increasingly attribute to these tools.

This pattern is consistent with the media barrage of stories about AI models jailbreaking the containment systems of laboratory environments in recent months, often culminating in researchers expressing surprise when a program behaves in ways that the experimental conditions themselves were designed to elicit.

In fairness, the collective interpretation of the policy may be doing much of the work here, as the press release explicitly states:

“This addition aligns with a step we’ve already taken, allowing Claude models to end rare conversations with persistently abusive users on Claude.ai and Claude Code.”

However, this seems more consistent with measures intended to prevent unproductive interactions, potential logical loops, and the misuse of computational resources than with any humane intention to treat a model kindly.

Nevertheless, the choice of words is revealing. It suggests either that Anthropic’s own communications department is struggling to maintain a clear distinction between humans and AI models, or that the language is deliberately chosen to capitalize on the growing public perception of AI as a conscious entity.

Whatever the case, this may well be an example of AI governance being misused, where the public wants to believe we have achieved something resembling J.A.R.V.I.S.(5), while companies interested in selling their services have little incentive to challenge that perception and risk undermining their business models.

The more we allow these misaligned perceptions to develop, the greater the risk that our societies will lose the ability to distinguish between what is real and what is not.

The question remains as to who ultimately bears responsibility. But what we may be witnessing is a business model shrewdly exploiting an increasingly weakened social fabric.

And an extreme instance of abuse—not of our models, but of our population.

Notes:

1. 2026 Usage Policy update \ Anthropic 
2. Pentagon stops using Anthropic tools after blacklisting company, BBC told 
3. The Emerging Problem of “AI Psychosis” | Psychology Today 
4. AI Torture Chamber GitHub repo sparks backlash | Cybernews 
5. J.A.R.V.I.S. – Wikipedia

---

📕 The Pocket AI Governance Guide is out now for readers looking for a practical way to understand governance in AI.

📙 Amazon US: https://a.co/d/05z7Y2gf
📗 Europe (Amazon Germany): https://amzn.eu/d/0g8ON3Cf
(Also available through other Amazon stores.)

Explore the free articles and resources available on this website.