Claude won't end chats if it detects that the user may inflict harm upon themselves or others. As The Verge points out, Anthropic works with ThroughLine, an online crisis support group, to determine how models should respond to situations related to self-harm and mental health.

Anthropic emphasizes that this is a last-resort measure, intended only after multiple refusals and redirects have failed. The company explicitly instructs Claude not to end chats when a user may be at imminent risk of self-harm or harm to others, particularly when dealing with sensitive topics like mental health.

Anthropic says the conversations make Claude show ‘apparent distress.’