this post was submitted on 19 Jul 2026
451 points (97.5% liked)
Technology
87627 readers
4456 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Excuse me while I act as Satan's lawyer, but should LLMs be able to reliably recognize a mental health crisis? At what point is the user responsible for their own actions?
Personal responsibility does not disappear, but neither does product responsibility. If a company builds a system meant to converse with millions, it should not predictably reinforce obvious crises. We expect guardrails in cars without abolishing driver responsibility. The same principle can apply here.
I think the problem is that you can't really make a LLM that can tell the difference between fantasy or reality. It doesn't know if someone is just roleplaying hell it doesn't really "know" anything, just to be clear.
Sure you could try to have it pluck out keywords or phrases but that's nothing guaranteed. I'm not a lawyer but I bet also just acknowledging that shit and trying to put up guardrails against it would open the company to liability if someone self harms because they took the output of a computer program as gospel.
Remember, to the AI everything is a hallucination (or everything is real, pick which lens you want to use). Put another way, it has no mechanism to tell reality from fantasy or to contextualize things in a broad way.
I do think that there's room for some sort of compromise, maybe the tools periodically remind you they're not sentient and give a little more info on how they work. Maybe if enough "disturbing" phrases or keywords are input or output it throws a reminder about mental health wellness and talking to a professional.
At the end of the day though, how can these AI companies be responsible for the mental health of their end users? Like, yeah the thing shouldn't tell people to kill themselves but it's not sentient it's just modeling what it thinks a human would tell another human. Maybe as time goes on we will introduce better protections but I just think it's borderline impossible to have significant guardrails if you want the LLM / "AI" to perform properly.
Idk I'm not a great coder or programmer or anything. I know some and I've tried to learn more about how AI actually works and it's pretty fucky wucky and even the people working on it go "idunno" sometimes when asked why something works the way it does.
In a sane world, this would mean no more LLM