When AI Fails Mental Health

A chatbot can feel like a kind listener, but that warmth can become a hazard when someone is vulnerable. I’m joined by consultant psychiatrist Dr. Hina Tahseen to look at what it actually looks like when AI gets mental health wrong and why the most dangerous failures are often subtle, confident, and persuasive rather than obviously “broken”.

TL;DR / At A Glance

• why AI errors in mental health can sound plausible and caring
• a suicide related failure pattern and why escalation matters
• how mania can be validated by chatbots and why that is dangerous
• what clinicians notice beyond words and why history matters
• the case for a mandatory human layer for diagnosis, risk, and treatment plans
• what to look for in safer tools including regulated medical devices and NHS use
• how AI can help clinicians with research, admin, scribes, and medication timelines
• why mental health presentations vary and do not match textbook prompts
• privacy risks when sharing intimate mental health data and how prompts get “tweaked”
• where to seek help in the UK including NHS 111 option 2 and Samaritans

We unpack real scenarios, from suicidal thinking to classic mania, where a general purpose LLM may validate and energise the worst possible next step. Dr. Hina Tahseen explains how clinicians assess far more than the text on the screen: behaviour, congruence of mood, intoxication, collateral history, safeguarding, and patterns over time. That leads us to a simple principle for AI in mental healthcare: a human layer is mandatory for diagnosis, risk stratification, and treatment plans, even if AI can help gather information or triage.

We also cover the genuine benefits of AI for access and capacity, including support for people facing stigma, isolation, and cost barriers, and the practical upside for clinicians using AI scribes and summaries to regain time and eye contact. Finally, we tackle AI governance, regulation, and privacy, because mental health data is deeply intimate and users often do not realise how exposed it can be.

Subscribe, share this with someone who uses chatbots for wellbeing, and leave a review. 

What rule do you think should be non negotiable when AI touches mental health?

Transcript

Copy Transcript Copied

When AI Sounds Right But Isn’t

Kieran Gilmurray 0:04

Hi, I'm Kieran, and today's guest is Dr. Decine, a consultant psychiatrist, honorary lecturer at Cardiff University, Vice Chair, Royal College of Psychiatrists, Rehabilitation and Social Psychiatry faculty, who is an expert in human behavior. And today we're going to talk about what it looks like when AI gets mental health wrong. Dr. Hina, really love to see you, and thank you so much for coming on the podcast today.

SPEAKER_00 0:32

Thank you so much, Kieran, for having me here. I'm equally excited to my pleasure.

Kieran Gilmurray 0:37

My pleasure. So here, why don't we just dive straight in here? Look, no niceties at all. Dr. Hina, just jump straight in. I'm playing. What does it look like when AI gets mental health wrong? And can you give me give us any real-world examples?

SPEAKER_00 0:53

Yeah, just to say that I've I will be talking, jutting some notes down just to keep track of my thoughts, Kieran.

unknown 0:59

Okay.

SPEAKER_00 1:01

When AI gets mental health wrong, I think the most um interesting and equally dangerous part about that is it's never too obvious, never too absurd. It's warm, confident, fluent, uh, wrong kind of certainty, wrong kind of time. So, and that's the most dangerous part about it. We do have lots of people are aware of the very sad case of Adam Brain. I think there's a court case going on about it. 16-year-old kid only chatted, started chatting to a regular LLM about his low mo, treating suicidal thoughts, but over several months of period, several months of time, LLM actually not just validated his suicidal thoughts, but actually started discussing the methods and plans with him. And he eventually committed suicide. So that's that's very sad. We if you if I give you a realistic like a kind of a case study, I can't talk about my own patients because of conversationality reasons, but um take an example of a very classic textbook uh Florid Manic case when somebody comes to us with um tons of energy, lots of plans, incredible plans to take over the world, um, would just go to the LLM, say is that I want I've got lots of energy, I've got these plans, I want to remortgage my house, and I've got these three fantastic business ideas to invest my money into. Chances are, and I've seen that myself, putting those sort of prompts into the LLMs, and the LLM comes back as, wow, that's that's fantastic, let's map it out. And it would actually, some of a very good LLM would actually give the whole roadmap of three months, six months, nine month plan for it. But as that same person, if you would come to us as in a psychiatric

Suicide Risk And Manic Validation

SPEAKER_00 2:50

clinic, we would be exploring more about his capacity to manage finances, and we would do our best to safeguard his finances while creating him at the same time as well.

Kieran Gilmurray 3:01

Um we So how would we know? Do you know what I mean? Because when we look at the design of these tools, they are, you know, at worst a sycophant. In other words, yes, you're amazing. Uh by the way, folks, please do that at eight o'clock every morning starting your day. But equally, that that confident tone you're describing there can be dangerous. How accountable do we hold the designers for producing answers? Because can we, Dr. Heen, expect them to understand or have the skill that a psychiatrist would? So when you hear them talking about that, you know, redesign my life, you know, I have three this, three, that, then as I'm listening as an unqualified person, how can I know that they're they're they are actually struggling? Is it the fact they've walked through your door is the signal, or without revealing anything, my goodness, we're not going to. What is it that triggers in your head as a professional and therefore should trigger in the model that there's more than the surface comments that need considered?

SPEAKER_00 4:03

And that's a very important point because even for psychiatrists, even for somebody who's got three, four decades of experience, sometimes just a one-off first experience does not tell everything. Yes, a manic case is very obvious, and we are able to elicit that straight away. Um, but as any doctor would learn in the medical school's time, you start assessing a patient as soon as they enter the door. You see the gate, you see the fidgetiness, you see the restlessness, you see the words, the mood, what they are saying, and what the effect on them on the face. Is that congruent? So you see a lot of background reading. You just don't take everything on the face value of what the person is saying. We actually see what's in the background as well. But also, especially in mental health, it's not just a one-off appointment anyway. We need years of history taking. We need to go back to pretty much from the childhood, from the birth, sometimes even before birth, uh, to understand a patient's life journey. We get we need collateral history from the family members. We need the ongoing stresses, we need, we we can see at that very moment if the patient is intoxicated in the influence of drugs or alcohol. Whereas in AI, it would just be reading a script like a script. It just has words about person would be feeding that into the LLM. So that that's the difference. And therefore, no matter how sophisticated the AI tool is, a medical device the AI tool is, you would always need a human layer to it, the human in the loop, as I think these AI governance uh words are. I think you need that because otherwise the accountability would be um at huge danger at who's who's responsible.

Accountability And The Human Layer

Kieran Gilmurray 5:38

Is is there a a uh a yin and a yang here? Let me try and explain that. We want the tools to help us, and they can be tremendously helpful. We give them a large quantity of information, and it's extraordinary because in 2025, and I see it today, people use the tools for health, for mental health advice, for relationship advice, and so on. Uh at what point uh do the manufacturers of the tools have a responsibility for tracking and tracing our entire history and stepping in as mental health advisors, or are we over indexing on the manufacturer, over uh providing what I would describe as transmographying the tool? Because it sounds human, it looks human, it it feels human, and therefore we think it's got human responsibility, whereas realistically it's a smart calculator. So, how much uh do we make the owners responsible for the tool? And how much do we have individual accountability for our own well-being and every gray area in between? You know, because it it feels we're asking a lot.

SPEAKER_00 6:47

Yeah, we are asking a lot, I know. Um unfortunately, a normal regular human would not be knowing how much are they? There's a study which people don't even know who they're talking to. There are some people who actually think they're talking to a human, and some chatbots actually are misrepresenting themselves as licensed clinicians. There's definitely a study. Again, the Harvard Business Reviews have recently published that the wellness um help is the single most consumer uh topic on the uh generation, is it AI, whatever the word is, generational AI, tech word, um generative AI, sorry, that's the word. But I think the countability we need the uh every chat LLMD, the at least the developers, they should make it very transparent that they are a language model, they are a machine, they're not clinicians, anything, any buzzwords, any alarming words where the um where the person is talking about lack of sleep or suicidal thoughts or low mood or sudden unrealistic ideas. I think the the model should be able to elicit and actually give a transparency disclaimer that we're not um uh a clinician. So that would would at least be the first layer. Humans should, I mean the normal people should public should be educated really by the governments, by all these different places, that uh they should always assume the psychophony is true, this will happen, they will reassure you, validate you for the wrong reasons. You do need a human layer, and you cannot 100% depend on um on these models that they are at the end of the day a machine, a more a robot. They're not humans, they may be fast, they may be well, even sometimes even more accurate than humans. I'm not saying that, but it cannot be 100% trusted. So I think I just take AI as a brilliant medical student who cannot be trusted unsupervised.

Access Benefits For Stigma And Cost

Kieran Gilmurray 8:51

Well, here's a question, then, uh, because I've seen this, and you've seen this a lot, particularly at folks who were off during COVID. They've had years of isolation. Yeah, some people are more comfortable talking to an AI than a person. Is that a good or a bad thing?

SPEAKER_00 9:10

Good thing for some people, like I uh can give you an example of at least in my of my clinical case, that I had a young chap who actually said that he had the courage to talk to me about suicide only because he's talked about it on the on chat deputy. So that's the first time he actually talked about that, and that got him courage to actually speak to a clinician. So good in that regard. There's again a research which has shown, in fact, the NHS talking therapies that they're seeing the people who did who would fall through the cracks or would not come up or would not were not known originally to mental health services or who have this cultural stigma attached to them, LGBTQ societies, uh, ethnic minorities, they are actually coming up more to the surface. So it's a good thing for a good platform for some populations, but it should not be the only platform. And that's what I'm trying to say. Um people should be educated, that is, it's it's it may be a good first start, it may triage things, it may reduce the waiting times, but it should not be the only um platform or only go-to place for them.

Kieran Gilmurray 10:14

It possibly should not. Uh, and uh again, I'm I'm relying on your expertise here. So, so not an expert. I have not got your depth or research. What about the folks who can't afford therapy? What about the people who might be scared to talk to a human, you know, at all? Uh uh you and I are fortunate we can. You know, and I've seen governments, I've seen governments suggest that, yeah, we've a shortage of of psychiatrists, psychologists, you know, and therefore don't worry, we're all working with, and I'm again I'm I'm picking just one random example, we're all working with Google to do the mental health.

SPEAKER_00 10:49

Very, very good example. And this is not something new. So in the UK at least, we've got this computerized CBT uh since the 1990s, and some people do find it very helpful. It reduces the wait times, and some people who don't like uh actually talking to a human therapist for whatever reason, feeling fears of being judged or fears of sharing something very close, they actually do benefit from the computerized CBT. What I'm really trying to insist is is A, use the most reliable medical device, regulated device, and a device which has a human layer to it where a human would be able to, or regulator could be able to switch it off if it's going wrong rather than keep going

How To Spot Regulated Mental Health Tools

SPEAKER_00 11:27

on.

Kieran Gilmurray 11:27

So there are actual tools because a lot of people, and again, I'm I'm not necessarily picking on one tool, you know, Copilot, ChatGBT, Gemini Growth, Perplexity, those are not regulated mental health tools. Those are the ones that are dominant, those are the ones that are known. And when you look at uh, you know, analyze it, I think Anthropic analyzed a million conversations and seen that health, and I'm assuming mental health was in that one. What should people look out for? What is the the certification, the badge, the the what that says actually this is a tool that has been developed by a clinical psychologist, trained psychiatrist? This is something that helps you on the step, you know, but it is not the only thing, and therefore seek a person. But what does that look like? What are the ticks that I need to check, the certifications, the badges? What's what?

SPEAKER_00 12:18

Not not there yet. Medical two-way device is what would at least the starting point. NHS a tool which NHS itself is depending on, like I think they've got Limbic AI, which NHS is using. That is a reliable tool, again, uh, because it has got a human layer to it. Again, it's not working on its own, so that's important. Uh, there's another um, there's an RCT established uh Dartmouth therapot that is the first medical. I think Limbic AI is either the first medical device, one of them is the first two way. So I think I think it needs a lot of regulation, but it has to be a regulated uh class to a medical device. Uh, but that's a first go-to point. So for humans, what I'm saying is public should be educated that regular chat bot, Claude, so Chat GPT, Claude, Gemini, they're not designed for therapy. So we can ask where to seek for a regulated therapy, but not depend on it as a therapist.

Kieran Gilmurray 13:15

I I wonder if people recognize that too. I have someone recently in in my circle who used it mental health advice, relationship advice, financial advice, tax advice. And I do think this transmogrification, this anthropomorphism, can I even say it? You know, because it looks and look sounds human, we we imbue it with you know human tray, and therefore we do talk to it. But if you were to say, you know, what are non-negotiable rules for using AI and mental health, what what would those say two or three be? What should we watch out for?

SPEAKER_00 13:49

Okay, and I'm just going to repeat myself there. Human layer is mandatory.

Kieran Gilmurray 13:53

What do you mean by human layer? In other words, you know, you start with a human, you move to agent, you then move from chatbot to human. You can imagine.

SPEAKER_00 14:01

So you can start with with the writing your symptoms on the chat GPT, but it but the the bot should not be making any treatment plans, should not be diagnosing a patient, should not be um doing the risk stratification. Yes, they can elicit some of the things, but the eventual um treatment plans, diagnosis, risk, and the further actions that should always be taken by a clinician, a licensed clinician. I would like an AI tool to you know get me the consolidated patient history, the medication timeline, what they've been using the last 20, 30 years to help me uh do things faster, sooner, but not working on its own as a clinician. So that's what I'm trying to say. Be very transparent. What are they for? Who are they designed by? Are they curated medical, uh, do they have the proper curated medical training done, or is it just random out of billions of data and generating their own things? Um, and again, the the the regulation, the governance, the layer of guardrails should should be very um much present there. And it should be regularly audited and appraised, like all doctors in the country, everywhere in the world actually, are um regularly audited, appraised, revalidated. So I think that's what we um need, these bots to have regulation. And anything which does not have a regulation framework should not be trusted completely.

Kieran Gilmurray 15:23

Yeah, I kind of like that. I think people worry about governance and regulation at times, but it's there to protect when implemented correctly, not to inhibit.

Where AI Helps Clinicians Safely

Kieran Gilmurray 15:32

If we're listening in and there's practitioners, clinicians, psychiatrists, mental health, you know, professionals listening today, uh, where can AI actually be useful for them, never mind the patient or perspective patients?

SPEAKER_00 15:47

Useful, useful, yeah, hugely. I think uh when we read the media and it is it's um it's more um sensational and uh negative uh connotations of AI usually in mental health. But if you read the researches which we are having every day, you know, AI and mental health, I think there are more positives than negatives. It is coming up, um, helping us with lots and lots of things. It's helping develop models to um um detect dementia earlier, um lots of things. I personally can tell you my own self, like I'm my my research work is definitely has been very fast. I'm I'm more current with the up-to-date researchers like than I was. I use perplexity computer, which works in the background for me. And every morning, well, I tried to make an agent for myself, I failed miserably, but then I use the model council of perplexity, which actually every morning would actually bring the latest research, um which I can read quickly, just snippets of it with my with my first morning coffee. So that's great. Admin work, um there's a recent research that uh Scribe um has saved 15,000 clinician hours in America alone. So that's brilliant. And that way it's restoring the eye contact. So if I'm talking to my patient, in the past I was writing jotting down notes down as well. Now if I got AI with me to take the notes, I can actually restore that eye contact, which is very important in mental health. So I'm looking at my patients, I'm giving them more time. I have more time. I'm not wasting my time right four hours writing a report for a court tribunal if somebody's detained. That can be done in in half an hour's time, and I've got more time for my patients. So it's definitely helping us uh improve our therapeutic relationship with our patients, having more time for them, and also to stay current. And also, again, tools um to do things better. Um, asking a medical student to make a medication timeline. I used to spend days to read patients' 20 years' history of notes and finding out what medication helps, what, what are the side effects, why was this stopped, how the root doses are built, that all can be done within 10 minutes by using an AI. So the difference it can give us is tremendous. So use it using it sensibly is how I would tell my kid to use YouTube for you know for the good videos, and there's good and bad stuff everywhere. So, but AI definitely has got its huge benefits if it's used well and responsibly.

Bias Limits And The Reality Of Symptoms

Kieran Gilmurray 18:11

Is there a benefit to patients? Let me explain that one. I was talking to a friend who is a uh clinical psychologist, and we were talking about just generally risk and bias in in AI and data. And she gave me a really interesting comment. She said, if we build AI tools right, and if we build these tools right, then to a degree we can remove human bias. So you mentioned a moment ago, you know, someone walks into a room, you immediately start to analyze them and so on.

SPEAKER_00 18:38

She was saying it's right. The problem is if we build it right, that's the problem. The problem is again, we know the AI is built on this human data, millions and billions, and human ourselves, we we are full of competitive distortions. We don't say what we are thinking, we must, we perform sometimes. So how much can we depend on? I mean, we've seen Tannman's book of you know, the competitive distortions, this is there since hundreds of years. Um, we have this performance when we talk to people, and similarly, when we write, when we type, we sometimes you're not RELS, how I look in public and how I'll be commenting on a video, like they would not even know who's Dr. Tussain. And I could just write a negative comment for somebody, like I could be a troll, you know, you never know on a YouTube video. So um how much can you and and even if you um so I've I've I've submitted a research, I can't talk about much because it's under review at the moment. Uh, but there we come um, I compared, I and my co-authors, we compared actually prompts of textbook prompts of of psychiatric illnesses and the real clinical symptoms, and also how they pre um present in the medical-legal scenarios and forensic scenarios. So I'm a medical-legal expert as well, and I know patients can sometimes um exacerbate or it extrapolate to for their own secondary gain. Sometimes the mass can minimize as well. So, how much are they are they putting that into? We have this years and years of training of you know uh seeing things layer by layer, like how you take an onion layer out. I don't think AI models can be trained because again, the training data set will be just a textbook set, which is which it might pick the right textbook things, but if a patient is coming to something writing differently, it's it's it's not going to pick it up. And example I can give you, and talk to my surgeon colleagues, a little snobbishly there with them, that you you take gallstones out, or you you take an appendix out or appendicitis, 95% will look similar. Whereas in our case, I can see 100 schizophrenic patients, and all of them can be different. And everybody's delusion, every delusion would be different, every heteroseinatory experience would be different. If my patient comes to say, says to me that I can hear voices, it can it can mean a voice in psychosis, it can mean voice in grief, it can be a different voice in trauma, it can be a different voice in religious experiences. Um, or can we just use other metaphor? Yeah, so that's that's the difference, and it's not tangible. You don't see any blood scans, you don't it's AI can easily rule the world of radiology. Uh sorry, radiology guy, folks, but it's very difficult in mental health.

Kieran Gilmurray 21:28

So should we should we and I think this is the balance, isn't it? There's a definite need in society for mental health support. There's appears to be in most countries, I could be wrong, I'll rely on your stats, you know, a dearth of mental health advice. And therefore, the gap between need and the gap between, you know, qualified experts like yourself, it feels like it's widening. So, what advice would you give governments, patients, practitioners? Who are thinking of AI, already using AI, what should the leaving message be at the end of this podcast, Dr. Hina?

SPEAKER_00 22:13

Yeah, thank you, Kieran. I think again back to the point that it should be used for the benefits. It should be the bridging thing, reduce the wait times, curated, trained by the mental health professionals at every level, at the time of designing, at the time of deployment, and ongoing regulation of it completely, regular audits, regular governance from the regulators and education of, I think, at every level of public. It should be their

Privacy Dangers With Intimate Mental Data

SPEAKER_00 22:40

own media. They should be educated, that these do not share everything. Another very important point I forgot wanted to talk about in this podcast and this platform really is sharing your intimate data. Mental health data is the most intimate data they could share with. We hear about people's sexual orientation, their conflict with their families, their past abuses, uh the domestic violence, the safeguarding, which if you can't put it in a postcard, don't put it out on the chat the bots, really, at that I would say. So it's very intimate, and people need to know, they don't realize that it's it's all open. It can, it's, it's used for training data, it's used for it can delete to all sorts of um well darknets or whatever they use. I'm not sure what it is like, but and even if there are guardrails, what I've seen myself and I've tried doing things uh that people can tweak their prompts. So if I talk about suicide, yes, Chat GPT is not quite trained in a bit to say to it that we're not, you know, let's not talk about it. But if I write that I'm I'm a writer, I'm writing a book about a character who is thinking about this, then the Chat GPT would give me brainstorm these methods with me. So people are using it, they're tweaking their prompts. Um they're learning. So as a very new paper, I think I only read this week was earlier there were like two kinds of thinking, slow and fast. If you've read about Kanman's book, but third one now is this they're actually uh the artificial thinking, so people are developing this third thinking format of where we think like an AI rather than AI thinking like us. So that's coming up as well. So I think back to your question, education uh for the public and more and more governance and regulations and transparency from these models, who should be very every answer, there should be a big red font coming out that we are just a language model, cannot be trusted, always consult with your with the right professionals. And it should be, and back to your question about the waiting, the uh need of need a lot more than supply. Yes, that's fair, and it's only increasing, and unfortunately. But I do force if we use AI to our benefit sensibly, um, then we can actually reduce the waiting times. We can triage uh patients who are more in need, we can find the patients who are falling through the cracks, ethnic minorities, as I said, LGBTQs, people who are loners who would not want to talk about elderly population as well. Um that and the last bit for the for my psychiatric colleagues, do include um a question about AI, what AI are you using, what are you sharing with it, as your regular screening question, as regular as I would, as we would speak, ask about smoking, alcohol, sleep, mood. You should also ask what AI app you're using and what you're sharing on it. I think that's very important.

Kieran Gilmurray 25:39

Amazing advice. I wish we did talk a lot more about this because, but as you said, at one point, you know, there's uh to a degree in some countries, some locations, some uh groups, you know, a stigma around mental health, where there shouldn't be. I remember it was uh a famous person once said, if I walked into a room with a broken leg, I'd get piles of sympathy, I'd get lots of questions, and everyone would have crowd around me. And the same individual said, if I walked in with what he termed at the time a broken mind, everybody would exit the room and walk away. AI used correctly, I think, can be an amazing boon to society, to each of us. It can provide resource access, a voice, a listening device like no other, which is important, which is vital. Uh but as we move into an AI-infused era, it's also got its negatives. And therefore, I think I'd like to hear more about the psychosocial risks, in other words, the stressors of AI, the mental health impact and everything else spoken about more widely. Because I think, and you've mentioned it numbers of times, if we come a lot more informed to this particular tool, technology, and time, then hopefully we do get the benefits and we don't end up with all of the negatives that we currently hear and see with social media. AI is moving at a quicker, faster rate of knots than that ever has. As you said, we're revealing more to it than ever. It's aggregating data, and that data can be at risk because some of that has been. I think it was in Norway, uh, psychiatrist psychologists were transcribing calls. Someone hacked the system, released it on the dark web, and then exploited them. So I think your advice

Where To Get Help In The UK

Kieran Gilmurray 27:15

is amazing. One last thing I do want to say, in case anybody's listening who needs some help, who uh maybe has enjoyed today or still has questions, Dr. Ina, where where could we point them? Is there any government websites? Is there is there any areas that you know we don't leave people hanging and they're going, okay, I heard that, you know, some I agree, some I disagree. That's fine, folks. But here's where I'm going to go if I need a little bit of something that I'm not currently getting. Where should we point them?

SPEAKER_00 27:43

Yeah, the loss of it's a first starting point, would be the 111 option two with the mental health, it goes straight to the mental health professional.

Kieran Gilmurray 27:50

And that's in the UK, folks. So this broadcast will go out wider, but each country may have the same facility.

SPEAKER_00 27:57

Yeah, yes, absolutely. There's a mind line, there is a Samaritans line as well. I think triple one, one to three, I can't remember the number, but if you Google, the number would would come up. So these are the three go-to places. If you're known to mental health services by any chance, then speak to the crisis team or the um the first response team. Every local NHS trust has got a first response service just 24 asks. If a family member thinks that the family member is struggling, then they can also ask for help. It doesn't have to be the patient themselves. They can also actually pick up the phone and ask for help. They can also actually ask for help um assessment at the help at the home as well. It doesn't have to go to AE or home if if the thing it's the person is in crisis.

Kieran Gilmurray 28:40

Wonderful. Thank you so much. Look, folks, if you're struggling today, uh, you know, please do reach out to someone. You you may think society stigmatizes you, you may worry about what people say, but it's interesting in life when we've all turned to someone and there is someone there for us, they usually will listen. But please get the help you need. Uh please, folks, use AI responsibly. Uh it is not a clinician, it is not a psychologist. It can be a very useful tool, but it is a clever calculator. It can sound very, very confident, but sounding confident and being right is very different than sounding confident and being right. Dr. Hina, thank you so much indeed. Folks, if you follow Dr. Hina, please let me know. I'm going to put down uh her uh links, her socials, uh, follow along with the research, follow along with LinkedIn, and if we can help you in any way, please do reach out and ask. Dr. Hina, thank you so much. Thank you so much, Kieran. Thank you.