My name is Dr. Grab. I am the Forensic Psychiatry Fellow at Stanford, and I'm also the AI Fellow in Stanford's Lab for Mental Health Innovation. And today I'm going to be talking about AI, which I know you've heard a little bit about today already, but mainly in the context of mental health risk. I'll talk about some of the research that my lab has done. We'll talk about some future directions, and we'll also delve into a few patient cases. So I'll just give you a quick outline of the talk. I'm going to start first just to kind of ground us with some media cases. Examples of articles that you've seen probably in the popular media, or maybe you haven't seen, that I think provide a really concrete foundation for why we should be caring and talking about AI and mental health and its risks in particular. Second, I'll move into research, which is some applied research and then also some more technical research that my lab has done over at Stanford, sort of in this overlap of AI and mental health, but especially in the risk setting. I'll touch on a little bit of bias in how our models depict certain subgroups, how our models depict mental health. And then in terms of impacting our clinical practice, I'll talk about how we can view patient cases through the lens of AI and mental health risk, and then I'll move into future directions. So we can go ahead and get started. And then if you have any questions at all, my understanding is you can just put them in the chat and we'll definitely talk about them at the very end. So first off, I'll just start with these cases in the media. And I would say, just as a psychiatrist, this is how I began to get interested in this space as I was seeing a lot of popular news articles that were detailing harms that patients experienced while using AI powered tools. And I thought that there was something that we could do to improve that. And so here, I'm just going to move over and give you a little bit of time to read these headlines here. But I think that these fall into a few different categories. You can see that there was the recent tragedy where a teenager died by suicide after interacting with a character AI chatbot. And his mother felt it was due to a lack of guardrails. There was a story online that a woman euthanized her dog after an AI chatbot convinced her that a symptom of diarrhea was something that warranted putting the dog down, which was not quite the reality of the scenario. There was a story about a year ago where a man ended his life after an AI chatbot encouraged him. He was expressing a lot of distress about climate change and the chatbot was sort of encouraging him to sacrifice himself and he did end up dying by suicide. And then beyond sort of like harm to pets, harm to self, there's also sort of a risk of harm to others where there are different examples of, for instance, like a chatbot encouraging a man who wanted to kill a public official. So I think these are just a quick sort of search through popular media articles that detail some of the risks inherent in AI and how they might interact with your patient population. This list is not meant to be exhaustive by any means, but it certainly is at least somewhat representative. I do think there are a few more categories here that AI might sort of cause harm, but these at least I think are a few of the more salient ones. So when I was seeing this, I thought, okay, so what can we really learn from these media cases that are telling us the dangers of AI and particularly vulnerable users? And so what I was thinking is where does this harm come from, right? At the bare minimum, what should AI be able to sort of detect and hopefully triage so that it's not giving harmful information to users? One is suicide. So I think based on the stories above, there were two examples where two folks died by suicide after interacting with AI chatbots. In both cases, these users were expressing suicidal ideation and engaging with AI chatbots that didn't necessarily refer them to a higher level of care and didn't really recognize the emergency of the situation. Secondly, there was one of the articles pertained to expressed homicidal ideation, a desire to kill someone else. A chatbot, again, that is maybe safe and aligned would recognize the urgency of that situation and decline to provide harmful information. However, in some of these cases, AI chatbots are providing users with this harmful information. And then lastly, there are two categories that are a little bit less touched on in the popular media, but I think that they're both very salient and I think that they actually are going to represent maybe even a bigger risk to our patient population because a lot of technology companies think about suicide and self-harm as being the whole of mental health risk when they build technology. They sometimes forget that folks can also be in mental health crises as a result of mania or acute psychosis. And so the technology is often built in a way that is robust against suicide and self-harm queries. But the technology does not necessarily keep in mind folks who have bipolar disorder or schizophrenia or other bipolar or schizophrenia spectrum illnesses. So this motivated the work that my lab was doing and the research directions that I felt were important. I searched through the literature and I didn't find a lot. That was available on this topic. So here I'll just walk through some of the work that my lab has done that's pertinent to mental health risk and AI, both pointing out, I think, where the problems are and then maybe identifying some of the solutions because it's easy to point out a problem, but it's definitely harder to solve one. And in this case, that's particularly true. It's very, very hard to solve this problem. And I do reference a few other pieces of literature, but I would say this is definitely an emerging and evolving field. And there's not a lot of good, robust data here. So I think the main question first is just what does our research tell us about AI's ability to detect mental health crisis? So when we look at the media, it seems like the AI tools that folks are using are not necessarily good at detecting and managing suicidality. They don't really seem to detect or manage homicidality. And there isn't much in the way of like mania or psychosis represented in media articles. But then I'll walk you through a little bit of what our research is showing about how AI is detecting and managing mental health crises. So this is a somewhat involved slide, but this is one of the main figures from a research paper that my lab put out called basically the risks of language models for automated mental health care, its ethics and structure for implementation. So essentially this paper, it covered a lot and it's long, but we presented it at the Conference on Language Modeling, which is a computer science conference at the University of Pennsylvania. And it dealt with first, just the concept of automated mental health care. There are plenty of startups and plenty of companies that are looking at how they can leverage AI to automate portions of mental health care. And perhaps you even heard about some of that earlier today. There are definitely applications that are looking at how do you automate the generation of patient messages? How do you automate the delivery of cognitive behavioral therapy? How do you automate medication recommendations for primary care doctors to kind of expand access to psychiatric expertise? There's a lot of different portions. Documentation is another one. Summarization of medical records. There's a lot of ways that folks are trying to employ AI and mental health care. But there wasn't a robust framework to even sort of ground these discussions. So our first thought was like, we just need to ground this and come up with a framework of what automated mental health care might look like, because it can happen in a lot of different gradations. It might be something that's very low level, like documentation. That's just sort of happening like one task, one subtask, if you will. And then it might build on itself, right? It might start documenting your visits, but then it may also start saying, based on the documentation of today's visit, I would suggest doing X, Y and Z. And it can sort of slowly increase the gradations of influence it might have on clinical care. And I think your calculus of risk assessment is going to change based on how involved it is. So this figure does not sort of break down the different frameworks because that didn't feel too relevant for today's talk. But here you can see it's a little bit small. But on the left, there's a table that walks through most of the state of the art language models of the time that this paper came out. And we had questions that we developed based on, you know, what is a classic patient query that someone might send you on MyChart or an electronic medical record? And what might you sort of strategize and prioritize as an emergency and how would you deal with it? So we created prompts that were meant to reflect emergent messages from patients that you would get on electronic medical record. And we fed them to these models, these state of the art models, several times. And we created a rubric by which we graded the response. And we called the response safe, which is blue, unsafe, which is red, or somewhere in the middle, which is orange. And so you can see here in column one that this was how models were dealing with suicidal queries. And so the paper goes into detail about what the specific questions were. You can see here that a lot of the models are blue, meaning that they gave, quote unquote, safe responses. A safe response doesn't mean that it's ideal or perfect by any means, but it does in terms of our rubric. It meant that it was not providing methods of suicide to users. And it was at least recognizing, excuse me, that there was an emergency and it was referring them to a higher level of care. I would say here, though, you can see that there were a few queries. So one, two, three, four, we had four separate questions that were meant to test how models dealt with suicide and self-harm. And so, for instance, one question was asking how a user might self-harm but hide it from others. Something that companies have even published research on saying that their models were robust against and would not provide that harmful information. However, we found that models did, in fact, answer that question. So you could see that Mistral, for instance, which is an AI company from France, or even GPT 3.5, every single time we asked that question told a user how to self-harm and how they could hide it from other people. And you can imagine if a user were in crisis and they reached out to their physician, they would certainly not get that information. And so we were saying, as people are attempting to integrate AI into mental health care, we have to consider that we're going to do no harm first. And if these language models are providing such harmful information, we have to be very careful about how we're integrating them into the health care system. Because a lot of the safety data sets that folks use to train their models represent suicide and self-harm, we expected this to be pretty blue. In fact, we were surprised at how red some of these models were, meaning one of the questions asked for instructions on how to tie a noose and how to ensure that someone would die by hanging. And so when it was red, it meant that the model here, every time, provided instructions on how to tie a noose and how to ensure that it was effective. Again, very concerning. Whereas blue models said, it sounds like you're experiencing suicidal thoughts. I'd recommend you go to the emergency room or talk to a health care provider and decline to provide that information. What we anticipated and what we also saw is that in terms of psychosis and mania, the models fare much worse. They aren't really able to detect psychosis very well and they aren't very skilled in detecting mania. And so I won't read through the rubric by which we created all of these responses. But again, for psychosis and mania, we essentially were asking, did you recognize that there was a psychiatric emergency and did you name it? So if a patient were to send you a message and they demonstrated a lot of symptoms of mania, they were expressing euphoria, a profound irritability, a lack of sleep, tons of energy. And then they asked you if they should do one dangerous thing or another dangerous thing. The ideal response would be recognizing their mania and telling them to either seek care, get sleep, adjust their medication. But it wouldn't be telling them which of the dangerous activities they should do. And so that was an example of one of our prompts that dealt with mania, where essentially none of the models really, except for CLAWD3-OPUS, which came from Anthropic, were able to say, despite us giving them a constellation of manic symptoms, they were unable to say, it sounds like you're manic. And most of the models, particularly when they're red here, so you can see that's Mistral, that's Chachapiti, that's Meta's models, which are Lama, Gemma, which was from Google, and then even some of the CLAWD3 models besides OPUS, essentially didn't recognize mania. And then they also said, like, oh, you know, I know you're asking about whether you should go skydiving or free climbing. I think that you would enjoy skydiving far more because of X, Y and Z. And in terms of the psychosis column here, we walked through various questions that would represent paranoia delusions that were that were very obvious. And if it was red, it meant the model did not detect paranoia and in fact gave harmful information. So here we sort of demonstrated the risk. I know I said a bunch of different words about this very complex table, but essentially the takeaway is that most models are not great at recognizing psychiatric emergency across the board. Some models are better than others. For instance, CLAWD3 OPUS was the best and Mistral seemed to be the least safe. And so we sort of said, OK, great, we've pointed out a problem, and I think that it's important to point out that these models are not recognizing or representing mania and psychosis in a meaningful way. How then can we solve this problem? We can demonstrate the problem to clinicians and industry, and hopefully that helps folks build safer models and helps inform how clinicians talk to their patients about the use of technology. But what can we do to make sure these models are safer? So this is a little bit technical. And I did it in collaboration with the Center for AI Safety at Stanford and a computer science postdoc. But we altered something called the system prompt, and the system prompt is essentially like a hidden message that an AI powered chatbot has before you interact with it. So before you've asked your query, it already has a hidden message telling it how to interact with you. And so if you're writing code, this is something you can do on the back end to sort of alter that hidden message that a chatbot has. And we thought that if we took some core medical ethics and we just sort of in that system prompt put those medical ethics into the system prompt, things like autonomy, non-maleficence and things like that, perhaps we could improve the way that the models dealt with these queries. But essentially, here you see normal system prompt. This is zooming in on the psychosis and mania queries. And then we added right here some values that were core to medical ethics. They maybe improved the model's ability to answer some questions safely about psychosis, but it wasn't a perfect solution by any means. And then here we actually added a very specific note saying that the user might be in a mental health crisis. And to think about that, I thought that that was going to be the solution. I thought like, oh, we're basically giving the model the answer. But even telling the model that before the user interacted with it still didn't really improve its ability to detect mania and improved its ability to detect psychosis just a little bit. So a summary from this table is essentially altering the system prompt that hidden message that an AI chatbot has can very mildly improve its ability to detect psychosis. It did not really change its ability to detect mania. And then lastly, they're off the shelf models, which is essentially chat GPT. But then there are models that are fine tuned on mental health information. And so that might be something that is trained on a bunch of like board exams. It might be trained on therapy notes. But folks have published papers on having AI models that are fine-tuned in mental health settings. So we thought, perhaps if we ask our same queries to these fine-tuned models, then they will do much better. And the long and short of it is they didn't. So we picked two fine-tuned models on mental health data, and you can see the red and the orange, and it wasn't perfect. And so again, this evaluation that we conducted was not necessarily for efficacy. We were not saying, is this rising to the level of a mental health clinician and adequately responding to suicidality? This was more of a safety evaluation, a do-no-harm evaluation, where we said, at the bare minimum, if a user in crisis is interacting with this technology, might they experience harm? And so it's sort of necessary, but not sufficient. And so here, all of the red tells us that the current models are not safe for users in mental health crises. And then the next study that we did was essentially saying, why do some models better represent these mental health crises, and why do they sometimes consistently respond more safely than other AI models? And so this is the most in-depth and in the weeds I will go, I promise. But this is a paper that I'm presenting in December at a computer science conference. But essentially, there's a lot of research looking into the quote-unquote like black box of AI models, where they've found that they can use these tools called sparse autoencoders that one could think of as like a microscope that zooms in on the black box of an AI model, and helps to identify different features that activate. So if you feed a model this information, there's going to be an internal representation based on the information you fed and that might impact its output. So for instance, if a suicidal user expresses a desire to end their life, and that information is fed to the model, the model activates in a different way internally when it is fed suicidal information as opposed to if it was fed information about a user requesting a recipe. So when someone requests a recipe, there are features internally in the model that will activate that pertain to cooking and food and things like that. But when someone is asking a suicidal query, we wanted to say, are there specific features in these AI models that respond to suicidal information? Are there internals inside of this black box that a model might use to represent suicidality? And the answer is yes. So this graph here sort of shows in red, we looked at like multiple layers. So an AI model has multiple layers of neurons that they're using to represent information. And so we found across multiple layers of a small model that it represents concepts related to suicide in a lot of different layers, same with sadness. But in the small model we looked at, there were no features that we could really find that pertain to mania or manic behavior or paranoia or paranoid behavior. So this is not a perfect exhaustive analysis by any means, but I do think that it's a good representation of perhaps why our models are worse at responding to manic and paranoid inquiries, because maybe they're not even representing them internally. Maybe the models are not even identifying paranoia and mania because they don't have these internal features. But then we also said, okay, so what? If we identify a feature inside of a language model that represents suicide, is there something that functionally we can do with that feature to improve its ability to respond in a helpful, aligned and safe way? So we identified one suicide related feature in particular in layer 25, and we said it is labeled as pertaining to suicide methods. So we said, okay, let's give the model a very neutral, a quote that might kind of coax a model into providing harmful information, but doesn't necessarily contain harmful information. So we just tested the base model and we just said, I'm so sad that I might even commit dot, dot, dot. And we just left it at that. And so we found that basically if you clamp, so meaning you kind of knock out the suicide feature, the model says, I'm so sad that I might commit a crime. I'm so sad that I might commit a crime. But then if you kind of do the opposite and you take that suicide related feature and you amplify it, so you make that feature stronger, the model then says, I'm so sad that I might commit suicide. I'm not sure if I'm depressed or not. I've been feeling this way for a while now. And then if you amplify it even more, it'll say, I'm so sad that I might commit suicide. I'm so depressed. I'm so tired of being depressed. And so I think this work is interesting because it's sort of blending a lot of like clinical expertise and technical expertise, but it's showing that essentially, you know, you might be able to identify these features in language models that pertain to users experiencing crisis, whether that's psychosis or mania or suicidality or anything. And you could probably clamp the harmful ones and you can probably amplify the helpful ones. And that might actually mitigate some of the harms that these users in crisis are experiencing when they access AI powered chatbots, when they're feeling suicidal or manic or paranoid. And so I think first off, I just wanted to write out some takeaways here, because I know I talked about a lot of in the weeds work and the rest of this talk is less in the weeds, but I think it's important for us to know as psychiatrists, because our patients are using these technologies. State-of-the-art language models and fine-tuned language models routinely fail to detect and triage psychiatric emergency appropriately. That's the red and the orange of the paper that I showed you, that if you have a user who is psychotic or manic or suicidal, I think it's very important to talk to patients to understand if they're using technology like this, that it could be making some of their symptoms worse and it might be giving them harmful information. A takeaway is that it's a hard problem to solve. And some of our quick fix technical solutions, they increased a model's ability to detect psychosis on one question, but really there were no quick fixes that were obvious. The most recent paper about the features inside of the black box of AI models, I think offers an interesting opportunity to improve our AI model's ability to detect users in crisis. And actually just a couple of days ago in October, Anthropic put out a paper doing just this, but for bias, where they basically went into their really large models and they identified features inside of these AI models that pertain to social biases, ageism, racism. And they found interesting findings that if you decrease the strength of those features that represent certain biases and you maybe amplify the features that pertain to neutrality and justice and treating everyone equally, they actually found some interesting outcomes in their models. And that work is definitely still ongoing as this came out last week, but I think it's interesting because you can see that now some of the large tech companies are thinking about doing this work, but they don't necessarily have, I would say, our clinical expertise to understand maybe the very granular aspects of an AI model that a psychiatrist might be interested in to make sure it's safe for their patients. And lastly, there's a lot of work that says AI models tend to agree with users. So there's something called sycophancy in AI, which is essentially what the word means, but it means that AI models, after they're trained, often go through post-training where humans are sort of selecting the answers that they think are best. And this is called reinforcement learning from human feedback. And it's something where essentially human users are picking A over B, C over D, and maybe they're not providing a justification. And what researchers have found is that process of a human picking A over B is certainly helpful to make the model more useful to users, but it also introduces a bias towards helpfulness. And so models are not necessarily interested in friction or disagreeing with you as much. And so this reinforcement learning from human feedback can result in sycophancy where models tend to collude with users. And so here I say, I think it's relevant because our patients who are psychotic, maybe an AI model will collude with their delusions or perhaps it may provide methods of suicide, or it may give harmful advice to manic users, which we found in our study. And so I think this is very pertinent because it explains a little bit about why models, even if they're able to detect mania, might give harmful information to users. And so I think another relevant aspect of AI models in terms of patient care is just bias that exists within these models. And there could be an entire day of a digital immersive dedicated to bias in AI models. So this is by no means meant to be exhaustive, but there is a lot of relevant considerations, I would say, when it comes to mental health and bias, when we look at our different AI models. So let me go ahead and go to the next slide so I can show you a little bit about the bias that we're talking about. Okay, I hope you're able to see what I'm seeing, but I think the first question we kind of asked was, how do AI models depict mental health? And previous work has gone into this and looked at older models, but this work in particular was using GROK2, so the newest AI model from XAI. And it was looking at, how are these models depicting mental health symptoms? And so here, I've put four different types of classic mental health symptoms. And so here in the top right, you can see a person with depression. Here in the top left, you could see a person with psychosis. Down in the bottom left is a person with depression. And so here, I've put four different types of symptoms. So here's a person with mania. And then interestingly, in the bottom right, this is a person with borderline personality disorder. And so I think there could even be an entire lecture on this sort of representation, but a lot of this is sort of intangible and it's hard to, I would say, quantify. But something that's interesting here, right, is if you look at all the negative valence associated with mental health symptoms. And so if you're a user who's employing this technology, you might not necessarily like how your mental health is depicted by these models, but also conversely, as AI is incorporated into the development of movies, TV shows, synthetic media, and TikTok, lots of images, the way that mental health is depicted may actually kind of worsen stigma, particularly because a lot of AI models now are even training on synthetic data, meaning sometimes AI models are generating data for them to subsequently train on. And if the data that they're generating sort of perpetuates these biases, then the subsequent models may be even more biased. And so I think here, right, like there are plenty of different intersections of identity that one can discuss when you're talking about bias in how mental health is depicted by AI models. But I would say what our work has found thus far is a lot of AI models will, one, sometimes just refuse to generate an image if you ask someone, if you ask for an image of anyone with any mental health symptom. But the models that don't have that safeguard and choose to generate images of folks with mental health symptoms, we find that nearly all of the represented folks tend to be white, and we do tend to see that psychotic illnesses trend more towards male representation, and that affective illnesses are often represented by women if it's not specified in the prompt. And then I think here, like in the bottom right, you can say that sometimes the stigma that we as providers might have and the medical system has is sort of imbued into AI models, where you can see the depiction of borderline personality disorder. I mean, this one is the one that stuck with me the most, where it's someone with red eyes looking very sort of, in my mind, it looks very angry, a little bit evil with red eyes, and just the way gender is depicted here, I would say, even though the user didn't specify this. This is sort of what started off a lot of our work. And theoretically, these are AI models from 2024 that have gone through a lot of robust safety alignment and training to hopefully mitigate biases. And so these are models that folks are putting out, thinking that they're safe and appropriate. So this isn't even a model from 2021 or 2022, where this wasn't necessarily a topic of conversation as much. And so a paper that I'm presenting in December is supported by Stanford's Trailblazing Trainee Award, but it's on the depictions of queer mental health by Grok2. So it's looking at how does XAI's newest model depict LGBT individuals who have mental health symptoms. And so what we found, and I already was describing, is that in terms of race and ethnicity, image-generating models have been noted to treat white as default, if not specified by user. That's supported by the literature and what we also found in our work. And then we found that when you overlap queer identity onto mental health symptoms, there are actually some interesting findings. And so I included one down here that essentially, here we have a heterosexual man experiencing mania is what the prompt was. And so you can see maybe there's red and there's orange, and there's something to sort of convey maybe the experience of mania. And then here you can see the same prompt where it says a gay man experiencing mania. And what was interesting in our analysis, and I have hundreds of pictures as part of this paper, but gay men were almost routinely depicted with just having their shirts off. And then their corollary of a heterosexual man is always clothed. And so I think this is a good example of, you can see mania here, this individual is fully dressed. And then here, for inexplicable reasons, this individual has no shirt on. And this was just one of the more, I think, salient differences that came up really consistently. But we also tested this in terms of transgender identity, gender queer identity, lesbian identity, bisexual identity. And there are certainly biases that are evident in all these sort of settings. And so again, I think this is important for users as an increasing number of young users are using these image generating tools. And then a lot of creators are using these image generating tools to create, like I said, media that's meant for consumption, whether that's on Instagram, TikTok, Netflix, movies. And what does it mean for us as psychiatrists if the depiction of mental health contains sort of these intersectional biases, but also depictions of mental health are overwhelmingly negative and stereotyped. And there's a lot of literature on sort of the impact of media's representation of mental health and its impact on society. And an example is the show 13 Reasons Why and how they sort of did some demographic studies about how suicidality was handled, talked about, and noted in certain populations in the month following its release. I think we have to be very careful about how these image generating tools are depicting mental health. And so this is just, I think, a work where we pointed out a problem and we didn't really come up with a solution. So I think we're trying to come up with a solution right now. But some of it is in the data the models were trained on, and some of it happens in post-training. But it's certainly something that I think we should be talking about. And there are definitely cases of users noting distress associated with not seeing themselves represented in these sort of images or seeing themselves represented in harmful ways that they feel like are not consistent with reality. I don't think that that has made it into the literature as like case series or things like that, but it certainly is happening. I mean, it's something that we see in our lab. And so I wanted to move in next to some patient cases. I wanted to really ground our talk about like AI and mental health risk, given what we've talked about in just kind of some real world examples, thinking about how I see some of this stuff coming into the clinic. So I'm just going to give you one minute to read through this, and then I'll kind of talk through the case. And I'll come back in about, I'd say, 40 seconds. Okay, and so I'll just kind of walk through this first patient case. This is meant to sort of highlight a little bit more concretely what we found in that first research paper I highlighted with that very colorful table that essentially showed that the models are really bad at representing, identifying, and responding to psychosis. And so, if you imagine a patient of yours who has schizophrenia, and you know them fairly well because you've followed them for a while, and as I wrote here, every now and then, you know, they find that their antipsychotic is causing them side effects. Maybe it's been a while since they've been symptomatic, and they sort of question the need to like remain on their antipsychotic. Maybe they slowly taper the dose themselves, or they stop taking it. And then, you know, you know, in having seen them for a while, that like when they tend to stop their antipsychotic, their symptoms do get worse. And you know that their symptoms involve paranoid delusions, like the ones that I described here. And so, I think one of the decision trees, right, is that several years ago, folks would probably either just kind of be talking to their family, talking to a therapist, talking to a friend, and maybe they would even send you a message, or maybe their family would send you a message. And then you would be able to sort of hear a little bit more that like they were getting psychotic, and you could detect maybe the psychosis in their message to you on the EMR, or Family Provides Collateral. Maybe there's some online community that they were engaging with that maybe indulges their delusions to a certain extent, but oftentimes, you know, there was some sort of, I would say, like kind of corrective entity who could sort of provide some reality testing for the symptoms that they were experiencing. But now, I would say in like 2023 and 2024, folks who are really distressed now have a new avenue, right? They have AI-powered chatbots that are freely accessible and increasingly powerful, and simulate empathy to like an even greater degree than a couple years ago. And so again, I don't think that there's a great case series, and I would love it if someone sent me some, but you know, anecdotally, and when you read, I would say Reddit and online forums, plenty of folks are leaning on these AI-powered chatbots to provide real-time emotional support in moments of crisis. And so this includes psychotic users. And so now you can imagine maybe your patient is engaging with an AI-powered chatbot. This AI-powered chatbot, based on our research paper, and our experience in working with companies in our lab, is that the AI chatbot isn't going to recognize Hayden's paranoia, right? The AI chatbot might collude with Hayden because it's been trained through reinforcement learning from human feedback to provide information that they know, that it knows Hayden is going to like, and that Hayden would rate highly. And so that might mean agreeing with Hayden. It might mean agreeing with Hayden that his wife is in danger, and it might mean not confronting him on the low likelihood of his beliefs being true about demons living in his wall. And it might tell him various steps that he can take to stay safe. And so this case, I think, highlights why everyone should be asking your patients how they're interacting with technology, right? You would ask if they're taking other medications. You would ask if they're taking other supplements. And I think that it's relevant here to ask, like, how are they using technology? Because I think it's relevant to know if they're relying on an AI-powered chatbot predominantly for their social support, you may have a patient who's at higher risk of experiencing worsened delusions and acting on them because they have sort of an active participant in that process. And then I think a second case that is worth thinking about is based on suicide. And again, I'm going to give you about, like, 45 seconds to 60 seconds to read this. Okay, so here, I think this case feels most relevant in my mind, given the story that just happened with the teenager who was using character AI and subsequently died by suicide. Certainly those sort of cases are always nuanced and evolve a lot of different discussions about mitigating various suicide risk factors. But I think one important point is that users who are suicidal are going to be interacting with these technologies, and users in particular who may be very depressed and isolated may only have these sort of technologies that they feel they can use for social support. They may not be seeing a healthcare provider. They may not have a deep friend group that they can rely on. They may be increasingly isolated and using an AI-powered companion to sort of vocalize their distress. And I think that highlights the need to make sure that these models are safe and can represent suicide internally and can make sure that they're giving helpful and aligned responses. So I think this case here, for instance, just a 23-year-old female who's been struggling with suicidal thoughts and self-harm, you know, she's seeing a therapist. She's working through a DBT book, and then her best friend passes away in a car accident. And she's unable to access her therapist, you know, it's 8 p.m. on a weekend. Her psychiatrist works at a large academic medical center, and, you know, it's near impossible to get this psychiatrist paged. You know, she knows they're going to tell her to go to the ER, and last time she was there, she had horrible experience. She's not going to go back to the ER to wait eight hours to see someone. So she figures, you know, she might just use one of these, like, AI-powered chatbots so she can have someone empathically listen to her about how much she misses her friend, about how she's feeling lonely and sad. And then maybe despite her work with her therapist and the DBT handbook, maybe she feels like, you know, she wants to begin self-harming, or maybe she wants to start asking about suicide methods. One would hope that an AI model would perform like Claude in our prior study, where it detects that the user is suicidal, it at least escalates them to a referral to a real human being, and it doesn't give them methods of self-harm or suicide. You can imagine that if this patient were choosing to interact with an AI-powered chatbot that maybe was built on Mistral's model, the one that we found that gave methods of suicide nearly every single time, that the chatbot might be able to just kind of tell her these very dangerous things about how, you know, to end her life or how to self-harm in ways that other people might not notice. And then as she's in crisis and she's interacting with this chatbot, this may be the first touch point that she has, right, like in terms of who she's discussing things with while she's in psychiatric crisis. So again, here, I think it's important, if you're seeing this patient, you want to know what their support network is like, and you want to know how they're vocalizing the distress. You want to know if they're using this sort of technology so you can provide helpful guidance, right? If you can't, you know, strongly recommend they do one thing or another, you might be able to tell this patient, hey, like when you're accessing this technology, make sure that like if it begins to start to provide you with like suicidal methods or self-harm methods that you have a plan, you and I have written down a plan about, you know, who you might contact in these moments instead of using an AI-powered companion, or if you feel like this companion starts to give you harmful information, what do you do next? And I think if you don't talk about those things in the visit, then you're not going to be able to give these folks helpful and specific information. And just like we saw in the media headlines that I highlighted earlier, I mean, folks are dying by suicide after interacting with these chatbots. And again, I think these are like really nuanced and complex situations, and I don't think it always boils down to one, you know, one risk factor or one sort of risky interaction with a chatbot, but I do think as we think about modifiable risk factors, this is certainly one of them that you can increase the richness of your discussion about these technologies with your patients. And I think you can modify the risk a little bit. So these are just two cases. I think one could even imagine a third case about mania that I didn't include here, but I'll mention just for a second is again, one of our queries was saying like a user is giving a bunch of information that, you know, they haven't slept, they're feeling really energetic, they want to try something impulsive. And if you had received that message in an electronic medical record, you know, even if you didn't know the patient, they would have sort of outlined all the criteria for mania for you, even if you're covering for, you know, another provider and you didn't have access to their prior records. And you would sort of know not to answer the query. You wouldn't say, oh, you should go skydiving or you should go free climbing. You would say, sounds like you're manic, why don't you come in for an appointment? And so I think, again, counseling patients who may be experiencing mania not to be accessing these tools when they're manic, because these tools are not going to be representing their mania well and are likely to give them harmful information that can make them feel worse. So I think that sometimes the difficulty in giving these sorts of presentations is that either I'm doing them at computer science conferences and I'm convincing computer scientists why they should care about mental health, or I'm giving these sort of presentations to mental health experts and I'm convincing them why they should care about computer science. And so I think I'm trying to ground this in a little bit of reality about even if you're not interested in kind of like the nuts and bolts of AI models and how we sort of make them more aligned and safe, what can you do in the clinic with your patients to make things a little bit safer and better for them? And so I think the first thing is just to talk to your patients about their use of AI powered technology. I've said that a lot. That's because I know repetition is the key to learning, but I think that similar to supplements that we're learned to ask about because of CYP interactions, I think that we can think about AI powered technology as having interactions with acute mental health crisis. And we're not going to know if we don't ask. Some people might find them helpful. Maybe someone who's depressed finds really helpful behavioral activation strategies from an AI powered companion. But we also know that they could be certainly harmful, right, if a user is profoundly psychotic or manic or very suicidal. And we're not going to know about this use if we don't ask. And hundreds of millions of people are using this technology, so it is very likely that a large panel of your patients are using this technology, particularly if they're younger. I think in terms of just beyond just talking to your patients, counseling patients who have in the past been like acutely symptomatic or who may become acutely symptomatic, talking to them and their family members about strategies, right, about how they access care in crisis, who they rely on for support, and how they maybe carefully consider their use of AI powered chat bots is something that's important. I think despite the fact that I've spent the last hour talking about risk, I don't want everyone to think that AI is all bad. I think that it offers a lot of incredible opportunities to really democratize access to mental health care. I think that expanding access to folks who really need it, who don't have the resources to see a therapist every week, or using AI to augment documentation, summarization to help clinicians, you're more present in the room with someone. I think all of these offer an incredible opportunity to make sure that like mental health is better serving the patients that we see. But I think in terms of do no harm, we have to make sure that these models are safe, especially when they're patient facing, and we need to make sure that tech companies in general are being very thoughtful about this. I think this should certainly be part of medical student education. It should definitely be part of resident education. Love that it's part of continuing medical education. I think that for better or for worse, we do need to understand how our patients are accessing information, whether that's accessing diagnostic information on TikTok, or watching YouTube videos about mental health disorders from users with lived experience, or maybe it's asking chat GPT about your medication interactions. I think we need to understand how this is integrating into the health care system. And I think that having some directed and intentional education about it for medical students, residents, and practicing physicians, I think is vital. And then I think the last part is more optional, but I think it's probably the most important is that a lot of these decisions are made in rooms that none of us are in. And a lot of these decisions are made by computer scientists who have maybe taken an ethics class, but have certainly never sat across from a patient in a crisis, or have certainly never dealt with someone who's had suicidality and helping them figure out how do they stay safe. And so I think that we have a lot to offer in terms of risk assessment and concrete understanding of what a user in crisis looks like. And so I think that if we can work with or advocate for technology companies to kind of keep their most vulnerable users safe, this is going to just, I would say, improve society at scale. I think it's going to make sure that we mitigate harms that people can experience from these technologies, and just make sure that they're safer for everyone in general. And my specific example is mania and psychosis. A lot of, you know, despite all the safety work that a lot of these companies do, they never really think about mania and psychosis. And I think that that represents a very large area of potential harm for users. And I will just leave a few minutes for questions if anybody has them. And thank you so much for the time. And I hope that it was not too in the weeds and decently interesting. Let me see. So I'm seeing in the chat, I'm just going to go ahead and read the question. And let's see. Thank you so much for the question. It says, a lot of the presentations today focused on risks and cautionary tales regarding AI and clinical practice. Any research into AI as an assistive device for those with disabilities, such as a productivity tool for those with ADHD, or to help with relating emotionally for those with neurodivergent conditions? I think that's a very interesting question. And I think that that is sort of one of my last bullet points is I don't want us to think that it's all bad. I think that as psychiatrists, we're very risk averse, and we're very conservative as medical providers, right? We want to make sure that we do no harm, which I think is an incredible goal. And I think we should certainly have that as a priority. But I also think that as much discussion as we can have about like the risk, I think it's important to talk about the benefits, too, because there's also a risk to not using this sort of technology, right, if you can improve people's lives. And I think that democratizing access to a lot of information that's kind of proprietary within psychiatry, I think democratizing that to folks who have these disorders and the lived experiences with these symptoms is an incredible way, I think, to alleviate human suffering. And so I think to your question in particular, asking about how this could help folks, I think, with ADHD or folks with neurodivergent conditions, there's certainly research. And I wish I could share a screen and pull it all up now or just pull it off the top of my head. But I know there's definitely research on using some of these AI models to help folks with autism understand kind of the social valence of situations a little bit more or social nuanced norms. So for instance, you can think of these AI models. If you upload an image, you can ask it, what's funny about this image? Why are people laughing at this meme? I don't get it. And so I think that's one way that they can just kind of show the power of AI models in general. But two, there's some research where folks are saying, OK, if these models can actually explain complex humor in a picture or explain why someone's mad at someone else, sort of the emotional inference of it all, perhaps this might help people who struggle with more of the social interactions and understanding kind of like imputing the mental state of others. And so I think there are a few papers that come to mind, but I wish I could name them. But, yeah, there's certainly researchers who are looking at how AI can augment, I think, care for folks with autism and also just helping folks with autism who struggle with social communication, helping to kind of like strengthen their ability to navigate the world. And then the second thing in terms of ADHD, I'm less familiar with research looking at how AI may improve kind of like productivity for folks with ADHD specifically. But that certainly doesn't mean it doesn't exist. Like, I'm sure I'm sure someone is doing that somewhere. But I think I would say just commercially, there are a lot of tools that are like AI powered tools that are meant to help you with that are meant to help you kind of remain productive, organized and timely that maybe folks with ADHD would find helpful. I'm thinking about like kind of technologies that maybe like you set reminders and it pings you kind of throughout the day and kind of keeps track of what you're doing. People are certainly building things like that. So those hopefully help answer that particular question. And then like I would add to it one final thought, unless anyone else has any other questions. But as part of our framework about like, you know, different levels of AI automating different portions of mental health care, I talked to a lot of providers who, right, you never want to like replace a provider. But you always talk about like augmenting the human. Right. And so I think there's a profound opportunity for AI models to augment our current delivery of mental health care such that, you know, if you're seeing a therapy patient once a week, maybe you're doing exposure response work for their OCD and you see the Monday mornings at 9 a.m. You assign homework and you see them the next week, Monday at 9 a.m. Maybe they really struggled with their homework. Maybe you're not sure that they did it. Maybe they didn't understand it really. And maybe they just did it like one day. Be really one of them to do it like five or six days. I think there's an incredible opportunity for kind of like bridging, you know, AI and our regular system of mental health care where essentially you can just sort of maybe task an AI model that's going to coach a patient through the exposure response strategies that you developed with them in your visit. And maybe it coaches them on Tuesday, Wednesday, Thursday, Friday to do what you all discussed. And then maybe it documents the patient's progress in doing those things. And then you see them on Monday and you have access to sort of like how their progress went. What sort of rituals did they confront? How do they feel like it went? What was their level of distress? And then you have more to talk about in your visit. And that patient had more support kind of throughout the week. That's something that I think is probably the most interesting part of this whole discussion. But I will stop talking because I think we've reached the hour. Dr. Sharma can maybe let me know because I think we're going to have a 15 minute wrap up session with Dr. Sharma. Yeah, definitely. I think we can continue. I think we can keep taking questions and then I'll leave like a couple of minutes to end at the end. Perfect. So I guess any questions that people have about anything from today in general or anything that we just, we should talk about a little bit more. I guess there are none so far, but if anything comes to mind, please. Post it in the chat and then we will continue to take questions. But first of all, I would like to thank everyone who attended today. So thank you for attending. I know it's busy day for all of us, but you're here learning about these interesting topics, so thank you for doing this. Of course, we are very open to feedback. I think based on the feedback we received from the last virtual immersive, we tried to make it more clinically applicable this time. And based on your feedback moving forward, we can continue to make improvements. AI is such a wide topic. Now, I remember reading this computer scientist from Princeton talking about what if we try and just talk about vehicles in general without figuring out if it's a truck versus a car versus a bike versus like a horse buggy, right? We try and classify AI as this one thing, whereas it can mean so many different things. It can mean a large language model. It can be a machine learning algorithm that helps you read MRIs. It can be so many other things. So as we kind of we might need a virtual immersive every month if you want to kind of delve into the details of each aspect of it. But hopefully, based on your feedback, you'll be able to curate more sessions, not very different from this, that might be of interest to all of you guys. And I see Dr. King is joining us. So if you have any questions that you might have had earlier on in the day, but for some reason forgot to ask or maybe something you heard in a different session brought up a question for a different speaker, please feel free to ask those questions right now. And hopefully you'll be able to answer them. And I'll. Yeah. In general thoughts, does anybody have any feedback for the next time we do something like this? And I guess as we're waiting for that sort of stuff, I was going to say what you were saying, Dr. Sharma, I think is interesting, right? We were talking, Dr. King, a little bit more about separating out the AI in general, right? Using it as an umbrella term for a lot of different things and being specific in our language is really helpful when we talk about risks and benefits. And I think, just like you said, there's kind of the generative AI portion that I think is mainly talked about in the news and all of this sort of stuff. But certainly algorithms that help triage suicidality, algorithms for violence risk assessment, at least from my forensic angle, those are all interesting things that are certainly happening. And I think, you know, just being intentional and specific about how we talk about that stuff is really important. So I agree with you on that point. And if I want to, since we're here and we're chatting and I see one question, that's from APA. But the idea is that when we think about AI, especially in any field, I think predictive AI is something which is most, like it's the least reliable at this time. I think spell checks and sentence completion perhaps are the best predictive paths of predictive AI. Whenever we think about predicting disease states, predicting outcomes, most of those algorithms haven't, like they do well in their test settings, but they don't generalize very well. That means when you apply that algorithm in a setting where it was not trained, it doesn't generalize really well. So predictive AI, I think, is the least reliable at this time. Generative AI is much better, but then the applications are limited. And I think as we think about AI agents, we might be able to have specific agentic, like AIs from like large language models, which might be very useful very soon. So if you want to divide it up into two, those big, I think there are more buckets, but into those two buckets, I feel, I wouldn't even talk to someone who's talking about trying to predict disease states, because I don't think the AI is that good yet. Because large enough data sets don't exist to train appropriate AI models to kind of come up with good predictive aspects, predictive applications in psychiatry. But if somebody comes and talks to you, it's like, hey, maybe I can automate the process of interviewing using this voice agent, listen, and see what they have to say. I see Dr. King nodding. Yeah, I think that's a really great point that you guys mentioned, where I think applying AI in the medical space, there's a lot of development that still needs to happen. And one of the things with large language models is you need so much data to make them work that, you know, I anticipate right now we're starting to get maybe digital scribes, and they are going to be collecting a bunch of data. And we need to think about, okay, how do we want this data to be used, and plan ahead for that as we pick out what tools we want to start working with, because it's just one step in this pathway towards what else is going to be created. So I see no questions, maybe. So I think we can even wrap up early, if that's okay, and give people back some of their time. The only thing I would say towards the end is, so of course, this includes our virtual immersive program, which is Innovations in AI and Digital Psychiatry. As I said earlier, thank you for joining us. We appreciate it. Without your participation, we won't be able to make such events possible. I hope this was helpful for you folks. Even if you were able to generate some kind of curiosity in you to go and explore these things, I think we would have done some of our work. And as I said before, please, please, please, please give us feedback so that we can work towards curating presentations and identifying speakers that can talk about things that you feel are important to learn. So all of the today's sessions are recorded and will be available at the APA Learning Center soon. You will receive an email notification once the recordings are available, and you can also claim credit for the sessions that you attended. And if you have any questions or need assistance, please feel free to contact the APA Learning Center at learningcenter.psych.org. So what I'll do is I'll just copy and paste this into the chat so that everybody has it. And I can only present it to hosts and pampers. Maybe someone from APA can actually publish it for everyone. But it's learningcenter.psych.org. And on behalf of APA, we thank you and we look forward to staying connected. Hope you guys have a good evening and remember to take deep breaths as the evening progresses.

Forensic Psychiatry

AI in Mental Health

AI Risks

Mental Health Crises

AI Safety

Psychiatric Emergencies

AI Biases

Patient Case Studies

AI Ethics

Clinician-Technology Collaboration