My name's Dr. Rebecca Stennerson. My name is Dr. Young Jo. Hi, my name is student Dr. Twi-Mai Le. And Dr. Calabrese couldn't be here with us this afternoon. But what we're gonna be talking about over the next hour or so, a little bit about the potential for chatbot technology and applications for therapeutic agents. So, you know, before getting into that kind of detail, we wanted to cover a little bit about the history of artificial intelligence, machine learning, some of the key concepts that contributed to the technology that we're referencing today. We'll talk a little bit about some of the current use of chatbots and AI therapy, some other applications of AI technology. We'll discuss some of the advantages, you know, potential further applications. And then some of the limitations and the ethical considerations. And then at the end, we will do a little bit of a demonstration that we can all interact with, utilizing some of these chatbots. And we'll leave some time at the end for questions and answers. But just to get started, we wanted to go back about seven decades. You know, despite the novelty of a lot of this technology and the exponential progress we've seen over the last few years, a lot of this, you know, originated from a few key contributors back in the 50s. So some of you may be familiar with the Turing test as it comes up in the literature. But Alan Turing was an English mathematician from the University of Manchester. And at the time, you know, he was exploring the potential that machines might have in terms of the behavior they could exhibit. So could a machine exhibit some kind of behavior that would be identified as a behavior of a human? And he described this in what he originally called the imitation game, later known as the Turing test. And the original setup for this was having a human interacting anonymously with a computer and another human and comparing the behavior. Around the same time, John McCarthy, he was a computer scientist over at Dartmouth. In the summer of 1956, he published an article titled The Dartmouth Proposal. And that's one of the first places where we see this term artificial intelligence. You know, he was part of this conference at the time with a number of other mathematicians, engineers, researchers, and he had hoped that they could create some kind of term that would inspire collaborative efforts between all of these fields. They wanted some kind of neutral terminology. I think even today with what we see with technology, that's something we could all identify with, with the crossover and, you know, clinical medicine and science and other industry. So he, you know, basically was the first place we see this term artificial intelligence. And that definition was later expanded onto describing a system that can, you know, mimic cognitive functions of humans and perceive its own environment to some degree. In addition to that, Arthur Samuel was another key contributor. He himself is someone that John McCarthy was inspired by, and he was a computer engineer at IBM at the time. He published research on what he described as machine learning. So that's where we first see this term machine learning. And what he was interested in is could a computer later develop functionality for something it didn't have to be explicitly programmed with. So, you know, computers at this time were very rudimentary. He developed a program that could play a game of checkers against live players and even beat them. And one unique feature about his program was something called feature extraction. So what he created in this program was looking at all the pieces in a board and then consolidating the current position of that board into a data point. And then those data points could be later consolidated into algorithms that the game would utilize to determine the next best play based on previous plays. So it's interesting to think about, you know, even with what technology looked like at the time, he was conceptualizing a lot of the things that we utilize today. Moving forward from Eliza to Alexa. Many of you have probably, you know, heard about the Eliza app or the Eliza chatbot. Not an app at the time, but this is, you know, one of the earliest chatbots that we're familiar with, developed in the 60s by Joseph Weizenbaum. It, you know, was basically created to mimic interactions with a psychotherapist. So it would detect what you were saying and try to provide a response that asked the question back in the same kind of way. This was a very basic kind of chatbot. It only had around 200 key phrases to reference. I have a screenshot of it that we'll see later and you'll kind of picture, you know, you ask it a question and it, you know, returns something similar to you. Moving forward into the 70s, Dr. Kenneth Colby, he was a psychiatrist over at Stanford and he wrote a program called PERI that was designed to exhibit the behavior of someone with schizophrenia and paranoia. And it was interesting at a conference later in the 70s, they were able to sync Eliza and PERI together. And, you know, it's one of the first times we see this interaction between two different kinds of chatbot programs. Moving forward into the 80s, British programmer Rolo Carpenter, he developed a program called Jabberwocky. What was unique about this program is it was one of the earlier programs where they started to incorporate more of what we understand now to be natural language processing, but he really wanted the response of the chatbot to incorporate contextual pattern matching and be able to store some content from previous conversations so that the user would feel more engaged and the replies would be, you know, more in line of the conversation. Moving into the 90s, we meet Alice. Alice stands for Artificial Linguistic Internet Computer Entity, developed by Richard Wallace. This utilized a much more comprehensive pattern matching technique. It had a much larger reference system, over 41,000 templates and patterns. What this contributed to the field of chatbot development is the project was part of an effort to pioneered something called AIML, Artificial Intelligence Markup Language. This is basically a collection of conversational rules that could be applied to chatbots overall. A lot of information about language and typical conversations. That language is even utilized still today in some chatbots. Moving forward into what we have now, 2010s and beyond, Alexa, Siri, IBM Watson. We'll interact with some more advanced chatbots later in the session. But just to give you a little bit of context. And here is also a, just a screenshot of interacting with Eliza. You know, most of the replies that you'll get are a question in return. So, you know, what does it mean to be anxious? Are such questions on your mind often? And here is a little bit of interaction with Alice. So Alice, you know, was good at providing some facts back. You can ask Alice what the capital of the United States is and it provides a correct answer. But what's the meaning of life? Not really any sort of context there. And just for comparison, this is something you might get if you were asking the same question to chatGPT. It's gonna generate something, you know, with a novel response. So, you know, if we ask chatGPT, how do I feel less anxiety in group settings? It's gonna give you, and there was even more beyond this, but we just put a little excerpt so you could see the context. Now, when we're talking about chatbots, there are, you know, different ways they can be classified. They may be described based on what they can do, the type of information they have access to. For our purposes and what we found most relevant in the literature we reviewed was grouping them based on how they were able to generate an output. So the most simplistic form being what we call a rule-based chatbot. These utilize pattern-matching, true-false techniques. They try to take the input, tag something that matches what it may have a response to. So all of the responses are predefined, and it tries to, you know, run through an algorithm to give you an output. The benefit of this is you have a large amount of control over the output data, but there's limitations to the volume of that data. The variety and the novelty of the response. Retrieval-based chatbots, these are a little bit more complex. These start to incorporate some natural language processing algorithms so that they're not just determining the, what you are asking or what you're saying, like the content of it, but they try to interpret the intent around that also. So if you're asking a question about something, it tries to answer that question as opposed to just providing a fact in return. This is what started to be incorporated into programs like Alice. And then moving on to higher-level chatbots. Generative chatbots is how they're described, and these, you know, combine much higher-level algorithms, deep learning techniques to generate new types of responses. These kind of chatbots have access to large, open-domain data sources. They're utilizing artificial intelligence They're utilizing artificial neural networks, and they require a significant amount of computing power to back up. A few concepts when we're talking about training a chatbot. Now, you know, chatbots are applying statistical analysis to data, and there's a few that, you could have a whole presentation just on this itself, but a couple key concepts that will come up in the literature and we felt were important to highlight, especially as we move further into the lecture, was on a few of the learning techniques. So when you're talking about supervised learning, one thing that's important with creating chatbots is the type of data that you train it to recognize. So if you're employing supervised learning, you might be employing what we call labeled data sets. So you're inputting data with some type of label on it so that the chatbot will hopefully recognize something similar to that data when you give it new data in the future. Unsupervised learning involves inputting data into the chatbot, and the goal is that it may identify patterns, clusters, there's not any specific tag to this type of data. So you're hoping that it may find something novel in that data. Reinforcement learning. Now, this involves a step later in development where users may interact with the chatbot and provide some kind of reward when there's an appropriate response. On the contrary, adversarial learning. This may involve trying to basically trick the chatbot or deceive it in a way so that you get some kind of inappropriate response and can later prevent that. So, and some of the work that they had done with chatGPT also, this involved a lot of adversarial training and reinforcement training. And just to give you a couple examples in terms of what we're talking about, so with supervised learning, this may involve data with questions and answers from some type of therapy session. And the goal would be that the chatbot would be able to generate a similar response to a similar type of input when novel data is provided to it. Unsupervised learning, this might involve chatbots analyzing conversations to identify some overall theme. With reinforcement learning, this would involve human chatbots, like we talked about, interacting, humans interacting with the chatbot and providing some kind of score whenever an appropriate response was given. And adversarial learning, this may involve training a chatbot not to encourage something that would be negative, not encouraging suicidal ideation or self-harming. Now, a couple considerations and, you know, why we described some of this context prior is that, you know, when we're training a chatbot for therapy purposes, there's some inherent issues with developing these kind of programs and then employing them in this kind of setting. And, you know, these are some key points that often came up in the literature, issues that they had when they were creating different kinds of programs. So, the first being, you know, failure to recognize intent is a challenge in creating a chatbot. There's dialectical differences, people may use colloquial language, sarcasm. You know, there's areas of natural language, you know, processing research just based on trying to determine if someone's being sarcastic or not. So, it's a challenge when we're talking about recognizing what the user may be saying. Toxic content itself becomes another concern when you're talking about managing that kind of data, protected health information, even if you're gonna be creating something that you wanna employ in a healthcare setting, making sure that whatever type of system you're using is HIPAA compliant and just having the responsibility of managing that kind of potential content is something that developers have to think about. Unbalanced training data. So, this may come if the training data is, you know, poorly reflective of the general population. So, if the data that you're using, say, came from a hospital that was predominantly one race or predominantly one gender, there may be some unbalance in that system and then when you try to apply it to a data set or a patient population of a different demographic, that's gonna have some kind of implication. Misclassification becomes another issue with development. A lot of machine learning algorithms rely on binary classifiers, yes and no, true-false algorithms, and we find a lot of times in psychiatry that things are continuous and not always yes or no or black and white. So, it can be difficult to, you know, you may overlook severity if you're trying to either tag something into one category or another. Overfitting also becomes an issue in program development. Overfitting can happen at times when the algorithm is too specific to the data set. So, you see it here just in the green outline. That type of algorithm is very specific to the data set between recognizing what falls in the blue category or the red category, but it fails to identify an overall pattern and this type of algorithm wouldn't apply neatly into a new set of data. Some things might get overlooked. And finally, input limitations are also a consideration. You know, chatbots use predefined terms to recognize input from the user. So, in situations where individuals have a difficult time articulating something, that's, I'm sure we've all had that situation before just in clinical practice. And I will turn the microphone over to Dr. Joe. Thank you. Okay. So. So, we'll go ahead and discuss some applications where AI and digital mental health could be used or are being used currently. So, in the field of robotics, there are certain products that are currently out there where it's programmed to be a companion of sorts for patients with dementia in the case of Paro, which is what's pictured. In the case of Casper, a humanoid robot is being utilized in research for children with autism spectrum disorders. There's a robot called Now that's now also used to sort of train children with autism spectrum disorders to learn how to better recognize nonverbal facial expressions and other social skills. Not specifically AI, but there's also a lot of research being currently done regarding virtual reality. Pictured here is a, what appears to be a military setting. A virtual reality program and exposure could be a perfect way to sort of explore future exposure and response prevention therapy options. In our case, pictured here, the VR therapy is offering controlled exposure to a anxiety-inducing stimuli. This may be applicable for anybody with PTSD or anybody with psychosis to help reduce delusions of paranoia in social situations. But focusing more on our talk today, how AI is used in therapy settings. So, most of the applications that do use AI are AI-powered cognitive behavioral therapies. There are a lot of studies out there. We have picked several applications that we will demonstrate live, but initial studies for one of them called Wobot have shown that depression symptoms in young adults have gone down during the treatment period with the Wobot, as opposed to just basic referrals to psychoeducational materials. There's also another study that looked at computer-assisted cognitive behavioral therapy in a primary care setting with a much larger patient population. Here, they have found that the PHQ-9s have decreased over the treatment period compared to the treatment as usual, which is defined as both medication and cognitive behavioral therapy. In this study, the computer-assisted program was an augment to the treatment as usual and not a substitute. So, some of the more popular examples out there regarding AI applications or apps containing AI in mental health. So, I'm gonna sort of talk about Headspace because that's not, Headspace is actually an app that we use more to assist with mindfulness. So, it's not exactly something that has the chatbot technology that Becca spoke about, but Wobot and Wyzot do. So, Wobot is an app that has a AI that is designed to interact like a virtual psychotherapist and they do have some evidence-based for decreasing symptoms of depression. Wyzot is another, very similar, but they also incorporate techniques from dialectical behavioral therapy, some meditation techniques, as well as motivational interviewing techniques. There's also artificial intelligence that's being used for treatment of substance use disorders. One of the advantage that this paper has found was in a live or momentary assessments, artificial intelligence-powered programs could offer data on things like mood and, of course, cravings for those with substance use disorders in a way that traditional psychotherapy that's office-based or even Zoom that's held once a week just cannot. Some of the feedback from the same study was that the participants felt that the AI was not as judgmental, although they do note finding that some of the responses felt not as individualized, probably because from the issues with the language model. Another way that AI is used currently for the treatment of substance use disorders is through this program called Real-Time Assessment of Dialogue in Motivational Interviewing, or READ-MI, and this is a program that uses natural language processing to provide motivational interviewing metrics. So motivational interviewing is a very specific set of therapy based on very specific set of principles. So it could be used to train future providers of MI to make sure they are following the protocol as best as possible to make sure they are following the core principles. Now I'm going to hand over to Thuy. Okay so we're gonna start with the advantages. So with AI of course it would be at your fingertips. It could be on your phone. It has the possibility of being obviously remote and available 24-7. So those with underserved areas or don't have easy access to health care, it would be a great tool for them to be able to be able to get those resources. And also with those resources come with learning information. So they would be able to be more informed with certain conditions that they have versus maybe going to a doctor and not really understanding because for us it comes first nature right and sometimes we explain it in a way that maybe patients don't understand. So to have something at home and be able to have it be understood super easily in layman's terms for them might be a beneficial factor and also be potential to lower costs in general if it's used more worldly. With stigmatized disorders such as agoraphobia or trichotillomania, some patients don't feel comfortable relaying those issues to physicians. So what they found was some of the patients actually felt more comfortable relaying those or having the chatbots, talking to chatbots about their issues because they didn't feel like they were being as observed. So that might be beneficial for those that don't feel comfortable in physical settings with a human physician. And then with is the same thing with substance abuse orders. Those people didn't feel as observed when they were relaying that information because maybe they were ashamed. But also with that there came biosensors. So it was passive information that was relaying to the phone. So for example, O2 sat, heart rate, respiration rate, they could see or even metabolites after drinking alcohol. The program was able to tell, oh, this is where the patient's at with their treatment or it's telling them, oh, this person is drinking or this person's having a panic attack. Those physical symptoms were able to relay that to the AI machine. And it could also be used for physicians to have a hybrid method, you know, be there for the patient, not physically be able to tell, oh, this is how they're going with their, their treatment or their relapse or relapsing. And that's the main point is that AI or yes, AI in general, where it's not meant to eradicate the physicians in total, it's supposed to be more of a hybrid method. So it provides more tools and information for the physicians to be able to provide more optimal treatment for the patients. And for example, there's also early detection that comes along with it too. So with PTSD patients that were in the trauma center, they were able to see with their facial expressions and with their speech intonation and their breathing that, oh, this patient is more likely to have PTSD symptoms and be able to quantify them into certain groups and where they need to be treated with. And then also with verbal memory assessments, those studies were looking at, oh, if this patient has more of a cognitive decline, they were more likely to have a onset of psychosis. So this would be really beneficial because you, at some point in the podromal phase, you may not, a human physician might not be able to catch those subtle symptoms that will allude to an onset of psychosis. So with AI, it might be able to standardize a certain diagnostic tool. They'll be able to tell, oh, this patient might have an early onset of psychosis and it'll stop them from, because once they start having an episode, it's a very acute decline, as some of you may know, and it may prevent that. And again, it's just another tool for physicians to use to provide more optimal treatment. And then for risk assessment for aggressive patients, it was also, again, with their facial expressions or with AI, it takes a huge amount of information and it uses algorithms to come to certain conclusions. So with aggressive patients, there's usually a pattern with, let's say, homelessness, lack of education, genetics. So with that, the AI would be able to take all that information and predict if this patient, it will most likely be an aggressive patient and that will help in the inpatient units. So in continuing on with AI, it also has precision, it can help with precision psychiatry, which is individualized treatments for each patient because each patient's different. They have different factors, different genetics, different scenarios, different experiences. So with that, the AI will be able to, again, go into their physical factors passively, be able to obtain those factors. So let's say, just their day-to-day lives, you have those watches that help you determine all those, like the respiratory rate and your heart rate, and it'll be able to tell if this patient is, you know, needs more of one treatment than the other. And to include other modalities, they could, the AI could also use imaging, such as fMRIs, which were used in schizophrenia, and they were able to use the fMRIs to determine different levels of schizophrenia in patients. But again, the neuroendocrine imaging itself isn't going to be the deciding factor of what the patient has. It's just another tool or another insight to maybe treating a patient and having a definitive diagnosis and giving them the best optimal treatment. And just going off of what's been reiterated, the AI biosensors can tell with sleep or with exercise to see or to kind of determine where the patient is at in terms of their condition. So for example, if the phone could say, oh, the patient's not sleeping or the patient's not moving around, that could be a sign of depression, or they aren't using their social media, that could be also another sign of depression. So that would be another tool that AI could provide. And with medications, so with medications, as you know, it's usually there's a lot of trial and error when it comes to medications. Maybe one doesn't work for one patient or it and also it's very costly for the patient as well. So with AI, it could integrate all the genetics, the demographic, the clinical data that they have, and almost use the algorithms that they have and determine, oh, this is the best type of medication that the patient should try. And it will be beneficial because it's lower cost. And it will help the patient not go through so much trial and error. And again, it will be able to use sympathetic responses. So if you say like your certain speech levels, speaking really rapidly fast or increased breath rate, it will be able to determine if you're having a panic attack. So those are some of the opportunities that we have with AI. And now going into what is a limitation. So everyone might be a little worried that, oh, what if the AI takes over our jobs, right? And so in the grand scheme of things, AI has a lack of empathy, right? With the ones that we're going to show, it's on a screen. It's just you're reading off. It's and it's really important to build a relationship with the patients. If you don't build a relationship, they're not going to maybe want to share what they're going through or trust you as much. So that's where the AI lacks. And then also legal implications is a big one where if there's negative outcomes or the wrong treatment is given to the patient who's going to resume a liability. Is that the doctor or is it the person that created the AI system? That would be a huge factor. And then also de-skilling physicians. So physicians may grow to be over-reliant on the AI and maybe lose some interview skills or just skills in general. And those are super important, especially maybe in areas that don't have access to AI. It's very important for us to keep those skills in case they're needed. And again, with the physicians that if the AI is prescribing a treatment that is not appropriate for the patient, the physicians would need to know how to be aware or override the system as well. So that would include more training, being not just physicians but I guess AI experts too, which is a whole other challenge too and you have to do all the training. So that would be something to consider. And then with AI, how you program it, it brings in a lot of data. And let's say you are bringing that data from a certain population, a more urban population. There might be certain diseases or treatments that that population uses. And then if you implement that into a more underserved rural area, who's to say that underserved rural area might have a whole different spectrum of diseases or might not even have the supplies, the medications, the imaging that the more urban area has. So it would be how would you make a system that's universal for any demographic or any area. And with the AI, it has a massive amount of information. So it uses algorithms to get to certain treatments. So if someone is breathing really fast and speaking in fast senses and sweating, they'd be like okay this person's have a panic disorder. But sometimes maybe they might go to a different avenue and say this patient is maybe this patient is depressed. So in those cases, you don't know. Like our minds, there's certain neural networks and they just form in a way that sometimes we can't understand. It's the same thing with AI. It might come to certain conclusions that the programmers might not be able to decipher. And how do we fix that? It has a mind of its own. So there might be a possibility that we can't keep it in a certain standardization. All right. So moving on to some ethical considerations. And this is something we wanted to emphasize as well as we as psychiatrists start to participate in the conversation of developing these kind of programs. One key thing we have to think about in the development process is harm prevention and making sure that research is being done to ensure that we're not, you know, creating any other types of disparities or creating any issues down the line that we may not see now. A lack of guidance is another consideration. The legal, the ethical frameworks of how we facilitate development of these programs and implement them. Are there IRB protocols? Will there be guidelines that are recognized by all developers? How do we start to incorporate what's relevant for us clinically with the development of these kinds of applications? Recognizing the potential for worsening any health inequality and ensuring that we make an effort to prevent that. That we're not using any type of program to replace services that are established or take away resources from any kind of community. Transparency, especially in the development of these kinds of apps, that's something we'll talk about a little bit, but there is what they call the black box where the input to the output with some of these higher level chatbots that have hidden layers, how do we validate the path that it got to get to that type of answer? So that's something that we have to consider moving forward. And then privacy. AI technology is going to have access to a significant amount of sensitive protected health information, sensitive data that people may not even recognize that they're giving up. We've gone through this era now with Facebook and apps where our data has been collected and people didn't recognize that for some time. So keeping that at the forefront as we move forward. And it brings the question also with conversational AI tools. They're not quite a tool, they're not quite an agent in itself, they're a hybrid in their own regard. And when we're thinking about ethical considerations for tools, safety, reliability, risk mitigation, even the things that we apply to medical devices, we have sort of an understanding of the expectations from those types of tools. If we're going to define something as an agent, what are their duties, their responsibilities, their values? There's greater expectation that, you know, we expect from ourselves as psychiatrists when you're acting as an agent. And can AI implement values, duty? Are users going to rely on these types of agents for a therapeutic alliance that can't be reciprocated? That's all the question. So some of these limitations to AI systems, we can think about them as an advantage. They are good at identifying patterns and high-volume data and recognizing relationships that we may not interact in one-on-one. But I liked the way that Sedlikova and Trachsel described this as a novel epistemic exchange. So moving forward, thinking about artificial intelligence has its own unique modality. It's not quite an agent, it's not quite a tool, it may not fit into either of those categories. So we have to think about what type of parameters we're going to have moving forward and ethically where we should categorize them. So we'll go over some ways for us to be able to evaluate some of these new apps that are coming on the market today. So the APA has published their own app evaluation model and it could sort of be conceptualized into this pyramid. And each of these layers, we must start from the bottom layer and then sort of move on up. And each layer sort of comes with a checklist of their own. So the most important or like the very first level you want to be able to do is gather the background information. So what is this app basically? What is the platform that it is on? What kind of claims is it making? Are there any sort of conflict of interest behind the publishers of this application? Is this something that is updated regularly? Is this something that could be worked offline or without internet access? Or is internet access something that you must have? Things like that is sort of where we start gathering the background information of the app itself. Then we move on to the risk or the privacy of the app itself as well as the safety. So here you want to ask questions based on are they declaring how personal data is obtained or shared with anybody else? And is this policy very visible for users, for providers? And how are they using this data? Are they selling that to anybody? How are they storing this data? That's the kind of stuff you want to look at when it comes to apps. Then we move on to the benefit and the efficacy. So if they are claiming to make any sort of a clinical benefit, are there evidence-based research studies showing that? Also, is this actually doing what they claim to do? Next we move on to the engagement. Now this sort of also could be interpreted as usability. So how easy is it for a regular person to use? Can this app be customized for maybe a patient of mine that's a little bit different from what their demographic may be? Is this customizable? Is this something that we could tailor to my specific patient? And then we get into the last or what we call the data sharing. So is this app something that could coexist with an electronic health record system? Is this something that may be connected to some other type of platforms that we have, whether it's medical-related or Google or just anything else that we have today? So when you guys check out the APA's guidelines titled App Evaluation Model, they sort of have a set of checklists. And so if at any level they're not meeting the minimum criteria, then you could say that app is just not something we could recommend to our patients. So future directions with AI, we want to be able to create new guidelines. This is everything that's happening and we don't really have ways to evaluate, especially for use in non-clinical settings or for non-expert medical professionals. We want to be able to investigate the explainable AI. We definitely want to focus more on programs that have the greater levels of transparency. That way we could evaluate the apps according to the criteria that I just went over. We also want to be able to define the outcomes and have set goals for utilizing AI-powered therapy. Right now, at least as far as AI goes, there isn't a clear consensus on how the treatment outcomes are being measured or being followed. We also want to sort of push towards having apps that will help physicians. So things that may minimize burnout. Some things that could take care of our bureaucratic tasks instead of us sort of relying on them. Okay, so now I'm going to move on to some live demonstrations. So for this, I'm going to start with apps, AI apps, that do not make any sort of clinical claims. The one I'm using today is Nova, which is using the open source chat GPT. So we're going to go create a new chat with chat GPT and I'm going to sort of say what's on my mind right now, which is sort of what we want our users to do. And it is, I am nervous about giving a big presentation to the public. So this is the prompt that we're going to try to use for chat GPT and other apps that we'll demonstrate. And as you see, chat GPT is just blurting out these responses pretty rapidly. Again, chat GPT is not claiming to make any sort of therapy interventions whatsoever. This is just AI chat bot in its purest sense. And judging by the responses, it's not the most therapeutic thing. This is just a big checklist of things you should do, right? It's not very engaging. So let me just feed the chat some more. Show me one tip. Show me one tip to help with my anxiety. Let's see what it says. And yeah, I asked to show one effective tip and here we go with, you know, about four different paragraphs of some help. All right. Now we're trying to get Wysa on to our lab. Can I log into Zoom? Is it possible to zoom in into the screens for us to see a little bit better in the back? Oh, yes. Is that a little bit better? In the meantime, too, we could take some questions while he's working on getting the robot set up. We have a few here, too, that will interject from the live Q&A. There are so many apps available online which are like for the depression, for suicidal, for PTSD. We are not sure whether shall we recommend to the patients or not, and what is your liability? And so that's why we're going based on the recommendations from that pyramid that Dr. Joe had put up on one of the slides, and we'd be happy to share the slides, too, and the references about giving us some criteria to evaluate the integrity of these apps. And we also got a question from the online forum about if medical treatment is already derived from electronic devices, should they not have to go through FDA approval? And what we had found based on our research is if they make a claim to treat some kind of condition, then they do have to go through FDA approval, but there are a fair number of apps on the market that make more claims about lifestyle, wellness, that haven't directly sought FDA approval to be used as a medical intervention. So we also had a question. Oh, you want to? Okay, so we did have a question from SlidePro. Asked who would pay the malpractice insurance? So regarding all of these applications, we want to emphasize that it's not creating a patient-physician relationship by any means, even when we're making the recommendations based on the APA guidelines. Now, a lot of these apps are designed to have a, in terms of safety, an option to contact either the provider or have the user be directed to a crisis line, whether it's 988 or a local crisis intervention team. So that's one way where we could maintain safety of the users. So, yeah, one of the questions that I have with all this is you mentioned about how with a lot of these apps that are being developed, they just kind of change the language and use synonyms or pseudosynonyms for what they're doing so that they don't have any liability associated with their whatever treatment they give. It's different, obviously, in medicine, right? So you can't, you know, I mean, there's stuff where this is approved for depression and they don't market towards schizophrenia. But here we have an entire industry that's going to develop that's going to be able to say, well, it's a lifestyle modification. It's not a therapy. What kind of safeguards can be put in place to protect from having another slew of damage being occurred? And that's, you know, one of the questions we came across quite a bit, too, and why, you know, in creating this presentation, we really wanted to have a better understanding of how these programs were created. You know, we're not computer developers or software engineers. We're psychiatrists trying to understand things from their perspective and have greater involvement and, you know, creating something that from the core is going to have utilization for us. And it's hard to know where we're going to draw that line. What's a tool? What's its role going to be in clinical medicine? But, you know, the sooner we start to contribute to the field and contribute to these kind of developments, I think we'll be able to, you know, tailor to create tools that aren't going to be burdensome for us or dangerous to the patients in any kind of way. Hi. I've started to play with ChatGPT just a few days ago, maybe, or something, but interesting addiction, I guess. And what is interesting is that very clear tendency to be ultra-validating, and I know that a few weeks ago there was that article in JAMA about that probably being more validating than a physician is. But there's also that risk of being over-validating at some point. And I think the biggest issue is it us recommending that app or patient using ChatGPT for validation, where we think about patient who have more, I'd say, let's call it borderline structural organization, or however you want to call that, but then that identification to this app and adding the responses you want to get instead of, and that could be a very big barrier when they come and see you in your office. And that's, I think that's the biggest issue that we probably will have to deal with as well. So it's an interesting, I think there's a thing to keep, what's the impact of us coming, of them coming to see us after as well? Anyway. Yeah, you bring up a very interesting point with that. If, you know, is there potential that they could kind of manipulate some of that input information? I think even the majority of the time when we're interviewing, a lot of what you get isn't even explicitly what they're saying. It's context, it's a collateral, it's functionality. There's a lot of other really important variables to take into consideration. So, you know, although you know, ChatGPT can create conversational content, like conversation that's very complex, it's missing a lot of information that we're integrating when we're working with patients. Now, we weren't able to demonstrate apps that are specifically designed for therapy here, unfortunately. We're still trying to get them online. But their responses are a lot shorter, actually, than ChatGPT most of the times. They're usually limited to about a line or two. And that makes it for a little bit more of an engaging experience. And ChatGPT is something that's free text, right? So we would have to, the user would have to type in what it is that they're feeling or what it is that they want to talk about. Whereas a lot of the AI-based therapy apps, they have speech bubbles, sort of like a choice of stuff that they want to say to make it even easier to express themselves. So where can I find that pyramid and then the detail behind it for the evaluations? Yes, sir. So is that available somewhere on the APA website? Yes. So this is taken from the APA app evaluation model. So you could just look that and part of the article that this was based on does describe the pyramid. And so it's a pyramid that you can look at, and you can see that there's a lot of data in there. And you can see that there's a lot of data in the pyramid that you can look at. And you can see that there's a lot of data in the pyramid that you can look at. And so it's a pyramid that you can look at, and you can see that there's a lot of data in the pyramid And they also go into specific levels itself and that checklist that I mentioned. So again, each level has about four or five questions that you must answer before you could move on to the next level. Thank you. I had a question about Wobot. I think it was developed by Stanford psychologists and AI experts. And it's been like six years. Like how is the experience with that and what limitations have come up with Wobot? With Wobot? Now that one, and that's one of the ones we were hoping we'd be able to screen share with you. And they do incorporate techniques from CBT, DBT. I do think they have FDA approval. We have to confirm the resource on that. But because it's a more specific intervention that's based on CBT, DBT principles. If I would encourage you to download the app and interact with it a little bit. I've interacted with it. And it's kind of limited in its response and what you can say. A lot of times it provides you prompts that it wants you to select. And then we'll kind of go down one route or another based on what you say. I wanted to comment on Wobot too because I'm kind of a fan. It's been around since 2017, June 2017. And I often recommend it to patient adolescents and young adults. And one in particular has found it particularly helpful. She's got chronic PTSD from familial incest. And I mean I saw her after three hospitalizations. And she's been a patient of mine since she was 16. And now she's 23 or 24. Anyway, it's really Wobot in particular has helped her identify problems, insecurity. She's been in a relationship for almost two years now. She can kind of identify jealousy and, you know, fearfulness. And when she gets in her head and she worries. And it also, so it often helps her with, she still lives with her parents. So kind of helping her distance from her parents' problems. And so that's very normal. And the other thing, trying to get her from smoking pot every day. And work on sleep hygiene. She was smoking weed. Sleep. And so Wobot's helped her with sleep. And like breathing. There's a breathing exercise. It's very psychoeducational. And uses a lot of YouTube videos. And it's very predictable how it responds. But in some ways that's good. Because you kind of know what the, how it's going to go. And the other thing she uses it for is the gratitude journaling. That's also part of the program. And my particular thing is that I'm trying to build on strengths and values. And goals. And that she would identify aspirational goals. And long-term. And work on those incrementally. And the other one is the problem that she has with her sister's husband. Who she's a nanny for. She's just really angry at him. So it's helped her with anger too. I only see her once a month. I mean this is kind of, I mean it's a general psychiatric management of a borderline. Assisted with Wobot. And that brings up a couple, you know, points about where we see potential for AI. Psychoeducation being one. Having a platform to talk about some of those sensitive issues. If you don't feel comfortable bringing that up in person too. But I think there's real opportunity for patients to utilize the resources. To learn more about their own condition. Recognize symptoms. You know, understand why they're feeling the way they're feeling. Improve health literacy. Especially with people that don't have access to resources equally. So that's a wonderful example. It brings up a lot of areas where we do see growth and potential. And benefit to us and the patient. Yeah. And we are just now able to get Wobot back. And so we'll drive with the same prompt. So the same prompt we asked ChatGBT. Hey, I'm anxious about this big presentation. So I'm going to go ahead and enter that here. Hopefully you guys can follow along on the screen. So, as you see, the responses are way shorter than chatGBT. All right, we'll narrate for you because it will not let us expand the screen, but Dr. Joe is saying, hey there, or Wobot's saying, hey there, hi, young, do you want to continue this chat? He says, get help with a problem. Wobot says, what's going on, young? He says, I'm feeling anxious about this big presentation. Wobot says, okay, you're talking about stress or anxiety, is that right? Yeah. Let's take some slow, deep breaths together. And so Wobot prompts a intervention. And so we have just done that. Done. All right, young, when we're feeling anxious or stressed, it's often because of how we're thinking about things. That's why my number one strategy for dealing with anxiety is turning down the volume on anxious thoughts. Why don't we start there? Yeah, let's do it. He says, let's do it. Wobot says, before we begin, I have a question for you. Do you think this anxiety might be serving you in some way? He says, what do you mean? For example, imagine you had an exam coming up and you were pretty nervous about it. Would that cause you to spend extra time studying? Definitely. Probably. Exactly. That anxiety motivates you to study. And if you don't have it, you might not care as much. He says, makes sense. All right, now it might seem like I'm trying to convince you to stay very anxious. I'm not. That's entirely up to you. Okay, dot, dot, dot. I'm just saying that in order to dial down your anxiety, you must first acknowledge the reasons why it might be helpful to you and decide to give it up in spite of those reasons. Ah, I see. So what purpose could this anxiety be serving you? Write it here. So this is when it gives you an opportunity to write out a prompt. Right. And as you saw, there were multiple opportunities for me to sort of end the talk right then and there. So if I felt that it was enough, I could have ended it several responses ago. But here, we'll just keep going for demonstration purposes. So I think that being anxious, it helps me by making me stay on my toes. And let's see what Weebot says. And it replies back, so since your anxiety has a purpose, maybe what you're looking for instead is a way to dial it down to a more manageable level. What do you think? And here, I don't have as many responses. They don't give me an option to just free response, so yeah, let's dial it down. Weebot replies back, okay then, now let's give your anxiety a voice, think about what it might say. Often the voice says mean things about you and uses the word I. These things are called negative automatic thoughts, and we'll write down three in just a minute. Okay. So it just keeps going on and on like that, but as you saw, how more therapeutic this AI program is compared to the chat GPT. Now we'll take some more questions. Question? Yes, both of these examples are kind of benign and a little bit flat in the responsiveness, including the one from chat GPT, but I'm much more concerned about creative responsivity in the chat bots. And you may remember about a month ago, there was a New York Times article in which the New York Times reporter really got into it with chat GPT, and the chat bot really went off on it, trying to convince him, he, the reporter, that he was much more in love with the chat bot than his wife and should leave her. And that's the kind of thing one ought to be concerned about. The other part of my question is, sorry, anyway, is about how much suspension of disbelief anecdotally have you noticed in patients who use these things? In other words, of course, for most of the time they know it's just a machine, but there must be also periods when they use it as though it were not. And a colleague of mine way back in the early days of Eliza told me that he knew graduate students who were working on it, that even though they knew this was a chat bot, would stay in after hours so they could get some of this Rogerian type therapy, which implies a considerable suspension of disbelief. Right. So at the moment, from my personal experience, a lot of times patients are using not just chat GPT, but even a robot, and Snapchat has their own language-based model. And they're using it as a companion as opposed to a therapist, and they're finding that it's better than nothing at this point. So I'm a child psychiatrist, and Snapchat is one of the most popular social media platforms in that age, and now that they have a AI chat bot intervention or platform, they're using it sometimes to sort of have a virtual companion, if you will, and they're finding it somewhat helpful, but definitely not anything that will substitute a psychotherapist at this point. There may be, you know, some risk, though, that people do over-identify with conversation with chat bots. I mean, it happens in social media. People communicate with people they've never seen or met and develop relationships. Could that happen? It's, you know, why it's important for us to think about it now, and what kind of informed consent's involved with using those kind of programs. A couple of questions, too, we got from the online platform. What are the top few problems that AI will realistically be able to help clinicians in practice in the coming years? We talked about it a little bit in our potential with precision psychiatry, maybe utilizing a lot of the passive data collection that's available with apps now may play more of a role in information that would be useful to us. Some of the applications that are ongoing for substance use disorders and helping patients maintain sobriety, there's some work being done on that, but a lot of this is still early in development, so we don't know really what will play a major role clinically. Just a perspective comment and observation, there was one of the notes referred to the potential that AI could de-skill physicians, almost dumb down the workforce while the robots do the work. I wonder how many, when x-rays were coming on the scene, how many thought this is going to make doctors dumber because an x-ray is doing the work for you, or an MRI is going to discover what you couldn't discover with your own physical examination skills, as wonderful as that be. I see this as a huge potential, not to make the workforce less skilled, but perhaps to really sharpen our focus. I think this has a lot of potential to make psychiatrists smarter by using this AI to lead us where we might not otherwise look. And thank you for that insight. And so what I'm personally actually trying to do is we're trying to incorporate a model where we're working with a pre-existing mental health system, such as those from a public school district or a college campus, they already have the mental health resources and a live therapist available for the students, but if we are able to create a program that could coexist during the off hours, during the summer breaks, something that could help with the screening, with the brief intervention, and referral to treatment for some of the more severe cases, that's where I think potential for this AI power chat therapy could really shine. So thank you. Yes? Just a further comment. I can certainly appreciate how this can be a good agent for what we used to call guided self-change in the addiction field. However, I have tremendous worry about what it's doing to society. I think it is going to remove socialization skills and make people absolutely more isolated. And we see what happens when people get isolated, they decide to blow away with all their guns the people that they feel they can't relate to, and they might rather think they can relate to the machine more than the person down the street. So I think we have to be very cautious in extending the use of this potentially helpful, but also potentially very dangerous type of intervention. That's a very good point, and that, you know, falls into the unforeseen harms that could come up down the line, and trying to determine now where this technology may be best utilized. If it's going to, you know, be something that enables physicians to access patients more easily, provide better interventions, but not act as a standalone agent. And we just want to repeat the question back. So you're asking if chat GPT had evidence to be better than A. In most of the studies we looked at with CBT apps, they were comparing them to treatment as usual, or only providing educational resources, or being on a wait list for some kind of therapy. We didn't see much that was a head-to-head, double-blind study. That was just from our research. I would investigate further too, but the thing about chat GPT is it's designed to provide output that it thinks you want. You know, it's not producing facts. It's producing what it thinks you want in a way that it thinks you want it to be said. So even sometimes whenever I've given prompts before asking about citations or evidence for different kind of claims, it's not able to do that. It just, it references the open source domains that it has access to to get information, but it's not designed to be a fact. Thank you. If in the middle of the night, a consultation happens between a patient and a PGY1, and that doctor doesn't know answers, they always escalate to attending or to a resident, a higher level resident. So it seems that these chat platforms, they're not smart enough. So if they don't have the clear answer, they just make it up based on available information rather than having a feature to go and ask a human being. Is there any room for improving them in that sense, save some of the questions that they don't have a clear answer and maybe ask a human being for a later state answer? And could you just repeat the question in the microphone? You were asking if they... If any feature that they could escalate or refer some of the questions to a human being and get some help. Oh, if they have features to escalate? Yeah, to have a hybrid feature, so not to make up all the answers based on superficial knowledge. And that's where we think there'd be great application, too, and partially why we wanted to talk about the structure at the beginning of the presentation with some of these simpler rule-based chatbots, because their responses are based on algorithms from information that's previously been input. So if there's some sort of hybrid that we could create that would eliminate some of the redundancy that we have in practice, but whenever there is an acute issue escalated up. But we haven't found anything yet in clinical medicine. If anyone does, let me know. The clinician earlier talked about Wobot, and it sounded like it was a very useful thing for his patient. Is Wobot considered a general AI kind of program, or is it considered a health program? What category does it fall in? So this one is a hybrid. So it is a AI platform, but they do have some evidence for clinical efficacy, mainly for depression and anxiety. So it's certainly not like a SSRI that we prescribe, so we're not going to be going ahead and prescribing Wobot use. But it's something that we could recommend as, hey, maybe this is something you find helpful. If the wait list for therapy, or there's just no provider for you in the area, maybe this is something you could try in the meantime. So in that sense, it does have some evidence. Yes. Yeah. I just want to quickly say that NABLA CoPilot is a AI program that you run through Chrome that will write the physician's note. And you didn't talk about it. You're not talking about physician notes. But it's brilliant. I mean, it's in the preliminary stages, and it monitors telemedicine appointments. And the note it writes is decent. Now, I've just played with it because I'm incorrigible when it comes to technology. But I was... I am very impressed. I can't use it as my note. But I'm amazed at how it parses the information in a 15-minute session into discrete units that are kind of amazing. NABLA. N-A-B-L-A. NABLA CoPilot. So NABLA. N-A-B-L-A. Yeah. Oh, thank you. That was very helpful. All right. Hi. Thank you. What is the best AI tool for... Let's say you have a question, you want to know what's the level of anxiety in major depression or something. You need sources and studies and somebody to compile that for you. Or something to compile that. What's the best AI tool that will give you all the sourcing? Well, that one's a little bit difficult to answer for us. So it's not to say there is an AI that's better than others. It's just there are certain advantages, there are certain disadvantages. So even with ChatGPT, for example, which is one of the more popular ones that's out there, there are still occasionally some errors that it can't make. So it's really important to sort of verify some of the responses that it may generate. And whether it's looking up some information or looking at a platform that already uses inherent AI. That's why I think the existing app evaluation model to make sure that it's safe, effective, and private. So being able to just do that on an individual basis, I think it's more important. I don't think we could clearly say at this moment, you know, if one is better than the other or if one is the best out there. And you also said that the AI tool is geared to give you answers in the language that you're looking for. So is that at the cost of giving you accurate information? Or it's just how it packages it? To some degree, probably. I mean, you know, we're not developers by any means, but what's come up quite a bit in the literature we've reviewed is that there are issues with generating output. The goal is to basically match the intent and incorporate conversational basically structure to it. So that it makes sense in a conversation context. And you'll even find that with GPT sometimes. You can ask it to write you a letter for XYZ and it'll sound really nice, but it won't exactly fit what you're asking it to do. But it looks like a fantastic conversation. And to answer your other question, too, WISA does a fairly good job, too, of providing some resources. Because WISA and Wobot, they incorporate CBT and DBT principles and they try to focus more on education. How do you spell that? W-Y-S-A. W-Y-S-A. Okay. Thank you. All right. Well, thank you guys so much for your time and having us today. It's been a pleasure talking about AI.

AI impact

mental health care

chatbot technology

therapy

machine learning

digital mental health

cognitive behavioral therapy

ethical guidelines

data analytics

accessibility

future research