So, welcome again. I hope most of the people who attended the last presentation are back here, but for anyone that's new, my name is Manu Sharma, I'm a psychiatrist and I'm the assistant medical director for research and innovation at the Institute of Living at Hartford, Connecticut, and I'm also assistant professor of psychiatry at Yale School of Medicine. So, in the last presentation, we mainly talked about how we can use speech and language measures as a biomarker that can help us with diagnostics, prognostics in terms of how they would respond to treatment and monitoring people over a period of time. In this presentation, we look at a different application of NLP and try and understand can we use phone-based apps and can AI help us creating chatbots or AI-powered apps that can provide one-on-one individual personalized therapy to individuals. So, when we talk about today, I think we'll talk about app evaluation models, what's the difference between chatbot and non-chatbot therapy apps, how do the apps work, like how they train these apps. We'll very briefly come up with some, talk about some of the studies and what they looked at and what they found, and then hopefully we'll spend enough time on the limitations of these apps as well. So, to begin with, I think digital apps are an excellent way of augmenting our clinical practice, right? We can, as I said in the last presentation as well, there's sometimes there's a lot of gap in between patient visits. There's not enough therapists to go around. I don't think there'll ever be enough mental health professionals to solve the mental health problems of the world. So, we have to find ways of digitally augmenting ourself, right? We can't, it'll be very difficult to create more therapists and more mental health workers and more psychiatrists. It will be easier to scale digital models, right? But we have to be extremely careful, right? Whenever we recommend digital apps or phone apps to our patients, I think we need to protect our patients. And I think we need to think about the kinds of apps that we're recommending, what those apps might do, are they effective or not? So, there are a couple of ways in which you can try and assess. So, the APA has done an excellent job of creating this tool for you guys. It's an APA app evaluation model. The link to it is actually at the bottom of the slide right here. But you can Google APA app evaluation model and you will hit this website. So, it gives you a set of questions that you can ask yourself as you evaluate an app to try and see if it is good enough to be used. And more importantly, is it good enough to be recommended to a patient, right? Now, some things to think about on which platform does it operate? Is it just iOS? Is it Android? Is it computer? When was it last updated? That is a very important thing, right? If it has been greater than six months since the app has been updated, that means there's something going on with the company. They might not be functional anymore, or they might have sold off to some other company and they might just be using your data for various things. So, it's very important to kind of look at when was the last app was updated. Of course, all of us, none of us actually need privacy policies, but are they transparent about the privacy policies? What do they talk? Does the privacy policy talk about how they share the data, the kind of data they can collect? For iPhone users, you can always kind of look at what they're tracking in the background. So, it's like it gives you a list of things that they track. Is it tracking your location? Is it recording your voice and all of those things? So, you have to kind of just review them before you start recommending folks to use apps. And as I said, there have been bad actors which have used the information they've collected in the guise of providing therapy for making extra money. And of course, that's why it's our responsibility to protect our patients and to kind of make recommendations that will be beneficial for them and nobody should be able to take advantage of them. So, I can go through all of this, but I think it'll be easier for you to actually just go on the website after this webinar ends today. If you've been recommending an app, just kind of look at that app and kind of go through this model and see what criteria it meets. And maybe you will find that you've been recommending the right app, or you might find that maybe I need to update my recommendations based on information you gather from this model. Another good resource to know about as we are thinking about apps is the OneMind Cyberguide. So, OneMind is a large mental health nonprofit. It funds a lot of digital startups. It funds researchers. They had a wonderful project where they were actually looking at apps and providing in-depth reviews so that if you go onto this website, unfortunately, they have stopped updating it. So, once you land on the website, you realize that they haven't updated it for some time now because I think, of course, with everything else, their funding issues, but the idea is they still have a good repository of the most commonly used mental health apps. And there are in-depth reviews and things that you can look into to help you decide which app might be useful in your practice or for your patients. So, these are two commonly used resources. I would recommend that everyone go back and actually take a look at these two websites. And if you're used to using apps or recommending apps, just running by the reviews and seeing what the challenges are or if there are privacy concerns or if there are some other concerns that you might pick up on based on these models or based on the reviews or evaluation that have been done by these folks for us. So, as I said, I think there are different kinds of apps that you can use for purposes of therapy or augmenting even your med management visits, right? So, oftentimes, we find ourselves in places where we are often, like, biggest complaint is we don't get enough time to spend with our patients, right? There's only so much you can do, and we have oftentimes very complex patients. So, there are lots of apps that you can recommend to patients that can help augment the therapeutic experience, right? So, there are non-AI-based apps, like, for example, Gamify CBT. So, there's an app called Happify, which it's basically education videos, and it walks you through negative thought patterns, what can be positive habits, some behavioral activation methods. It can actually track a few things for you. So, there's technically no AI involved, right? These are sets of things that you can do on the app, and they can customize your experience based on preference a little bit. But generally, it's like there is no chatbot involved. There's no personalization of therapy. There are modules which you can go through, and you might or might not benefit from it. But again, great for people who are not clinically severely ill, like suffering from situational anxiety, depression, or just want to improve themselves, right? A lot of us can benefit from DBT and CBT-based methods in our day-to-day life, right? So, apps like these are a good place for that. Then, of course, everybody's heard of these mindfulness apps, right? It's like they raised billions and billions of dollars during COVID, and they are big companies now. And they can help with other activities of daily living, like meditations, deep support, mindfulness. Again, if you have someone who's struggling with or has found mindfulness helpful, you can recommend these apps to them, and they can run through some modules that might help them develop a skill. Again, no AI involved in this. No chatbots involved in this. No personalization. Some personalization, but definitely some customization, but no personalization. So again, these are non-chatbot, non-AI apps. Then there are lots of apps out there that you can use for mood tracking and symptom tracking. Example is MoodFit. It's like, they call it fitness for your mood. So, for example, you can track your mood. You can kind of rate your mood every day. You can set up reminders. You can set goals for yourself in terms of how often you might do a certain habit, and then you might get some feedback off of that, right? And then some prompts, like your Apple Watch pinging you. It's like, hey, you've taken only 2,000 steps today. You generally average 6,000 steps. What's going on, right? Or if you might have a string of bad days, it might pick up on those things and give you some reminders and feedback. So apps like MoodFit. Again, these are representative apps. This is not an exhaustive list by any means. But then, so there are apps which can kind of use CBT or DBT-based techniques or mindfulness-based techniques. They can train you on them. There'll be predefined modules which you can access. You can kind of pick and choose what you want to do, and they might track your progress in a gamified way. And of course, anybody who works at the VA knows about PTSD Coach, CBT Coach. There's a bunch of self-assessment tools, educational resources, crisis resources. So these are apps which you can use. They might enhance your experience of treatment a little bit. But again, not a lot of artificial intelligence involved, not a lot of customization and personalization. So these are the non-AI, non-chatbot apps, right? Now, from there, we have advanced a little bit more, and we have chatbots. So I'm pretty sure you've heard the word chatbot several times today. There are several such apps which are already existing on the market, which you can actually download and try. It's a fun experience. Just try their free versions. It's very insightful sometimes, if I would recommend you guys doing it. Apps like Wobot, VISA, Uber, Quark are some of these apps. So what are chatbots? Chatbots basically simulate, right? The key word here being simulate human-like conversations and can provide some psychological support and assistance, right? So what are chatbots? Now, these are software applications, right? And they're designed to simulate human-like conversations. And they use AI methods like natural language processing and natural language understanding to create these narratives or create these dialogues between humans and machines that can replicate the interaction you might receive during therapy. And as we've talked about in the past, like someone asked a question around Alexa and Siri, yeah, these exist whenever. So for example, right? It's like if you're buying a new car, you go to a dealer's website, a message screen pops up and says, hey, I'm so-and-so, your virtual assistant. How can I help you with your car buying experience today? So traditionally, these chatbots have been used for answering questions, right? Some automated questions, providing customer support, like service recommendations. And user interaction. So again, mainly by commercial companies to help them make better profits or provide better customer experience. Now, we're using the same technology to kind of try and develop apps that can help us with therapy, right? So again, not all chatbots are created equal. Now, when I say that, there's levels of complexities involved. And as I said, right? So it's like from last presentation, there's rules-based analysis, right? So linguists go in and they kind of have some predefined rules. And based on those rules, you can kind of pick up some measures. Similarly, you can have predefined rules and predefined questions and then predefined answers. These are rules-based chatbots, as they're called. So what ends up happening is you program a set of instructions. So if they ask this, this is the answer. And you kind of type out that answer. And all the machine does is picks up on those keywords and then gives the right answer based on that. The problem, it's very simple. It's not advanced technology. It's easy to deploy, might be great for certain applications. But again, there are lots of limitations. So for an example, this is how a predefined rules-based chatbot may work, right? So the user asks, how can I reset my password, right? So now, reset my password would be a keyword that the program picks up on. Now, because it picks up on reset my password, it gives them a predefined answer. So the predefined answer with the keyword, with the question of the keyword, reset my password, is to reset your password, please click on forgot password link on the login page, right? So if you ask the question, can I reset my password? It will look at the keywords again, reset my password. And it will give you the same answer, which is not technically answering a question. But the predefined rules, the rules are such that if it picks up on certain keywords, it gives them those answers. And then again, the next keyword would be reset email. And it will ask you to kind of look at your spam folder. But then randomly, for some reason, as you're talking about passwords, you might decide it's like, hey, I might want to know about the weather as well. And if you might say, it's like, hey, what would be the weather like today? And it kind of, again, this is a predefined answer. So if it cannot pick up on any of the predefined keywords, keywords that might exist in its library, it will have customized answer. It will say, I'm sorry, I can only assist with account-related queries, right? So there's never any room for providing wrong answers. It might give you non-relevant answers. Or it might say, oh, I'm not capable of doing this. And this is the frustration we experience with Siri and all of these things on an everyday basis, right? It's like, for example, imagine me with my accent trying to ask Siri to play a not popular Bollywood song, right? That's a challenge. I can't do it while I'm driving. Because again, that's a frustrating experience. Because again, most of what Siri does is rules-based. It picks up on keywords and gives you answers based on those keywords. It has a set of defined functions that take place in the background once it picks up on those keywords, right? So again, Siri will never kind of, if I ask it for directions, if I ask Google for directions and give me directions, it'll never send me to the wrong place. So it might, but again, the chances of errors are less. The answers are predictable. But it doesn't do well with complex sentences and nuance, right? So that's the problem. So then there are chatbots, which are based on natural language processing and deep learning, right? So they rely on NLP methods to try and fully understand not only the keywords, but the context in which those words are spoken, and then generate nuanced answers. And this kind of simulates human conversations in a much better way as compared to a rules-based format, which might seem quite artificial. And you will pick up very quickly that there's an algorithm on the other side and not like an actual human, if you're especially interacting in a text-based way. So how would that change? So it's like, if you say, okay, fine, how do I change my password? So you can change your password by visiting the settings page and selecting, would you like me to direct you to the link? Then you say, yes, please. It picks up on the fact that you asked it not only to how to change your password, you're asking the algorithm to provide you with a link. And by the way, while you're doing that, can you tell me the weather as well? So the chatbot understands differences between asking for a password versus asking for the weather forecast. I'll tell you, it'll ask you, okay, fine, which city do you want, which city's weather do you want? You say New York, it'll give you the link and also tell you what the weather in New York will be. So it understands nuance. It understands context. It is not limited by predefined rules. And that's what makes it more human-like, right? So how do we train a natural language processing algorithm to do therapy, right? So there are various steps that are involved in this, right? To start with, you have to define the therapy content and framework for the app. Now, what does that mean? So if you want to train a chatbot to do CBT specifically for depression, what you would do is you would reach out to a CBT expert and read CBT textbooks, right? And talk about what are some of the major methods or techniques that you might use to help you become a good therapist or provide CBT for depression. And then what you would do is you might think about a define, you might do cognitive restructuring, behavioral activation, and then you would give it examples, right? And then you will design a conversation. So you create an example for training this chatbot. So you feed the chatbot and say, hey, this is what cognitive restructuring means, right? So basically, you're training it to understand the framework of CBT for depression. So for example, anybody who does clinical work knows that this is an example. If the example that follows, if life was so easy, I would be like 50 pounds thinner and not, I won't have as many gray hairs as I have. But the idea is to kind of provide a framework. So what would cognitive restructuring framework look like? You would create an example. So user says, I always mess things up. I'm such a failure. So the chatbot, now these are, so a therapist has sat down and said, the user will say this, the chatbot should respond by this. Then the user says this and the chatbot responds to this. So basically, it goes from talking about how the patient, the person who's approaching the chatbot says that they're a failure. The chatbot asks, why do you feel this way? It talks about what happened at work. And then it tries to help the person convert the feelings of failure to a feeling of opportunity. You'll do some cognitive restructuring and voila, right? The patient feels better already and the therapy session is over. But so they would say, the therapist would say that, hey, this is what cognitive restructuring looks like. Here's one example. And it might not be just one example. So this is just for illustration purposes. And jokes aside, the clinician and the therapist might actually, or the expert person who's programming this might use hundreds and hundreds and hundreds of examples and label these examples with the technique. So this might be cognitive restructuring. This is behavioral activation. And within these, this might be catastrophic thinking. This might be generalization and other things. So it labels all these aspects which give the chatbot a framework of what is involved in a good therapy session. Next, what they do is, as most of these companies do, they collect anonymized therapy sessions. So for example, the company out of UK, which was the provider for largest text-based therapy for the NHS, they kind of set, developed a program that's similar like this, which they kind of provided some basic concepts for CBT for depression. And then what they did was, OK, fine, here's all this extra data where the patients are receiving CBT treatment for depression. Go look at patterns. So they let this algorithm lose on millions and millions of words and hundreds and thousands of hours of therapy data, which is collected in an anonymized way. And this time, what the difference is, they're not labeling anything. They're not saying this is good therapy, this is bad therapy. They're not saying this is cognitive restructuring. They're not saying anything. They're just saying, we gave you this framework. Go pick up these patterns in all this existing data. So what the algorithm does is it goes and it trains itself and picks up on patterns. And they're very good at it. And they might develop, OK, fine, this works, this might not work. If someone says this, this might have been a good response. Just like chat-GPT works, based on the context of prediction at work. And it's so good that it kind of almost resembles human language. But chat-GPT is very general. It's kind of trained on a lot of general data. These chatbots are generally trained on, again, you start with the framework, which is provided by the therapist, and then go on and train this data on large volumes of actually CBT-based treatments. Now, what happens next is you kind of train the model to understand as well. So it's not only giving answers, but also trying to understand what the person is saying, right? So you might start early testing and say, hey, they might hire people or college students to just sit and have interactions and kind of give pre-decided-on questions to the app, right? So the question is, you might give feedback to the developers, like, hey, this feels artificial. Or maybe it's giving wrong responses. Maybe it's giving non-therapeutic responses. Maybe it's giving bad advice, right? So all of that gets taken into account. And when that happens, you end up making adjustments to the algorithm. And then you give it feedback and say, it's like, hey, this response is good. This response is bad. And you keep tagging it, right? It's not very different from, I don't know if you guys know the example of how people are trained. So when ducklings hatch, it's very difficult to differentiate between if it's a male or female. So what ends up happening is that the person who's getting trained is sitting there. And the person who's already trained and is a master of figuring out if the ducks are males or females after they're hatching just stands back. And this person is sitting here and guessing if it's a male or female. And you get feedback from behind saying if they're correct or not. And eventually, that's how the brain works. It's picking up on patterns, right? You become better and better and better at it until you become the master and you train the next one. These algorithms work similarly, right? Based on the example, based on the feedback, they keep learning from every experience. And every time you go and say, hey, this was good, this was bad, it adjusts itself, right? It kind of takes that feedback and learns to become better and better and better, right? So for example, it might start understanding. And it might kind of pick up on nuances, not only about depression, but some feelings of inadequacy and other things in the example that's on the slide, right? Now, so again, as I said, understanding feedback loop, we covered this in the last slide. So let's talk about some of these examples, right? All these companies that have actually gone out and deployed these apps in the real world, right? So for example, this was a study that was done by Wobot in which they picked up 34. So again, these were college age students and they self-identified as struggling with issues with depression and anxiety. Now, 34 of them were randomized to interacting with Wobot for two weeks. And then 36 of the participants were given an information booklet, right? This was an NIMH booklet about depression and college age individuals, right? So they were expected to read that in a period of two weeks. Of course, it's not like a very greatly designed study, but then intervention lasted for two weeks. And what they found was 83% of the people completed trial. So it's like, it was acceptable. So people stuck with the app for the duration of the study. And then they did see some changes in symptoms. At least these were self-reported. But you have to keep in mind that the reductions in depression symptoms or anxiety symptoms as they measure using PHQ-9s, GAT-7s, and PANAs, which is a negative affix scale and a positive affix scale. These were people who self-identified as having depression and anxiety. These were not diagnosis based on BSM criteria. We don't know how severe these symptoms were to begin with. And of course, it had a very small sample size. But a good takeaway is, yes, like if you have a person who might be experiencing some situation-based depression or anxiety, they might use this app and in two weeks feel better, right? So that's the takeaway from this study. So then Wobot went ahead and did something similar. So again, so what Wobot is doing is it's using different chatbots for very specific applications. So this one was for depression. They created a similar version for substance use disorder. So in this, basically what they did was they had close to like 100 individuals who were like between 18 and 65 who had a high score on KH-8. And they kind of started using these apps. Now, what they did was after the period of engagement, they realized that they, all of them kind of self-reported where they reduced, expressed reduced cravings and they were more confident and they, like they could resist use. And they also showed improvements in their depression and anxiety symptoms. Of course, there was no significant change observed in pain scores in these patients. And again, all of them said, yes, I would use this app. This app was good. And one interesting finding was the more a person used the app, the better the outcomes were. So if the person downloaded the app and engaged with it, say 15 to 20 minutes a day, their response was much lower as compared to someone who might use it, say an hour a day, religiously every day and work on the things and kind of engage in the program itself. Which you see, which makes sense because you see that in therapy as well and engage patients, then you will have better outcomes. But again, another acceptability study, you have to keep remembering, these are all self-reported measures. They also did a similar study in patients with postpartum depression. And I think this was a larger and a better designed study also. And they found improvements in people who were self-reporting having struggles with depression in the postpartum period. Another study was done by a group of, again, another app, which is known as VISA. And this, what they did was, again, so if you look at the pattern, they're not going after the core clinical population that we might see as psychiatrists. They're going towards catering to populations which might not have clinically significant depression and anxiety symptoms and might present in non-psychiatry offices. They might go to their primary care physician. Or they might be experiencing some depression, anxiety secondary to the neurological condition. So that is the target population, or college age students who might be struggling with depression and anxiety, which are milder forms, might be situational and might not reach a level of clinical significance. So that's the population that they're targeting. I don't think they will, I haven't seen any work done in co-clinical population yet. So in this study, what they did was, what they did was, they kind of recruited patients who were attending a musculoskeletal clinic. They might have had a variety of musculoskeletal disorders or immune disorders, and they were reporting symptoms of depression and anxiety. And for two months, they were given access to, again, an automated chatbot, which this company has. And they showed significant improvements in anxiety. But not a lot of changes in depression symptoms. And again, the same pattern. The more people use their app, the more people engage with the chatbot, the better the outcomes was. And not surprisingly, their physical symptoms were also improved because their depression and anxiety was better. Again, so another application in a non-core psychiatric setting that shows promise. Of course, small sample sizes, self-reported measures, and not a lot of robustness in the way they were measuring symptoms. But again, at the end of the day, if you think about it, how the patient feels and how they report feeling is a very important aspect of therapy and treatment. And these apps are doing well in that phase. Of course, these are not very clinically, I'm hoping these are not very clinically severe patients who are just getting app as therapy, because I don't think there's a space for just clinically using it in patients with severe depression, not even moderate depression. Then another such company is Youpur. So you can download these apps today on your phones and ask your patients to download it as well. So Youpur actually has been downloaded millions of times, apparently, based on their website. So what they did was some of the participants agreed to be part of this research study as well. So they looked at the patterns of using Youpurs and the app itself and how much they were engaging with the chatbots. And they basically followed them over a period of time through standardized testing. And they basically talked about how the depression and anxiety symptoms improved as well. So the same story where you're seeing self-report of improvement in symptoms, right? But you have to keep in mind, when we look at these studies, none of these, I keep saying that none of these are clinical populations. Most of them have self-identified with having depression and anxiety symptoms, which would be the case in clinical population as well, but a clinician with their clinical judgment and acumen might be able to determine the severity of the symptoms and might be able to kind of talk about, hey, maybe this is just an acute stress reaction or an adjustment disorder versus like a true major depressive disorder, right? So that level of nuance was not there when they were recruiting for these studies. And again, the measures of improvement were self-reported as well. And these were contained short-term studies, right? So we don't know what happened with long-term use of these, like long-term use of these apps. Did the symptom improvement persist? Were there exacerbations? What happened during those exacerbations? And all of those questions still remain unanswered. So the promise is there, right? But unfortunately, it's even before infancy. I would say this is, in its, like, it's even before infancy because we do not have enough data to confidently use this in a clinical population. Whereas if you have cases where you're on college campuses and you do not have a lot of resources and you have, like, anxious people who might be situationally anxious and having adjustment disorder, maybe that's where you can still think about using it in a very controlled research setting, right? So the promise is there. It's very early on. And most of it is around acceptability, right? So people are okay with using these text-based therapy apps to receive their therapy support. And to some extent, it provides a lot of ease, right? You don't have to have pre-set appointments, right? You don't have to have... You can kind of text with them at 3 a.m. in the morning or 2 o'clock in the afternoon, on the weekends, whenever. So you're not limited by the presence of an actual therapist on the other hand and their availability. You can use the app wherever. And algorithm set appropriately, it's scalable. You can cater to as many people as you want. It's like, it all depends on how big your servers are and how many, like, what your server limit is, right? But technically, you can cater to the entire population with the same app, of course, once you have a good enough app for that. But as I said, like, there'll never be enough therapists. There'll never be enough psychiatrists. There'll never be enough people providing mental health care. So these might be opportunities to help deal with patients in a proactive manner where the symptoms might not be very severe and preventing them from developing more severe symptoms and needing access to psychiatric services, right? So there's a lot of promise there. So the promise of scalability, the promise around accessibility, the promise around ease of use, the promise of, like, privacy, right? Like, I don't have to drive to a psychiatric office, of course, to tell a psychiatry that's become easier, but nobody has to know that I'm in therapy, right? I think that's a big barrier and stigma associated. Still, unfortunately, the patient receiving psychiatric services. So there's promise there, but there's lots of perils as well, right? So there's lack of privacy, right? Again, without naming names, there have been instances where companies who have used therapy data and have sold some of the data for profits, right? And of course, when you think about these companies, right, all of these companies are venture capital-backed for-profit companies, right? So at the end of the day, not saying that that's a bad thing, but they have certain financial pressures. And when push comes to shove, and if you're not careful, there's a lot of risk of privacy and data security, because again, when you're doing therapy with an app, you're revealing your deepest, darkest thoughts to the app, and that gets stored in some server, and that can be used against you, that can be used for a variety of things. Now, when you think about, like when I went back and kind of absence of human empathy, right? So for example, when I talked about the training part, right, where you're using these examples to train folks, right, where I did this one. So using these examples to train the app. Now, this might be, the app might become excellent, right, in picking up these patterns and simulating human-like conversations, right, conversations like this, but what it lacks is human empathy, right? When, through med school, through psychiatry training, through training, like for people who are therapists, going to LTS, some of these social work schools, family practice schools, or psychology schools, right, that's not the limited training you have. You also have the experience of being a human being that has existed on this world, in this world, for the number of years you've been alive, and that experience counts for something, that helps you understand the human condition, that helps you understand human insight, that gives you insight into certain behaviors. It will be extremely difficult to train an AI app to do that, I'm sorry. Of course, with large enough samples and enough experience and modeling across millions and millions of hours of therapeutic interactions, it might get good, but I, at this time, with the way the technology is, maybe it's a failure of my imagination, but I feel that it might never reach the level of human empathy, especially in a therapeutic setting. But again, we never thought something like CHAT-GPT would exist, and it does exist today, so who knows? And in fact, there have been studies which have shown that people who prefer, like GPT models trained in therapeutic techniques, some people actually prefer those answers to human answers, right? So again, we might be proven wrong, but it'll take a long time for that to happen. And as I had mentioned before, it's limited clinical scope, right? So it's important to kind of think about, like, where can we deploy such apps, right? Certain applications at this time are people on wait lists, right? People in primary care clinics who might not have access to therapy. Oftentimes, the comparison we make for these apps is an unfair one, because we compare these apps to actual therapists, right? Are they as good as an actual therapist? Where there is millions and millions of people out there who have zero access to therapy, who either don't have the right insurance, or the therapist appointment is two years down the line, and what about those people, right? Is something better than nothing? I think that's the question we need to answer. Are these apps better than no interventions at all? And I think that's the gap which these apps will eventually fill. So these will be apps that might be used in primary care clinics, people suffering from chronic conditions, might be having an adjustment disorder, or might be dealing with difficult life situations, or to some extent, people having actual clinical depression or clinical anxiety, and then are just on a wait list, and they're just waiting to see a therapist because there's no appointments available. So maybe this, although the scope is limited, but this is the gap it will fill. And of course, we talked about privacy and data issues. Regulating ethical conditions, right? So we have to think about how regulations might need to change to allow something like this to happen, right? Because when we think about therapists or psychiatrists or others, not only we do have rigorous training, we have licensing requirements, we have CME requirements. I'm guessing that's why a lot of people are here, it's like because they're getting the CME, CEU credits as well, right? So there's an expectation for ongoing therapy and upgrade, there's supervision and there's improvement of skills, there are, if you don't act in the right way or if you're making mistakes, there are boards that can take action against you, right? So we have to come up with a similar regulatory mechanism for artificial intelligence and apps as well, right? Because in the absence of that, we might end up in a situation where a lot of companies will end up taking advantage of people seeking help, right? And that might not be fair for the people. So we have to definitely think about regulatory and ethical considerations as we think about these apps, right? Now, obviously there's a risk of misdiagnosis, right? Which is always present and misinterpretation of symptoms. So for example, I'm depressed, but I'm typing into the app saying that, hey, I've had a great day, right? Let's start the therapy session. Of course, if this was in person and I'm a human therapist, I might be able to pick up on other things and say like, hey, what are you saying and how you're presenting might not make sense, but the app might not have that ability. And a big thing has been engagement and retention challenges, right? So for example, people get bored, right? How many times have people downloaded apps, gotten excited about them and then stopped using it in two weeks, right? Happens to me a lot, generally around 1st of January, right? When you kind of download an app and say, it's like, I'm going to have a very productive year and I'm going to develop this new productivity system and develop a second brain where using Notes and Evernote and all of these apps, three weeks in, sorry, back to paper and pen. So the idea is there's a challenge with engagement and retention. There's a certain level of fatigue that people will start experiencing once they start using this app. And then of course, the most important thing, there's no real-time crisis intervention, right? Of course, you can program these things in where if a patient kind of starts talking about suicide or self-harm, you might automatically end up calling 911 or local mobile crisis services. But again, the app in itself will not be able to handle crisis. Or like you can't give a tablet to a patient presenting to an emergency room, like, hey, yo, talk to a chat board. I'll come back in an hour, right? So that's not possible again. And as I mentioned earlier, there's absence of nonverbal cues, right? So the way we rely on a lot more than communication cues, we think about body language, like not only the patient's body language, our body language, our nonverbal cues, they're such an important part in the therapeutic process that it's difficult. So then there was some suggestions. So they actually looked at how can we kind of account for some of these barriers, right? So the suggestions that people give was there's repetitive content and limited free versions. So you might actually might have to have a larger content and larger features in the free version of these apps, right? So that was one of the biggest concerns that patients had, because when they used the free version of the app, they got kind of bored by it. And for anything else, they had to pay for it. And then they might need to kind of update their algorithms more frequently so that the personalization happens, the modalities change, and hopefully the way they're interacting with individuals change, right? So they have to think about long-term engagement strategies, refreshing content, adapting to user interactions over a period of time. And this might not happen only at a group level where they're making certain changes to the app itself. They might have to change strategies and they might have to reach a level of nuance where they're changing strategies on an individual level based on the feedback it is getting. So maybe you might have an app trained on CBT methods, but CBT is not working for this patient. They might need something else, right? The app needs to be able to understand when something is not working, especially if it's getting the same kind of response over and over again. And that's the level of nuance it needs to reach. And that's what patients asked for in this paper. So in conclusion, accessibility is important. I think I keep saying this, we are in a crisis of sorts. The house is burning right now. And I don't think we'll ever have enough firefighters to fight the house fire, right? It's like, I'm quoting passages out of a book I read about the mental health crisis. And I think there'll never be enough of us. So how do we get people overcome that gap? And I think technology is an answer. Chatbot's an answer, right? We can engage with these patients at any time, day or night, weekends, holidays, and that'll be very effective, right? And we have seen some initial, very basic evidence that they might work, but they're definitely acceptable. We know for sure that they're acceptable. People are using it. Like hundreds of thousands of people have downloaded these apps. So there's something there, right? But we don't know what makes it work, right? Is it just the fact that I feel that I am engaging with someone? Is it true therapeutic interaction? Or is it just placebo effect, right? Of me feeling better because I am just, at the end of the day, I'm just unloading all my emotional baggage to this app. And this app keeps giving me, has unlimited patients and keeps giving me nice kind words and therapeutic responses no matter how much I talk to them, right? So we don't know how these apps work, right? What part of, like, is it specific things within the app? Is it just the engagement? Is it placebo effect? We don't know that. We have to be careful about who has access to these apps, right? It's important. It might, in fact, to some extent, on one hand, it might increase accessibility, but on the other hand, it might make it tougher, right? Like not everybody has access to smartphones. Not everybody is comfortable using smartphones. Not everybody is willing to engage in therapy for smartphones. So if you over-rely on these apps and just use those to kind of try and fill the gaps, you might be missing a very important part of the population, which has been overlooked for a very long period of time as well. And of course, there's nothing, like, I don't know how, how do we solve the problem of personalization and empathy? I, again, the answer I always keep hearing is keep training the model over and over again for a long period of time. But I'm not quite, I think there'll be a limit to how much personalization and empathy an AI algorithm might be able to develop. But I'm, I can remain optimistic. I'm skeptical at this time, but let's, I'm optimistic, skeptically optimistic that can happen. And of course, complex conditions, right? I work in a schizophrenic clinic, right? And I know that they love using their phones. They're okay with phone based interventions, but I've never seen a single company that wants to target the serious and persistent mental illness population, unfortunately. I don't, I won't go into the reasons why I have a hypothesis, but complex conditions, imagine a person with complex personality traits, in addition to depression, complex trauma history. At this time, there's no app to kind of talk about that. And of course, you have to think about data privacy integration and how do we integrate these things into our regular care? And of course we need lots and lots and lots of research. So in inclusion, promising, more perils than promise at this time. But I think in the future, there'll be an important part of treatment, mainly because we'll never have enough mental health professionals or professionals to treat all of the patients. So with that, I'll open it up for questions. All right. I have seven questions already. Will you be able to share your email? For sure. I can type it out. I can type it out in the chat, not in the chat, but chat box. So there you go. All right. So that takes care of the first one. A new app has been approved by the FDA for depression along with medication. Can you please comment on that? That's a good question. I'm sorry. I don't know which app that is and what went into the FDA approval, but one trick I have seen lots of companies use, it's not an FDA approval in the sense that you would have a pharma company approve it. FDA actually is working very hard on creating a framework to help approve digital therapeutics. So digital apps are considered digital therapeutics. FDA is working very, very, very hard in trying to find ways to create a framework to help approve digital therapeutics. Trying to find ways to carefully assess the efficacy of these apps on par with what they would do for something like if you want to get a new drug to market or you want a new indication on the label. I don't think their mechanism is as robust. So I would be careful when I say approved by FDA. I think it's bad therapeutics. There was a computer-based app that was spent for FDA approval, but that approval is not as robust as you would expect for medications. So you might not need the same level of evidence. Don't quote me on this. I might be wrong. I haven't looked at what they submitted for their FDA approval. It might be something we might have to look into. But one word of caution is companies use FDA approval in a variety of ways, and FDA is still actively working on developing a robust system assessing this. So I would take that with a grain of salt when somebody says that, oh, we are FDA approved for treating depression. And it's a digital therapeutic, especially. But that's an important question. Thank you for that. And I might need to update myself on that. Thank you. Is there any use for APPs in clinical sessions? That is AI-assisted therapy. I'm not sure. Is there any use of these AP, oh, apps. Oh, sorry. So here's the nuance part of the language, right? When I see APPs, I see advanced practice providers, right? So APPs in clinical sessions. That is AI-assisted therapy. Yes. So I would say AI, so you can augment your session. So in between session work, if you want to kind of direct homework, and if you want to kind of work on some techniques, and then you want to reinforce during in between, like between session work, I think you can use them. There are a bunch of apps out there. My recommendation would be to look at these, like just use Google, go to the APA app checklist, see if they're good enough, and go from there. There are too many for me to kind of point one or two. Which AI for therapy would you recommend and is free? So most of these apps have free versions. I don't think I can make recommendations, but I would say the same thing. I mentioned a couple of few apps here. You can download them. All of them have free versions of it, and you can try out and see how it works. But my guess is for the real stuff, you will need to pay. You need to have a paid version. The versions which are free are quite basic. Are therapy chatbots generated AI or based on neural networks? So interesting question. If I'm understanding this question correctly, so even with neural networks, they technically don't have true understanding of what you're saying, right? They can evaluate inputs in a variety of ways, just like generative AI does, which is context specific. But if I'm understanding the question correctly, or if I'm saying it correct, none of these algorithms, even if it's deep neural networks, do not have true understanding of what they're looking at. They're just good at picking up patterns. And we keep picking up patterns in a very large context, right? Context that might span multiple dimensions, things that human brain is not capable of. So a simple answer to that question is most of them can be compared to something like a generative AI chatbot, where they are actively predicting what the next word can be based on the context provided by the previous interaction and all the other words. And they're doing it with great accuracy because they've been trained on very therapy related data. Remember I said, you start with the framework, then you train these models on therapeutic interaction, then there is multiple iterations based on feedback, and then they're deployed. So at this time, most of them will function just like generative AI. But I doubt that even with, and somebody can correct me, but I doubt that even with deep neural. So again, generative AI, to some extent, might use deep neural networks, right? So yeah, I don't know that answer the question, but they're like chat GPTs at this time, but specific for therapy. There are new studies out there that have shown that chatbots have more empathy compared to humans. They do not get tired, they do not get quick. If you can give your opinion, where will this be likely developed in private or versus public sector? So I would say, I think private sector is the way to go. And so here's where I think things will change. So when you go from fee-for-service model to a more value-based model, you will start seeing that these apps are used more and more and more. Because then the question will not be that I'm getting paid to provide a service, but then the question will be, I'm getting a sum of money and I need to do the best of this patient to get the best outcomes for this patient. And that's where you will start seeing apps and digital therapeutics being more utilized in the public sector. In the private sector, yes, there is early adoption. So in fact, you can kind of think about lots of companies using these chatbots, or at least these companies are targeting employee assistant programs, wellness programs in a lot of private companies. And again, there you are very unlikely, at least in a population level, the incidence of in a big company. For example, if you sample a psychiatric clinic, the rates of depression, psychosis, schizophrenia, and everything else will be very high. But if you're looking at the population as people working for Amazon, say, or at a high corporate level, you might not expect very high levels of, but at least very severe levels of depression that are just using employee assistant programs. So there's a lot of application. Even now, I think that's the kind of application these companies are targeting. But I think this will change once we go from a field service model to more of a value-based system where we are getting paid for patient outcomes more so than just the service you're providing, right? Because right now, public sector companies do not have any incentives to kind of provide extra care because they get paid for every time they see the patient, right? So that's scary to me that we would send patients to interact with chatbots instead of encouraging them to long-standing supportive relationships. AI could be collecting assessment and not for the support until we get long-term diagnosis. I agree with you 100%, right? But then, of course, we don't live in a perfect world. And I can talk over and over for a very long time talking about how a society in general, it's getting tougher and tougher to develop those long-standing supportive relationships, right? So again, we oftentimes compare these technologies with the best of the therapists and the best intentions of the therapists, not even the best actions of the therapists. But what we need to be comparing them to is complete absence of any kind of personal support, any kind of support system, any kind of access to self. So are they better than nothing is the question we need to ask rather than are they as good as the therapists? All right, I'm going to leave the next question. The app is re-originating. Yeah, I'll take a look at it. I haven't. FDA authorized. So there you go. So FDA authorized versus FDA approved. They play with words, so be careful. I think those two words have different meanings. What do patients of these apps lack evidence-based indicate that they are safe, effective to use in specific clinical populations? So good question. So again, I would go back because I think these companies are publishing new papers every day. I think they were just, if you think about specifically what Wobot, I think they were completing a big trial and their results should come out soon. I would make these decisions based on how I make decisions about anything else around questions around new medications coming out. I would kind of use the same level of questioning and skepticism while we kind of think about this. I think we're running out of time. Am I able to take one or two more questions, Helen? Sorry, Hannah? Yes, we have about a minute left. Okay. So I wouldn't as of now, as of my last review of the literature, there is not enough evidence to support this, their use in clinical population and outside of research setting. I might be wrong. They're coming up with new research papers every day. They might even challenge me on these questions, this answer, but I would use the same level of skepticism I would use if a new drug comes to market and evaluate it. Do you know if any of these therapy bots have used screening procedures before accepting patients like Sam? I don't think so, but they would do a bunch of standardized tests because all of them have inbuilt measurement based care system where they're able to try and assess how severe the conditions are and they might be able to kind of keep looking at and keep kind of measuring that over a period of time and show it to yourself that is like, hey, you're using this app and your scores are getting better. So they do. I don't think they use it strictly in the form of screening, but they do it for measurement based care purposes for sure. And I think that was the last question. So thank you so much. I hope you guys learned something. I definitely learned a lot while creating the presentation. So thanks so much. I'll see you guys at 4.45.

artificial intelligence

natural language processing

mental health apps

chatbots

app evaluation

data privacy

therapeutic interactions

anxiety and depression

AI in mental health