Okay, welcome everyone. My name is Justin Baker, and I'm very pleased to be here today to participate in the APA's virtual immersive on AI and psychiatry. I'm an associate professor of psychiatry at Harvard Medical School and the co-director of the Institute for Technology and Psychiatry at McLean Hospital. And today I'll be talking about sensing psychosis, how we can use AI or machine learning to advance psychiatric assessment. Here are my disclosures. So how do we assess that someone may be experiencing a serious mental health condition? The way that this has traditionally been done in research is that there's the concept of a control and a case, and that if you put all of the people who have different biological variables into some sort of space, you might be able to distinguish the people who have a condition versus the ones who don't. However, in reality, making a case versus control distinction is more complicated. So often in psychiatry, as with many other conditions, there's many forms of caseness that can complicate differentiating those individuals from healthy. Also, we have many conditions that are thought to exist on a continuum from health to disease, where finding the right breakpoint between illness and health complicates this assessment. And finally, there's this notion of comorbidity, where the same individual might be experiencing two different forms of caseness that we would have to be able to distinguish. Further complicating it, we often think of assessing mental health across biological scales. So going from risk for a condition in terms of genetic variance, all the way up to variance of the syndrome itself requires going across multiple levels of analysis from how those genes would impact cellular assemblies, brain structure, brain functioning, ultimately signs and symptoms that eventually make their way into the syndromes. And typically, this is something that's learned across thousands of individuals that therefore would complicate the assessment of any one individual along those different dimensions. There's also the element of time. How do we assess that someone might be having a condition will depend on where they are in their own normative development. We know that the brain is developing well into adulthood, and therefore, any abnormal behaviors or syndromes might be considered normative at certain stages of development. There's also time on a more, on a finer time scale, where many of the conditions we study are episodic, and someone might be coming in to see, for an evaluation while they're in, let's say, a manic episode or a depressive episode, but they also might be euthymic, which could complicate how we assess the underlying condition. And so the way that we like to come at this is using the notion of latent construct models. So what is a latent construct model? It's simply something that is an estimate of something that we can't directly measure, either because it's infeasible or because it's a complex construct or concept. But we know that the brain uses closed-loop systems to derive estimates of latent constructs. So in particular, the brain, you know, the nervous system has sensor networks that allow for certain kinds of inputs to come in, and then we have the entire brain and the rest of the computing system is there to extract just the relevant features from those sensory systems that inform upon the latent constructs of most interest for survival for the organism. And then moreover, the brain can cause movements to occur so that those sensory systems can interact more with that environment that it's trying to estimate something about, and by doing that, derive feedback that can then be used to iteratively narrow in and hone in on the true value of that latent construct. In addition, clinicians are constantly using closed-loop methods, but they're often using them sporadically and inconsistently. So clinicians learn to use their visual systems, their auditory systems, and other sensory systems to pull in information about their patients, and they use their training to extract the most relevant features from those sensory inputs to inform upon latent constructs such as, what is my patient's diagnosis or what is their mental state? And then clinicians also learn to use movements, whether it's nonverbal behaviors or verbal behaviors, to generate new inputs from that patient to, you know, whether that's questioning or certain ways of posturing that can, again, iteratively allow the clinician to hone in on whatever that latent construct is that they're trying to estimate. And finally, what I'll tell you about today is how we're starting to use actual sensors that we can put onto patients in different kinds of settings, whether it's inpatient settings or outpatient settings, to start to derive information about their mental state and their diagnosis, again, using the same concept that we're going to be pulling in new kinds of data from wearables, cameras, phones, and microphones, and then do intelligent feature extraction that allows us to map constructs of interest to psychiatry. And then also, keeping a closed loop, we want to be able to develop new ways of prompting or probing patients so that the data we collect from these devices is the most informative. So this is just another way of viewing what I was already talking about. There's a raw data layer that contains many of the data types that we use today, from surveys to blood samples, MRI scans, electronic health record, as well as many of those digital data types that we're incorporating increasingly into research and eventually practice. But again, what's really critical here is the feature extraction or feature detection layer, where we can start to bring in tools from computer science to be able to parse and make sense of the inputs from many of the devices that are not so commonly used in today's clinical practice. But again, the idea here is we want to map these features into the latent psychiatric variables that we're most interested in assessing as part of our treatment. And so then, to zoom out further, the way we see this fitting together in the grand scheme of things is that we would have comprehensive phenotyping, as shown at the top, where we would be able to extract estimates of many of these latent psychiatric variables, but then we would send those estimates into further synthesis and change detection and be able to combine it with software that puts this patient at the center and provides insights both to the patient, but also to people in their care environment, such as their treaters and their family members and peers. And we also hope, as part of this work, to be able to develop more sophisticated tools for biobanking, clinical support, and also supporting research. So what I'd like to take you through over the next half an hour or so is these three different use cases of bringing AI and machine learning into our clinical assessments. And so the first one, inpatient sleep and activity, is probably the simplest. And so for this one, we're going to be just taking data from wearables and combining that data with data from the EHR, the electronic health record, and then being able to not only detect sleep and circadian rhythms using a wearable, but also then building this into a visualization that is intended for clinicians to use as part of their assessment on an inpatient unit. So over the past several years, we've been at McLean Hospital using a device similar to the one on the bottom left. It's a gene active watch. It doesn't tell time. It just provides an accelerometry signal showing how much the patient is moving. And so the idea is that we record this wrist actigraphy or activity movements during the hospitalization, and then we compare it with clinical events such as medications, mental status, to try to gain insights. And so what we'll be able to do is process the raw signals to look something like what you see in the black, which gives you an overall activity level. And then using some simple machine learning tools, we can map these raw activity levels into states, our estimated states, based on the percentage of activity that that person is moving relative to their own baseline. And then following that assignment, we can then take those smaller chunks and connect them into longer periods of wakeful activity versus sleep. And then once we can do that, we can start to visualize this over time for the clinicians and generate visualizations somewhat like this, where you can see that time is moving from left to right, and you can see that this is showing this baseline, and we can and you can see that this is showing this patient's movement pattern. This is closer to the raw data, and then we can extract the sleep, the estimated sleep period, such as this. So in this example, the patient was not sleeping very much when they first come into the hospital, and then as you can see, their sleep somewhat normalizes, and then they have a couple days, which is especially long sleep. So once we can do this, we can start to use the same algorithm to apply across both the entire inpatient population, but also over much larger data sets, such as population-level data sets like the UK Biobank or the All of Us data set. And so here I'm just showing that we can begin to really build normative ranges for what some of the features that are extracted by the pipeline, and that way we can begin providing insights to patients or doctors relative to the norms we calculate. We can also assess how different sleep medications affect the sleep parameters under different conditions. So for instance, comparing how melatonin versus trazodone affected sleep duration or sleep fragmentation, just based on pulling those medication records directly from the electronic health record and comparing it to the sleep derived metrics. But today I'd like to focus more on this idea of generating useful activity reports for clinicians. And so we embarked on this project with input from the psychiatrists, the nurses, and as well as researchers on the units. And so what I'll take you through is what that is looking like today. So what we provide is visualizations such as the one I'm showing here that have large numbers at the top of the page for easy, quick reading. And then underneath that, we have more details about how the trends over time change over the course of a hospitalization. And then as you can see here, we show the medications for this particular patient over the course of that brief hospitalization. And then as you can see below that, we also then show exactly when sleep occurred relative to specific doses of each of the medications, the psychiatric medications this patient was receiving. So for now, this is really just a tool for the clinicians to get a quick read on sleep, sleep duration, sleep quality, and waking activity. But eventually, the idea is to actually derive response functions for the common psychiatric medication so that we'd be able to determine whether someone is experiencing any side effects from a particular medication or also whether the medication might not be as effective or whether it's seeing an efficacy signal that we would have expected based on population norms. And so here, we made an effort to visualize the medication record in a way that is easy for people to interpret in terms of increases or decreases in the dose so that for the busy clinician, they can get a quick read on both how medications have changed during the hospitalization but also maybe how that has affected the patient's sleep. So this is just zooming into a particular example of how sleep and the medications might be looking like in a patient who's experiencing many medications at different times of day. So sleep is just the tip of the iceberg with activity classification that can be done with wearable devices. Using similar devices, one can also derive methods to detect activities of daily living such as brushing teeth, washing hands, combing hair, but also things like taking medications or other types of behaviors such as smoking or drinking that might be relevant to someone's clinical treatment plan. We're still a ways away from being able to do this robustly, but that is one future direction of how we might be able to use wearables in clinical care. So next, I'll shift gears and talk about capturing and characterizing dyadic interactions. I think as everyone can appreciate, whatever is happening in terms of sleep and activity is certainly one component of the psychiatric assessment, but the real bread and butter of when of how we do a psychiatric assessment today is in dyadic interactions typically between a patient and a doctor. So we're going to talk about the dyadic interaction of sleep and activity typically between a patient and a doctor or a patient and other kinds of care providers. And so for this part of the talk, we'll be showing how we can combine data from surveys with data from video and audio, and again, extracting different latent dimensions this time more related to executive function, language, affect, regulation, stress. And then there is also a component here of how we can begin to visualize these signals in ways that a clinician or researcher might be able to use them. Here, I'll focus on a visualization that really is designed more for the researcher, but these techniques can be adapted for clinical situations as well. So why do this? Well, today, I would say that capturing and characterizing interactions is done in a very human resource-intensive way, meaning that an individual who's been trained sometimes over years is asked to evaluate someone. If you incorporate all of the training, this component of the evaluation is really quite expensive. So in research studies where you might be doing MRI scans, the cost, the embedded cost of doing the clinical evaluations is often much more than any one MRI scan, even though MRIs are often noted to be expensive relative to other types of measures one could acquire. Even with that expense and the human resource time, it's still a highly subjective way of assessing individuals, and at least in research context, it really doesn't have the right…it's got poor intertemporal and interrater reliability for many of the features that the evaluator is going to be assessing in that patient. And so, in our view, these in-person dyadic interviews are probably not ideally suited for capturing a robust point estimate of how someone is doing, their symptom severity across multiple dimensions, especially when you want to be doing that over time in the individual and across many individuals. And so, we reasoned that we would start by simply recording these clinical interactions, and rather than having a human have to do manual annotation of what happens during the evaluations, we would get help from computer scientists to help to code the entire video. And so, we worked with a group out of Carnegie Mellon University who had developed something called MultiSense. This is a software tool that allows a video stream collected off of a standard camera, such as the ones on the left, to be able to pull the facial landmarks and head orientation from the video stream, and ultimately to map that onto the facial action units, all the different parts of the face that move when you're expressing emotion. It also is able to look at gaze, you know, how you're orienting your gaze, whether you're making eye contact, and also could measure things about how you're speaking. And so, in collaboration with this group, we thought to design a simple and short assessment that would allow us to really tap into what information the clinician is able to get even in a short period of time. And so, for that, we decided to use the Psychiatric Clinical Rounds Encounter as our model. We chose that because, as any psychiatrist who's worked on inpatient units will be able to attest, you often have a very short amount of time to assess your patient along multiple dimensions of psychopathology, and you have to be able to do that in a way that feels authentic to the patient at each day of the hospitalization. And so, we decided to, with Dr. Liz Leibson, who's my collaborator at McLean, we came up with a semi-structured interview. The first question of which is intended as an open-ended response, what brought you, you know, what brought you into the hospital, and followed by a number of specific prompts that Dr. Leibson found to be useful over the course of her career working on the inpatient unit. Things like, has anything been on your mind recently? What are your goals for the hospitalization? How are people treating you? These questions are, in some cases, trying to indirectly probe things like paranoia, depression, anhedonia, et cetera. And so, then what you can see on the bottom is a heat map showing how the participant and the evaluator were responding in terms of their facial movements throughout that brief encounter. And so, what you can see here is that the participant was alone at first, and that was intentional to keep them, evaluate them a little for two minutes by themselves. And you can see that during that period, they're doing a lot of brow raising, possibly looking around the room. Once the doctor enters the room, you can see that they smile. These particular action units show that the eyes and the mouth are showing movement patterns consistent with a smile. And then you can also see in response to each question how the participant was responding and also how the evaluator was responding. So, you can see here, for example, that this evaluator is not smiling much, but she is raising her brow. She is squinting to indicate interest. And, you know, this is a very well-trained psychiatrist. So, you know, maybe for her, this is her optimal way of behaving throughout an interview. But you can imagine this could be a training tool to help more junior clinicians to learn how to provide the right kind of behavior throughout an interview such as this. So with these kinds of data, we can then try to show with machine learning how we can start to predict some of the conventional items. So for instance, the brief psychiatric rating scale includes a depression score. So here we found that people who had a higher depression rating actually had changes in their brow furrowing. However, what we found was not that brow furrowing goes up as you've become more depressed, but rather that the standard deviation or the variance of brow furrowing actually goes down. And this is consistent with a constricted affect where someone just isn't making very many movements anymore and the clinician might've been able to notice that. But this is just showing that this particular finding is somewhat face valid with how clinicians would assess someone who's got constricted affect. We also found that individuals who would be rated as having high unusual thought content on the positive and negative syndrome scale were more likely to show brow raising and smiling, particularly during that alone period before the evaluator was in the room. So again, this was another clinical pearl that seemed consistent with how we would evaluate someone who may be having self-dialoguing or responding to internal stimuli, even in cases where they're by themselves on the unit. And then finally, we can use these kinds of aggregate measures to, again, using machine learning techniques. And I won't go into exactly how we computed this here. You've already been hearing about these techniques earlier today, but we can then begin to link optimal combinations of these features to get measures of, let's say, the negative symptoms from the PANS or other measures that might be useful in a clinical trial setting. So conceptually, what we're trying to do is take all of these more conventional items. Here, I'm showing a network graph where the distance between any two of these colored circles shows how correlated those items would be on the standardized assessments of the PANS, the Madras, the BPRS, across a sample of about 50 individuals. And what we're gonna be doing here is I'm gonna superimpose what we can then extract directly from the automated video analysis. And so here in the red, yellow, and green colors are facial action unit, magnitude, head rotations, and then this idea of expressivity, which is a feature that is derived from multiple action units. And so you can see here how expressivity, for instance, is correlated highly with elevated mood, whereas motor retardation is correlated more with this particular action unit. So eventually, we can establish more criterion validity by collecting more and more datasets and then being able to refine these estimates with further and further contextualization, such as which part of the interview was being assessed as well as which specific question was just being asked. So as of today, we have real-time reports that can summarize each interview in real time, not in real time, but within basically a few hours of getting an interview, we can generate a report such as the one shown at the left where we can quickly visualize the affect of the participant, the affect of the interviewer, as well as information about their audio data. And at the bottom, you can see there's a word cloud of what specific words were used during the interview, as well as normative ranges for speech rate, coherence, which is a measure of whether participants were using uncommon words. We're sorry, uncommonness is that, and then coherence is the similarity of that person's speech over time. And so we can begin to generate these kinds of measures and be able to see not only how the time series looks, but also with a nod towards neuroimaging, be able to see how all these features are linking together as well as these group and participant level norm data. So you can see not only how the patient is performing relative to the group, but you also in this visualization can see how they look relative to other interviews that that patient has been part of. Okay, so next I'd like to move into the third component, which is bringing these elements together into a remote patient monitoring or passive sensing. So again, when you're working with patients and you want to know how they're doing, at the moment, you mostly get that information when they're coming into the visit with you. However, of course, all patients also have other parts of their life. Our group works primarily with young adults, so they're either going to work or school, they're trying to navigate social relationships on top of dealing with their illness and dealing with relapse, recovery, and so on. And so we decided to undertake what we call the year in the life study, which was to really try to follow a cohort of individuals who had severe mental illness to see over the course of an entire year, what do the ups and downs look like in a way that we might be able to intervene in the future in more granular ways, but also just to be able to understand what kinds of changes are health related and what kind of changes in their life might just be related to their life unfolding. And so for this project, we were fortunate to get a grant from the National Institute of Mental Health as part of what's called the Intensive Longitudinal Health Behavior Network. And our project was focused on enrolling 100 individuals with either depression, bipolar disorder, or a psychotic condition and following them for up to or more than one year. And then the goal here was to track patient self-reports through surveys that we can use the phone, clinician evaluations when people would come in for either in-person or video-based interviews, as well as those objective measures of sleep and activity like I was showing in the previous slides. And then really to understand when in this population that's enriched for likelihood of having episodes during this one year timeframe, be able to observe when there are changes in clinical severity, what do the changes in self-report and objective outcomes look like? So this is the individual participant study timeline. And what we sometimes also call the bipolar longitudinal study. I will say that we weren't only looking at bipolar patients, but we would look at anyone who had, again, more of a severe mental illness. Everyone was asked to wear this gene active watch for the entire year, bringing it in to us to swap it out at each study visit. They also installed an app called Biwi on their phone that allows us to record different signals that I'll mention in a moment. And then they would come in for a monthly study visit, perhaps similar to a patient who's coming in for a monthly clinical visit. And then finally we run some simple web-based neuropsychological testing using something called Test My Brain to get at individual's processing speed, as well as their cognitive control using a simple reaction time and a choice reaction time task that I won't be describing today. So just to flesh this out, here's a representation of which kinds of sensors and behaviors we can assess with each of these four modalities. So with the gene active, we can assess movement patterns, light, as well as a button press that people could press to indicate certain behaviors. The Biwi app allows us to collect movement, location, calls and texts, as well as participants would be asked to fill out a survey each day, as well as an audio journal. And then finally the study visit could be used to extract those audio visual features I was mentioning previously, as well as the conventional scales such as these four that I mentioned here. The MCAS, by the way, is a Multnomah Community Assessment Scale and it's designed to assess functional functioning. And then finally, we're mapping these sensors and behaviors into these particular latent constructs to try to understand whether, particularly changes in energy level could be something we would be able to measure or infer based on these signals that we're collecting here. And I'll get a little bit more into this in a moment. So first I wanna take you through what one participant in the study looks like, give you a sense of the richness of the data. So this individual was in the study for over 700 days. And what you can see here is these are some of the items we asked her to answer each day on her app. How energetic do you feel? How alert? And you can see that at the beginning of the study, she's doing really well. So here I'm representing each of those positive items and negative items and the colors here. And you can see that she's got very high colors for all of these items and low colors for the negative items. But then look what happens around day 120 or so, she shifts very dramatically into a negative state where she's endorsing feeling hostile, irritable, ashamed, upset, et cetera. And then look again around day 200, she shifts back into a positive state. And again, in this particular patient, her remission and relapses are lasting quite a while. And then notice that as she moves further into the study, that the episodic nature starts to become more frequent. So the episodes are shorter lasting, but we're able to clearly detect them using these ecological assessments from the phone. And the bottom is just representing an overall positive or negative score based on combinations of these different items. So now I wanna show you what the other data types are doing in this particular patient. So here I'm showing now the Madras and the PANS negative scores. And what you can see here is that the Madras score, the Montgomery Asperger depression rating scale goes up when she is in fact depressed as you would expect. But notice how far the lag is. So after 30 days or so after she starts into this negative episode, she comes into the visit. And so we correctly, the research staff correctly labels her as having a high Madras value. But then notice what happens when she's coming in less frequently towards the later part of the study. She reported feeling so depressed during some of these high points that she was unable to come into the visit. And so when she actually makes it into the visit, she's actually doing well. And yet we're rating her as high on the Madras because the Madras asks the rater to rate over the past 30 days, how depressed have you felt? So you can see right away that we get a mismatch between how she's doing in front of us and how the rater scores that evaluation. And then finally, I wanna bring in the objective data. So in this particular patient, note that she's able to wear the watch that we provided very consistently. So I didn't explicitly mention, but each column of this visualization is a single day. And the only times that she took it off are when you see the white colors. And so she was basically very consistent doing both these ecological surveys, but also wearing the watch. And so then I wanna drive to two other points about the actigraphy data. One is that when she gets depressed, what you can see is that her sleep, actually she starts to wake up much later in the morning. These graphs go from early to late. And so you can see that her sleep duration becomes a little bit longer, but also moreover, notice that the activity pattern during the day, which was this bright red color when she was doing well, shifts into this kind of orange, even green color when she's not doing well. And that signifies that she's moving much less on days when she's feeling depressed, even despite sleeping more. And I think that is again, consistent with what we see in severe cases of depression where people are really feeling lethargic, despite in some cases increases in sleep. And so this is where we can begin to draw a latent variable, derive a latent variable from what we think of as the differential between how much sleep you're getting and how active you are during the day. With the idea being that people who are moving a lot during the day and not getting enough sleep would actually be kind of going down on a more global energy balance, whereas people who are sleeping a lot and not moving much during the day would actually be coming up on their balance. And so that's what we represent here. This is an energy level that we're deriving directly from this difference between day-to-day sleep and wake activity. And so this led us to this idea that perhaps when we see these kinds of coherent movement of both subjective ratings and changes in people's activity profile, that perhaps what we're observing is that the body is actually going into a different energy state where rather than just spending energy, it actually tries to conserve energy to bring the body back into a homeostatic norm. And so this is still a theory, and this is something that we're continuing to investigate in the larger data sample. One way that we're looking at this is in the UK Biobank dataset. So UK Biobank, over 100,000 individuals from UK were asked to wear a wearable for one week. And so we took that data and we applied the same algorithm that we applied in the previous datasets to derive these average maps of the human week of activity. And so you can see here Monday through Friday on the left, and then the weekend is represented on the right. And so you can see that sure enough on the weekend, people tend to wake up a little bit later and be a little bit more active on average. By having such a large dataset, we were able to break this into discovery and replication sets. And you can see that the pattern is almost identical in the two discovery and replication datasets. Next, what we can do is ask questions about this data in terms of how does this average week of activity change over the lifespan? And then how does it change with respect to certain conditions like depression? And so on the left, what I'm showing you is the average movement profiles over a week for both females and males as someone ages from 45 up through 82. And what you can probably visually appreciate is that there are a number of noticeable changes by age, much less distinction by gender. But what you see here is that, I just want to draw your attention to really one feature here, which is as you age, you actually start moving much less during the day. And that's what this greener color looks like compared to the red hotter colors down here in the earlier age. And so what I'm showing you on the right now is that we can then take all of those data and ask, how does it differ based on whether someone reported feeling recent depression or not? And so in red is that as someone ages, these are people who had depression versus who didn't have recent depression and changes in sleep onset time, when they go to sleep, when they wake up and when their sleep duration. But notice in particular, this waking activity value that I was drawing your attention to. What you can see is that in healthy non-depressed individuals, there's a almost linear decrease in the level of waking activity that is again normative. But look what happens when someone's depressed, that whole curve gets shifted to the left as if that individual or people who have depression on average are moving roughly like someone who's about five years older than that individual would be. And so something like this, again, this is at a population level of tens of thousands of individuals, but we can still use this kind of approach to then start bringing it back into a more normative context like you might think about for like a growth curve. So when you have a pediatrician would look at a child's height, weight and head growth and be able to chart that individual relative to their own percentile over time. And so what we can do here is bring in all of those UK biobank datasets, each dot here representing an individual or individual week of data. And then we can begin to superimpose other individuals. So here the red dots are from that bipolar longitudinal study that I was mentioning to you a moment ago. And what you can see is that this individual who I was, every dot here represents a single week of this person's data. And so what you can see here is that on days that she's doing well, she tends to have a waking activity that's much closer to her age appropriate median, whereas when she's doing very poorly, her percentile drops into the low teens in some cases. So next I wanna focus on a different type of data, which is geospatial data. So location data is tremendously useful in terms of establishing the context of individual's lives. We all go different places over the course of our week and those places often signify important social factors and functional factors, whether we're going to work or meeting with people. And so here what I'm showing you is a privacy preserving representation of two years of spatial recording from one participant. And so you can understand how to read this. Time is moving from, or I guess days are moving from left to right, and then time of day is on the Y axis. So you can see, and I should say that the home location here is the gray location. So you can see that this individual, when that person is spending the time at home, and then also when the person is going out into the world. I wanna draw your attention quickly to just three aspects here. The first is this blue location. This is somewhere that the person ends up going for extended periods of time twice during this recording, as well as a bunch of times during the week as well. So you can see that some of you might be wondering what that location is. And so this is a hospitalization. So when they go in, they're hospitalized, we can directly deduce that from the data. We don't have to infer it really. We can directly tell that. We can also see this particular patient's therapy sessions when they go to that same hospital location during the regular daytime hours. And we can also, in this case, see the person's Alcoholic Anonymous meeting, their AA meetings, because they also come back to the hospital on certain days of the week. We can also start to identify part-time jobs. So this Teal location is a job that this person is going to, and you can sort of tell that this is not a five-day-a-week job. It becomes even easier when I just visualize this as a function of day of week. So now you can see that this Teal location, at least at the beginning of the study, was on Monday and Fridays. And then towards the end of the study, the patient got a different part-time job at this Black location, primarily on Thursdays and Fridays. And then the last thing I want to draw your attention to is this Orange location. So another aspect of social functioning is whether you're engaged in social relationships. And so in this case, the Orange location is a romantic relationship that this individual had while in the middle of the study. And you can see that they start spending more time at this individual's house, and then eventually sleeping at that individual's house for a period of time before the relationship eventually stops and then starts again later. So again, I want to just highlight here that we can use these kinds of geospatial data to essentially do automated, minimally burdensome, and continuous assessment of individuals' community and social functioning, even in individuals who have high levels of clinical severity, as was the case in this population. So here I'm showing you seven examples that give you a sense of the richness, the location diversity of each of these individuals, and a rough sense of what kinds of locations these people might be visiting based on the patterns that you're able to see. Now, in practice during this study, we would use our audio journals as well as our clinical interviews to actually validate some of our assumptions and get more specific about what types of locations each of these places were for each participant. And then finally, I just want to highlight that these kinds of data are very intensive, and in some cases, you might even say invasive. And so privacy is often something that is raised as a potential risk for collecting these kinds of very granular data and across multiple modes of data in the same people, especially vulnerable populations like those with severe mental illness. And so this is something that our group takes very seriously and if you're interested, you can follow the link here. We did a work with Harvard Law School and had these series of articles on different aspects of ethics and legal and social implications as part of an NIH bioethics supplement. We also generated an ethics checklist for digital mental health. And in particular, this is meant to be for researchers, but really just to take researchers at the beginning of the project and really require that they go through these different dimensions of informed consent, equity, diversity, access, privacy partnerships, regulation and law, return of results, and as well as duty to warn and duty to report, all of which are gonna be different in these kinds of deep phenotyping studies where you're collecting so much information at the level of individuals. So what we're really trying to use here is getting ways to capture behavioral variation across multiple time scales. And so we can use things like wearable sensors and phone sensors and video to capture the really granular changes in behavior that occur at the level of seconds or even less than seconds, as well as being able to zoom way out to the level of years, since we know that patients who experience episodes or hospitalizations often see their recovery phase as not lasting days or weeks, but really in many cases, lasting years. And so we think that these kinds of approaches by being relatively low burden can actually enable us to be able to follow patients and get rich informative signals at both of these timescales. And the hope here is that we can begin really moving beyond the more caricature style visualizations that I started with to something that's more computationally defined and that can use all of the benefits of ubiquitous sensing as well as ubiquitous computing. So AI and machine learning helping us to really make sense of the mountains of data that we can acquire using these types of approaches. And finally, I'll just end on this question of will AI eventually replace psychiatrists? I think this is something that is really obviously sort of a hot button topic at the moment, but I wanna encourage everyone here to think about how do we decide that a particular tool or a set of tools is ready for that particular use case? Also, who gets to decide this? It's just something that the marketplace will decide, that companies will generate these things and if people use them, then we'll use the ones that are useful to people? Or is it something that the government or the FDA needs to decide? Or is it really just the treating physician who can employ some of these techniques as they see fit as long as it isn't degrading their clinical work with patients? So with that, I will wrap up and I'd be happy to answer any questions. So I'll go through the questions that I see here in the Q&A box. So I'll just read the question and then I'll answer it. So this is excitingly scary, I feel, especially I wonder what a psychotic paranoid patient would think, what is the data on patient experience on AI observation on their behaviors? Yeah, this is a great question. Obviously we need to be sensitive to the fact that some patients aren't going to be interested in having their behaviors tracked, whether that means they don't want their interview to be recorded or they don't wanna have a sensor tracking their location. In my experience, while I've certainly encountered patients who have high psychosis who've felt that way, in my experience, it's sort of not what you would expect. In other words, I have patients who don't have any paranoia who really don't feel comfortable with the recording for whatever reason, and other patients who are quite paranoid and have dense persecutory delusions and believe that other people like the government are surveilling them. When I've directly asked them whether they feel comfortable with us following them, I got a really interesting response which was, are you saying that I shouldn't trust you? Which I think was sort of just belies that in the healthcare context, patients already have a strong trust element with the clinician that is different than when you're sharing data with like Google or with the government. So I think there's something obviously privileged about if we can start to do this in a clinical setting where even a research clinical setting where there is an implicit trust between either the patient or the participant and the study doctor, which through all of these studies, we do try to have a relationship, to maintain a good relationship with patients both so that we can help them understand why we're recording each of these signals, but also so that if they are uncomfortable, we wanna know that, we wanna understand what their hesitancies would be and what kinds of mitigation strategies we could use that still allow us to collect granular informative data without necessarily causing them to have concerns about that. And I see another response that, same concern with telepsychiatry. Yeah, I think people under the care of a clinician are surprisingly willing to allow that clinician to see into their life as long as it's perceived as being able to benefit their care. All right, so I'll move to the next question. As one of those busy clinicians who's working on the front lines, I'm curious to your thoughts on the implications technology will have on clinical jobs, understand the utility of more objectivity and efficiency, but I wonder if this is going to make clinical community-based psychiatrists like myself less relevant. Yeah, this is a great question. I think in that question of will AI eventually replace psychiatrists, it wasn't on the slide, but I was part of a British Journal of Psychiatry debate on exactly this topic, which I would encourage, it's from 2019, so I would definitely encourage anyone with access to that journal to maybe read that debate. I came down on the view that AI would not be replacing psychiatrists, but that there's a lot of things that technology could use to improve efficiency and also potentially to help clinicians who are maybe more isolated or even maybe clinicians who are early on in training to get information that could help them improve their clinical evaluation skills or be able to provide their patients with other tools that just reduce their workflow, like in terms of if you're following someone on a new medication, such as lithium or clozapine or something like that where you might wanna follow, side effects, profiles using wearable, you would be able to use that information even during your regular brick and mortar clinical evaluation. I think it will be a long time before psychiatrists are less relevant. That would be my opinion, but, yeah, I think the comment here is a good one. Probably clinicians won't lose their jobs to AI, but clinicians might end up starting to be displaced by people who use AI in certain contexts. I think there's potentially some truth to that, but I would also just say that at the end of the day, I think psychiatry and psychology are still very much a human to human, humans who are in distress wanna be around other humans, not necessarily AIs. That could change, but I think we're not nearly there yet. So I think it's more about using AI to improve the evaluation and help people connect better between the patient and the doctor. So another question, do you have any comments about people using off-the-shelf GPS or law enforcement using the same information? Iverson, maybe you can explain this question a little bit more, but I think there's benefit about, obviously law enforcement uses GPS now. I mean, that's what a home, when you have a house-based arrest, that's basically what that is. It's a GPS device that notifies law enforcement when you leave a certain area. I think that these kinds of approaches with better software could eventually help law enforcement to use less restrictive settings with more information and more context. For instance, if someone needs to go pick their kid up at school, there would be unallowable places where that could get used. Yeah, so I think the question about inappropriate use. So I think, yeah, I think if a law enforcement was wanting to, for instance, access the data that you would collect it as part of clinical care, I think that's a separate concern and that we would want to obviously design systems that would limit what could be shared in some of those cases. And obviously there's lots of people using off-the-shelf GPS as part of other mobile applications and things like that, which is not necessarily in the healthcare context. And those kinds of signals can be used for inappropriate targeted advertising or other things. So I would say, in general, when working with these kinds of data, how you store the data, how you process it. For instance, I didn't show any visualizations today of actual locations on a map because we feel that we wanna make sure that we're de-identifying data as soon as we can as part of the data analysis pipeline. And I think it's a really important point that outside of healthcare contexts, people are often using these kinds of signals for other things that are not health-related. And so here again, my take would be that should prompt us not to avoid these things, but really to proactively find health use cases that are health-promoting so that people aren't just used to their data being sniffed out by other people. Okay, question from John Young, great presentation based question of clarification. These three use cases, it seems the main breakthroughs here so far is using technology to capture phenotypes. What is the contribution to this so far using AI to interpret and generate the dashboard? And should that come later? Right, so in the presentation I presented today, it's really about AI in quotes, machine learning and artificial intelligence being able to take data and be able to infer latent states from those data, be able to use distributions of the data and be able to, for instance, identify the sleep pattern from a rock accelerometer. That's a form of AI, it's not a, that particular one is a very straightforward one. And then in the dyadic section, we're still using machine learning and AI both to identify on a video frame by video frame component, like what are the different facial action units doing? That's a machine learning algorithm. We also use machine learning to take those summative values and relate it back to other clinical assessments such as the clinical total scores or any of the other aspects of a mental status exam. And then, yeah, there's always more AI we could be doing. Further downstream to generate dashboards or come up with insights, but here I've really just focused on using machine learning and AI to take these really complex multimodal data streams and use computational techniques to derive more discrete estimates from those multimodal time series data. So next question, I think the question is not if, but when the daily use of AI becomes part of our life and work, do you have a sense it wouldn't become ubiquitous in clinical practice? Yeah, that's a great question. You know, ubiquitous implies, you know, maybe a generational shift, right? Because once it's standard of care of psychiatry, I think that that's gonna take, you know, everyone in the fields buying into these kinds of approaches and I think just the nature of psychology and psychiatry practice, we still have, you know, people who are, you know, proponents of techniques that have been around for many decades. So I think that isn't gonna change right away, but I would say that some of these techniques are much closer to be ready than others. So for instance, the first presentation I showed about the actigraphy being used on the inpatient unit, we're actually already moving that from a research context into a clinical quality improvement context at McLean Hospital, where physicians are able to generate, you know, be able to use these kinds of reports in their patients. Still early days, but I think, you know, the hope is that those kinds of more straightforward ways of bringing in sensing into clinical practice will come sooner than later. You know, whether it's ubiquitous or not, you know, we'll have to see, but I think some of these are definitely much closer to being ready than others. So next, you're accumulating a vast amount of data, new statistics to arrive at your diagnosis. Isn't that the antithesis of individualized medicine? Yeah, I'm not sure about this question. So we're not trying to arrive at a diagnosis. You know, what we're trying to do is we're trying to use data to personalize the assessment of each individual so we can detect, you know, regardless of maybe what the diagnosis is, we can detect when that person's, you know, both subjective and objective measures are changing in relation to when treatments are being provided. So to me, that is what precision medicine should be about is basically, and there's really, it's not so much using statistics. It's more about being able to use different kinds of computational techniques to detect change at that individual level. Hopefully that answers that question. Yeah, so for those healthcare systems that don't have the resources to do all that you're doing, which of the data collection analysis have you found to be most useful? That's a great question. And mindful of the time here, it's 1245. So maybe we can, this will be the last question I answer, but what I will say is that although, you know, some of the methods we're using are more resource intensive, having someone install an app on their phone or using a wearable is not so difficult. And I would say that that's probably the most useful place to start. You know, you can obviously start if you have the resources and you're using video-based encounters for teletherapy, and it's something of interest to your care practice, you know, being able to start collecting and storing those videos for future analysis is something that also can be done with a lot of additional resource, without a lot of additional resources. Okay, well, these are great questions. I wish we could have gotten to all of them, but I really appreciate the input and I really appreciate everyone taking the time.

AI in psychiatry

machine learning

psychiatric assessment

latent construct models

closed-loop systems

wearable devices

ethical concerns

patient trust

individualized care