How to Scale Mental Health Care Without Losing the Science with Dr. Soo Jeong Youn

[Music] In the first part of my conversation with my guest, we talked about how a psychotherapy researcher can start to see the private sector as a real possibility, not as a fallback, but as a place where rigorous clinically meaningful work can happen. In today's episode, we get into what that can actually look like. My guest tells the story of moving into a large healthcare organization and helping build a new behavioral healthcare pathway from the ground up. We talk about what it means to develop a product while also building the evaluation infrastructure around it, how to think about outcomes from the beginning, how to work with business stakeholders, and how to maintain scientific and clinical rigor while moving quickly. For people interested in psychotherapy research, implementation, science, digital mental health, or non-traditional career paths, this episode gives a rare behind-the-scenes look at how this work can happen in the private sector and what it takes to do it well. But first, if you're new here, I'm your host, Dr. Dan Cox, a professor of counseling psychology at the University of British Columbia. This is psychotherapy and applied psychology where I talk with leading researchers about what actually matters in practice, what's behind their findings, and what they wish clinicians new sooner. And if you enjoy the show, please subscribe when your podcast player or on YouTube, like and subscribe. That small click makes a surprisingly big difference for the podcast. This episode begins with my guest talking about the job that moved her into developing and evaluating a new behavioral healthcare pathway. So without further ado, it is my pleasure to welcome back my very special guest, Dr. Sue Jong-Yoon. So I ended up joining a Reliomedical Group, which is part of OptimCare, which is the care of the University of Health Group. It was a really, really cool opportunity for me to pass up. There were a couple of different perks. One, I got to work with one of my former grad school, grad school lab mates, his name is Sam Norberg. And he was absolutely like, you know, even think about stereotypical leader of like a vision, understands everything that you say in day and right now. And in terms of characteristics of individual characteristics, he has them. So the biggest perk for me was that I get to work with Sam again, which is great. And then the other perk was that there was, we were getting funded to develop a new product. So it wasn't like coming to fix, change, anything. It was coming, like, develop something new that already is thinking about scaling from the get go. So again, the funding was coming through this like innovation fund. We had a wonderful business lead, also her name is Georgia Hoyler. Well, spoiler alert, actually working with her now, seeing another capacity. But she's also incredibly wonderful. And the two of them had got in this funding to develop a new behavioral health care pathway, like a new product to develop to really address the emerging like behavioral health needs. But not being able to of course like train enough of you and me kind of people like specialists and mental health. So let's develop something alternative that we can give to providers as a way to triage patients to something else that might be digital in nature. So that we can quote, quote, free up human providers to see different kinds of cases. So that was a mandate. What really, you know, again, what made it such a cool opportunity was like research of course, rigor science was what was at the one of the selling points within that they, the two of them sold as a key infrastructure element that was necessary in order for this to be effective and for the ultimately swayed the funding source to actually give them the money. So I get even the was coming from like, you know, for profit company that the what really swayed them to be like, okay, here's a bunch of money for you to do this is because they were bringing this scientific rigor as a differentiated factor. So what was I was asked to do was like, okay, within three months, publish. Like that was the first big deliverable we had to see. We had to think about, okay, how are you going to use implementation science in this case to really set the stage for large scale, large scale implementation within the one behavior health department. Then publish on it, think about what are the metrics from the get go build them out, set the data collection infrastructure, what are we going to be our outcome measure for our patients, for our providers, what are the metrics of success that we're going to be evaluating our product on our program around so that we can then, you know, really share that out. And unlock the next set of funding. So, so the so was this so as you were as the team was developing the product, they were sent your simultaneously integrating the evaluation components. It was to yeah, that was like at the beginning, we set that up and that was like part of the my role. Like what's to bring in that evaluation framework, think about what are we going to evaluate, how are you going to think about implementation, like for we're going to trade how are we going to roll this out, how are we going to be evaluated if the role that is going well or not so that we can then really feel confident about like whatever our patients are telling us outcome wise is actually valid or if it's like, nope, this is noise that we're getting or is like routine, you know, placebo effects, different factors that might be contributing to the outcome. But the request, the request was just to, it sounds like this very nebulous broad request to create this product that's going to be, it sounds like mostly asynchronous, so it doesn't require an actual therapist or healthcare provider that's going to be this tool. And so you have to, so your team has to develop simultaneously develop this tool and develop the evaluation of the tool in, it sounds like on several dimensions, both is it helping the patients, how is the role like all this sort of stuff and you're doing all this develop. So you had to put together, it sounds like some probably very lengthy proposal that includes all the aspects of the tool, the evaluation and then submit it to see if you get to the next step of funding. No, so the funding was given, I was hired, we developed this and then we rolled it out and then we reported out on it. Again, the trust was there that we, the three of us as a team, were going to be able to deliver and all these things with this and different river. Of course, we were keeping everyone updated as we went. But it really was, you know, there was the conversations where I'm running, aligning what the business needs were as they're thinking about scaling within Optim. If this works, the idea is again, like we want to bring distance, many of our patients is possible. So keeping that alignment with that like business goal in mind, like what are also the other rigorous metrics that you want to bring to the table and why and making sure that they're all aligning vertically with each other so that it's not in a position where like, well, we can do this because, you know, like for our patients, it's not good or for, for my business process, it's not good. Like there was, there was no misalignment, which I think is like a common, sometimes assumptions that people make. Like, of course, you're working for, for, for a profit company. Like the intentions are going to be not for the patient. I'm sure there are many cases where that's true. I don't want to say, because I don't know all the cases. I'm pretty sure that that is true. I'm sure that came from somewhere. But it wasn't my day to day, like, and it wasn't my day to day for years, like the whole time that I was there that was not something that we were. There wasn't ever a decision that we felt that we had to make that will compromise clinical care that will compromise scientific rigor that was like the research needs were at all with the business needs, like that, that was, that was not a thing, like, you know, you were lying, you collaborate, you work together. And then the, what is different is like the language, right? And like, how do you think about outcomes in a different way? How to talk about it? How do you present it? What are the metrics that are of interest? So two, two questions just to, so one question, this is a, I know the word vertical is like jargon in business where old and I hear people say it, but I really don't know what vertical alignment actually means. So what does that mean? In this case, I'm actually thinking from an implemented science perspective, it's like aligning all, if you think about the different layers, right, within an innovation expansion or implementation, you have the patient, the provider, the department that the patient is part of, like, then the other layers, like what is that department housed within what kind of a setting, then you have like, you know, other layers like the policy, so that I'm thinking when I say vertical, I mean, that all those layers from an implementation perspective are aligning. They're all wanting the same thing. There's no competing needs to provide it once is, but the patient wants to be. So there's, there are, there are odds, but that the end goal of the product and what we're evaluating and tracking towards, there's, yeah, everyone wants the same thing. And it's not that like this layer wants something else completely different or slightly different enough that will put anything else at odds. So that felt very strong from the get go and that was something that we were, what we kept aligning on as the product did really well on its scales, like we expanded to primary care, we expanded to another organization. By the time I left, we had trained so like hundreds of primary care providers, we had seen thousands of patients go through a program. And that was like what really helped address potentially any like, well, like we can expand right away because blah, blah, blah. Like yes, and like what if we don't have those elements ready on the products side, but we have these other workaround ways, can we still do it? So it made the problem solving much faster and easier because everyone knew why we were doing the things and the output, the program itself was going to be addressing all the needs of all the people that were involved. And one of the, in listening to you talk, you know, I've done some work with like groups working with the government and this sort of thing. And I think one of the challenges is often that the sort of the stakeholders being like the funders and this sort of thing, it's like today they want X and Y. And then so you go and collect those data or do that sort of thing. And then you go back and go to report back on it nine months later. And now they want C and D. And it's like, so you're kind of constantly, I don't know if chasing your tail, I don't know if that metaphor works, but you're kind of like, it is sort of like, it is a little bit like the cat going after the laser pointer. Yeah. Where it's like, you're like, you get there, but then it's gone. You know, and it's like, and it's just like, it's not actually a thing. It's just this nebulous thing. It has, has that, it sounds like you didn't have that experience. No, we did. Oh, listen, the chaotic way of the cat falling the laser. But yes, in the sense that like the outcomes that we were tracking towards did change and they purposely change. I think that's the main difference that I'm hearing, maybe even putting words in your mouth, Dan, a little bit. No, go for it. Yeah. Is if the, if the, my job was to bring up any potential misalignment that was happening. Right. Like if the patients are telling us they hate this, I think thing, right? But then top the top people are saying, like, well, again, we're not going to hire a thousand bajillion therapists because they don't exist. So that you need to make this work. That will be pure misalignment, right? That will just not work. So, but as the program is growing, the goal post has to change that we cannot just remain static around like these are the metrics that were evaluating success or make sure that we're aligning around those, the assumption. And again, this is where the framework was very helpful. The assumption is that as you're thinking about implementation of face-based way, you think about piloting, you think about feasibility testing and that's a main goal. What does that mean? What is it not? Then great, you met the schools. Then the next layer that you unlock when I quote has different things that you want to be able to track towards. And purposely evaluate because you're then at a different point in your implementation phase, your scaling, for example, or maybe you're in the system in face of whatever face you may be, the goal post purposely changes. And then what you're evaluating, then adjust purposely and is additive, but it adjusts purposely to that moving goal post. But again, it's in a very systematic way. And that's why having these conversations across the different stakeholders in a frequent enough way was very helpful. I was at one of the core pieces that we kept intact as part of our program development. We wanted to make sure to bring, want to go all the stakeholders along for the ride, for like a better description so that you they could understand, okay, why am I purposely not doing D E F, but I am focusing on A, B, C right now. No way that hey, if we do ABC right now, we are absolutely going to do D E F next. And if we don't hit A, B, but we do see then we're going to shift to whatever the next letter and the alphabet is as like an extra next thing. So that there's conversations upfront rather than reactive. And oh, this is what you found, this triggers this blah, blah, blah idea on me. So let's try that. Like that is a very different way of thinking about the development of our product, which sometimes happens. And I think it's wonderful. And for some different purposes, that is absolutely the way we should be thinking about things. But for the way that we were thinking about this product development and again, having scalability from the get go was very helpful, it just helps map out your trajectory in a much different way from day one. So two questions. So obviously in a lot of the different steps along the way, you're measuring outcome, patient outcome in different ways. So two questions. One, did you ever use any sort of control, either control groups, office control, any of that sort of stuff? Or, and then two, were you measuring any of the process variables that we're often interested in? So we purposely made the decision not to do an RCT. Like when we're doing the product development, because we, my bias, of course, this is very crystal clear in this conversation today. We wanted to develop something that was going to stick. And in my head, if you do an RCT, then you still have to do this kind of like more effectiveness based like evaluation anyway, because there's so many variables that will change that you need to get to know if it works right now. So we did, we included it and implemented it as part of routine care from day one. That was a purposeful decision that we made. So we did a quasi experimental design where we did a purpose in image controls that we then captured because we had the data available for them. So having made that decision from day one allowed us to have this like, you know, matched control setup. So just, just real quick, so just because some folks might not know what propensity score matching is. But basically, and so these were basically you had existing patient data for patients who didn't have this program, but you basically pick a whole bunch of other relevant characteristics and then you wait them and do all these fancy statistical things to try to match these folks who didn't go through the program who you have this historical data on with the folks who did go through the program. Yes. And we had the control that was historical as well as the control of patients that didn't go through for whatever reason, maybe because he was not available. For example, right away, like for example, we were planning to launch to the next and expanded site, like, you know, one of the things that we have front it was data collection. Great. And we have that, you know, like, put a, put a naturalistically creative wait list control for like a better word that you can do all these like, you know, oh, Sue has blah, blah, blah characteristics. Life's fine. Somebody that kind of matches in those characteristics, but didn't get the intervention in this case our treatment product and then see what's up with those patients in terms of their change. So you're still collecting the same data on those folks. They just aren't getting the treatment. Got it. And were you doing any process, but we would consider more process research? Yes, and no. So less on the traditional like psychotherapy ones, some of the products that we actually ended up partnering with included some of those in house. So there was some like a lion's measures that they were capturing separately that we wouldn't then partner and then get access to to evaluate ourselves as well. We did a lot of like more engagement metrics like are the patients like using it. So decision treatment utilization things like that. So and then there was like other variables like satisfaction, like, you know, self efficacy things like that that we were not again, not captured for all of them, but it was captured by some of our partners where he made sense. So it's again, it's that it's our why on investment, right? Like it's where are we collecting a lot of data knowing that the patients are going to be frustrated? Like, you know, I'm here to do, especially if it's an async kind of a product, like I don't have my therapist that is going to be telling me like, I have reviewed what you have given me as an outcome measure. And this is what it means. And let's talk about it on the second session. This is like three in the morning. The patient is like pulling out their app because they can't sleep because they're having like worries. And then the app is helping them in that moment. So like, you know, who is then going to collect what data for what purpose? Like some of the product themselves like wanted to understand like, how is the patient relating to the digital tool? Like, you know, so the modified under like version of the alliance, which they're measures for. Whereas like for us, it was more around like, is the whole program that we developed to the ecosystem of a program that we developed that houses all of these different digital tools that have for enough to you. So the questions are a little bit different because of the product that we were also developing. Yeah. And then based on some of the feedback you got, were you, did you use some of that to tweak the tools and then did you find that some tools maybe weren't very helpful. So you started in a jacket. Yeah. Okay. All of it. We just started with the last question that you asked. We actually developed an evaluation framework in house to decide which framework, which are the products we're going to bring into the ecosystem. And then also which are the products that we're going to get to quote unquote stay. So there was like a benchmark that we had set out that the products have to meet before they could even, we approved them to bring onto our ecosystem. It was one of the again, rigorous science evidence being one of the key engines for what we developed. We wanted to make sure that there was trust that the providers, the patients could just trust the product that we were bringing that like we had done the vetting for them. You didn't have to go on research or you're like we did the in house researcher team, PhD level researcher team that was encompassed by the clinicians, physicians, like all the different members like had done this vetting that was rigorous was IRB approved. Like there was all this like compliance conversations that we had at the beginning that you know funnels a very few select into our ecosystem. And then also we continue to evaluate them to keep them more known and there were some that we ended up asking them that you know not the best match for our patients. So we're not going to keep you. So we ended up like you know removing the ecosystem. And then there was a lot of people that we heard from are provided for patients. Hey, I love that you have blah blah blah targeting blah blah blah. I'm here. What about our PTSD patients? What are our global patients? What are our global patients? So that also then expanded like the scope of what kind of conditions are ecosystem targeted as well. And we actually have a paper out that describe the human center design process that we engage to adapt the product from a behavioral health provider oriented product to primary care. So like that was also a very rigorous way that we did the adaptation process to solve in three months. But like again, it's like the rigor and the speed do not have to be at odds with each other. They can absolutely work together. And you can still also share all the lessons learned. So hopefully other people will not make the mistakes that we did. And then we'll be able to share some of those like you know nuggets of knowledge. Yes. So so my right to say that like as so like there's sort of you can think. Tell me if I'm right in this. You have like the the ecosystem that's rolled out and then you have a whole bunch of tools modules, whatever you want to call that are like products that would like go into this ecosystem. And so that you're the there's some sort of like there's the development and evaluation through several steps of those products. And basically if they pass muster in those first several steps, then they get chucked into the larger roll out, but not into so they're sort of a you know, business might just call that like a pilot, even though we might call think of a pilot in terms of a study something a little bit more specific. But like they have to make it through that out of beta phase before they get into the larger roll out that goes to everyone. So yes towards the end, as we go bigger, there was some some situations where we again exactly what you just said that we were launching some of the products to some of the providers or teams, but not the whole people. The evaluation was based on historical information and rigor that the company is provided. So for most of them actually with the evaluation based on what the information and data that they already capture. We were being very biased in our selection. We were purposely sending the bar like if you were just coming to us as part of the conversation with like marketing engagement metrics, not enough. So these were these are people outside of your organization that were basically trying to sell you products. Yes, yes. And we were the ones that were saying yay or nay based on like okay, what is the record with which you're ready doing a lot of the work on development on your side so that we could really partner with those teams and companies that were again our own bias. We're sharing that that we believe that science and research and data have to be at the infrastructure of what it is that we provide. Yeah, okay. So I want to go to sort of like the I think would be your next role, which is your current role. But before I do like how you know when you're working in industry, these products aren't available to everyone, right? They're available within. So like how was that or is that for you sort of knowing that the you can talk about the product and do research on the product, but that's some of that like you know you can't just say hey go to my website and you could download this app and check it out and try it. You know like it is proprietary. So like I don't know what was that like for you? What is that like for you? So you're absolutely right, like you and some of these like they're there's even like extra nuances like some of these are actually not even available direct to consumer. Like you know the business model of the company is such that like they only partner with like pairs or you know so you only get access through it through somebody else. And some of these are like yeah you got to pay up funds in order to access them. And the product that we were developing in was also you know way even more I will say adding another layer like we were developing for the patients that were part of this like ginormous infrastructure but still an infrastructure called all them. So if Master no one wanted it, we we we part a lot of conversation, right? Like it wasn't that it was like we were developing something on an academic lab and then oh it's good. Like great we want to make it freely available right things like that. So I think it's like for me it was okay because again part of the conversation was the intent is to scale this largely to as many people as possible at some point. And it's a matter of like then somebody pays for something at some point like it's a matter of like when in the journey of a patient's experience is doing the payment. Like so if you're a patient who is sick in traditional therapy services like you look up somebody and you have your insurance pay for it. Technically they are still paying for it because they're paying their insurance right and they have a copay or whatever but somebody is making a transaction monetarily to pay for the services. Same thing here like it's just a matter of like who is doing the upfront cost you know like payment. The good thing about what we were developing at the beginning was that because again we were funded by this innovation grant. It was free of charge for patients and providers. As we were developing this part of the big discussion was like okay what is the business model going to be that is going to support this so that we can continue to grow it can continue to develop. And then who is going to be them making the payment is the patient directly. What patients even be willing to pay this to your point of hand like okay if they are how much five ten fifteen dollars or no two dollars one dollars like what is that sustainable. Are we going to have the provider groups pay for it I guess an upfront cost there's a lot of models that do that. Are we going to go the pay or app knowing that it's going to take a look forever but that we want the insurance companies to be able to cover for something like this is your billing code that already exists that we could then just say yeah we're going to use billing code blah blah blah and then be done or is this something that some of the companies do this we're going to develop a brand new billing code and then use that and then that takes a different level of conversation so it's those conversations might have they're not at odds with what you develop like right if the thing that we develop was we purposely use like a research and rigor because and all the publications looking with it was because then it really can be a business advantage for your product you are showcasing to the market this is different this not just like everyone can put in a web page and say like I do blah blah blah no no no like we are going through the highest level of screening and telling you that it works so then from whoever is going to be paying for whether it's the patient themselves the provider group the peer group they will then use that and you can use that yourself as a way to say I deserve a premium price and then be part of the negotiation conversations as you as you're talking one of the things that I'm thinking about that's very cool about what you're talking about is you know when a researcher at a you know a more traditional researcher develops some tool some product and does you know does all this development and actually does the rigorous evaluation and all this sort of stuff at a often time to just kind of dice that it's you know it's great idea and like you know and I'm talking about you know the legitimate very serious great things that are developed which there are many yeah that but what's really cool about what your experience was at least how I'm maybe tell me if I'm wrong but is that you all spent years developing this thing refining it improving it blah blah blah and like the organ the very large organization that you were part of as you like they're still using it and will continue to use it and maybe refine it improve it adjust it whatever and so it continues to help countless people over the years even though some you know in some cases everyone on the initial team is gone right they're not a part of this anymore but it's still their work is not in vain you know it's still helping people which is very cool yeah there's some core team members are still there but to your point that is not the part of what is not dependent on each individual person right so yeah so yeah and they but they don't have to be there like they could be gone yeah right and like exactly and it's still in the pipeline yeah and it's inherited and that it's still helping you know dozens of people a week what hundreds of people year thousands of people whatever it is right like that's very that's got to feel good yes this yeah that's cool tell me about where you went after that so after that I again at some point I knew I was gonna leave like I told my partners from day one um if I've done my job properly like you don't need me at some point very soon right like again I was brought in as this like person that's gonna help to build something and then implement it and scale it once the ingredients are there the blueprint is there you don't need somebody like me anymore and that's great but that means like it's like when we tell our patients right like you know uh we'll know that we've done a really good job together when you don't need me need to see me anymore and we're done so um so as we were reaching that endpoint again very very fortunate in where it was where I started to look for my next position with that with that being in a place where I didn't have a job which was again very very fortunate same process I was looking for something that um maintained that like rigor and and it's even anything even things even more because I had just been rewarded right like I think the couple years the two three years that I spent there like they rewarded the rigor of science that we brought to the table as a key differentiator for the product so this mentality of anything was like even more stronger like we there exist other people that are doing this because the market was telling us that like yep this is a thing um so I ended up finding this position um I'm part of Equip Health um which is a company that is providing virtual treatment comprehensive virtual treatment for um eating disorders um for all age groups or for children, families, adults um and the um my title literally is BP of clinical excellence and my job is to do psychotherapy process and outcome research the questions that my team asks are very much aligned with how are patients doing are they doing well or not in what way how do we even measure well like what does that mean is that really actually capturing what our patients care about like or is this capturing something else where I'm missing the boat how do we understand this what is the um best way that we can optimize data collection how do we introduce measurement risk here within the infrastructure or not introduce it but like strengthen it because it's already here and that's literally what I'm getting paid to do right now um and Equip Health this is not a brand new position in the company this has been at the company from day one this is one of the things that have really really attracted me to the company we've been um I had known about them for a long time um and it's been I've been there about five months or so um it's been really really fun like it's it's really again maintaining all it feels like I'm bringing all the knowledge for the last many many years that I've accumulated um and get to really um use it in a way that is helping literally thousands of patients because Equip Health has scaled so large and is so bait it's like in every single state in the US um it's a different company than the one that I just talked about um because it's a provider-led company so it's telehealth provided services you as a patient see humans so it's all it's all synchronous it's all yeah there's some async components but those are like supplementary to the um like provider-facing work that you're doing so like some like applications or these sorts of things that are like yes some like you know for example videos that you would watch to learn around psychoeducation things like that um but the bulk of the treatment the majority of the treatment is done through the work that you do with your providers and you have therapists you have psychiatrists you have your med team you have um also your registered detections like you have mentors you have peers like peer mentors um and family mentors so it's a really comprehensive set of like very dedicated humans that get to know you and your family what's going on with you and then they provide that care in-house so that you don't have to also leave you know like stop pause your life and then go somewhere and then come back like you can actually continue with your life and then get um evidence-based treatment care is it is it like a relatively uniform like protocol is it is like the patients that we see are all a struggle with all in-disorder so the protocols are um you know geared towards whatever eating disorder symptoms that patient is experiencing and then the patients get the evidence-based treatment for that eating disorder so for example if you're um struggling with an arrest interval so under 18 you will get blood treatment that is the gold standard like a family-based treatment if you're over 18 depending on the age you might get CBTE for example so it's really it is protocolized it is evidence-based treatment and it is personalized in the sense that like it's not just like one treatment is treatment that we know the field is telling us that um is the gold standard or one of the gold standard treatments for that eating disorder so so more or less maybe not exactly but these would be the that that the treatments are based on the treatments that show up on like the division 12 uh the clinical site APA division of like the uh I don't even remember it's called now but like the evidence-based treatment the treatments at work evidence-based treatments it it every five years the name changes like that's sort of the foundation like the gold standard that you're very much pulling from and obviously it would vary you know a patient needs to meet with like this patient seems to need more uh work with the psychiatrist medication management found the gold law but like that's sort of the foundation of it that's absolutely right and again the the you will get the holistic team in-house ad equips so that you don't have to basically have a therapist but then you have to go find your own you know psychiatrist or mid-provider or dietician like you know I've you have this like coronation problem like everybody is in-house of the company so got it and so it's all remote so there well so there must be must be a lot of what there must be folks within the company that their job is just kind of logistics right so like uh uh uh uh uh uh uh uh uh uh john smith just started the treatment so then there's a whole lot we're going to have to do in terms of evaluation and then to base on that evaluation that means we need to pull we need to get this psychiatrist and this psychologist or this social worker or this what are this nutritionist or not right depending on so there's just a lot of logistics and the ideas it's all one it's all in-house so that that one kind of case worker i'm sure that that's not what it's called but case worker type of person is working with john smith to help make all of that happen in the way that would be best for john smith yes cool okay so what so what's so talk a little bit about your role there what you do i mean i know you said very briefly but yeah so there's kind of two very complimentary um but few pieces that we bring that that the team is in charge of like again one is all things outcomes like we are the team that is um evaluating our patient's getting better or not what does it mean to get better or not what are even evaluating better or not properly how are we even evaluating our patient's getting better like is it the right appropriate measure what is wrong with the measure should we improve the measure what do we do with a measure all the things so all the things that normally we'll follow in the number of traditional psychotherapy psychotherapy process and outcome research um we do in-house um we have a wonderful data science team that we also in-house that we work very closely with we also collaborate with a lot of different academic institutions and labs um to be able to answer a lot of discussions and gain a lot of the you know expertise that all these other team members bring to the table and then we bring that data and then we do two things like one is we absolutely bring it and socialize it in-house so that the clinical program team which is the one in charge of like making sure that the protocols are being um you know delivered in a in high fidelity that the ones that are training all the providers the ones that are like making sure that all those things are working properly we bring that back to that team and say hey you're you're doing treatment A for patient blah blah blah group great we're seeing one and we feel confident that what these outcomes are going to stick they're working well let's do more of it what can we do and then we also identify the non-respawners because obviously there's going to be non-responders um in certain cases so we under try to understand what's going on with those patients like what are the things that are leading them uh leading them to non-respond what are what are we learning from the data that we bring back to the clinical program scene so that they can make uh using the best clinical knowledge available again also partner with the outside um experts as needed like make the edits that will be most beneficial to really help every single patient the goal is to help every single patient that comes into our doors um so that's where we spend asking way too many questions and spending way too much time thinking of that and then the other very fun part of it is that our uh team is in charge of like dissemination mostly through academic conferences presentations we do a lot of like paper writing um so just to like make sure that we are again using that science rigor as a differentiating piece to the market but also as a way to keep our in-house work like the cycle that's what people look like really at the highest rigor as well like you know if um our goal is to publish on this like we are going to be very rigorous and how we evaluate ourselves we're not gonna cut corners there's no like uh you know none of that stuff like no there's established benchmarks there's establish information in the literature we're gonna want to disseminate that showcase that so of course we're gonna be feeling confident and also that when we say blah blah blah it really is blah blah like whether it's a good blah blah or not something blah blah and then what do we do with it so do you guys do any evaluation of therapists um we do yes so we are especially because you know the we do a lot of the in-training in-house so we want to identify like okay these are the providers that are looking at like so what do we what can we learn from those providers that we can then make it part of the standardized training program in-house and then also we want to understand like okay some providers need extra help here or there and that's totally fine so what can we do then as part of the ongoing training I'm going to provision model like what are the extra resources that we can provide to our providers for them to actually be feeling confident in the work that we do everyone adequate that is a provider is a full employee that we don't have contractors which means that it's um it's like the company is motivated and there's alignment again that vertical I mean that we're just talking about like there's alignment from a business need to make sure that the providers are feeling confident that they're taking care of that they're getting the support of resources and training in order to do their job because then of course that's going to help our patients which is going to help the company do well which then we can then bring it back and do that evaluation to really again not in a punitive way um but to really help the provider hey you're doing awesome with these patients with these patients maybe what is going on here like what can we do to help you feel like you can do more of at the same level of outcome achievement as your other patient groups and we do the same level of like rigorous analysis with our patients at the aggregate level of course like like what are the patients that we're doing really good at and like what are the factors that are contributing to that versus like you know some patients were like doing okay so where's it what's causing the delta and what are the patients that we're like we have a lot of learning to do still and that's great and then what can we learn that it's actually going to then translate to actionable next steps um for us to do better yeah I was thinking that you know one of the you know we have so much evidence that the therapist matters and that you know some therapist are better than others some therapists are better with some patients some types of folks more so than other types of folks but we do know that there's a lot of between therapist differences and we do know some of the some of what those between therapists differences are right like you know obviously the facilitator of interpersonal skills is getting a lot of traction right now and everybody's talking about and like that that differentiates good therapists from less good therapists more and less effective therapists perhaps I should say I still think there's there's a one of the things that we really don't understand is like how how we can help therapists who are struggling either overall or of a certain types of patients like how we can actually help them become better like I think there's a huge gap there just because most people don't have the kind of um ecosystem yeah that you're working with in where you actually have the opportunity to do that work and actually try different things be like oh this really helped these therapists this didn't help these therapists and then also to find out where sort of the limitations are where maybe first you know I always say like the world needs accountants and so like some therapists are just suck like they just do and like that just like people suck at every job so that's okay so like to figure out where those where we butt up against those walls where it's just like yeah this therapist just this they need to be an accountant they need to be an engineer they need to be whatever this is not for them but then also like where are the places where we can help therapists become better and then how to do that more effectively so it seems like there's a very cool opportunity there to actually do stuff that's really going to help the field that that's a hope for sure like and the wonderful thing is that they infrastructure to do this kind of a research again was was a ready one from day one was there adequate um I'm not gonna take credit I've been here again five months I'm not gonna take credit for like oh yeah then I said it they're like you know learning health is the infrastructure in five months I am that one it was one of the key reasons why I took this job is because the potential for all of this and the company has already been doing a lot of research to address some of the key questions in the field anyway um so it's a matter of continuing to use that research learning health system that is existing to ideally answer some of these questions that will then help some other you know setting some other organizations some other providers in other practices so yes for sure yeah hey are you guys like one of the things I often think about is that one of the things is in psychotherapy that we don't have like people develop these tools that automate uh you know they use machine learning and all these sorts of things to train these you know train technology to be able to do this coding right and like but usually it's just kind of like one-off projects which are cool projects and it seems like and the type of stuff you're doing now particularly because it's all online that you could create some of these tools I mean it's a lot of work but I mean I don't know I'm with all the AI stuff it's getting easier I really is like I've I've coded several apps already and a website two websites I don't know like you know I'm having fun with it so if I can do it anybody can do it um but uh that like have have you guys thought about any of that in terms of like developing some of these automated tools that could just do some of the coding which could be used to give feedback to therapists or to other treatment teams about clients to mod of but then also to then you know be able to take some of this incredibly laborious coding stuff that we often do and just to automate that or even you know you could train a tool with patients ratings of the working alliance or whatever happens to be yeah so then the patient no longer has to fill out those three questions at the end of the session do you guys thought about we're doing all of the three things that you just mentioned oh great different capacity um there's also different companies that are doing different aspects of what you just said yeah and some of them are members of NASPR um shout out to the listen team so um absolutely that is in just to kind of end the loop of what you were saying and then we're evaluating it right to then to see is it working is it not working where is the model failing then we use that to then retrain and then regather data and then like continuously improve the models that were developing in-house but also then see is that making any changes to how the providers are delivering care is it having an impact on outcomes in what way um we are planning to submit something for the next year's NASPR so a SPR conference so be in the look at for that too but absolutely and this is I think one of the wonderful things is that I get to just part of my job is like we have so much data which is a rare problem to have insectotherapy process and outcome research I think one of the few um places where we had this like problem was when it was part of I was I had the fortunate experience of working as part of the Center for Collegiate Mental Health like a practice research network which again has so much data similar situation here so um one of the wonderful things that I get to do is get to meet new wonderful researchers that have wonderful ideas and would like to partner together to ask and research and answer some of these ideas together um so totally like all the things that you said we're starting to do it is we're not obviously done and so if there are other people that are interested in working with us together to evaluate and research any of these aspects we will love to work together too so all right um okay I have like two hours more of questions but let me just since we're since we're pretty much out of time I just wanted to like so what advice would you give for folks who want to who are interested in doing psychotherapy research but want to do it in the private sector there's um I really like what you said so I'm gonna say two things and borrow um something that you said earlier if that's okay the we I think still end up in this bubble by the end of our training that like yes of course I know all this but like a lot of people know this like that is absolutely not an assumption that people may should make and you you have said it beautifully then so the first starting point is like learning how to educate other people outside of our field that this is something that we bring to the table that it is truly a differentiating factor of skills that we bring and that there is a ginormous space for this skills and need for this skills especially again where the uh digerman to health field is and I use the word verb "body" um in terms of needing people that can use those skill sets in order to do a lot of the stuff that I'm doing right now so that is like the first thing that people I think make assumption like oh yeah I'll either not explain properly or maybe they don't need that anymore or they don't necessitate that they don't they make that assumption and shut doors automatically in their brains like absolutely not like you know totally um know that that is a key benefit it's a key knowledge base that we are experts on that not that many people have and that it really is very very necessary and you can have real world applications in very tangible ways that still maintain again if you're biased like my own like the love for research on all those all those questions that come with that and then the second part that I have is just if you know like at least when I was going through training like I was told to pick like you know you got to do this or this you got to academia or private practice or this or that absolutely not like I have my full-time job I have a small private practice I'm still supervising at MGH like I'm on the board of other nonprofits like there's a lot of different avenues with which you can also apply the skills so if someone is not ready to make like full transition great you don't have to there's a lot of like different ways and models where you can actually do this work um so just keeping the the options open and as long as they're okay and they're flexible like there's a lot of different options that can be that are already available or can actually be constructed which is like kind of a little bit of what I ended up doing for myself with a little hodgepodge of here there thanks right for sure yeah that's this great advice and it is interesting I you know I often say to my students either my supervisors or you know that I'm working with or in my classes often say you know I have lost all perspective on like what is common knowledge and what you know and it's like because you just do right like when you when you're doing something for 10 20 30 years however long right and that it's like you're just like that thought everybody knew this but then when you actually start talking to people even people in the field right even other even people who are psychotherapists or whatever you start talking about something and they're like I I don't know anything about that you know and it's just like so that you're you're right I think we do often think that a lot of the stuff that we do just regularly and assume is relatively common knowledge isn't so yeah to give yourself more credit that's a great a great point that so the last thing I always ask folks and I'll I'll have several links to papers and websites and stuff but is there anything in particular that you want to point listeners to resources or anything for that either want to learn more about your work or just what's happening in these areas in general or anything like that um absolutely I'll send you all the papers um they can also message me they will like um and um I think like nsbr and isp are again we've been doing a lot of presentations um for the work that we're doing right now like we'll be at nsbr and isp are presenting on an quick work as well as like on the previous work that I've done uh rely on a group and opt in so um it's a wonderful network of wonderful people and just attending the conferences like you know um that that's going to be also wonderful resource I think like there's a lot of panels that will be of interest yes for sure now I'll link I'll link to both nsbr and sbr thank you yeah so yeah i appreciate well let's see I can't tell you how much I appreciate this oh thank this is so fun thank you so much for inviting me it's been wonderful to chat that's a wrap on our conversation as I noted at the top of the show be much appreciated if you spread the word to anyone else who you think might enjoy it until next time and see you next time.[Music]