When Research Gets Closer to Practice with Dr. Soo Jeong Youn

[Music] For many psychotherapy researchers, the path can feel pretty narrow. Graduate school, internship, postdoc, academic job, grants, papers, repeat. But there are other possibilities. In today's episode, my guest tells her story of how a psychotherapy researcher starts to become interested in and works outside of the traditional academic path. We talk about psychotherapy process research, implementation science, the frustration of moving slowly inside of large institutions, and how the private sector can become a serious and meaningful place to do clinically relevant research. This is a conversation about career possibility, but also about impact. How do you take what we know from psychotherapy research and build something that can actually reach people? That's what today's episode is about. But first, if you're new here, I'm your host, Dr. Dan Cox, a professor of counseling psychology at the University of British Columbia. This is psychotherapy and applied psychology. Right, talk with leading researchers about what actually matters in practice and what's behind the findings and what they wish clinicians new sooner. If you enjoy the show, please subscribe when you're a podcast player or on YouTube, like and subscribe. That small click mixes surprisingly big difference for the podcast. This episode begins with my guest talking about how she stumbled into psychotherapy process research. So without further ado, it is my pleasure to welcome my very special guest, Dr. Sue Jong-Yoon. So I kind of stumbled upon it. So in college, I was doing more kind of traditional MDD focus treatment innovation research. So I was a research assistant on different types of trials that were looking at innovative ways to address MDD, so like new medication types, new like dream mobility, so like RTMS, like one of the first trials that was going on. That was a research assistant that like back in the day, totally dating myself here. But that was like, okay. Let me just stop you just real fast. So acronyms, MDD, most listeners will know a major depressive disorder, but then otherwise. Yeah, RTMS is repetitive, transpranum, magnetic stimulation. So it kind of looks like this flat thing that you put on your brain, and then it sends waves that supposedly stimulate the brain, and it's alternative to medication for treatment for resistant depression. So the idea was like I wanted to like depression-focused research. Like that was something that I was very interested in doing like research from the beginning. And I just continue with that. So did a lot of research assistantships when I was in college. Moved to Boston, post college, to work at the Depression and Clinical Research Program at Masternau Hospital, which is a big R1 institution in the US. And that was literally focused on doing a lot of innovation research and trials, focus on treating depression. As I was working there, like one of the rules of the RAship is to like talk to the patients, get them set up for data collection, you know, like make sure that they show up for their appointments, walk them through where they need to go for, you know, the various aspects of the trial. And like what stood out to me was, you know, they started to ask for specific providers. Like, you know, again, as part of the trial, you know, they're just literally sometimes doing assessments because they sound like pharma trials. So it's not that you're doing therapy per se either, but I actually want to meet with blah, blah psychiatrists or blah, blah provider as my part of my like 10, 15 minute check-in that you they had to do as part of receiving this like medication. So that really, like, ping something in my brain, like, so I started asking them, they just like, oh, why are you asking that? You know, knowing that obviously we were not able to move them, but I was just personally very curious. And they started to talk about like, oh, I really feel like connect with this provider differently. They seem to really get me. I feel like I can actually talk to them about what's working I was not working and not just feel like I'm just getting rushed through. So I literally did have no idea that this concept had been around called the therapeutic alliance for a long, long time completely stumbled upon it as a 90 if 2017 thing. You know, I mean, like, what is this? This is so cool. Do people know about this? Obviously people knew about this, but it was like brand new concept for me that really resonated with me. It was also an interesting time because the placebo effect in the farmer world was like exploding at the time. So there was a lot of conversation about like, what are the different factors that might be enhancing placebo effects that are obviously mitigating the impact of medications and blah, blah, blah. So it became this whole like, it started to kind of intertwine itself in a very interesting way. So like, I started doing a lot of research because part of the job was to do like presentations as part of the continued education for the different RAs. And started reading a lot about the therapeutic alliance and just started to get really into it and started to learn about this world. I was like, the therapy process research and started to learn more about like the different mechanisms of change. What is that? What do we do? What do we know? What are the pitfalls? So midway, I pivot it. Like I was like, okay, if you looked at my resume until then it was like very heavy MDD, like major depression disorder, like focus research track. And towards the second year of my like job in this case, like I was like, I don't I don't want to do that anymore. Like I wanted to do this very like, like, delete it, but different focused area. So, did as much as I could with what I had, I like asked to volunteer to different labs that were within the hospital that were focusing a lot of this kind of research. And then, you know, just literally asked for a miracle, basically. I applied for PhD clinical site programs. And I was very honest in my statement. I was like, I know I don't look traditional. I don't really have experience. I don't know if that's the right way to have done that. But, you know, like it just went for it. I was like, you know, they're going to see this discrepancy of what I'm saying. I'm interested in and what my CV looks like. So I'm going to just put it out there. And my advisor, the person ended up being my advisor, Luis Castingay, literally took a chance. He invited me to interview. He and I talked about it like from the very beginning of my conversation. Because, you know, obviously, I was so nervous. I just like sleep learning. I know I'm not a traditional researcher in this area. I have no experience. But this is why I think you should talk to me and, you know, think about me. And it was a really, really fun conversation. Like that hour flew by. I did not feel like an interview at all. I don't know if it's a bit of a bad thing in retrospect. But like, you know, just like we fell like war vomiting. Like we're sharing ideas. This is awesome. My brain is so stimulated. So that's how I like I got into it. Yeah. Okay. Okay. So how many years? Yeah. Like in hindsight, you recognize that very few people coming out of undergrad actually have like psychotherapy process research as like part of their CV. Well, yes and no. Like I want to pen undergrad. So I actually don't have an excuse. Because there's the Beck Institute. There was also psychotherapy process and outcome research center that literally the offices were right next to the offices where I worked as sort of the depression anxiety team there. So that's very nice Dan. And I have no excuse to share except for the fact that I was I should have probably looked at right a little bit more actively in undergrad. But it was like very much like one one check mine in at that time. I was like, I want to do this. This is exciting. This is interesting. Until I found something else that I was like, oh, that is that seems to be kind of filling in some of the blanks around like what I was starting to notice in the type of research and some of the findings that we were finding and the limitations that we were finding that type of work. So part of what you were trying to notice, they were starting to notice was that the therapist seemed to be particularly, or the psychiatrist in some cases seemed to be particularly important and relevant and seemed to make a difference in terms of treatment patients experiences. For sure, the patient's experience, like that and he was definitely like how they felt after doing an after the trial. And even in a very controlled setting, like really where you're just going literally through measures, right? Like that are these are providers that were part of a very control trial and the fact that that was still something that the patients really felt enough to make to make ask, right? Knowing that these are the parameters in which they were receiving this care, like really stood up to me. Yeah. So then you went to Penn State for graduate school. And so your pursuit of psychotherapy process research continued but extended in some ways. It did. So we started with a lot of process research questions expanding to outcome, of course, as well. And asked all the questions that fall in that big umbrella, especially related to like what makes therapy work. Obviously continue with this interest of mine. Like a lot of my focus was on therapists of hex, like understanding like what makes them therapist different better, not better. What are the contributing factors? How can we train therapists a little bit different or augment the trainings that they have to really enhance outcomes? And then also learn a lot about practice or any research. And so that's like another big expansion. In the availability of research knowledge, field, way of thinking, even like a paradigm of way of understanding how do we even evaluate research, research outcomes, or how do we even think about research in a non traditional controlled way. And that really, really was just like, yeah, this is for me. Like this is the research that I really, really want to do more all. So what was that like? Because I mean, it sounds like you came from at Penn and then when you move to Massachusetts, it's like much more of a controlled research context, right? Do it in my control trials, all that sort of stuff, right? And then you go down to Penn State and all of a sudden, there's this guy who's working with people in the community, trying to figure out like what are the practitioners in the community care about, work with them to figure out the research questions that they care about, and then doing research in the in the real world, right? In the consulting room in the real world, with practitioners in the real world, not in a controlled context. What was that? Yeah, what was that experience like for you? It was, it really felt like, it just felt very natural. Like the kind of questions that came about the way that you would do research, all the problems that come doing, you know, that's part of that research umbrella, the limitations. They didn't feel like things that we had to solve. They felt like, well, duh, like this is representative of the real world, so why would we try to change it or fix it? Like we should understand how if anything that is part of the real picture and how that potentially impacts what we're seeing in the questions that we're asking. So it felt like a very, also very biasedly, my brain is like, it's, I don't know, my brain likes to have like something tangible that you can hold onto to be like, okay, I've done this. So this felt even closer to that. I'm like that whatever outputs were going to come out, all that research, the answers that were going to come out to the research came with less with different limitations, but less limitations around like, okay, in order to actually then take this and give it to myself as a training, for example, at the time or a practitioner that is out there for them to actually use it in the real world. Like there's way less many edits that they need my help, but I'm like, well, if I edit it in this way or that way, I don't know if the things that we just were reading about or still in a hold or not. So it felt very very like much more unlining with like my, how my brain works. Yeah. That's interesting. As you were talking, I was thinking that when I was an undergrad, like my supervisor was a cognitive psychologist. And so we did very lots of the types of studies, the cognitive psychologists do, reading lists of words and this sort of thing, right? But it was very, very controlled. And that when I went to graduate school, I had some difficulty adjusting. Like I had some of that experience of like, man, some of this stuff is just so uncontrolled. Is it, is it valid? Like is it worth doing? But doesn't it look like you had that experience? I definitely didn't have a question around like, is it valid? I think the questions were, there was totally an assumption and my bias that it was bringing to the table because again, my brain just loves, just attached to this, me like this, I don't have to work hard to like understand this information that is coming my way. So the validity was something that I was like, well, yeah, like, you know, like all the problems are there and within all the things and all the noise and all that stuff, we're still going to be able to extract something that is giving us directionally. Okay, this is helpful for your patient if you do blah, blah, blah. Great. Then it just felt a little bit more that I could read something that would come out of that kind of a research and then take it to the room with my patients more readily without any like talking to my supervisors in the gym and being like, okay, wait, so if I make this or that change with that result still hold or do I need to then think about this other stuff and how that kind of connects together. So it felt again, the validity, maybe it's an assumption that I was making in how it was consuming that information, but it just felt very easier for me to grasp. Yeah, just because the distance from like research to practice was so much closer. Yes, yeah, again, bias, maybe again, I can talk about like, does that real or no? And what does that mean? My brain was saying like, yeah, that is there is no space. So you should totally just accept it. So yes, yeah. So I want to, you know, ultimately get to how you got into doing private sector stuff, doing psychotherapy research and the private in the private sector. But so where did you go? But I'm curious about sort of your journey, sort of where you went after Penn State and sort of the path that you took. Yeah, so after Penn State, I came back to Boston, which is where I'm currently living. I did my internship at a DMH site, which is to permit me to help publicly funded it's severe mental illness, like focused kind of a practicum or internship. And for postdoc, what I ended up doing is joining an implementation science lab at Mass General. So it was kind of like full circle for me coming back to my academic home, maybe you want to call it that way. And that was like the last piece of the puzzle in my brain. So it was like, what case if I'm interested in really understanding like what are the patients experiencing, what is happening in the room, like, how do we think about research that is going to be ideally action move right away, like, you grab it and you bring into the room without needing to do any changes in modifications, then multiplying that to make it be impactful at a public health level, the implementation science was like the natural next extension of that research. So you ask very similar questions, right? You ask questions around what works for which patient, like what is effective. And then the field is asking then questions around like, okay, then how do we bring that truly to the real world settings and different kinds of settings. So it was kind of like, again, that assumption that I was just telling you about, like, oh yeah, I'll just read it and then bring into the room. There was a whole field that was looking into like, how do we actually make that happen in a way that is going to be not in this individual way, but at the public health level system change. So a lot of questions are like, what are the dynamics of an organization that we should be thinking about in making innovation happen and stick around for a long time. What are the system level changes that we need to put in place in order to really maintain and support this kind of infrastructure changes that potentially we need to put in place for innovation to really stick. So that was super fun. Like that was so so fun. I worked with my mentor at the time and who became my very close colleague Luana Mackeys there. We had so much fun. We, what we ended up doing was really thinking about the fact that like, there's a lot of stuff in psychotherapy research and then to help that we know works. There's enough evidence for us to feel confident that we're not doing harm and that enough confidence for us to feel like, okay, we're actually doing something that is helpful. So then why keep it within this very specialized relationship only kind of a setting, right? That was also the time when the youth mental health crisis started to really balloon and became like a really widespread knowledge thing and like really a lot of attention was being paid around like, what do we do directly help you make a dent in the US, the health system is still very fractured. So thinking through together, like what are the ways that we can then bring some on this knowledge to non-specialized settings, like you know, really thinking about what is all the jargony stuff that is in our field, that we have a lot of it, that is helpful for so many things and what kind of order all the things that we can get rid of without impacting the core mechanisms of change. And then rethinking who can deliver these core mechanisms of change to have a really more widespread impact, both at the early intervention and maybe even like prevention at times, depending on the work that we did, and having a different level of focus. So that was also super, super fun. So it felt like the two worlds were like the leap blending in a wonderful way and you know, really thinking about, okay, how do I bring the knowledge that I had from all the years of training around like these are the core mechanisms of CBT, for example, that work when we're treating, I don't know, depression or anxiety or something. And then how can we rethink those as like modular skills, for example, or classes that somebody can teach us a teacher? What are the elements that we want to make sure that a primary care provider really includes as part of the conversation with the envelope not to actually doing just an intervention, but even maybe getting the patient set up enough to really consider, you know, seeking a specialist mental health provider later. So that was super fun too. Yeah. So at this point, had you started thinking at all about working in the private sector? Nope. Nope. That was still like, I was still very happy in the work that we were doing. I didn't, I always knew that like I wasn't going to be staying forever. I ended up staying way longer than I thought. I thought I was going to last like until my post-doc and then I was going to have to leave. I just, you know, green writing non-stub, it's just not something that I particularly enjoy. Obviously given that, you know, my interest line, something that usually is not easily fundable by grants. But we've got a lot of traction through alternative funding sources. So like, there was a lot of philanthropy that was supporting this work that there were some alternative grants that were becoming available that were not focused on just RCT, but were really interested in this more like translational research type of work. So I did have lasting truly. So I think about it. I lasted a long time. And then what made me start to rethink a little bit was, I think I was like five plus years into this work. Things are just moving very slow. You know, like as much as we, the lab we were working very quickly, we were set up to do things very fast. It's still in a good, then medical center, right? Things work still pretty slowly. And even if you try to run really fast within this ginormous infrastructure that is pretty slow, there's so much that you can get done. And so that started to really get at me. I was like, okay, why are we still talking about doing this, went all the people that want to do it have agreed that we're going to do this. We're ready to go, the designing is ready. Everyone is excited. And there was a lot of different things that we had to just take care of, like, administratively, that just felt like I understand and also why, why, why, I don't, I'm starting to feel impatient. And then COVID, which was horrible, 95, 96, 97 percent of it was absolutely horrible. The two, three percent that was, I think, good about it was that it started to destigmatize mental health conversations. It started to really draw a lot of attention around mental health and in what are solutions that we can then quickly innovate around, then truly mean quickly so that we can do something about this. So I think it was like, there, I'm not going to say that COVID started this whole industry movement, but it really was like you know, it gave it supercharge. So a lot more opportunities started to emerge in terms of like companies that were focused on this. There was a lot of money that was being thrown into this as a solution. And they're started to emerge different companies that you could tell they were doing it differently. They were, they were doing it in a way that was scientifically sound, rigorous data driven decision making. That was not just like, oh, I'm going to put a label on a website and say we're going to do this and then figure out how we do it, but that it was really well thought out like, okay, we know very similar to what we were doing, like knowing, starting with like what does the field already know that works. What is the research telling us that we should or should not do? And then how can we bring that then on the delivery side a little bit differently so it's direct to consumer? So it's not that you need to go through a provider directly in some of the company ways that they're set up, but we can give it directly to the consumer and not having to facilitate them finding a therapist. Although a lot of those also platforms started, yeah. So just to, yeah, so I understand. So what you're saying is rather than like them having to go through some sort of mental health professional to get to these services that they could just like the services could market themselves and people could directly contact those services that you're describing and then start working with them. So there's like different ways that, so there's some companies that are focused on matching, you know, therapists with a patient, right? Then you're looking for a therapist here, look at our ways, the directory, we'll have from whatever will help us to take that match. And that would be like, that would be like, I think that would be like better help, which I hear advertised constantly. Yes, for example, there's other companies that do a little bit things a little bit differently. They provide help for the provider to do their work more efficiently and more effectively so that they can then still be the main conduit for delivering the therapy to the patient, but they're really supporting the provider from a different angle, whether it's like getting rid of all the paperwork, getting rid of all the insurance billing stuff, helping with some like clinical decision support development, so like a lot of the fee monitoring stuff, like, you know, bringing that into the platform so that the provider has access to that information readily, you know, without needing to be, for example, admitted within like a ginormous healthcare setting. So there's companies like that. And there's also other companies that are literally repackaging therapy and then delivering directly to the patient, either as an as an async kind of way that they consume that information and they're literally helping themselves. Or it's kind of like you're helping, you have a few different touch points with a real human, whatever that human may be a therapist or a peer or something or coach something in the between, but the bulk of the work that you're doing in terms of the therapeutic work is the patient through this app or through this device or whatever. So VR headsets, for example, will be a great example. There's also different devices like the Camigo device that it looks like an inhaler that you breathe in and helps really promote the DV, like different diaphragmatic breathing. So there's a lot of different direct to consumer devices and or digital tools that have been developed that were again, be doing really wonderful work. And so that picked my interest. It helped me be flexible in what I was assuming like, okay, research, it only has to be done in this one kind of a setting, but really broke also my assumptions that I was making that industries bad. They only do for profit research, but that there's like companies that are actually doing both in a very rigorous way. And I think the like the pandemic helped really propel these companies like more rapidly, which again, personally coincided with my frustration starting to increase even further. Like I was already frustrated, COVID made it even worse because really we were at this point where then the system that we were getting out reaches so much. And the thing that was as a lab and the thing that was like slowing us down was not our intention or readiness or anything like that or the partner's readiness. He was like, well, we got to slow down, we got to figure it out first. And like a lot of like infrastructure administrative stuff that that again, I understand why it's there. It's important and we need to abide by it. And personally, I was like, I don't know if I want to stay in this kind of a setup anymore. Yeah, so two things. First of all, that's not all important. Some of it is, some of it is, but not all of it. And secondly, though, and I've said this one here before, but I think what you're saying, I don't, it's hard to even remember how extreme the shift was. When the pandemic hit that, you know, I was described, you know, pre pandemic, particularly among my senior colleagues, the idea of doing psychotherapy remotely over Zoom. It was no, I mean, like there wasn't like there wasn't an appetite for it. And it wasn't something that was being pushed very much. But like, it just there wasn't really a thing that much. And then probably a lot of the therapy that real world therapist did that was remote was probably more on the telephone than it was because like Zoom was the, you know, we had Skype, but you know, a lot of people, you know, it just wasn't really a thing. And then the pandemic hit. And you know, I, I had several, you know, folks or colleagues of mine who are now retired, who I saw, it took 15 minutes for them to pivot and go, if we're going to keep on training students, we need to do this remotely. Let's figure it out. I mean, it was, I mean, it was the weeks, maybe a month. I mean, it was no time at all until we figured out how are we going to change all of our training, you know, our training clinic. And it was and those shifts have so much sustained like that, that, you know, the people, it's like, they're going to be those of us who were trained and practiced pre pandemic and those of us who trained and practiced post pandemic and the people post pandemic will just, they'll just accept it as the field is just, it is changed. It is totally changed. A hundred percent. Yeah. Yeah. So, so, so anyway, so, okay, I did mean to, but just, no, it just gets, it's, you know, nostalgicly as an interesting sort of time to look back on it. Even pre pandemic, what practice looked like, what training looked like is different. Okay, so the shift happened. These, these private enterprises popped up and you saw that at least some percentage of them were working really hard to do sophisticated research, or you know, sort of research-based practice. And then you're in this place where you're sort of, you like where you work, but you're also feeling frustrated by the lack of movement on some of your projects. Okay, so take it away from there. So, from the, I think the, the, the, into lecture reasons for why to move like, were there, like they have been there for a while. I'm not a reactive person either for better or worse. So, you know, it took my, my brain a while to be like, am I really going to give this up? Like, am I really going to make this change? This is still a really big pivot. You still an unknown, you know, in, like, you know, thing. What, what would it mean? Like, if I leave, like, are people going to think like, oh, I, she sold her soul. Like, she's living in industry now, right? Like, there's something, you know, maybe in the circles that I lived in at least, there's credibility that you already automatically kind of get from coming, being saying like, I'm working at like, Master in the hospital. It's like very, you know, like, established institution that is already associated with like, rigor and really well thought out research and thoughtfulness. Like, what is going to happen if I join like, ex company, like, that people are automatically say, like, oh, yeah, she's doing it for the money, which also was part of the reason. Also, let's be transparent, like, personally, like, also, it was a big decision for our family to be like, okay, we're at a place where, uh, we're mobility, something to be thinking about. Um, and so the, the very grateful thing that I had going off for me, and I'm truly grateful for this because I know it's not a, it's not an, uh, an option for a lot of people was that I had time. There was nobody, like, I had a job, nobody was kicking me out. I had told my, uh, PI, um, that I was thinking about this move. They were incredibly supportive and they made it very clear. Like, there is absolutely no rush. Like, you don't have to leave, there is funding, there is demand. We're going to be good. If it takes you five, 10 years, you get, do you keep exploring no problem, which I can, it's totally, totally, um, you know, like a very privileged place to be in making this big, ginormous decision for myself. Um, so what I did is like, I, I did a lot of exploration. I was very, very picky. I met with a lot of people, you know, people say, network, I had no idea what that is. So I don't know how to network with people outside of conferences in academia. I don't know what that means. So it really literally meant messaging people and say, hi, I'm being very vulnerable. I am thinking of making this transition. And again, for better words, this clearly seems to be a theme in my life. I over, um, cheer. So I was like, I have no idea what I'm doing here. I am thinking about making this transition. I clearly understand that I'm, again, don't look like this prototypical person that has no finance background, no tech background, no product background, no nothing. Um, but I'm coming with this like need and desire to make a big public health level impact. I have the rigor of what I'm knowing what it means to do good research, good science, what is good evidence. And I also come with a skillset of like knowing what matters, like, you know, like what matters for different stakeholders, not just like what matter for clinical trials, but what matters for peers, what matters for, um, like department heads, what matters for providers, what matters for patients. And I know how to speak that language and have a badminton and I think about it. So I had a lot of people not respond. I had a lot of people meet with them very nicely and tell me like in so many different words like, yeah, no, we're not gonna work together because the world that we live in are so different that it would just take literally too much training to bring you on board, which is totally fair. And I also got a lot of interviews that were very positive. And there was a lot of excitement around like what I was bringing that was a little bit different than what they had on the table, how that could be leveraged and potentially new positions get created in order to bring the skillset into the, um, company and what was going to be the, um, ROI, like the return on investment in bringing someone like myself and my way of thinking my skillset that will be complimentary and help expand the development of whatever thing was getting developed. So, so, so, just a couple, so what these were just like informational interviews that you were doing. You weren't, you were sort of like, hey, I want to chat with you about what you're doing potentially. I'm just trying to figure out what's going on in this sector and this field and if I can play a role. At the very beginning, so this, my exploration started like I think overall from beginning of it, we're in a first send out that email to I went and actually decided to take, accept an offer. It was about nine to ten months at the very beginning, although they were cold emails, like some of them were um, friends from undergrad that I knew, some were um, people friends of a friend, some were like, you know, um, I had people that I knew because my partner is in finance, we knew somebody, we knew somebody, and you know, like again, all we were all home, so we were all emailing anyway. So I was like, what do I have to lose? So send out emails left and right. So these were just, and these were just people working at these different companies? Yes, yes, yeah. So I was, again, I was even picky about like who I was emailing to begin with. There were some companies that I was like, I'm not sure if, if, just the way that they're thinking about the product, the way that they think about what they bring to the mental health field is something that I actually agree with. So like those like just did not give a message. But the ones that looked at least from what I could tell, like promising, or there was something there, I literally just cold emailed or cold message, throw linked in. Um, and then I think about four months into it, then specific positions started to get posted from this company. So it's, so there was a little bit of shift in terms of like how many cold emails that was sending proportionally versus like, well, were the ones that I was actually applying to? And towards the end of my, like searching, it was, I would say half and half, I would say about like 60 40, still like cold outreaches, like, and then that person be like, okay, you know what? We don't have something for you right now, but like you should talk to a blah, blah, blah person, and then meeting that blah, blah, blah person, because of, you know, like all the companies know each other. So, um, so that's like, I would say, what's 60% of the conversations that I was having, 40% was me applying to like the traditional application way, um, to these post-apositions and then, you know, going through like a formal interview process in that way. So in these conversations, when they were, when you, when, when the two of you, and various diets were talking about what the return of investment would be, like, what were the, what, what was it? Like, how did, how did, how was that conceptualized? So, there, there was some of them were like, we're good. Yeah, sure, because they're good, right? Yes, and again, some of them were doing like very thoughtful clinical program development, which, um, if you say you're a clinical psychologist, you get very quickly boxed into that, like, oh, you can be in a clinical team and then help develop what goes into the clinical product. Um, and I can do that. It's just not, you know, it's not like where my juice in my brain goes. Like, it's not super just because I can do it. It's not something that I want to do. Full time, at least. So, um, so then he was like around steering the conversation. I'm like, yes, I can do that. And let me tell you what all the other things that I can do. I can help, um, you know, like we can, I can help you understand whether this works or not. And this is how we would be thinking about how to measure what works or not. And then we will be also asking this question to understand why don't you want to know where to invest your dollar monies in the product development side so that you can really emphasize the elements of the product that are actually going to give you that impact on the patients improving or keeping the patients engaged or coming back, right, to using whatever product you're developing. So helping them think through those because now the field is also a very different juncture. Like that is an expectation at this point. Like the industry is also moving that direction where, um, you know, just engagement metrics, not enough. Um, when I started looking, it was like, there was sort of where even reporting on engagement because they were actually tracking it. They were like very novel, innovative. Like, the field and quickly adapted like, okay, those are like what, um, the funders are now requesting us like standard data metrics. And then now we're moved into the direction of like, okay, standard metrics also, that means outcomes. Like it's not just enough for you to tell me, blah, blah, percent of your patients are still using all your products. Like I want to see, is this actually helping? And then how are you assessing helping? How are you gathering the data? How are you evaluating it? So there's been an evolution in the field itself. Um, but when I was talking, I was like trying to help them see that that was a possibility. Um, which again, if you think about it, I'm not saying anything, you know, like literally I'm trying to take credit for decades of psychothermia from the Malcolm research, but it's bringing to a brand new audience a little bit. And then how to think about assessing it. Like, you know, we can't just ask 4,000 questions to our patients. Obviously we wouldn't do that normally. It's like a theory process and outcome research anyway, but like really being judicious like around like you need to justify each and every second that the patient is spending answering a question is a second that they will be spending not engaging with the product that I'm developing. So where's the ROI there? Like, how do I justify that? So, um, so just a, because I, you said several things that I want to come back to, but I just want to put a pin in those, um, about like sort of where the field is now, but zooming back to that experience. So it sounds like when you're having these conversations, a large part of what you were doing was educating the people that you were talking to. Absolutely. Yeah. And some of them were like, I get it. And, you know, again, not within the scope of what I need right now, very fair, totally okay. And it up wonderful, still connected to a bunch of those, um, founders and then they will like put me to somebody else, right? So, okay, before I have to start, yeah, so like what was like the coolest project or like product that you saw that like never got off the ground? Like, what was the thing that you were like, oh, this is so cool. But it's sort of like as far as you know, anyway, kind of died or opposite something that was so cool that was that ended up being really successful or sort of going the distance. Um, let me answer the question a little bit differently. I think, um, because I think the, the, um, not to name specific names, but like what struck me was, um, there were companies that I talked to that were literally at the pre-seed money. Like they had done around of like funding where he was like friend and family, like, or they had like just started gotten enough money to get a lit of like a 14 going and I was like, oh, are they, are they going to survive? And some of those are the junkwreit and there were companies that were seriously, they had like millions of dollars of investment and ready that I had was interviewing with them and meeting with them and are no longer with us. So like, yeah, so I think it's like what was surprising for me was less around like the product itself, but was more, um, and again, this was me learning and I'm still learning, um, a lot about like what keeps, there's, you know, what makes the company stick, right? And like, what makes it actually work in a long term? And what, what, Dave, do you, do you have any sort of thoughts about that? No idea. Like obviously I'm going to be very biased and say, yeah, the ones that did a lot of scientific research and rigorous research are still surviving, not the case. Um, like there's, for example, um, some companies that, that pursued, um, that, the FG route to get clearance, um, to be a medical device right in the US. That's a fact that some of these companies can take you are required to do a lot of very systematic trials or some very specific guidelines that you have to follow. Um, and they put in their bets there. They were too early. Um, it takes a long time. The time frame of evaluation from an external source to give you that kind of a, um, stamp of approval, like a designation and then for the peer, so adopt that billing code and they put the infrastructure so that you can actually then like, you know, it started to like send, send it out and then get reimbursed. The, the time frame is a long one. Um, so just for a startup company, some, some of those just don't have the funding to be able to last that long. So, um, so yeah, like I don't have a, I don't know if there's like a yeah, these are the things that you need. Um, but, I don't know what you think about this, but, but I think I always think that it's kind of, there's all these random factors, right? So all that stuff is real that there are these random factors that play into it. But I always think it's just, it's the one person, typically the one person, maybe it's two, but the one person that like, do they have the conscientiousness, stick to itiveness, one track, mindedness, you know, like just like they're just going to go, you know, like my dad had, you know, growing up, he had his own business and, you know, we saw many, many other businesses sort of fail, you know, people that he would work with and that sort of thing. And like, I just think that he was just dog-id in his pursuit and he just, you know, the willingness to work 70 or 80 hour weeks some weeks and just go, you know, and I think that I always think that that, that sort of, that, that trait, you know, I always think that's what is the biggest difference between success and failure. It's not as much the idea. It's like the one or two people at the top and they're just willingness to like, give it all for the company. I, I would have reviewed the execution is where the things are like, it seems to make a big difference, right? Like there's a lot of people that have wonderful ideas and then maybe just don't have the skillset to execute whatever that execute means. Like sometimes it will be like you're to your point, like the, the ability to like stay put and like, you know, hold on to that idea and not get swayed by like, well, what you could do this and you could expand here and you could do that and like, what about this problem, that problem and like try to do all of it but like stick with it and then like be like, no, this, this is my vision. We're going to stick with it, whether it works or not, this is it. Sometimes that's what gets into the way of execution. Sometimes it's like, you're not adaptable actually like and then whatever you started with like the, again, we're talking about with it. The world changed so fast. If it didn't adapt, you are not around anymore, right? Like you literally are not. So there's sometimes that adaptability comes into play in a different way. Sometimes I think as the company scale, like, you know, some funders are able to really develop themselves also to develop other skillsets that helps the company sphere and changing to all these different ways of growing because different skills are needed at that. And sometimes the founders are what you need is a founder that is very insightful and is saying like, no, I don't know all these things. So I'm going to bring a wonderful team of people that can help me complement all this. So but it's really like, you know, execution is where things are going to go not work. So so what what what job did you end up taking? That's a wrap on the first part of our conversation. As noted at the top of the show, be much appreciated if you spread the word to anyone else who you think might enjoy it. Until next time.[Music]