ChatGPT Breakout Technology
This month’s episode of “News you can Use” on HealthcareNOWRadio features news from the month of March 2023
As I did last month I am talking to Craig Joseph, MD (@CraigJoseph) Chief Medical Officer at Nordic Consulting Partners.
With the big news of the release of ChatGPT4 and the progress made with this model, we talk about Craig’s experience with ChatGPT v4. So far he has not noticed the difference. We discuss the exam performance and in particular how well ChatGPT4 did on the Phsycian Step 2 exam. And we finish the segment with a few lines poorly performed by me of the ChatGPT in Healthcare Rap that I asked the engine to create in the style of Eminem to a little background music.
ChatGPT in Healthcare Rap
As generated by ChatGPT and performed poorly by me:
Yo, it’s ChatGPT in healthcare, we’re on a mission
To revolutionize the game with machine intuition
AI in healthcare, we’re breaking through tradition
Digital health and telemedicine, we’re making transitionsOur program’s got the latest trends, it’s hot and it’s fresh
We’ve got the experts on the mic, they’re the best of the best
They’re spitting out knowledge, leaving nothing to guess
You’ll learn so much, your mind will be blessedChorus:
ChatGPT in healthcare, we’re taking it to the top
Our program’s so fly, it’ll never stop
From AI to telemedicine, we’re making a swap
ChatGPT in healthcare, it’s the cream of the crop
Listen in to hear our thoughts about the Silicon Valley Bank implosion and its impact on healthcare, especially Digital Health investments, and our review of the Epic Research that continues to flow out of this database of shared anonymized patient data from Epic Clients (Most Primary Care Telehealth Visits Unlikely to Need In-Person Follow-Up).
You can read more about the series here and the concept of keeping up with innovation in healthcare. Please send me your suggestions on topics you’d like to see covered. You can reach out direct via the contact form on my website, send me a message on LinkedIn or on my Facebook page (DrNickvT), or on Twitter by tagging me (@DrNic1) and #TheIncrementalist or you can click this link to generate a ready-made tweet to fill in:
Listen live at 4:00 AM, 12:00 Noon or 8:00 PM ET, Monday through Friday for the next week at HealthcareNOW Radio. After that, you can listen on demand (See podcast information below.) Join the conversation on Twitter at #TheIncrementalist.
Listen along on HealthcareNowRadio or on SoundCloud
Raw Transcript
Nick van Terheyden
And today as I’m joined each and every month, I’m here with Dr. Craig Joseph. He’s the chief medical officer at Nordic consulting partners, Craig, thanks for joining me.
Craig Joseph
Well, thank you for having me for yet another month.
Nick van Terheyden
Yes. And another well paid month. I’m just gonna say, of course,
Craig Joseph
I appreciate it. The new cars looking great that you sent me and I want to I want to thank you for that. Next time. One with a motor would be great.
Nick van Terheyden
Oh, well, you got a Tesla.
Craig Joseph
I was that way I couldn’t find a place to put the gas in. All right. Okay. Let’s take a look at it again. I thought you gave me something without a motor. I was very upset. But apparently that’s the new way of doing things.
Nick van Terheyden
Well, so the big question in my mind is, Am I really talking to the real Craig Joseph? Or is this just chat GPT generated content? Because you’ve stepped into the chasm, haven’t you, your your you have upgraded and paid paid the fee for the latest and greatest? Well,
Craig Joseph
listen, I was worried about open AI not making enough money. And they’re scrappy startup. Industry really hasn’t acknowledged their existence. And I really felt the need to do something to make sure that they survive. Because, again, they’re so small, and really, no one’s heard of them. So I did, I upped and went for the $20 a month option, which is, I think, a relatively new thing. Where you get, the small thing I think you get is priority access. So when others are seeing that message, hey, you were busy right now, but check in later. I won’t see that as much. But I do have the option now to choose the latest and greatest GPT.
Nick van Terheyden
So that’s an option you don’t actually automatically get for?
Craig Joseph
No, no. You get to select. At least I do and Far be it for me. I don’t know if they’re doing a bunch of you know, a B options. But when I when I log in, I get to select which engine for lack for I’m not sure what the right word is what which version of
Nick van Terheyden
that engine is missing from your car? Yes. Yes,
Craig Joseph
exactly. Right. So, so I have tried playing with the latest and greatest, which is GPT for at least the one that’s publicly available. And I so far, in my very limited and focused testing have not been amazed at the difference. I’m told, I’ve been reading from people who are again, smarter than me saying like, wow, it makes it makes chat GPT the current version that most people are on that version three, just looks stupid compared to this new version four. I have not really seen that yet. Maybe because I’m not asking the right questions, but I don’t know. $20 a month is not done. Not a ton. But I think it’s still worth it for now just you know, kind of play with it. And I suspect if I got a little bit more sophisticated and were able to compare answers, you know, on then on the screen at both times, I might, I might start to see the difference. But it’s still, it’s still got the wow factor for me. That chat GPT I’m still still amazed. In fact, I was taking a quiz, I think it was the Wall Street Journal just the other day gave a quiz. And it gave seven different things that generative aI had created from, you know, gave you a picture and said, Hey, is this a real picture? Did we or was this created? Hey, is this a real cover letter? Or was it created? And I don’t want to ruin it for anyone. But I suspect by the time this comes out anyone that was interested in having seen this would already do it. I think there were seven questions. The first three I said it’s real, it’s real. It’s real. Those were all were all wrong. They were all generated by an AI and then the last four I was like yeah, okay, generated generated and that’s what it was. So they gave you seven examples all of which were fake were generated by an AI of some sort or another and you know, I could there was a picture and it was clearly the picture was a little little off anything was fingers still seems to be a major problem. But it you know, still super interesting and still I think the potentials just unlimited. It’s really just a wow factor. Just like when when in my mind when the when I first saw the iPhone or when I first realize what Google can do. It’s that’s where I’m at with, with GPT and other generative API’s.
Nick van Terheyden
So first of all, I’m impressed that you had the $20 to spare per month, you were obviously doing well. I’m glad to hear that. And then I’m disappointed to hear that you didn’t get the value proposition from this because I’ve been pining after him. In fact, I was even exploring whether I could get into the API because I think you can get you can be at get put on the waitlist. Now, I haven’t managed to sort of progress that. But I’m the fact that you got nothing out of it. It’s just really disappointing. You’re not selling it well for you know, trying to support
Craig Joseph
and and hopefully no one from open AI is listening for for 750 $7.50 per use, I will let you use my username and password. And, you know, if you if you only do it twice, you’re you’re saving some money.
Nick van Terheyden
As you know, I don’t need your permission to use your username and password because I’ve broken that. So no problem. But thanks for the Franks for the offer. So use cases, let’s let’s talk about one specifically in azo. So it took a bunch of tests, the number four and I I’m with you, I am totally hyped up about it. I’ve heard people say that it’s on the peak of X have over inflated expectations on the classic image of whatever, I don’t think so I really think it’s past that. Certainly based on some of the experiences. And I recognize not all of this is applicable, let’s be clear, there’s some things that it just doesn’t do. Or do terribly well. But a lot of it it does. And one of the things that stood out to me, aside from it, passing the law exams, and all of those, you know, MCAT tests, and whatever was, it was presented with the step two, which is the medical exam for medical students, fourth year, they take it. And that says that they’re a doctor, although there’s Step Three as well. But you know, I don’t know the US very confusing in terms of qualifying, they have all these exams, and who knows where you are. But anyway, step two, and it was in the 95th percentile, which means it qualified it. Absolutely. That’s really hard to do to be clear, obviously, and it qualified. And of course, you know, created quite the stir well, do we need doctors? What’s the my response was a little bit different, I think. And it was, Well, that tells you how useless personal view, the testing mechanism is for doctors, because is that really testing? Anything that we need? Well, not anything, obviously, there are elements to that. But what are we testing for? Do you think this will be the end of testing and we’ll move to something else?
Craig Joseph
Well, I hope it’ll be the end of memorization. Yes, kind of rote memorization testing. Because back in the day, a million years ago, when you and I took these kinds of tests, a lot of it was just like, hey, can you spit out knowledge back to us? Can you regurgitate stuff, and that was, it was just some extent important. I get it. And that was the only way that they could, you know, separate folks out I guess, during during their training, but with the internet, and now with AIS and other tools that help us organize the information on the internet presented in a thoughtful and logical way. It’s become less and less important to know how certain enzyme pathways work, right. That’s what we needed. That’s what I remember being tested on and steps one and two. And that’s less important, I think, what’s more important and what’s also much more difficult to test is the Hey, how do you interact with a human Hey, how do you make sure not to forget the the the answer or the disease process, which is either more commonly seen or less commonly seen? You know, in medicine, we talk about zebras being the the uncommon diagnosis that everyone has to remember. So, you know, how do you you have to know that that exists? Certainly, you have to know what’s out there and have some idea, but you don’t really need to have all of that some of all of that information in your head anymore. And that I think that’s been the case for for decades, even before we had robotic AI is to to feed us information. It was all out there. I didn’t need a book, I could just go into Google, but I might have to search for it a little bit. Now that’s becoming even less and less a requirement. I with you. I’m with you. I don’t see this as being hype. I see this as being real. And every week we come up with more examples of how this thing can be helpful. And again, and just like we I think talked about last month, to be clear. If it’s not real smart, it’s good at predicting what the next word should be. And it puts that all in in order in a meaningful way. But it can be very confident of the very, very wrong answer. And so you have to, you have to know how to interpret this stuff. And I think it’s like any tool, if it’s not used in the right way it can cause it can be very harmful, whether that’s a hammer, or a CT machine in the wrong hands. It’s it’s, it’s not good.
Nick van Terheyden
Yeah. So one thing I would add to the testing of the requirements, so we also have is not only your interaction with humans, but your interaction with technology. You know, I’ve I’ve said this before, the sort of chat GPT whisper and the ability to extract information. There’s some approaches to that. And, you know, I always, I’ve always found that, you know, practice is the appropriate tool set, for me at least to find the right way, especially when I’m looking for information that I have a decent sense of whether it’s appropriate or not, to your point. I mean, I think it’s worth highlighting this because I heard a really interesting story. So I was asked to put together a relatively short notice for a local group, a chat GPT and healthcare session. And that was handed to me. You know, I didn’t have a lot of time. So I went to chat GPT and asked who would be a good expert and the initial expert that I asked for one in Washington, it understood Washington state, and I said, No, no, I want Washington DC. It found me this professor Bronner, Toski, David, and I invited him and he came, and actually one of his first questions was, how did you find me? And I said, Well hold that thought, I’m going to actually tell you that in the press.
Craig Joseph
That’s amazing. It was very cool, you
Nick van Terheyden
know. But he told the story of the use case, and his name is a little bit unusual. And he wanted to see where it was from, and he dived into chat, GPT went through this whole thing, and it says, Actually, you’re from this small village in this country. And he said, Well, could you provide reference anyway, three or four sessions in, it comes up with really solid data to say you’re from this village. And this is why and you know, and when he asked for references, it just made up things. And it was complete fabrication. Yes, but you had to know. And you had to keep going to sort of get to that. So that represents sort of, you know, I guess the risk or the challenges. But equally, in terms of the generation. I mean, it did some extraordinary things, it produced a whole outline for me, which I didn’t follow exactly, but it was, you know, pretty comprehensive in terms of what I built for it. And, you know, let’s talk a second if we could about the fun news cases, I think, you know, having it do things in the style of and, you know, I asked it to create a chat GPT in healthcare, in the style of m&m which, you know, I’m hopeless at this. But let’s be clear, it did a phenomenal job. And essentially, our programs got the latest trends. It’s hot and it’s fresh. We’ve got the experts on the mic there the best of the best, Chad GPT and healthcare, we’re taking it to the top our program so fly, it will never stop.
Craig Joseph
We’re done now.
Nick van Terheyden
I think so. We won’t torture anybody any further with GPT.
Craig Joseph
We’ve won the internet for when this show airs. We’ll need to make sure local and national authorities so that they can plan for the Internet to either explode or implode. I’m not sure which is going to happen. But that, that that’s it.
It’s there. It’s
Craig Joseph
full. If you’re if you’re listening to us while driving, pull over.
Nick van Terheyden
Yes. Please have no accidents.
Craig Joseph
acknowledge what happened. We were all part of that.
Nick van Terheyden
And no nasty groves either, please. So for those of you just joining, I’m Dr. Nick the incrementalist today. I’m talking to Dr. Craig Joseph. He’s the chief medical officer at Nordic consulting partners. We’re doing our news you can use and of course chat GPT is top billing. And I think we’ll continue to be we’ll continue to explore it. But let’s move on to some other things. Obviously, from a financial standpoint, I think the big news is and you know, Silicon Valley Bank, which you know, major funding bank for a lot of the innovation and specifically to digital how Have, and it imploded, I saw a great explanation of that, essentially, you know, they get money, then they re lend the money, and they don’t essentially have the money times seven, I think was the number of all this imploded because it was a traditional run on the bank. But the impact of this is going to be enormous, right? I mean, we’re, we’re seeing it already, I think, or are we? What are your thoughts?
Craig Joseph
Well, you know, it’s, it’s scary, that that could happen with a bank that’s as important and unimportant at the same time, right. It’s like the I think it was the 17th largest bank, it depends on how you look at it. But you know, a big regional bank, but so much of the IT infrastructure and startup world banks there, that when they had trouble, and we’re going under that, that affected a lot of people, and it also affected a lot of groups that have nothing to do with technology. I was reading about a pediatrician on on Twitter who said that his office, the all of their stuff was through Silicon Valley Bank, and, and someone’s like, why would you do such a thing? And so why would I do something? I think they were in Oregon, or, you know, something, something stayed far away? Or having nothing to do with with Silicon Valley? And his answer was, well, we banked with this bank that got bought by SVB. And I didn’t see any reason why I should change just because it was a different name. Everything seemed to be the same. And, of course, that’s right. There’s no way for most of us to know if our banks are being properly managed or not. But it’s certainly instantly overnight, you saw large firms and small firms looking for money so that they could make bank to make payroll. And, and whether you approve or not, the federal government has stepped in and said, Hey, we’re going to we’re going to make sure that everyone is whole, everyone who has money in the bank. Certainly shareholders are out of everything that they had. But even the you know, the FTC will insure everyone up to a quarter million dollars, but for businesses that have a million dollars or millions of dollars, potentially that was a big loss of, of money that to through no fault of their own. Right. And so I think we’re well, I understand that were those all of those depositors are made whole is it was scary, and it’s clearly going to affect the healthcare IT world for years to come. I think the the times of easy money and trying to grow fast and break things, as you’re, as you’re doing it are certainly not over. But there were this is kind of a reality check. And we’re bringing back bring being taken back to reality. See where I’m going with that. Back to react? No, I’m not doing it.
Nick van Terheyden
Oh, yeah, you’re gonna do the I’ll get the music. And then again, if you’d like.
Craig Joseph
I’m not I’m not gonna I can’t do it. But I, you know, I do think we don’t really know what what the bank implosion that started, we don’t really know where it’s going to end. But clearly digital health is is is going to be on a lower trajectory for a while. I think that’s, that’s clear. And so companies that maybe we’re going to be formed are not going to be foreign companies that were looking to grow, either organically or in organically, we’re going to have to take a little have to look more closely, because the money’s not going to be as easily as easily there for a bit. Hopefully, that’s a small amount of time. I don’t know if it’s I suspect it’s not forever. At least I hope it’s on.
Nick van Terheyden
Yeah, no, it can’t be forever, we can’t allow for that. I it’s such an important area of innovation. And I know there’s lots of different feelings about you know, how much of this innovation actually generates, you know, utility in health care. i In the early days, I’ve been involved, I learned a couple of things, one that, you know, is worth sharing here that it’s about a one intent for that early investment for actual payout, you know, delivering against whatever money they put in, you only expect one intend to be not just successful, but actually return your money. So that’s a low order of success, which, you know, you wonder about those other nine companies, I’ve said this, you know, certainly at the conferences, you see these folks and you go walking through the halls and you go the next year and you don’t recognize any of them because they’ve all changed. So maybe that’s a good thing. But also, we need to experiment. I mean, this is you know, not to pull up chat GPT again, but it’s that experimentation that sometimes creates These new innovations that you go, Wow, that really makes sense. I mean, I’m, you know, I’m excited about that potential and what that does, you know, when the money’s flowing, but if it’s not, we’re losing that. So I’m with you I was a bit, I certainly taken aback, I learned something about some of the finance that 7x factor was, for me a surprise. And all I could think of, I’ve got to be honest, was the old Westerners, Western movies with people, you know, the line outside the bank. And that’s what we seem to have banks with lines outside, even, you know, going outside this country, I’m with you. I hope that’s not the sort of the the end, I’m sure it’s going to pause and make people think harder, which, you know, not a terrible thing. But we also don’t want to sort of stifle innovation. So I, you know, limited time left to sort of talk about things. You know, I saw a piece of research, I have to say, I’ve subscribed to the epic research, notifications, and I’ve had some, you know, it’s actually producing some interesting results, I think you’ve got more experience with this. It’s anonymized data that if you’ve got access to I don’t know how this maybe you can explain how this works. But they showed that telehealth visits were unlikely to need in person follow ups, which, you know, seems like a small thing, but that’s actually a big deal. Because if that’s the case, then we were delivering value. Assuming that, you know, it was a lower cost interaction, and you know, they were getting what they needed.
Craig Joseph
Yeah, so epic is one of the vendor one of the biggest in the world for electronic health records and their customers can opt in to sharing as you say, information, putting it all you know, patient information, in a de identified or as de identified as you can into a big system. And if you share that information, if you if you opt in, as I think most epic customers are doing now, then you have access to that for research purposes. So it’s
Nick van Terheyden
a two way street to be clear. So you can’t access it unless you agree to contribute. Absolutely. Oh, that’s good. Okay, great, it
Craig Joseph
is tightly tightly controlled. So again, there’s no there are no names. So there’s no identifying information in there. But there’s lots of data and, and one of the things that Epic is trying to do is get some of that information out, get knowledge out of just data and, and they acknowledge that the peer review journals that that you and I look to for, for Sterling evidence, it takes a long time, right to get something out. And so one of the things that they’re doing with these, with these data are asking questions, and then putting information out pretty quickly, over the course of months, and saying, Hey, this is not peer reviewed, we are not telling you that we are cutting edge scientists, but we have all this data, and we want to we want to share some of the findings. And, and, and they they give you the information as much as they can, and then say, hey, researchers, this might be something that you want to see if you can duplicate. So one of their articles that came out a few weeks ago, was exactly what you described, where they showed how how they looked at patients that had a telehealth visit, and then followed them for the next couple of months and said, hey, who of these patients needed an end an in person visit for what we could tell was a same or similar problem. And it was less than half it broke up in different ways. But less than half of the time, they needed a follow up. And that’s been one of the arguments against telehealth forever. So that, you know, you can have your online visit with your doctor but push comes to shove, they need to listen to the back to your to your back to listen to your lungs to see if you have pneumonia, can’t figure that out without special tools and, or without just you know, examining you and and so the theory went that well, most of those patients are just going to have to have another visit that’s going to cost more and take more time and it’s no good forever. Well, preliminary data seems to show that that is not the case that the majority maybe not everyone but the majority of people are satisfied and do not require a follow up in person visit for the for their telehealth visits, some of which is because we’re good at what we do, I guess as physicians or clinicians and others might be that people are smart enough to say, hey, if this is something where I think they’re not going to be able to to diagnose me or give me answers, I’m just going to make an in person because it’s so they’re not wasting their time asking people asking doctors to make decisions that they already know can’t be made without without an in person visit. So it’s certainly moving us in the right direction and pretty exciting.
Nick van Terheyden
Yeah, I think I’ll round it out because unfortunately as usual, we’ve run out of time But, you know, the polling for me not only that piece of research, but I think you know, well constructed in the you’ve got to contribute to be able to access I think that’s a great way of innovating, recognize all the challenges of anonymization. But you know, I’m hoping that they’re doing the right or managing that because it’s very difficult. Let’s be clear. But then importantly, you can sign up for these, the output, or at least I’ve been able to, and I’m not an epic customer as such, which is great. I get these emails. And, you know, sometimes it’s more interesting to me than other times, but you know, it’s widely available. And I think, to your point, it’s providing real world evidence, which I think is great. So that’s, that’s good news. Unfortunately, as usual, we’ve run out of time, so just remains for me to say Yo, brother, thanks for joining me.
Craig Joseph
Yo, yo, yo, it’s been a pleasure. You know, that’s horrible. I feel badly that people have to listen to this
