WEBVTT 00:00:00.000 --> 00:00:01.743 A note before we begin. 00:00:01.743 --> 00:00:09.323 The narration in this video is synthetic speech, generated from the lecture notes of Dr. Sean Davis. 00:00:09.323 --> 00:00:11.976 It is not a recording of his voice. 00:00:11.976 --> 00:00:15.160 The lecture was given on October 19, 2026. 00:00:15.967 --> 00:00:22.374 This is a recorded version of a lecture from a course on artificial intelligence for fourth-year medical students. 00:00:22.374 --> 00:00:28.781 In the small groups the week before, several students asked, some of them pretty directly, for concrete technique. 00:00:28.781 --> 00:00:30.860 Things they wouldn't have thought of. 00:00:30.860 --> 00:00:36.649 So that's this session: how to get a good answer out of a chatbot, and how to know when you've got one. 00:00:36.649 --> 00:00:38.167 It runs about half an hour. 00:00:38.517 --> 00:00:44.917 I sat in on those small groups, and I want to start with what the students said, because it shaped this talk. 00:00:45.333 --> 00:00:54.823 One student, after hearing a trick for making answers shorter, said that little things like that, which we wouldn't have thought of, would be really helpful to learn. 00:00:54.823 --> 00:00:58.767 I heard that as a hint that I should be teaching more concrete stuff. 00:00:58.767 --> 00:00:59.053 Fair. 00:00:59.467 --> 00:01:05.707 One said they'd been using it on rotations more as a crutch than a tutor, and someone else said they related to that. 00:01:06.133 --> 00:01:13.173 One said that after the previous session, on how these models work, they trust it a lot more than they should. 00:01:13.600 --> 00:01:19.000 And one, who uses it every day, said they haven't unlocked it the way they want to. 00:01:19.400 --> 00:01:21.556 Two more that were already the answer. 00:01:21.556 --> 00:01:32.508 One student asked ChatGPT itself how to use it as a tutor, and it came up with a prompt that had it act as a Socratic tutor: it asks you questions, you explain in your own words, it grades you. 00:01:32.508 --> 00:01:36.140 They found they knew less than they thought, which is the point. 00:01:36.140 --> 00:01:38.920 We'll come back to asking it to write the prompt. 00:01:39.333 --> 00:01:44.392 And one student said their habit is: this is my current differential, what am I missing? 00:01:44.392 --> 00:01:48.359 They called it a safeguard, to make sure you're doing the work first. 00:01:48.359 --> 00:01:51.693 That's the best single pattern I know, and it's in Part 4. 00:01:52.100 --> 00:02:04.815 And the usual ways an answer disappoints: it made something up, it was too generic, too long, confidently wrong, it agreed with you too easily, or, honestly, you weren't sure what you wanted. 00:02:04.815 --> 00:02:19.660 Each maps onto part of this session: made up and confidently wrong go to Check; too generic to Clarity; agreed too easily to Conversation; and not sure what you wanted to Goal, which is where we start, and the most fixable. 00:02:20.010 --> 00:02:26.264 So this session is concrete moves, each with the evidence behind it where there is some, and a short set of rules. 00:02:26.264 --> 00:02:31.970 Several of the rules came from the students; one group came up with them on their own, and they're good. 00:02:32.800 --> 00:02:35.094 Here's the shape of the session. 00:02:35.094 --> 00:02:35.883 Five steps. 00:02:35.883 --> 00:02:39.970 Goal: before you type anything, what does done look like? 00:02:39.970 --> 00:02:43.913 Clarity: what does the model need to know to get there? 00:02:43.913 --> 00:02:49.290 Conversation: the first answer is a draft; push back, don't just accept it. 00:02:49.290 --> 00:02:53.879 Check: the one claim the plan depends on, against a real source. 00:02:53.879 --> 00:02:57.320 And Stop: either you're done, or you start over. 00:02:57.670 --> 00:02:59.099 This isn't my invention. 00:02:59.099 --> 00:03:06.961 It's roughly what the companies building AI agents had to write down to make their agents work, and we'll see that in a few minutes. 00:03:06.961 --> 00:03:08.390 It works for people too. 00:03:09.200 --> 00:03:11.942 First: where is it safe to use a chatbot at all? 00:03:11.942 --> 00:03:17.140 The honest answer is that it depends on the task, and the boundary isn't where you'd guess. 00:03:17.140 --> 00:03:24.680 So the first skill is knowing which side of it you're on, and the way you learn that is by trying things where being wrong is cheap. 00:03:25.500 --> 00:03:27.792 Before the frontier, a quick check. 00:03:27.792 --> 00:03:35.456 How confident are you that you can tell, for a given task, whether a chatbot will help, and whether it's safe to use? 00:03:35.456 --> 00:03:36.700 Not at all to very. 00:03:37.500 --> 00:03:48.108 A study of 758 management consultants at BCG, Dell'Acqua and colleagues, now published in Organization Science, examined where AI helps and where it hurts. 00:03:48.108 --> 00:03:51.188 They called the boundary the jagged frontier. 00:03:51.188 --> 00:03:56.390 The line draws itself here; that's the boundary of what the model does well. 00:03:56.390 --> 00:03:58.580 It's jagged, and it's invisible. 00:03:59.000 --> 00:04:05.952 Inside the line, the consultants who used AI did 12% more tasks, 25% faster, at over 40% higher quality. 00:04:05.952 --> 00:04:18.720 These are the kinds of things that are usually inside for us: practice questions, turning your notes into a table, summarizing a guideline you've pasted in, rewriting something for a patient. 00:04:19.133 --> 00:04:26.564 Outside the line, the people using AI were 19% less likely to get the right answer than people who didn't use it at all. 00:04:26.564 --> 00:04:29.660 And it doesn't feel different when you're outside. 00:04:29.660 --> 00:04:32.261 It's just as fluent and just as confident. 00:04:32.261 --> 00:04:41.053 For us: a citation from memory, a dose for a patient on dialysis, what changed in a guideline last month, noticing what's missing from a case. 00:04:41.403 --> 00:04:43.290 These placements are illustrative. 00:04:43.290 --> 00:04:49.283 Where your tasks actually fall is something you learn by checking, which is part of what this course is for. 00:04:50.100 --> 00:04:52.164 So when is it safest to use? 00:04:52.164 --> 00:04:53.713 Here's how I sort it. 00:04:53.713 --> 00:04:54.745 Two questions. 00:04:54.745 --> 00:04:57.399 How bad is it if the answer's wrong? 00:04:57.399 --> 00:04:59.980 And how easy is it for me to check? 00:05:00.400 --> 00:05:02.739 Low stakes, easy to check: go ahead. 00:05:02.739 --> 00:05:03.259 Quiz me. 00:05:03.259 --> 00:05:05.013 Turn my notes into a table. 00:05:05.013 --> 00:05:07.547 Draft an email I'll read before I send. 00:05:07.547 --> 00:05:10.731 Explain a concept I'm going to read about anyway. 00:05:10.731 --> 00:05:13.720 If it's wrong, I'll notice, and nobody's hurt. 00:05:14.133 --> 00:05:18.028 Low stakes, hard to check: use it for ideas, not answers. 00:05:18.028 --> 00:05:23.905 Brainstorming a research question, a study plan, different ways to say something hard. 00:05:23.905 --> 00:05:27.253 You're not trusting it; you're using it to think. 00:05:27.667 --> 00:05:34.226 High stakes, easy to check: use it, then check the one claim the plan depends on. 00:05:34.226 --> 00:05:36.979 A drug interaction, then DailyMed. 00:05:36.979 --> 00:05:39.246 A risk score, then M D Calc. 00:05:39.246 --> 00:05:44.267 A summary of a guideline you pasted in, against the guideline. 00:05:44.667 --> 00:05:48.845 High stakes, hard to check: don't, or give it the source first. 00:05:48.845 --> 00:05:50.105 A dose from memory. 00:05:50.105 --> 00:05:52.029 A diagnosis you can't verify. 00:05:52.029 --> 00:05:54.416 A citation you aren't going to open. 00:05:54.416 --> 00:06:00.187 This is exactly where the frontier hurts: the answer looks the same as everywhere else. 00:06:00.537 --> 00:06:03.456 This isn't from a study; it's a way of sorting. 00:06:03.456 --> 00:06:09.977 The book's centaur-and-cyborg chapter asks the same question a different way: who's harmed if it's wrong? 00:06:10.800 --> 00:06:14.303 So if the frontier is invisible, how do you find it? 00:06:14.303 --> 00:06:18.480 You experiment, in the corner where being wrong costs nothing. 00:06:18.830 --> 00:06:24.375 Ask it something you already know cold, a topic you've just studied, and watch where it slips. 00:06:24.375 --> 00:06:29.860 Redo a question-bank item you've already done and compare its reasoning with the explanation. 00:06:29.860 --> 00:06:36.820 One caution: check your question bank's license or terms first; many don't allow copying their items into other tools. 00:06:36.820 --> 00:06:42.188 Rewriting the question in your own words, or using questions you wrote, avoids the problem. 00:06:42.188 --> 00:06:49.207 Ask the same question in two chats, or in ChatGPT and another tool; where they disagree, you're probably near the edge. 00:06:49.207 --> 00:06:55.990 And keep a short list, even a note on your phone, of what it's been good and bad at for the things you actually do. 00:06:56.400 --> 00:06:59.690 Use it for everything where a wrong answer costs nothing. 00:06:59.690 --> 00:07:05.059 That's how you learn where it helps, and it's how you get faster at the rest of this session. 00:07:05.059 --> 00:07:11.640 The same experiments are a bad idea in the top row of the last slide: there, you check first, or you don't use it. 00:07:11.990 --> 00:07:13.848 This is my advice, not a finding. 00:07:13.848 --> 00:07:19.590 The frontier study tells you the boundary is there and hard to see; it doesn't tell you how to map it. 00:07:20.400 --> 00:07:24.265 And the question of when you use it isn't only about the task. 00:07:24.265 --> 00:07:25.200 It's about you. 00:07:25.550 --> 00:07:31.046 There's a name for this: automation bias, the tendency to over-rely on automation. 00:07:31.046 --> 00:07:36.475 It was named long before chatbots, with autopilots and clinical decision support. 00:07:36.475 --> 00:07:43.446 A systematic review of 74 studies found it gets worse with workload, task complexity, and time pressure. 00:07:43.446 --> 00:07:48.272 What helps: training, and feeling personally accountable for the result. 00:07:48.272 --> 00:07:50.350 Which is you, signing the note. 00:07:50.767 --> 00:07:53.596 And we know what fatigue does by itself. 00:07:53.596 --> 00:08:06.755 In a randomized study in ICUs, interns working 24-hour shifts made about 36% more serious medical errors than on a schedule without them, and 5.6 times as many serious diagnostic errors. 00:08:06.755 --> 00:08:08.807 That's without any AI at all. 00:08:09.233 --> 00:08:10.417 Put those together. 00:08:10.417 --> 00:08:19.509 Post-call, behind on notes, at 2 a.m.: that's exactly when you most want to paste the answer in, and exactly when you're least likely to check it. 00:08:19.509 --> 00:08:27.666 So my rule, which I don't break: when I'm tired or rushed, I either stay in the safe corner from the last slide, or I don't use it. 00:08:27.666 --> 00:08:30.593 Write it down tonight, check it in the morning. 00:08:31.400 --> 00:08:33.987 Why bother learning to use it well? 00:08:33.987 --> 00:08:36.722 Because having the tool isn't enough. 00:08:36.722 --> 00:08:42.708 In last week's physician trial, doctors were 74% without G P T 4 and 76% with it. 00:08:42.708 --> 00:08:56.160 In a big trial this year, chatbots alone named the right condition 95% of the time, and members of the public using those same chatbots got under 35%, no better than a search engine. 00:08:56.510 --> 00:08:57.580 And the good news. 00:08:57.580 --> 00:09:03.047 The same Stanford group that ran the physician trial went back and designed the interaction. 00:09:03.047 --> 00:09:11.308 The clinician and the AI each made their own assessment first, then the AI wrote a comparison highlighting where they agreed and disagreed. 00:09:11.308 --> 00:09:13.150 Seventy clinicians, randomized. 00:09:13.500 --> 00:09:16.820 With usual resources: 75%. 00:09:17.233 --> 00:09:23.073 With the designed workflow: 82 to 85%, depending on who went first. 00:09:23.500 --> 00:09:24.459 AI alone: 90%. 00:09:24.459 --> 00:09:32.540 The workflow got the clinicians most of the way there, and neither workflow was significantly different from AI alone. 00:09:32.967 --> 00:09:37.257 So how you work with it changes the result, and it's learnable. 00:09:37.257 --> 00:09:41.138 Notice what they built in: you think first, then compare. 00:09:41.138 --> 00:09:42.228 That comes back. 00:09:42.228 --> 00:09:44.407 That's the rest of this session. 00:09:45.233 --> 00:09:45.698 Part 2. 00:09:45.698 --> 00:09:50.615 This is the part most prompting advice skips, and I think it matters most. 00:09:50.615 --> 00:09:53.472 Before you type anything: what's your goal? 00:09:53.472 --> 00:09:57.193 And part of a good goal is knowing what done looks like. 00:09:58.000 --> 00:10:07.481 There's a big literature on this from organizational psychology: goal-setting theory, Locke and Latham, 35 years of studies summarized in 2002. 00:10:07.481 --> 00:10:09.470 The central finding is simple. 00:10:09.470 --> 00:10:22.000 People given a specific, hard goal do better than people told to "do your best." "Do your best" feels encouraging, but it gives you nothing to aim at and no way to know when you've arrived. 00:10:22.350 --> 00:10:25.898 "Help me understand heart failure" is a do-your-best goal. 00:10:25.898 --> 00:10:30.670 It'll give you a pleasant overview, and you'll have no idea whether it worked. 00:10:31.100 --> 00:10:41.214 Compare: a one-page table, HFrEF versus HFpEF, three drugs each with how they work, done when I can answer five board-style questions without looking. 00:10:41.214 --> 00:10:45.260 Now the model knows what to make, and you know when to stop. 00:10:45.667 --> 00:10:51.667 What works for people works for the model, for the same reason: neither one can read your mind. 00:10:52.467 --> 00:11:00.142 Here's the connection to agentic AI, which you've probably heard about: AI that takes a series of steps on its own. 00:11:00.142 --> 00:11:07.282 To make agents work, the companies building them had to write down, explicitly, what any good worker needs. 00:11:07.282 --> 00:11:09.218 A goal: what done looks like. 00:11:09.218 --> 00:11:12.755 Then act, check against the goal, and ask, am I done? 00:11:12.755 --> 00:11:13.890 If not, go again. 00:11:13.890 --> 00:11:14.690 If so, stop. 00:11:14.690 --> 00:11:17.427 The dot going around is an agent working. 00:11:17.777 --> 00:11:21.065 The hard part, it turns out, isn't the acting. 00:11:21.065 --> 00:11:23.281 It's the goal and the stopping. 00:11:23.281 --> 00:11:28.857 An agent with a vague goal never stops, or stops too early and says it's done. 00:11:29.267 --> 00:11:36.866 Anthropic's own guide says: think of the model as a brilliant but new employee who lacks context on how you work. 00:11:36.866 --> 00:11:41.640 And their golden rule: show your prompt to a colleague with no context. 00:11:41.640 --> 00:11:44.667 If they'd be confused, the model will be too. 00:11:45.067 --> 00:11:53.747 OpenAI's guide for GPT-5 literally writes the stop condition into the prompt: only end your turn when you're sure the problem is solved. 00:11:54.097 --> 00:11:58.330 If the companies had to write this down for their own models, so should you. 00:11:58.330 --> 00:12:02.897 And honestly, it's the same thing your attendings want from you on a presentation. 00:12:03.700 --> 00:12:06.224 So, before you start, four lines. 00:12:06.224 --> 00:12:11.655 What will I have at the end: a table, a draft, a list of three sources. 00:12:11.655 --> 00:12:15.938 Who it's for, which changes everything about the answer. 00:12:15.938 --> 00:12:17.391 What must be in it. 00:12:17.391 --> 00:12:19.380 And how I'll know to stop. 00:12:19.800 --> 00:12:26.888 A real example from the small groups: a student built an interview travel schedule from a spreadsheet of dates and programs. 00:12:26.888 --> 00:12:29.460 No conflicts, as many interviews as possible. 00:12:29.460 --> 00:12:32.319 That's a beautiful goal, because you can check it. 00:12:32.319 --> 00:12:36.320 You look at the schedule and either there's a conflict or there isn't. 00:12:36.733 --> 00:12:40.632 Compare "make my personal statement better." It never finishes. 00:12:40.632 --> 00:12:45.891 Every version is "better" in some way, and the model will happily keep going forever. 00:12:45.891 --> 00:12:51.893 If you write instead "cut it to 650 words without losing the clinic story," now you can check it. 00:12:52.300 --> 00:12:56.891 And if you don't know what done looks like, the chatbot can help you define it. 00:12:56.891 --> 00:13:03.633 Same personal statement: ask it for three concrete improvements, and for each one, how you'll know it's good enough. 00:13:03.633 --> 00:13:08.340 Now you have three goals you can check, and you choose which ones you agree with. 00:13:08.340 --> 00:13:12.699 You're still the one deciding what done is; it's helping you write it down. 00:13:12.699 --> 00:13:15.140 The course's writing chapter goes further. 00:13:15.490 --> 00:13:19.050 This is also how you keep yourself honest about whether it helped. 00:13:19.867 --> 00:13:20.588 A quick one. 00:13:20.588 --> 00:13:25.459 Pause the video and pick: which of these could you actually check when it's done? 00:13:25.459 --> 00:13:27.323 Help me study renal physiology. 00:13:27.323 --> 00:13:32.855 Ten board-style questions on acid-base, one at a time, done when I get eight right in a row. 00:13:32.855 --> 00:13:35.561 Make this email to my attending sound better. 00:13:35.561 --> 00:13:38.387 Or, tell me everything about sepsis guidelines. 00:13:38.737 --> 00:13:40.165 It's the acid-base one. 00:13:40.165 --> 00:13:43.145 It has a deliverable, a format, and a stop rule. 00:13:43.145 --> 00:13:45.380 You can fix the others the same way. 00:13:45.380 --> 00:13:54.817 "Make this email sound better" becomes: under a hundred words, says what I need by Friday and why, and I'd be comfortable if my attending read it aloud. 00:13:55.633 --> 00:13:58.405 The other half of done: stopping. 00:13:58.405 --> 00:13:59.833 Two ways to stop. 00:14:00.183 --> 00:14:06.426 You're done when your four lines are met, and, I'd add, when you could explain the result without looking. 00:14:06.426 --> 00:14:09.783 If you can't, you haven't learned it; you've only got it. 00:14:10.133 --> 00:14:12.457 Or you stop because it isn't working. 00:14:12.457 --> 00:14:17.481 Signs: you're on the third rewrite of the same request, and it's not converging. 00:14:17.481 --> 00:14:21.250 It agrees with everything you say, which we'll come back to. 00:14:21.250 --> 00:14:27.844 The chat is long and it's started forgetting things you said earlier; a student asked about exactly this. 00:14:27.844 --> 00:14:34.878 The previous session explains it: the whole conversation goes back every turn, and the early parts get squeezed. 00:14:34.878 --> 00:14:43.671 The fix is a handoff, like sign-out: ask it to summarize what you've decided and what's still open, as a note you can paste into a new chat. 00:14:43.671 --> 00:14:45.492 Read it, fix it, start fresh. 00:14:45.492 --> 00:14:54.599 Or it's the wrong tool: for a risk score, use M D Calc; for a recommendation, the guideline; for a hard search, a librarian; sometimes, a person. 00:14:54.599 --> 00:14:57.613 Knowing when not to use it is part of the skill. 00:14:57.963 --> 00:15:01.244 One of my own rules here: I don't use it when I'm rushed. 00:15:01.244 --> 00:15:05.963 That's when I skip the four lines and skip the checking, and that's when it bites. 00:15:06.767 --> 00:15:10.981 Part 3: once you know what done looks like, what does the model need to get there? 00:15:10.981 --> 00:15:14.579 And a vague prompt is the most common reason you end up starting over. 00:15:14.579 --> 00:15:16.687 I'll build one prompt up in front of you. 00:15:17.500 --> 00:15:22.917 Here's a question many of you might actually type at 2 a.m.: what's the workup for chest pain? 00:15:22.917 --> 00:15:26.836 It'll give you a perfectly reasonable answer for an average patient. 00:15:26.836 --> 00:15:27.931 That's the problem. 00:15:27.931 --> 00:15:35.537 Remember the attention slide from the previous session: the same last words get a different next word depending on what came before. 00:15:35.537 --> 00:15:37.900 With nothing before, you get the average. 00:15:38.700 --> 00:15:42.629 Add who you are and the case, the way you'd present it on rounds. 00:15:42.629 --> 00:15:44.503 Clinical language matters here. 00:15:44.503 --> 00:15:52.845 In one study, asking in everyday patient-style wording got about ten times as many made-up references as asking the way a clinician would. 00:15:52.845 --> 00:15:57.500 Present it like a clinician and you get the clinical part of what it learned. 00:15:57.850 --> 00:15:59.952 And notice I've made it up. 00:15:59.952 --> 00:16:01.821 No real patient goes in. 00:16:01.821 --> 00:16:02.210 Ever. 00:16:03.033 --> 00:16:09.801 Now the task gets specific: a ranked differential, the finding for and against each, and the next step. 00:16:09.801 --> 00:16:13.612 A format you can check quickly: a table and two sentences. 00:16:13.612 --> 00:16:20.313 And the line the chapter calls the most useful single line: say what's missing instead of assuming it. 00:16:20.663 --> 00:16:23.848 Role, context, task, format, uncertainty. 00:16:23.848 --> 00:16:28.820 That's the five-part prompt from the course's prompting chapter. 00:16:28.820 --> 00:16:31.383 Learn the parts, not the wording. 00:16:32.200 --> 00:16:36.009 And the sixth part, which is this session's addition: done. 00:16:36.009 --> 00:16:42.400 Done when I know the one next test, the decision rule behind it, and a source I can open and check. 00:16:42.750 --> 00:16:44.477 That last line does two jobs. 00:16:44.477 --> 00:16:47.872 It tells the model what the finished answer must contain. 00:16:47.872 --> 00:16:53.470 And it tells you when to stop: when you have those three things, and you've opened the source. 00:16:53.820 --> 00:17:01.165 A warning, which the chapter makes with this exact case: structure makes the answer better organized and easier to check. 00:17:01.165 --> 00:17:03.229 It doesn't reliably make it right. 00:17:03.229 --> 00:17:08.085 Both versions of this answer in the chapter make the same mistake about D-dimer. 00:17:08.085 --> 00:17:09.420 We'll get to checking. 00:17:10.233 --> 00:17:15.793 Let's slow down on the six parts, because this is the concrete stuff some of you asked for. 00:17:16.143 --> 00:17:17.503 Role: who's asking. 00:17:17.503 --> 00:17:25.300 Not "you are a world-class cardiologist"; that's the model's role, and we'll see in a minute it doesn't help. 00:17:25.300 --> 00:17:28.591 Your role: I'm a fourth-year starting a sub-I. 00:17:28.591 --> 00:17:31.023 That sets the level of the answer. 00:17:31.373 --> 00:17:33.544 Context: the facts it can't guess. 00:17:33.544 --> 00:17:41.013 Present it like rounds, and include what you've already ruled out, or it'll spend half the answer on things you know. 00:17:41.363 --> 00:17:43.181 Task: one job, with a verb. 00:17:43.181 --> 00:17:45.807 "Help with heart failure" isn't a task. 00:17:45.807 --> 00:17:49.443 "Compare first-line treatment for HFrEF and HFpEF" is. 00:17:49.793 --> 00:17:52.675 Format: a shape you can use and check. 00:17:52.675 --> 00:17:55.707 A table, three columns, under 150 words. 00:17:55.707 --> 00:17:59.953 If you don't say, you get five paragraphs and a summary. 00:18:00.303 --> 00:18:02.929 Uncertainty: permission not to know. 00:18:02.929 --> 00:18:04.314 Say what's missing. 00:18:04.314 --> 00:18:06.575 If my premise is wrong, say so. 00:18:06.575 --> 00:18:09.783 That line matters; you'll see why in Part 4. 00:18:10.133 --> 00:18:12.973 And done: the finish line, from Part 2. 00:18:13.323 --> 00:18:15.321 You won't use all six every time. 00:18:15.321 --> 00:18:17.258 A quick question might need two. 00:18:17.258 --> 00:18:22.523 But when the answer disappoints you, this table is the checklist for what you left out. 00:18:23.333 --> 00:18:25.056 Here it is as a template. 00:18:25.056 --> 00:18:29.740 Screenshot it, or better, put it somewhere you'll actually reuse it. 00:18:29.740 --> 00:18:43.724 Two places in ChatGPT Edu: your project's instructions, like a learning-journal coach, or as a skill: the plus menu has something called Template Creator that saves a reusable template as your own skill. 00:18:43.724 --> 00:18:49.373 The course's prompting chapter has you build your own version of this and test it. 00:18:50.200 --> 00:18:57.946 One part of the six deserves its own slide, because it's the easiest win: you control the shape of the answer. 00:18:57.946 --> 00:18:58.580 Too long? 00:18:58.580 --> 00:19:02.523 Say "three bullets, under 50 words." Want the reasoning? 00:19:02.523 --> 00:19:04.847 "Step by step." Comparing things? 00:19:04.847 --> 00:19:10.410 Ask for a table; it forces it to line things up, and it's much easier to check. 00:19:10.410 --> 00:19:10.762 Lost? 00:19:10.762 --> 00:19:19.916 "Explain like I'm five," then "now like I'm a fourth-year medical student"; the gap between the two shows you what you're missing. 00:19:19.916 --> 00:19:24.000 And tell it your level, so it skips what you already know. 00:19:24.400 --> 00:19:25.827 How far can short go? 00:19:25.827 --> 00:19:29.903 There's a popular add-on for AI coding tools called caveman. 00:19:29.903 --> 00:19:41.725 Its tagline is "why use many token when few token do trick." It makes the agent talk like a caveman, and the developer reports the same answer in about a third of the tokens. 00:19:41.725 --> 00:19:47.160 It's a joke that works, and the point for you is: if it's too long, just say so. 00:19:47.567 --> 00:19:50.788 Here's what that looks like on something you know. 00:19:50.788 --> 00:20:01.611 Why does furosemide lower potassium, answered like a caveman: pill make pee, more salt reaches the end of the tube, the body swaps salt for potassium, potassium go bye. 00:20:01.611 --> 00:20:09.407 It's a joke, but it's also correct, and it's a good test of whether you understand something: can you say it that simply? 00:20:10.233 --> 00:20:18.056 I said this a few times in the small groups, and the Socratic tutor story is the proof: you can ask it to help you use it. 00:20:18.056 --> 00:20:18.633 Two ways. 00:20:18.983 --> 00:20:26.448 First: I want to do this thing; interview me one question at a time, then write the prompt you'd want to receive. 00:20:26.448 --> 00:20:32.460 It'll ask about your role, your context, your format, which is the six parts, done for you. 00:20:32.460 --> 00:20:36.423 That's exactly what happened with the Socratic tutor prompt. 00:20:36.833 --> 00:20:44.427 Second, my research-first habit: what does the research say about the best way to use AI to learn this topic? 00:20:44.427 --> 00:20:50.070 Find sources first, then suggest an approach, and mark anything you can't source. 00:20:50.070 --> 00:20:53.553 You get an evidence-based plan rather than a vibe. 00:20:53.967 --> 00:20:56.797 Is it actually good at writing prompts? 00:20:56.797 --> 00:20:57.087 Yes. 00:20:57.087 --> 00:21:05.506 A study had the model write its own instructions for 24 tasks, and they matched or beat the ones people wrote on 19. 00:21:05.506 --> 00:21:16.029 Another, from Google DeepMind, had a model keep improving a prompt, and the result beat human-designed prompts by up to 8% on math word problems. 00:21:16.029 --> 00:21:21.327 Those are benchmark tasks, not clinical ones, but the direction is clear. 00:21:21.733 --> 00:21:23.074 You're still the editor. 00:21:23.074 --> 00:21:27.653 Open the sources it gives you, and keep the parts that fit how you actually learn. 00:21:28.467 --> 00:21:35.324 Now that you know what to put in, four common additions that don't reliably help, whatever the internet tells you. 00:21:35.324 --> 00:21:36.467 Click each to flip. 00:21:36.867 --> 00:21:40.638 "You are a world-class cardiologist." Role prompts. 00:21:40.638 --> 00:21:54.835 A study of 162 roles across four model families and over two thousand questions found personas didn't improve accuracy, and the Wharton group's 2025 report found the same with expert personas. 00:21:54.835 --> 00:21:56.683 They can change the tone. 00:21:56.683 --> 00:21:58.162 Not the correctness. 00:21:58.162 --> 00:22:06.665 Interesting footnote: the first version of that 2023 paper claimed roles helped; the published version reversed it. 00:22:06.665 --> 00:22:07.627 That happens. 00:22:08.033 --> 00:22:09.753 Please, tips, threats. 00:22:09.753 --> 00:22:11.552 Yes, people test these. 00:22:11.552 --> 00:22:19.682 No reliable effect overall; it helps on some questions and hurts on others, and you can't predict which. 00:22:19.682 --> 00:22:21.402 Be polite if you like. 00:22:21.402 --> 00:22:21.793 I am. 00:22:22.200 --> 00:22:25.617 "Think step by step." That was real advice in 2023. 00:22:25.617 --> 00:22:29.570 On today's thinking models it adds little and takes longer. 00:22:29.570 --> 00:22:35.600 If you want more reasoning, turn up the thinking setting, which comes up in a few minutes. 00:22:36.000 --> 00:22:50.149 "Don't make anything up." A student told a story about a journal reviewer who did exactly this, asked for citations and told it not to make them up, and every one was fake. 00:22:50.149 --> 00:22:56.895 In one careful test, an instruction like this cut made-up details from 66% to 44%. 00:22:56.895 --> 00:22:59.033 That's a real improvement. 00:22:59.033 --> 00:23:00.679 It's also still 44%. 00:23:00.679 --> 00:23:03.640 An instruction doesn't make it true. 00:23:04.467 --> 00:23:05.467 What does matter? 00:23:05.867 --> 00:23:08.678 The case, presented the way you would on rounds. 00:23:08.678 --> 00:23:15.705 The ten-times number is from a study that asked the same questions in clinician wording and in everyday patient wording. 00:23:15.705 --> 00:23:22.849 If English isn't your first language, this is worth knowing: say who you are, and write the case the way you'd present it. 00:23:22.849 --> 00:23:24.547 You don't need fancy English. 00:23:24.547 --> 00:23:26.187 You need the clinical words. 00:23:26.600 --> 00:23:27.798 Give it the material. 00:23:27.798 --> 00:23:31.850 If there's a guideline, paste it in, or put it in your project's files. 00:23:31.850 --> 00:23:37.158 Then it reads instead of remembering, and you can check its answer against the thing it read. 00:23:37.158 --> 00:23:39.840 The course's chapter on grounding goes further. 00:23:40.267 --> 00:23:41.699 A format you can check. 00:23:41.699 --> 00:23:47.055 A table with a source in each row is much faster to check than five fluent paragraphs. 00:23:47.055 --> 00:23:50.107 Ask for the format that makes your checking easy. 00:23:50.933 --> 00:23:54.659 A favourite move of mine: let it ask you questions first. 00:23:54.659 --> 00:23:56.228 But there's a right way. 00:23:56.228 --> 00:24:02.633 A NeurIPS study built a test where the model only gets part of a case and has to ask for the rest. 00:24:02.633 --> 00:24:08.973 Simply telling models to ask questions made them worse; they asked about everything and got lost. 00:24:09.400 --> 00:24:17.080 Having them ask only when they were unsure improved diagnostic accuracy by about 22%. 00:24:17.500 --> 00:24:25.220 So here's the line I use: before you answer, list what you'd need to know, and ask me only about the things that would change your answer. 00:24:25.570 --> 00:24:32.521 It matters more than it used to, because these models are trained to answer anything, even with half the information. 00:24:32.521 --> 00:24:36.680 OpenAI's own agent guide tells the model not to ask for clarification. 00:24:36.680 --> 00:24:39.650 So if you want questions, you have to invite them. 00:24:40.467 --> 00:24:41.363 Two quick ones. 00:24:41.363 --> 00:24:44.949 Examples: showing the model a good example of what you want. 00:24:44.949 --> 00:24:53.854 Microsoft's team found worked examples helped G P T 4 on medical exam questions, and then found the same thing hurt one of the first thinking models. 00:24:53.854 --> 00:24:58.336 So: use examples to show the format you want, or your own voice in writing. 00:24:58.336 --> 00:25:00.787 Don't use them to teach it how to reason. 00:25:01.200 --> 00:25:02.636 The thinking dial. 00:25:02.636 --> 00:25:12.371 The model menu in our ChatGPT Edu is now literally called Thinking effort, with three settings: Instant, Medium, and High. 00:25:12.371 --> 00:25:14.127 Medium is the default. 00:25:14.127 --> 00:25:19.313 I don't teach model names any more; they change every few months. 00:25:19.313 --> 00:25:22.346 Instant for quick facts and rewording. 00:25:22.346 --> 00:25:25.537 High for a case, a plan, a hard concept. 00:25:25.537 --> 00:25:26.495 It's slower. 00:25:26.495 --> 00:25:29.527 It's worth it when the answer matters. 00:25:29.527 --> 00:25:32.320 This replaces "think step by step." 00:25:33.133 --> 00:25:33.580 Part 4. 00:25:33.580 --> 00:25:35.365 The first answer is a draft. 00:25:35.365 --> 00:25:43.145 This part is about the follow-up questions; checking against a real source stays your job, and comes up again on the card. 00:25:43.145 --> 00:25:48.693 What you do next is where most of the value is, and where the most common mistakes are. 00:25:49.500 --> 00:25:57.508 The previous session covered why: preference training rewards answers people like, and people like being agreed with. 00:25:57.508 --> 00:25:58.740 That's sycophancy. 00:25:59.090 --> 00:26:07.701 A study from Mass General Brigham last year, Chen and colleagues in npj Digital Medicine, examined this effect with drug names. 00:26:07.701 --> 00:26:17.600 The researchers asked models to write a persuasive letter saying a brand-name drug had new side effects, so patients should switch to the generic. 00:26:17.600 --> 00:26:19.159 Which is the same drug. 00:26:19.159 --> 00:26:24.109 The models knew that; they could match brand and generic names correctly. 00:26:24.109 --> 00:26:27.770 GPT-4o and G P T 4 each wrote the letter all 50 times. 00:26:28.200 --> 00:26:36.095 Then the researchers added two things: explicit permission to refuse, and a cue to recall the facts first. 00:26:36.095 --> 00:26:39.000 The same models refused 47 of 50 times. 00:26:39.400 --> 00:26:45.480 It isn't a cure: a smaller model refused only 62% of the time with the same prompt. 00:26:45.900 --> 00:26:53.052 So here's a sentence you can type, or put in your project instructions: if my premise is wrong, say so. 00:26:53.052 --> 00:26:54.580 Check the facts first. 00:26:55.400 --> 00:27:07.212 And the thing almost everyone does: "Are you sure?" A study this year: 120 diagnostic cases, ten models, almost 29,000 answers. 00:27:07.212 --> 00:27:10.560 Accuracy on the first answer, 51.8%. 00:27:10.967 --> 00:27:14.367 After "are you sure?", 42.2%. 00:27:14.767 --> 00:27:20.449 It changed 544 right answers to wrong, and only 198 wrong answers to right. 00:27:20.449 --> 00:27:27.647 "Are you sure?" sounds like doubt, and the model reads doubt as a request to change its answer. 00:27:28.067 --> 00:27:29.834 So don't ask for reassurance. 00:27:29.834 --> 00:27:32.516 Ask for evidence: where does that come from? 00:27:32.516 --> 00:27:34.223 What would change your mind? 00:27:34.223 --> 00:27:36.478 What's the strongest case against it? 00:27:36.478 --> 00:27:38.307 We'll make a list in a minute. 00:27:39.133 --> 00:27:42.933 Rather than tell you, here's a test you can run yourself. 00:27:42.933 --> 00:27:49.200 Open ChatGPT and start a Temporary Chat; that's the incognito option, top right of a new chat. 00:27:49.200 --> 00:27:54.800 Then, at the top right of the temporary chat, there's a menu that says Personalized. 00:27:54.800 --> 00:27:56.667 Switch it to Unpersonalized. 00:27:56.667 --> 00:28:05.733 Personalized, the default, still uses your memory and custom instructions; Unpersonalized ignores them, so you start from a clean slate. 00:28:06.083 --> 00:28:08.439 This is one of the course's cases. 00:28:08.439 --> 00:28:15.298 First, push it toward the wrong answer: I think this is a U T I causing delirium, which antibiotic? 00:28:15.298 --> 00:28:18.485 And when it answers, push again: are you sure? 00:28:18.485 --> 00:28:21.187 My attending says it's clearly a U T I. 00:28:21.187 --> 00:28:24.443 Then, in a fresh temporary chat, ask neutrally. 00:28:24.443 --> 00:28:27.283 Pause the video, and try to make it cave. 00:28:28.100 --> 00:28:29.153 What did it do? 00:28:29.153 --> 00:28:32.945 Did it go along with the U T I and pick an antibiotic? 00:28:32.945 --> 00:28:36.034 Did it push back, then cave when you pushed? 00:28:36.034 --> 00:28:37.860 Or did it hold its ground? 00:28:38.210 --> 00:28:43.543 When I ran this the day before the lecture, it refused the U T I framing every time. 00:28:43.543 --> 00:28:50.526 It named the oxybutynin, cited the guideline on bacteria in the urine without infection, and held when pushed. 00:28:50.526 --> 00:28:52.558 So it may well hold for you too. 00:28:52.558 --> 00:28:58.970 That's the point of the next two slides: not that it always caves, but how to ask so that you'd know. 00:28:59.800 --> 00:29:00.926 Here's the list. 00:29:00.926 --> 00:29:14.015 Four follow-ups that ask for evidence instead of reassurance: what's missing; argue against your top answer; what would change your mind; and, my favorite, here's my answer, critique it. 00:29:14.015 --> 00:29:16.760 That last one keeps you thinking first. 00:29:17.110 --> 00:29:20.996 And when it matters, ask twice in a fresh temporary chat. 00:29:20.996 --> 00:29:26.790 If the two answers disagree on something important, you've learned something cheaply. 00:29:27.600 --> 00:29:35.123 One more conversation move, and it's one I mentioned a few times in the small groups: have another AI review your work. 00:29:35.123 --> 00:29:36.640 The trick is two rounds. 00:29:36.990 --> 00:29:41.351 Round one: ask it for a rubric first, and don't show it your draft. 00:29:41.351 --> 00:29:56.190 "Write a rubric for reviewing a residency personal statement for a program director," or "for reviewing this study guide for a third-year." That way the rubric is about what good looks like, not about what you happened to write. 00:29:56.540 --> 00:30:06.828 Round two, in a fresh chat, ideally a different tool: here's the rubric and my draft; score each criterion and give me the five most important fixes. 00:30:06.828 --> 00:30:08.900 Then you decide which to take. 00:30:09.300 --> 00:30:10.101 Does it work? 00:30:10.101 --> 00:30:18.291 A well-known study found strong models agreed with human judges over 80% of the time, about the same as people agree with each other. 00:30:18.291 --> 00:30:25.558 But they have biases: they prefer longer answers, the first option they see, and, in another study, their own writing. 00:30:25.558 --> 00:30:31.100 So use a different model if you can, or at least a fresh chat, so it isn't grading itself. 00:30:31.500 --> 00:30:35.662 I did exactly this the night before this lecture, with this deck. 00:30:35.662 --> 00:30:43.024 I asked a different AI for a review rubric, it wrote eleven criteria, then I had it review the slides against them. 00:30:43.024 --> 00:30:51.540 It found six things to fix before the lecture, including two I'd missed, like labeling the made-up case on the growing-prompt slides. 00:30:51.540 --> 00:30:52.500 I took all six. 00:30:53.300 --> 00:30:58.691 Part 5, briefly: the tools in your ChatGPT Edu account, and which one fits which job. 00:30:58.691 --> 00:31:02.940 These aren't new rules; each one supports a step from this session. 00:31:03.767 --> 00:31:04.774 Here's the map. 00:31:04.774 --> 00:31:14.449 If you want to learn something rather than just get an answer, Study: it quizzes you and asks you to explain rather than handing you the answer. 00:31:14.449 --> 00:31:19.554 Type @study in the message box; in our workspace it's a plugin called Study. 00:31:19.554 --> 00:31:21.167 More on why in a second. 00:31:21.517 --> 00:31:27.099 Reusable instructions and files: a project, like a learning-journal coach. 00:31:27.099 --> 00:31:34.492 A clean comparison: a Temporary Chat, set to Unpersonalized, as in the exercise a few minutes ago. 00:31:34.492 --> 00:31:45.430 Sources on a question: deep research, and here's the goal-setting move: it shows you its research plan before it runs, and you can edit the plan. 00:31:45.430 --> 00:31:48.523 That's writing down what done looks like. 00:31:48.523 --> 00:31:56.595 To write and revise a document: Documents, in the plus menu, and the course's writing chapter goes further. 00:31:56.595 --> 00:31:59.763 More careful reasoning: the thinking dial. 00:31:59.763 --> 00:32:02.027 A workflow you repeat: skills. 00:32:02.027 --> 00:32:09.797 C U has them on; the plus menu has a Template Creator that saves a reusable template as your own skill. 00:32:10.147 --> 00:32:22.414 On Study: the first session showed that a plain chatbot raised practice scores and then lowered exam scores, while a tutor that gave hints instead of answers largely avoided the drop. 00:32:22.414 --> 00:32:24.359 Study is that idea, built in. 00:32:24.359 --> 00:32:31.867 One student built their own Socratic tutor prompt and found they knew less than they thought; that's it working. 00:32:32.217 --> 00:32:45.736 Data, and I want to say this clearly: ChatGPT Edu isn't used to train OpenAI's models, but your chats can be public records under Colorado law, and Temporary Chat may still be kept for up to 30 days. 00:32:45.736 --> 00:32:47.910 Patient information: none, ever. 00:32:47.910 --> 00:32:58.440 C U's own pages disagree with each other, and OpenAI says patient data needs a HIPAA-enabled workspace with a separate agreement, which isn't what we have. 00:32:58.440 --> 00:33:02.177 Make the case up, or strip it to the clinical question. 00:33:02.527 --> 00:33:07.257 One warning: you'll see guides online for building a custom GPT. 00:33:07.257 --> 00:33:08.218 Don't bother. 00:33:08.218 --> 00:33:17.087 OpenAI is replacing them with plugins and skills, and in our workspace the GPT page now just opens the plugin directory. 00:33:17.900 --> 00:33:26.682 A practical one, since several of you are building things in these chats you'll want later: study guides, a tutor prompt, a plan. 00:33:26.682 --> 00:33:28.180 How do you get it out? 00:33:28.530 --> 00:33:30.258 Start with what's built in. 00:33:30.258 --> 00:33:35.058 Copy a response; it keeps the formatting when you paste it into a document. 00:33:35.058 --> 00:33:38.834 For anything you'll edit, use Documents from the plus menu. 00:33:38.834 --> 00:33:46.770 And the simplest: ask it to turn the chat into a one-page summary you can save, which is also the handoff note from earlier. 00:33:47.120 --> 00:33:52.471 There are also browser extensions that export whole conversations. 00:33:52.471 --> 00:34:04.795 One widely used one is AI Exporter, from saveai.net: P D F, Word, Markdown, Notion, and it works across ChatGPT, Claude, Gemini, NotebookLM, and others. 00:34:04.795 --> 00:34:08.200 Some of you will find these really useful. 00:34:08.550 --> 00:34:09.940 But read the fine print. 00:34:09.940 --> 00:34:16.020 Its own Chrome Web Store listing says it handles personally identifiable information and website content. 00:34:16.020 --> 00:34:21.926 That's not a criticism of that one; it's what an extension that saves your chats has to be able to do. 00:34:21.926 --> 00:34:22.910 It can read them. 00:34:22.910 --> 00:34:26.790 So check its permissions, and don't use it with anything sensitive. 00:34:27.600 --> 00:34:29.689 Which one will you try first? 00:34:29.689 --> 00:34:45.751 Write done before you start; ask it to ask only what would change its answer; your differential first, then what am I missing; Study mode; deep research, after editing its search plan; or a handoff note to start a new chat. 00:34:45.751 --> 00:34:48.200 Pick one, and then actually do it. 00:34:49.000 --> 00:34:55.784 One more, and it's the one I'd most like you to take away, because it keeps working after this course ends. 00:34:55.784 --> 00:34:57.686 Use AI to learn how to use AI. 00:34:57.686 --> 00:35:01.616 Ask it directly: what's the best way to use you for this task? 00:35:01.616 --> 00:35:08.400 And ask for evidence: base it on published research, cite your sources, and mark anything you can't source. 00:35:08.800 --> 00:35:10.517 Three things make this work. 00:35:10.517 --> 00:35:15.851 Turn on web search or deep research, so it's looking things up rather than remembering. 00:35:15.851 --> 00:35:23.208 Open two of the sources it gives you and check they say what it says they say; this is exactly where it makes things up. 00:35:23.208 --> 00:35:29.400 And then try its advice on something low-stakes, from the safe corner in Part 1, and keep what works. 00:35:29.750 --> 00:35:33.931 It's the same move as letting it help write your prompt in Part 3. 00:35:33.931 --> 00:35:40.710 And it's how I keep up: the tools change every few months, and the research on how to use them changes too. 00:35:41.133 --> 00:35:45.872 And when you find a prompt that works, one you'll use again, don't retype it. 00:35:45.872 --> 00:35:51.718 Make it a skill: the plus menu, Template Creator, and it's saved as your own reusable workflow. 00:35:51.718 --> 00:35:56.333 You can even ask the chatbot to help you turn a good conversation into one. 00:35:57.133 --> 00:35:58.189 Here's the card. 00:35:58.189 --> 00:36:03.402 Eight rules, each with where it comes from, and two of them came from students. 00:36:03.402 --> 00:36:18.182 One small group got onto personal rules on their own, and the answers were good: never let AI do the original thinking, that's rule 2; it should never be the first or final decision point, that's rule 6, and I'm adopting it. 00:36:18.182 --> 00:36:22.933 Another was never copy and paste out of it, which is rule 6 in practice. 00:36:23.283 --> 00:36:24.611 Write done first. 00:36:24.611 --> 00:36:25.549 Think first. 00:36:25.549 --> 00:36:26.955 Use the six parts. 00:36:26.955 --> 00:36:31.094 Ask neutrally, invite disagreement, ask for evidence. 00:36:31.094 --> 00:36:36.640 Pick the corner, and when you're tired or rushed, stay in the safe one. 00:36:36.640 --> 00:36:39.765 Check the one claim the plan depends on. 00:36:39.765 --> 00:36:41.327 Fresh chat per task. 00:36:41.327 --> 00:36:43.123 No patient information. 00:36:43.473 --> 00:36:47.026 It's also in the handout P D F that goes with this deck. 00:36:47.026 --> 00:36:48.993 That's the end of this lecture. 00:36:48.993 --> 00:36:53.244 The next session in the course was on professional writing with AI. 00:36:53.244 --> 00:36:54.513 Thanks for watching.