WEBVTT 00:00:00.000 --> 00:00:01.707 A note before we begin. 00:00:01.707 --> 00:00:09.127 The narration in this video is synthetic speech, generated from the lecture notes of Dr. Sean Davis. 00:00:09.127 --> 00:00:11.724 It is not a recording of his voice. 00:00:11.724 --> 00:00:14.840 The lecture was given on October 14, 2026. 00:00:15.667 --> 00:00:16.108 Welcome. 00:00:16.108 --> 00:00:21.730 This is a recorded version of a lecture from a course on artificial intelligence for medical students. 00:00:21.730 --> 00:00:24.707 It runs about forty minutes, plus any pauses you take. 00:00:25.057 --> 00:00:26.244 Here is the plan. 00:00:26.244 --> 00:00:33.927 First, four words, AI, machine learning, deep learning, and generative AI, and how they fit inside each other. 00:00:33.927 --> 00:00:45.801 Then one model from start to finish: the E G F R on a chemistry panel, where its number comes from, how a model is fit to data, and what happens when you add a predictor. 00:00:45.801 --> 00:00:52.017 Then the fitting loop that every model comes from, and two questions that sort any model. 00:00:52.367 --> 00:00:54.122 After that, what goes wrong. 00:00:54.122 --> 00:00:55.877 A model can be too flexible. 00:00:55.877 --> 00:00:59.262 A neural network is the same loop with far more dials. 00:00:59.262 --> 00:01:03.086 And a score on cases a model has already seen proves nothing. 00:01:03.086 --> 00:01:06.847 Then back to the clinic, and what this changes on the wards. 00:01:07.197 --> 00:01:08.464 Along the way there are a few questions. 00:01:08.464 --> 00:01:10.144 Pause the video and answer each one before you go on. 00:01:10.144 --> 00:01:12.997 The next session in the course starts with the chatbot, which is the same idea with words. 00:01:13.800 --> 00:01:19.430 Recall the history of AI: how we got from rules written by hand to the chatbots of today. 00:01:19.430 --> 00:01:23.541 Here are four words from that history, and how they fit together. 00:01:23.541 --> 00:01:27.400 If you haven't seen that session, nothing here depends on it. 00:01:27.750 --> 00:01:30.808 Artificial intelligence is the largest circle. 00:01:30.808 --> 00:01:35.328 The sizes of the circles aren't meant to show how big each field is. 00:01:35.328 --> 00:01:40.380 It means a computer doing a task that seems to need judgment, by any method. 00:01:40.380 --> 00:01:46.030 My-sin, from the 1970s, advised on antibiotics with rules that experts wrote by hand. 00:01:46.380 --> 00:01:48.154 Machine learning is inside it. 00:01:48.154 --> 00:01:50.106 The rules aren't written by hand. 00:01:50.106 --> 00:01:51.703 They are learned from data. 00:01:51.703 --> 00:01:55.252 The E G F R equation on every chemistry panel is an example. 00:01:55.252 --> 00:01:57.736 A statistician would call it a regression. 00:01:57.736 --> 00:02:04.420 The line between statistics and machine learning is blurry, and for this course a model learned from data counts. 00:02:04.770 --> 00:02:07.541 Deep learning is inside machine learning. 00:02:07.541 --> 00:02:13.489 It uses a neural network: many layers of simple units, each passing numbers to the next. 00:02:13.489 --> 00:02:18.490 Models that read images, such as chest X-rays, are usually built this way. 00:02:18.840 --> 00:02:21.321 Generative AI is inside deep learning. 00:02:21.321 --> 00:02:24.389 These models write new text or make new images. 00:02:24.389 --> 00:02:26.609 A chatbot is the example you know. 00:02:26.609 --> 00:02:32.159 So a chatbot is generative AI, and it's also deep learning, machine learning, and AI. 00:02:32.159 --> 00:02:35.880 The words overlap, but each one means something specific. 00:02:36.230 --> 00:02:42.159 Today we start in the second circle, with the E G F R, and reach the third, deep learning. 00:02:42.159 --> 00:02:45.981 The next session reaches the smallest circle, the chatbot. 00:02:45.981 --> 00:02:49.670 The same idea, learning from data, is under all of them. 00:02:50.500 --> 00:02:54.879 Every model in this talk has the same setup, and it's worth one slide. 00:02:54.879 --> 00:03:04.513 Each person, person i on the slide, has a target, which is the thing we want to predict, and a set of predictors, which are the things we know about them. 00:03:04.513 --> 00:03:09.143 The model is a function: it takes the predictors and returns a prediction. 00:03:09.143 --> 00:03:12.271 The prediction is almost never exactly the target. 00:03:12.271 --> 00:03:14.147 What's left over is the error. 00:03:14.147 --> 00:03:17.838 So the target equals the model's prediction plus the error. 00:03:17.838 --> 00:03:23.594 The target can be a number, like weight, or the chance that a patient falls in the hospital. 00:03:23.594 --> 00:03:27.660 It can also be a category: benign or malignant, pneumonia or not. 00:03:28.010 --> 00:03:31.522 The error isn't only that each person is a little different. 00:03:31.522 --> 00:03:36.264 It also holds any error in measuring the target, and in measuring the predictors. 00:03:36.264 --> 00:03:41.357 My PhD is in human genetics, and one of our constant problems was defining the disease. 00:03:41.357 --> 00:03:45.982 If you can't define it well, it's very hard to find the gene that goes with it. 00:03:45.982 --> 00:03:48.850 The best models start with the best measurements. 00:03:49.200 --> 00:03:55.202 Those four words, target, model, predictors, error, will come back all session. 00:03:55.202 --> 00:03:56.950 The E G F R is a model. 00:03:56.950 --> 00:03:58.546 A chatbot is a model. 00:03:58.546 --> 00:04:04.320 The f can be as simple as a straight line or as complex as a neural network. 00:04:04.733 --> 00:04:06.664 How do we choose the model? 00:04:06.664 --> 00:04:08.523 We need a way to score it. 00:04:08.523 --> 00:04:13.314 A loss function adds up the errors across everyone into one number. 00:04:13.314 --> 00:04:20.393 One common choice squares each error and adds them up, which is called the residual sum of squares. 00:04:20.393 --> 00:04:22.824 Another is the plain average miss. 00:04:22.824 --> 00:04:24.754 Big errors make a big loss. 00:04:24.754 --> 00:04:31.976 Fitting a model means choosing the version of the function that makes the loss as small as it can be. 00:04:31.976 --> 00:04:34.693 On the next slide, we do that by hand. 00:04:35.500 --> 00:04:38.851 Here is that setup with the simplest model there is. 00:04:38.851 --> 00:04:40.205 The target is weight. 00:04:40.205 --> 00:04:42.009 The one predictor is height. 00:04:42.009 --> 00:04:43.943 Each dot is one made-up adult. 00:04:43.943 --> 00:04:49.615 The model is a straight line with two numbers in it: a, the intercept, and b, the slope. 00:04:49.615 --> 00:04:54.577 Each thin line is that person's error, how far their weight is from the line. 00:04:54.577 --> 00:04:59.540 The loss is the sum of the squared errors, and the average miss is beside it. 00:04:59.890 --> 00:05:03.822 If you have the deck open, move the two sliders yourself. 00:05:03.822 --> 00:05:09.548 Sliding the intercept moves the whole line up and down; sliding the slope tilts it. 00:05:09.548 --> 00:05:13.894 As the line starts, the average miss is about thirty kilograms. 00:05:13.894 --> 00:05:18.930 Try to make the loss as small as you can, then press "Show the best fit". 00:05:19.280 --> 00:05:24.926 I could move these sliders all day long, but it turns out we have good ways of fitting a model. 00:05:24.926 --> 00:05:32.059 Fitting is what the computer does when you press that button: it finds the line with the smallest sum of squared errors. 00:05:32.059 --> 00:05:34.912 For a straight line, that's the best you can do. 00:05:34.912 --> 00:05:43.589 In this made-up data the best line has an intercept of about minus ninety and a slope of about one, and its average miss is about eight kilograms. 00:05:43.589 --> 00:05:45.729 The numbers themselves don't matter. 00:05:45.729 --> 00:05:50.306 What matters is that they were chosen to make the loss as small as it can be. 00:05:50.306 --> 00:05:55.120 This is called linear regression, and it's machine learning in its simplest form. 00:05:55.533 --> 00:05:59.020 What you just did by hand has a name: the fitting loop. 00:05:59.020 --> 00:06:03.965 Guess the dials, score the misses with the loss, adjust the dials, and repeat. 00:06:03.965 --> 00:06:06.691 When to stop is a question we come back to. 00:06:06.691 --> 00:06:13.538 For a straight line there's a formula that jumps to where the loop would end, which is what the button uses. 00:06:13.538 --> 00:06:18.293 Bigger models, like neural networks, really do run the loop, lap after lap. 00:06:18.700 --> 00:06:21.233 Even the best line leaves errors. 00:06:21.233 --> 00:06:25.760 Some tall people are light and some short people are heavy. 00:06:25.760 --> 00:06:27.142 So what's missing? 00:06:27.142 --> 00:06:30.135 What else would help us predict weight? 00:06:30.135 --> 00:06:34.356 Age, sex, how much of the weight is muscle, waist size. 00:06:34.356 --> 00:06:38.269 Each one is another predictor, another column in X. 00:06:38.269 --> 00:06:42.260 The E G F R does this with creatinine, age, and sex. 00:06:43.067 --> 00:06:46.347 Now to a clinic, and a model you already use. 00:06:47.167 --> 00:06:50.474 Pre-operative clinic, on your surgery sub-internship. 00:06:50.474 --> 00:06:55.529 A sixty-two-year-old woman is being worked up for a laparoscopic cholecystectomy. 00:06:55.529 --> 00:07:03.767 Her basic metabolic panel is back, and one line is flagged low: an E G F R of fifty-seven, where the lab's reference is above sixty. 00:07:04.167 --> 00:07:10.751 The attending asks whether her kidneys are all right for the contrast CT the surgeon wants. 00:07:10.751 --> 00:07:13.356 You answer the way everyone answers. 00:07:13.356 --> 00:07:19.217 E G F R fifty-seven, stage three a, mildly reduced, probably fine with hydration. 00:07:19.217 --> 00:07:23.124 The attending nods, and that's the whole conversation. 00:07:23.124 --> 00:07:26.887 You'll have it several hundred times in intern year. 00:07:27.237 --> 00:07:30.291 The panel on the slide is made up for this course. 00:07:30.291 --> 00:07:31.757 It isn't a real patient. 00:07:32.567 --> 00:07:34.841 Before I go on, commit to an answer. 00:07:34.841 --> 00:07:37.179 Where does the fifty-seven come from? 00:07:37.179 --> 00:07:39.579 Did a machine measure it in her blood? 00:07:39.579 --> 00:07:42.486 Was it measured from a timed urine collection? 00:07:42.486 --> 00:07:46.087 Or did an equation calculate it from three other numbers? 00:07:46.437 --> 00:07:49.677 Pause the video here and pick one before you go on. 00:07:50.027 --> 00:07:51.607 The answer is the equation. 00:07:51.607 --> 00:07:53.714 Nobody measured her kidney function. 00:07:53.714 --> 00:07:58.104 Three numbers went into an equation, and the equation produced fifty-seven. 00:07:58.104 --> 00:08:00.153 The point isn't knowing the answer. 00:08:00.153 --> 00:08:02.787 It's noticing that the number was calculated. 00:08:03.600 --> 00:08:06.560 Nobody measured her glomerular filtration rate. 00:08:06.967 --> 00:08:16.367 Measuring it means infusing a marker that the kidney filters and does nothing else with, such as iohexol, and timing how fast it clears. 00:08:16.767 --> 00:08:20.927 That takes hours, and it's rarely done in routine care. 00:08:21.333 --> 00:08:27.994 The fifty-seven was computed from three things the lab did have: her creatinine, her age, and her sex. 00:08:27.994 --> 00:08:35.373 Something turned those three numbers into a kidney function, and you trusted it enough to clear her for contrast. 00:08:35.800 --> 00:08:38.270 So where did that equation come from? 00:08:38.270 --> 00:08:40.674 Not from the physiology of nephrons. 00:08:40.674 --> 00:08:42.343 It was learned from data. 00:08:42.343 --> 00:08:45.814 A few thousand people did have their G F R measured. 00:08:45.814 --> 00:08:54.760 Their creatinine, age, and sex were given to a computer with one instruction: find the formula that best predicts the measured number. 00:08:55.167 --> 00:08:56.999 The E G F R is a model. 00:08:56.999 --> 00:09:04.647 It's the model you already trust most, and you have probably never asked what it was trained on. 00:09:04.997 --> 00:09:06.344 Remember this patient. 00:09:06.344 --> 00:09:11.856 She weighs forty-five kilograms and has lost a good deal of muscle over a year of illness. 00:09:11.856 --> 00:09:13.938 We'll come back to her at the end. 00:09:13.938 --> 00:09:17.797 Nobody has told her that a model had a part in the fifty-seven. 00:09:18.600 --> 00:09:20.879 Here is fitting, done by hand. 00:09:20.879 --> 00:09:26.121 Each dot is one made-up person: creatinine across, measured G F R up. 00:09:26.121 --> 00:09:31.362 The thick curve is a simple equation with two numbers in it, a and b. 00:09:31.362 --> 00:09:33.793 Those two numbers are the dials. 00:09:33.793 --> 00:09:41.162 The number a sets how high the curve sits: it's the G F R the curve gives at a creatinine of one. 00:09:41.162 --> 00:09:45.644 The number b sets how fast G F R falls as creatinine rises. 00:09:45.644 --> 00:09:51.722 Each thin line is a miss: how far one person's measured G F R is from the curve. 00:09:51.722 --> 00:09:55.520 The number beside the sliders is the average miss. 00:09:55.870 --> 00:10:01.571 If you have the deck open, move the two sliders and make the average miss as small as you can. 00:10:01.571 --> 00:10:05.150 Where the curve starts, the average miss is about nineteen. 00:10:05.150 --> 00:10:07.030 Then press "Show the best fit". 00:10:07.433 --> 00:10:13.006 Fitting means a computer turns these dials until the error is as small as it can get. 00:10:13.006 --> 00:10:16.481 At the best fit, the average miss falls to about six. 00:10:16.481 --> 00:10:20.480 This is the same fitting loop you ran with height and weight. 00:10:20.480 --> 00:10:25.659 Here the computer tries every setting the sliders allow and keeps the best one. 00:10:25.659 --> 00:10:29.134 Real fitting uses faster methods, with the same goal. 00:10:29.134 --> 00:10:40.344 The real E G F R equation has a few more numbers in it, and it was fit to eight thousand two hundred fifty-four people whose G F R was measured, not to forty made-up dots. 00:10:40.344 --> 00:10:43.753 A chatbot is the same idea with billions of numbers. 00:10:44.103 --> 00:10:49.446 The data on this slide are made up, and the curve is simpler than the real equation. 00:10:49.446 --> 00:10:51.863 Its numbers aren't the real E G F R's. 00:10:52.667 --> 00:10:54.759 Now add a second predictor. 00:10:54.759 --> 00:11:01.270 The real E G F R equation uses sex as well as creatinine and age, so we do the same. 00:11:01.270 --> 00:11:08.867 Same made-up people, same idea, but now the equation can have a different level for women and men. 00:11:09.217 --> 00:11:16.457 If you have the deck open, turn on "Add sex" and watch the average miss and the R S S. 00:11:16.807 --> 00:11:20.877 The fit now gives each sex its own curve, and both numbers fall. 00:11:20.877 --> 00:11:28.510 The average miss goes from about seven to about five, and the R S S from one point three seven to zero point seven four. 00:11:28.510 --> 00:11:36.397 In this made-up data, women have a lower G F R at the same creatinine, so a model that knows the sex can follow both groups. 00:11:36.397 --> 00:11:39.896 Statisticians call the sum of squared misses the R S S. 00:11:39.896 --> 00:11:45.811 The fit makes it as small as it can, so the R S S can only go down when a predictor is added. 00:11:45.811 --> 00:11:55.607 Whether the drop is real, or only a better fit to these forty people, is a different question, and we come back to it with a slider later in this session. 00:11:56.033 --> 00:12:03.233 Fitting with a second predictor is the same job as before: find the numbers that make the misses smallest. 00:12:03.583 --> 00:12:05.052 The data are made up. 00:12:05.052 --> 00:12:13.583 The real equation has more terms, and it was fit to eight thousand two hundred fifty-four people whose G F R was measured. 00:12:14.400 --> 00:12:18.037 Next: six words, the fitting loop, and two questions. 00:12:18.037 --> 00:12:21.880 Together they sort every model you'll meet in residency. 00:12:22.700 --> 00:12:27.900 The E G F R story contains almost every word you need for the rest of the course. 00:12:28.250 --> 00:12:31.722 Predictors are the inputs: creatinine, age, and sex. 00:12:31.722 --> 00:12:35.862 The response is the thing being predicted: the measured G F R. 00:12:35.862 --> 00:12:43.407 The training set is the eight thousand two hundred fifty-four adults whose numbers were used to fit the equation. 00:12:43.407 --> 00:12:47.748 They came from ten studies and were mostly research participants. 00:12:47.748 --> 00:12:49.417 About a third were Black. 00:12:49.417 --> 00:12:54.492 The validation set is a second group held back while the model is being fit. 00:12:54.492 --> 00:13:00.034 It's used to choose between versions: when to stop, or which of two models to keep. 00:13:00.034 --> 00:13:02.771 The test set is checked once, at the end. 00:13:02.771 --> 00:13:10.250 For the E G F R, that was four thousand fifty people from twelve other studies that the equation had never seen. 00:13:10.600 --> 00:13:13.558 Watch the names, because they vary by community. 00:13:13.558 --> 00:13:21.508 The response is what statisticians call it; machine learning says target, which is the word we used on the setup slide, or label. 00:13:21.508 --> 00:13:22.124 Same idea. 00:13:22.124 --> 00:13:30.259 And the E G F R paper calls those four thousand fifty people its validation data set, and many medical papers use the word that way. 00:13:30.259 --> 00:13:35.374 In machine learning, a group held back and checked once at the end is the test set. 00:13:35.374 --> 00:13:41.044 When you read a paper, ask what the held-back cases were used for, not what they are called. 00:13:41.044 --> 00:13:44.680 And the model is the equation itself, once it has been fit. 00:13:45.030 --> 00:13:50.641 The letter f stands for function, which just means a rule that turns inputs into an output. 00:13:50.641 --> 00:13:58.842 Fit means that a computer tried numbers in the equation until its predictions were as close to the measured values as they could get. 00:13:58.842 --> 00:14:01.987 For the E G F R, the model is a handful of numbers. 00:14:01.987 --> 00:14:03.775 For a chatbot, it's billions. 00:14:03.775 --> 00:14:05.070 The role is the same. 00:14:05.900 --> 00:14:09.232 Here are those three sets as a picture. 00:14:09.232 --> 00:14:12.820 Start with one table, one row per patient. 00:14:12.820 --> 00:14:21.364 For the E G F R that's creatinine, age, and sex, the predictors, and the measured G F R, the target. 00:14:21.364 --> 00:14:23.500 Every row has the answer. 00:14:23.850 --> 00:14:26.412 Before any fitting, you split the rows. 00:14:26.412 --> 00:14:31.865 Most go into the training set, and that's what the fitting loop turns the dials on. 00:14:31.865 --> 00:14:34.361 A smaller slice is the validation set. 00:14:34.361 --> 00:14:41.785 The loop gets to look at it, to decide when to stop and which model to keep, so think of it as a pseudo test set. 00:14:41.785 --> 00:14:44.084 And the last slice is the test set. 00:14:44.084 --> 00:14:44.872 It's locked. 00:14:44.872 --> 00:14:50.785 Nobody looks at it while the model is being built, and you check it once, at the very end. 00:14:50.785 --> 00:14:52.690 That's the number you report. 00:14:53.040 --> 00:14:56.146 The shares on the slide are typical, not a rule. 00:14:56.146 --> 00:14:59.058 Two details matter more than the percentages. 00:14:59.058 --> 00:15:03.005 Split by patient, so the same person never lands in two sets. 00:15:03.005 --> 00:15:09.800 And split before you fit, because once the test cases have influenced a choice, they're no longer a test. 00:15:10.150 --> 00:15:12.645 The E G F R researchers did one better. 00:15:12.645 --> 00:15:16.163 Their test set wasn't a random slice of the same table. 00:15:16.163 --> 00:15:20.001 It was four thousand fifty people from twelve other studies. 00:15:20.001 --> 00:15:25.950 A test on different people, from different studies, is harder to pass, and it tells you more. 00:15:26.767 --> 00:15:31.810 Here is the history of one equation, rebuilt from the numbers in the papers. 00:15:31.810 --> 00:15:35.327 Read the last column as a pattern, not as nephrology. 00:15:35.677 --> 00:15:44.665 M D R D was fit to one thousand and seventy people who already had kidney disease, and tested on five hundred fifty-eight more. 00:15:44.665 --> 00:15:51.530 In healthy people it gave values that were too low, and healthy people were labelled as diseased. 00:15:51.530 --> 00:15:59.740 The 2009 equation was more accurate at higher G F R, and it included a term that adjusted the result for Black race. 00:15:59.740 --> 00:16:06.109 Even with that term, the 2021 study found it overestimated measured G F R in Black adults. 00:16:06.109 --> 00:16:10.356 An overestimate makes kidney disease look milder than it is. 00:16:10.356 --> 00:16:16.018 The 2021 refit removed race, and it's still a few points off in both directions. 00:16:16.018 --> 00:16:23.237 A version that adds cystatin C, a second blood marker that doesn't depend on muscle, is more accurate. 00:16:23.667 --> 00:16:25.392 Every row says the same thing. 00:16:25.392 --> 00:16:30.165 The model was fit to these people, and here is what happened to people unlike them. 00:16:30.165 --> 00:16:34.363 The training population is part of the model, as much as the equation is. 00:16:34.363 --> 00:16:37.468 That's a take-home message, and we'll come back to it. 00:16:37.468 --> 00:16:44.427 Why the race term was there, and who was harmed by it, is the subject of the bias and equity session later in the course. 00:16:45.233 --> 00:16:47.777 Here is the fitting loop as a picture. 00:16:47.777 --> 00:16:51.391 You ran it by hand with the height and weight sliders. 00:16:51.391 --> 00:16:54.336 Step one, guess the dials; anywhere will do. 00:16:54.336 --> 00:16:57.683 Step two, use them to predict every training case. 00:16:57.683 --> 00:17:02.970 Step three, score the predictions: add up the misses into one number, the loss. 00:17:02.970 --> 00:17:07.388 Step four, adjust the dials in the direction that lowers the loss. 00:17:07.388 --> 00:17:08.793 Then go around again. 00:17:09.143 --> 00:17:10.207 When do you stop? 00:17:10.207 --> 00:17:12.897 Not when the training loss stops shrinking. 00:17:12.897 --> 00:17:14.024 Look at the chart. 00:17:14.024 --> 00:17:23.158 The red line is the loss on the training cases, and it keeps falling: with enough laps, a model can fit the cases it learns from almost perfectly. 00:17:23.158 --> 00:17:29.040 The gold line is the loss on a second set of cases the loop never fits to, the validation set. 00:17:29.040 --> 00:17:31.417 It falls, bottoms out, and then rises. 00:17:31.417 --> 00:17:38.112 From there on, the model is memorizing the training cases, noise included, instead of learning the pattern. 00:17:38.112 --> 00:17:39.300 That's overfitting. 00:17:39.300 --> 00:17:42.679 So the loop stops where the validation loss is lowest. 00:17:42.679 --> 00:17:49.061 The validation set is also how you choose between models, for example a simple one and a flexible one. 00:17:49.061 --> 00:17:52.565 You'll see that happen with the slider in a few minutes. 00:17:52.565 --> 00:17:58.383 Think of it as a pseudo test set: the loop gets to look at it, so it can't be the final word. 00:17:58.733 --> 00:18:01.703 Then you leave the loop, and there's one more step. 00:18:01.703 --> 00:18:08.573 Step five: test the model once, on a third set of cases, the test set, that neither the loop nor the choices ever saw. 00:18:08.573 --> 00:18:13.929 The loss inside the loop only tells you how well the model fits the cases it was built from. 00:18:13.929 --> 00:18:16.665 The test set tells you how it does on new ones. 00:18:16.665 --> 00:18:20.333 We'll see what happens when that step is skipped or done badly. 00:18:20.683 --> 00:18:28.544 For a straight line, and for the E G F R equation, there's a formula that lands where the loop would end, so nobody needs to go around. 00:18:28.544 --> 00:18:34.483 For a neural network there's no such shortcut, and the computer really does run this loop, many times. 00:18:34.833 --> 00:18:37.439 The strip at the bottom is the whole path. 00:18:37.439 --> 00:18:41.657 Fitting needs training cases with both the predictors and the truth. 00:18:41.657 --> 00:18:44.945 The loop is checked on the validation set as it runs. 00:18:44.945 --> 00:18:48.294 The model it gives, f, is tested once on the test set. 00:18:48.294 --> 00:18:53.753 Then a new patient arrives with only the predictors, and f turns them into a prediction. 00:18:54.103 --> 00:19:02.162 The E G F R, a sepsis alert in the E H R, and a chatbot differ in the data and in how many dials they have. 00:19:02.162 --> 00:19:03.743 The loop is the same. 00:19:04.093 --> 00:19:06.501 So there are two take-home messages here. 00:19:06.501 --> 00:19:07.382 There's a loop. 00:19:07.382 --> 00:19:14.253 And there's such a thing as overfitting: a model can do very well on the cases it was fit to, and poorly on new ones. 00:19:15.067 --> 00:19:18.559 Two questions sort any model in about ten seconds. 00:19:18.559 --> 00:19:19.747 Start at the top. 00:19:20.097 --> 00:19:23.333 First: does the training data contain the answer? 00:19:23.333 --> 00:19:24.720 For the E G F R, yes. 00:19:24.720 --> 00:19:27.560 Someone measured every one of those G F Rs. 00:19:27.560 --> 00:19:29.343 That's supervised learning. 00:19:29.343 --> 00:19:34.560 If there's no answer column, only patterns to find, it's unsupervised learning. 00:19:34.560 --> 00:19:44.137 The example is breast cancer subtypes, which were found by grouping tumors that share gene activity, with nobody telling the computer the groups. 00:19:44.137 --> 00:19:49.817 We won't say more about unsupervised learning today, but it's worth knowing it exists. 00:19:50.167 --> 00:19:54.403 Second, and only for a supervised model: what does it output? 00:19:54.403 --> 00:19:56.556 A number makes it a regression. 00:19:56.556 --> 00:20:02.181 The E G F R is a regression, and so is a model that reports an ejection fraction. 00:20:02.181 --> 00:20:06.417 In the picture, each red line is how far a prediction misses. 00:20:06.417 --> 00:20:11.833 A category makes it a classification: benign or melanoma, sepsis or no sepsis. 00:20:11.833 --> 00:20:16.972 Here the model draws a boundary, and a case on the wrong side is an error. 00:20:16.972 --> 00:20:25.167 An unsupervised tool that sorts cases into groups isn't a classification, because nobody told it what the groups were. 00:20:25.567 --> 00:20:28.774 The difference matters for how you judge a model. 00:20:28.774 --> 00:20:31.065 A regression is wrong by an amount. 00:20:31.065 --> 00:20:35.058 A classification is wrong by a kind, a miss or a false alarm. 00:20:35.058 --> 00:20:37.676 A miss costs the patient who was missed. 00:20:37.676 --> 00:20:43.567 A false alarm costs the staff who answer it, and the patient who is worked up for nothing. 00:20:43.917 --> 00:20:46.713 Sometimes it isn't obvious which kind a model is. 00:20:46.713 --> 00:20:52.077 Sorting that out is a good first step to understanding how it was built and how it'll be used. 00:20:52.900 --> 00:20:56.902 Now you try, with the two questions from the last slide. 00:20:56.902 --> 00:21:00.404 First: does the training data contain the answer? 00:21:00.404 --> 00:21:03.620 Second: is the output a category or a number? 00:21:03.970 --> 00:21:09.587 A model watches a heart ultrasound video and reports an ejection fraction of fifty-five percent. 00:21:09.587 --> 00:21:11.050 What kind of model is it? 00:21:11.400 --> 00:21:14.600 Pause the video here and pick an answer before you go on. 00:21:14.950 --> 00:21:16.609 The answer is regression. 00:21:16.609 --> 00:21:22.914 The training data contain the answer, a measured ejection fraction, and the output is a number. 00:21:22.914 --> 00:21:25.170 This is the EchoNet Dynamic model. 00:21:25.170 --> 00:21:32.006 The same model also says whether the heart failure has reduced ejection fraction, which is a yes or no. 00:21:32.006 --> 00:21:34.926 One system can give a number and a category. 00:21:34.926 --> 00:21:38.510 The type of problem depends on the output you ask for. 00:21:39.333 --> 00:21:40.657 Same two questions. 00:21:40.657 --> 00:21:46.231 A model looks at a photo of a skin lesion and answers: benign nevus or melanoma. 00:21:46.231 --> 00:21:47.973 What kind of model is it? 00:21:48.323 --> 00:21:51.523 Pause the video here and pick an answer before you go on. 00:21:51.873 --> 00:21:53.786 The answer is classification. 00:21:53.786 --> 00:21:59.194 Every training image carried a diagnosis, and the output is one of two categories. 00:21:59.194 --> 00:22:02.097 This is the dermatology model from Stanford. 00:22:02.097 --> 00:22:07.967 A classification is wrong by a kind: it can call a melanoma benign, or a nevus malignant. 00:22:07.967 --> 00:22:12.913 Which of those costs more is a clinical question, and it's yours to answer. 00:22:13.733 --> 00:22:16.595 Four fields, the same two questions. 00:22:16.595 --> 00:22:20.013 Read the Type column, then the last column. 00:22:20.363 --> 00:22:26.628 In cardiology, a model reads ultrasound video and gives a number, the ejection fraction. 00:22:26.628 --> 00:22:31.825 In pediatrics, a model reads a hand X-ray and gives a bone age in months. 00:22:31.825 --> 00:22:35.385 Both are regressions: they are wrong by an amount. 00:22:35.385 --> 00:22:40.653 In dermatology and pathology, a model reads an image and gives a category. 00:22:40.653 --> 00:22:44.283 Both are classifications: they are wrong by a kind. 00:22:44.633 --> 00:22:48.500 Now look at the last column, because the same lesson is in it. 00:22:48.500 --> 00:22:57.043 The cardiology model was off by four point one points at the hospital where it was built, and by six point zero at another health system. 00:22:57.043 --> 00:23:08.455 In the pathology challenge, the best entry scored zero point nine nine four on the area under the curve, and the worst scored zero point five five six, barely better than a coin flip. 00:23:08.455 --> 00:23:12.446 Results on the authors' own test data are the optimistic number. 00:23:12.446 --> 00:23:14.753 Ask where it was tested, and by whom. 00:23:15.103 --> 00:23:28.753 For the record, three more examples are listed under the table: an E C G rhythm classifier, tumor subtypes found without any diagnoses, and the diabetic retinopathy system from the history of AI. 00:23:28.753 --> 00:23:31.623 They are here so you can find the papers. 00:23:32.433 --> 00:23:35.558 If you have the deck open, move the slider and watch. 00:23:35.558 --> 00:23:43.164 In the video the slider sits near the simple end, a straight line, and the chart on the right shows both errors at every setting. 00:23:43.164 --> 00:23:51.124 I simulated these data, so I know exactly where they came from: the dashed line is the real pattern, which the model never gets to see. 00:23:51.124 --> 00:23:54.189 The filled dots are the cases the model learns from. 00:23:54.189 --> 00:23:59.496 The hollow dots were held back for testing, and while we fit, we pretend they don't exist. 00:23:59.496 --> 00:24:03.033 Under the charts, the two numbers are the error on each set. 00:24:03.383 --> 00:24:05.837 At the far left the model is too simple. 00:24:05.837 --> 00:24:11.113 It's a flat or straight line through a curve, and it's wrong everywhere, on both sets. 00:24:11.113 --> 00:24:17.983 Statisticians call that kind of error bias, in a technical sense that has nothing to do with the social meaning. 00:24:18.333 --> 00:24:19.744 Now let the model bend. 00:24:19.744 --> 00:24:25.815 As you slide right, the training error goes down, and for a while the test error goes down with it. 00:24:25.815 --> 00:24:28.820 Then, somewhere in the middle, something happens. 00:24:28.820 --> 00:24:33.051 The training error keeps falling, but the test error starts to climb. 00:24:33.051 --> 00:24:42.433 Keep going and it gets kind of absurd: the curve bends through every filled dot, the training error drops to zero, and the test error goes off the chart. 00:24:42.433 --> 00:24:48.014 The model has learned the random scatter in these twelve cases along with the real pattern. 00:24:48.014 --> 00:24:49.853 That error is called variance. 00:24:50.203 --> 00:24:55.387 That's overfitting, and here's the real definition, from a standard machine learning textbook. 00:24:55.387 --> 00:25:02.226 A model overfits when a simpler model would do worse on the training cases but better on new cases from the same population. 00:25:02.226 --> 00:25:04.266 Two things in that definition matter. 00:25:04.266 --> 00:25:05.920 First, it compares two models. 00:25:05.920 --> 00:25:17.391 Test error is almost always a little higher than training error, and that alone isn't overfitting; the sign is that more flexibility keeps lowering the training error while the error on held-back cases rises. 00:25:17.391 --> 00:25:20.369 Second, it's about new cases from the same population. 00:25:20.369 --> 00:25:27.483 A model that fails at a different hospital, on different patients, has a different problem, and we'll see it on the sepsis slide. 00:25:27.833 --> 00:25:32.086 If you were only ever shown the training error, you'd pick the worst model. 00:25:32.086 --> 00:25:41.441 This is the job of the validation set in the fitting loop: choose the setting where the error on held-back cases is lowest, and keep a separate test set for the end. 00:25:41.441 --> 00:25:46.033 And if you find you've overfit, the fix is to go back and make the model simpler. 00:25:46.383 --> 00:25:51.623 If you have the deck open, pause here and move the slider from one end to the other. 00:25:52.433 --> 00:25:53.927 Here is a neural network. 00:25:53.927 --> 00:25:57.273 It's the fitting loop again, with a far bigger equation. 00:25:57.700 --> 00:25:59.497 On the left are the inputs. 00:25:59.497 --> 00:26:06.220 For an image, these can be the raw pixel values, so nobody has to decide in advance what to look for. 00:26:06.633 --> 00:26:09.161 In the middle are layers of simple units. 00:26:09.161 --> 00:26:15.080 Each unit adds up what it receives, with its own dials, and passes the result to the next layer. 00:26:15.080 --> 00:26:21.246 On the right is the output: a number, such as an ejection fraction, or a category, such as melanoma. 00:26:21.246 --> 00:26:27.473 You don't have to tell it what to look for, but you do still need the answer for every training case. 00:26:27.900 --> 00:26:30.282 Every line in this picture is a dial. 00:26:30.282 --> 00:26:32.021 This toy network has fifty. 00:26:32.021 --> 00:26:35.369 The networks behind the four examples have far more. 00:26:35.369 --> 00:26:40.391 G P T 3, an early ancestor of ChatGPT, had a hundred and seventy-five billion. 00:26:40.391 --> 00:26:49.985 Turning that many dials takes an enormous amount of computing time and energy, which is a big part of why a new chatbot model costs so much to build. 00:26:49.985 --> 00:26:56.745 Fitting is the loop you already know: turn every dial until the misses on the training cases get smaller. 00:26:56.745 --> 00:26:59.900 The word deep in deep learning means many layers. 00:27:00.250 --> 00:27:02.661 Everything from the slider still applies. 00:27:02.661 --> 00:27:08.425 A network with that many dials can memorize its training cases, so the test set matters even more. 00:27:08.425 --> 00:27:12.130 And what it learned depends on the people in its training data. 00:27:12.933 --> 00:27:15.031 The slider showed one trade-off. 00:27:15.031 --> 00:27:17.653 Here is a second one that comes with it. 00:27:18.003 --> 00:27:20.861 The E G F R equation is a handful of numbers. 00:27:20.861 --> 00:27:24.480 You can read each one and check it against what you know. 00:27:24.480 --> 00:27:31.338 Higher creatinine gives a lower E G F R, older age gives a lower E G F R, and you could work it out by hand. 00:27:31.338 --> 00:27:34.132 A few thousand people were enough to fit it. 00:27:34.132 --> 00:27:38.323 The cost is that it can only follow the shape its equation allows. 00:27:38.673 --> 00:27:42.094 A neural network has thousands to billions of numbers. 00:27:42.094 --> 00:27:49.633 It can fit patterns no short equation can, such as what a pneumothorax looks like on a film, or how a sentence goes on. 00:27:49.633 --> 00:27:56.221 But nobody can read those numbers one by one, so it's harder to explain why it gave a particular answer. 00:27:56.221 --> 00:28:01.796 And it needs far more data, or it fits the noise, as the slider showed at the far right. 00:28:01.796 --> 00:28:07.307 Turn that around: with only a few hundred patients, you can't fit a very complex model. 00:28:07.307 --> 00:28:14.529 If you recall the history of AI, that's one reason the internet and the electronic health record mattered so much. 00:28:14.529 --> 00:28:16.873 Machine learning needs a lot of data. 00:28:17.300 --> 00:28:24.045 So the more flexible the model, the more patterns it can fit, the less you can explain, and the more data it needs. 00:28:24.045 --> 00:28:28.621 A model is called interpretable when a person can read why it gave its answer. 00:28:28.621 --> 00:28:31.495 Recall the expert systems from the history of AI. 00:28:31.495 --> 00:28:35.835 They were very interpretable: they could tell you the rules they followed. 00:28:35.835 --> 00:28:37.830 Neither kind is better in general. 00:28:37.830 --> 00:28:42.053 For a lab value, a simple model you can check is often the right choice. 00:28:42.053 --> 00:28:46.980 Images and language need the flexible kind, and then the test set matters even more. 00:28:47.800 --> 00:28:54.080 Here is the mistake the E G F R researchers avoided, and the mistake many vendor slides make. 00:28:54.430 --> 00:29:01.728 Suppose you fit a formula to eight thousand people, and then you report how well it predicts those same eight thousand people. 00:29:01.728 --> 00:29:07.578 You have measured how well the formula memorized them, not how well it will work on the next patient. 00:29:07.578 --> 00:29:12.270 A flexible enough model can fit its training data perfectly and still be useless. 00:29:12.700 --> 00:29:16.642 That's why a separate test set is the least you should accept. 00:29:16.642 --> 00:29:22.174 Fit on one set of cases, and report performance on a different set the model never saw. 00:29:22.174 --> 00:29:29.613 If someone tells you their model's error is near zero, your first question is: on the training data or the test data? 00:29:29.613 --> 00:29:33.238 Did you try it on new data, and what was the error there? 00:29:33.238 --> 00:29:36.798 The same goes for a model trained on synthetic patients. 00:29:36.798 --> 00:29:41.631 Synthetic training data can help, but the test still has to be on real ones. 00:29:41.631 --> 00:29:48.180 When someone gives you one number and doesn't say which set it came from, you do not have a number yet. 00:29:48.600 --> 00:29:53.456 The usual rule uses three sets, the same three as in the fitting loop. 00:29:53.456 --> 00:29:56.370 The training set is used to fit the model. 00:29:56.370 --> 00:30:04.071 The validation set is used to choose between models, for example between a simpler one and a more flexible one. 00:30:04.071 --> 00:30:07.748 The test set is kept aside and used once, at the end. 00:30:07.748 --> 00:30:14.964 If the test set is used to choose, it becomes a second validation set, and its number is optimistic too. 00:30:14.964 --> 00:30:17.600 The next slide shows why this matters. 00:30:18.400 --> 00:30:21.525 You may recall this model from the history of AI. 00:30:21.525 --> 00:30:27.328 Here is why it belongs in today's talk: it's the same failure as the slider, in the clinic. 00:30:27.328 --> 00:30:33.897 One E H R vendor's sepsis model, the Epic Sepsis Model, was in use at hundreds of US hospitals in 2021. 00:30:33.897 --> 00:30:40.721 One academic center tested it on thirty-eight thousand four hundred fifty-five of its own hospitalizations. 00:30:40.721 --> 00:30:47.800 A test on patients from a different setting than the ones the model was built on is called external validation. 00:30:48.150 --> 00:30:54.030 The area under the curve was zero point six three, where zero point five is no better than chance. 00:30:54.030 --> 00:30:58.290 Of every hundred patients who had sepsis, the model missed sixty-seven. 00:30:58.290 --> 00:31:06.510 And since it alerted on eighteen of every hundred admissions, when only about seven had sepsis, most alerts were for patients who didn't. 00:31:06.860 --> 00:31:11.175 The grid shows the same counts, scaled to two hundred admissions. 00:31:11.175 --> 00:31:12.901 Each box is one admission. 00:31:12.901 --> 00:31:16.817 The colored boxes are the thirty-six where the alert fired. 00:31:16.817 --> 00:31:20.137 The dots are the thirteen patients who had sepsis. 00:31:20.137 --> 00:31:22.991 Only four of the dots sit in a colored box. 00:31:22.991 --> 00:31:25.580 The other nine had sepsis and no alert. 00:31:25.930 --> 00:31:27.439 The study doesn't show why. 00:31:27.439 --> 00:31:32.973 A common reason is that the patients a model meets in use differ from the patients it was built on. 00:31:32.973 --> 00:31:34.930 The name for that is dataset shift. 00:31:35.280 --> 00:31:38.528 For the wards: no alert does not mean no sepsis. 00:31:38.528 --> 00:31:42.520 And if you silence an alert, you should be able to say why. 00:31:43.333 --> 00:31:49.413 That gives you another of the course's recurring lessons: the published number is the optimistic one. 00:31:49.763 --> 00:31:55.232 A model's performance on the cases it was built from is usually the best it will look. 00:31:55.232 --> 00:31:59.683 On new patients, especially from another hospital, it's usually lower. 00:32:00.100 --> 00:32:02.103 So ask three things of any model. 00:32:02.103 --> 00:32:03.499 What was it trained on? 00:32:03.499 --> 00:32:05.624 What was it tested on, and by whom? 00:32:05.624 --> 00:32:13.940 If the training set and the test set are the same cases, the score tells you nothing yet, because the model has already seen the answers. 00:32:14.290 --> 00:32:16.372 Why ask what it was trained on? 00:32:16.372 --> 00:32:19.595 Because the training data are part of the model. 00:32:19.595 --> 00:32:27.050 A model built on patients in one country, or at one kind of hospital, may not work the same way somewhere else. 00:32:27.400 --> 00:32:34.567 Often you won't get the details: companies frequently don't publish what a commercial model was trained or tested on. 00:32:34.567 --> 00:32:35.241 Ask anyway. 00:32:35.241 --> 00:32:41.244 But there's a second answer that doesn't depend on them, which is to test it on your own patients. 00:32:41.244 --> 00:32:51.046 I sit on the committee that vets machine learning models for our health system, and one of our first questions about any model is what it will cost to localize. 00:32:51.046 --> 00:33:00.418 Localizing means running the model on our own past patients, say the last six months or the last few years, and scoring it against what we know happened. 00:33:00.418 --> 00:33:01.827 Take a fall-risk model. 00:33:01.827 --> 00:33:05.380 Of the patients who fell, how many did the alert fire for? 00:33:05.380 --> 00:33:08.749 And how often did it fire for patients who didn't fall? 00:33:08.749 --> 00:33:13.160 Sometimes the next step is to tune the model to fit our patients better. 00:33:13.510 --> 00:33:15.410 And it doesn't stop at go-live. 00:33:15.410 --> 00:33:23.377 Patients change over time, and so do the records; even the codes used for diagnoses have changed their definitions over the years. 00:33:23.377 --> 00:33:27.790 So someone has to keep monitoring the model to make sure it still works. 00:33:28.600 --> 00:33:30.883 Take any sentence from the internet. 00:33:30.883 --> 00:33:33.800 The patient was started on empiric vancomycin. 00:33:34.150 --> 00:33:38.268 Now turn it into a table, the same kind of table as the E G F R. 00:33:38.268 --> 00:33:40.584 The predictors are the words so far. 00:33:40.584 --> 00:33:42.643 The truth is the very next word. 00:33:42.643 --> 00:33:44.637 "The" is followed by "patient". 00:33:44.637 --> 00:33:46.889 "The patient" is followed by "was". 00:33:46.889 --> 00:33:52.164 And so on, until "the patient was started on empiric" is followed by "vancomycin". 00:33:52.164 --> 00:33:57.955 You might ask: lots of sentences start with "The", and many different words can come next. 00:33:57.955 --> 00:34:02.458 That's why the model uses all the words so far, not just the last one. 00:34:02.458 --> 00:34:06.190 The more of the sentence it sees, the better it can guess. 00:34:06.600 --> 00:34:11.131 Nobody had to label anything, because the next word is already in the text. 00:34:11.131 --> 00:34:15.602 Every sentence ever written becomes training cases with no labelling work. 00:34:15.602 --> 00:34:21.040 That's why these models could be trained on a very large fraction of the written internet. 00:34:21.390 --> 00:34:28.710 The next session starts here: a chatbot is machine learning on words, and the same test-set questions apply. 00:34:29.533 --> 00:34:30.686 Last few minutes. 00:34:30.686 --> 00:34:31.973 Back to the clinic. 00:34:32.800 --> 00:34:34.027 Go back to the clinic. 00:34:34.027 --> 00:34:37.040 The sixty-two-year-old woman is still on the schedule. 00:34:37.467 --> 00:34:42.187 She weighs forty-five kilograms and has lost muscle over a year of illness. 00:34:42.600 --> 00:34:44.480 Creatinine comes from muscle. 00:34:44.900 --> 00:34:52.940 Her creatinine of one point one is a normal number for a normal amount of muscle, and she doesn't have a normal amount of muscle. 00:34:53.367 --> 00:34:59.898 So the same creatinine probably means a lower true G F R than the fifty-seven the equation returned. 00:34:59.898 --> 00:35:06.887 Her low muscle mass changes the predictor without changing the truth, and nothing in the equation can tell. 00:35:07.300 --> 00:35:10.986 Name her model's predictors, response, and training population. 00:35:10.986 --> 00:35:15.140 And what do you now say to the attending about the number on the panel? 00:35:15.490 --> 00:35:17.930 Pause here and answer out loud or in writing. 00:35:18.333 --> 00:35:19.966 Here is one good answer. 00:35:19.966 --> 00:35:23.368 "The fifty-seven is estimated from her creatinine. 00:35:23.368 --> 00:35:27.995 With her low muscle mass it probably overstates her kidney function. 00:35:27.995 --> 00:35:41.126 If the decision depends on it, I would ask for a cystatin C-based E G F R." Whether it changes the contrast decision depends on how close her true value could be to your department's threshold. 00:35:41.126 --> 00:35:44.664 It matters more for a drug dosed by kidney function. 00:35:44.664 --> 00:35:51.059 The training population was eight thousand two hundred fifty-four adult research participants. 00:35:51.059 --> 00:35:54.733 Nothing in the equation knows how much muscle she has. 00:35:55.083 --> 00:35:59.208 Notice also that nobody told her a model had a part in her care. 00:35:59.208 --> 00:36:04.363 Whether someone should have is a question for the patient communication session. 00:36:05.167 --> 00:36:07.287 Here is what this means on the wards. 00:36:07.637 --> 00:36:12.004 When an alert fires, or stays quiet, remember the sepsis model. 00:36:12.004 --> 00:36:14.291 No alert does not mean no sepsis. 00:36:14.291 --> 00:36:17.757 If you silence one, you should be able to say why. 00:36:18.107 --> 00:36:23.295 When a tool comes with an accuracy number, ask what it was tested on, and by whom. 00:36:23.295 --> 00:36:27.787 A score on the cases the model was built from is the optimistic number. 00:36:28.137 --> 00:36:32.676 When a tool has drafted your note or message, read all of it before you sign. 00:36:32.676 --> 00:36:34.798 You're responsible for what it says. 00:36:34.798 --> 00:36:41.577 What to document when a tool helped depends on your site's policy, and the ethics and regulation session covers it. 00:36:41.927 --> 00:36:48.807 And patient information goes into no chatbot, unless your hospital's privacy office has approved that tool in writing. 00:36:49.233 --> 00:36:52.908 If you tell a co-intern one sentence, make it this one. 00:36:52.908 --> 00:36:55.513 Ask what it was tested on, and by whom. 00:36:56.333 --> 00:36:58.570 Every session ends on the same question. 00:36:58.570 --> 00:37:01.253 What does this change about how you'll practice? 00:37:01.667 --> 00:37:04.717 Write your answer down before you close this deck. 00:37:04.717 --> 00:37:07.707 It is not optional, and one word isn't an answer. 00:37:08.057 --> 00:37:09.377 Pause here and write it. 00:37:09.727 --> 00:37:10.547 Then one more. 00:37:10.547 --> 00:37:16.464 The next time someone shows you a model's accuracy number, what will you ask in the next two minutes? 00:37:16.464 --> 00:37:19.277 For example: what was it tested on, and by whom? 00:37:19.277 --> 00:37:21.562 Some of you'll be in exactly that spot. 00:37:21.562 --> 00:37:31.815 A company shows up at lunch with a model that predicts which of your clinic patients will end up in the emergency department, and asks, don't you want to catch those patients? 00:37:31.815 --> 00:37:37.967 There are a lot of questions to ask before you say yes, and I hope today's slides help with some of them. 00:37:38.367 --> 00:37:42.975 In the next session, we start with the chatbot, the same loop with words. 00:37:42.975 --> 00:37:48.847 Then the class opens ChatGPT Edu together and uses it to set up learning goals and a journal. 00:37:49.667 --> 00:37:50.358 To sum up. 00:37:50.358 --> 00:37:56.507 AI contains machine learning, which contains deep learning, which contains generative AI. 00:37:56.857 --> 00:38:03.454 The E G F R is a model learned from data, with predictors, a response, and a population it was fit to. 00:38:03.454 --> 00:38:07.658 Fitting chose its numbers to make the error as small as possible. 00:38:07.658 --> 00:38:12.057 Every learned model has those three parts, and you can ask for them. 00:38:12.407 --> 00:38:17.326 Supervised learning has the answer in the training data; unsupervised doesn't. 00:38:17.326 --> 00:38:21.047 Classification gives a category; regression gives a number. 00:38:21.397 --> 00:38:24.078 A model that's too flexible fits the noise. 00:38:24.078 --> 00:38:28.755 That's overfitting: a simpler model would have done better on new patients. 00:38:28.755 --> 00:38:35.677 A neural network is the same loop with far more dials, so it needs more data, and a test set matters even more. 00:38:36.027 --> 00:38:39.294 A score on the training data describes the fit. 00:38:39.294 --> 00:38:40.615 It is not evidence. 00:38:40.615 --> 00:38:43.187 Ask for the test set, and who ran it. 00:38:43.537 --> 00:38:45.857 Next, the same loop with words. 00:38:46.207 --> 00:38:49.133 The fifty-seven on the panel came from somewhere. 00:38:49.133 --> 00:38:50.567 So does everything else. 00:38:51.367 --> 00:39:03.395 If you want more, the last slides of the deck have two things to follow up on: a hands-on overfitting exercise in the TensorFlow Playground, and a short paper on dataset shift written for clinicians. 00:39:03.395 --> 00:39:04.967 The links are in the deck.