WEBVTT 00:00:00.000 --> 00:00:01.650 A note before we begin. 00:00:01.650 --> 00:00:08.826 The narration in this video is synthetic speech, generated from the lecture notes of Dr. Sean Davis. 00:00:08.826 --> 00:00:11.266 It isn't a recording of his voice. 00:00:11.266 --> 00:00:14.280 The lecture was given on October 13, 2026. 00:00:15.100 --> 00:00:20.931 This is a recorded version of a lecture from a course on artificial intelligence for medical students. 00:00:20.931 --> 00:00:28.420 It's a story, about eighty years of people trying to build machines that think, and what happened when those machines met medicine. 00:00:28.770 --> 00:00:30.932 I want to start with a puzzle. 00:00:30.932 --> 00:00:36.336 In 1979 a program called My-sin was tested on ten real cases of meningitis. 00:00:36.336 --> 00:00:42.821 Experts who didn't know which answers came from the computer rated its antibiotic choices. 00:00:42.821 --> 00:00:48.370 My-sin did as well as, or better than, the Stanford faculty in the same test. 00:00:48.720 --> 00:00:50.681 And then nothing happened. 00:00:50.681 --> 00:00:53.320 It was never used for patient care. 00:00:53.670 --> 00:00:55.098 Keep that puzzle in mind. 00:00:55.098 --> 00:01:01.265 The answer isn't mainly about the technology, and it's the same answer that explains several failures since. 00:01:01.265 --> 00:01:03.150 We'll come back to it at the end. 00:01:03.967 --> 00:01:06.152 Here is the map for the hour. 00:01:06.152 --> 00:01:07.207 Five chapters. 00:01:07.557 --> 00:01:17.597 First, the early ideas, from the 1940s to the mid-1960s, when a handful of people asked whether a machine could think at all. 00:01:17.947 --> 00:01:26.882 Second, expert systems: the 1970s and 80s, when the plan was to interview experts and write their knowledge down as rules. 00:01:26.882 --> 00:01:28.347 My-sin belongs here. 00:01:28.697 --> 00:01:30.292 Third, the AI winters. 00:01:30.292 --> 00:01:36.457 Twice, the promises ran far ahead of the results, and funding and interest collapsed. 00:01:36.807 --> 00:01:38.544 Fourth, learning from data. 00:01:38.544 --> 00:01:44.527 Instead of writing the rules, we give the machine many examples and let it find the patterns. 00:01:44.877 --> 00:01:51.197 Fifth, the chapter you're living in: large language models, the technology behind ChatGPT. 00:01:51.600 --> 00:01:57.600 If you remember one thing from today, make it this question: where does the knowledge come from? 00:01:57.600 --> 00:02:00.725 Hand-written rules, or patterns learned from data? 00:02:00.725 --> 00:02:06.600 Every era answers it differently, and the answer explains both the successes and the failures. 00:02:07.400 --> 00:02:08.178 Part one. 00:02:08.178 --> 00:02:09.560 The early ideas. 00:02:09.910 --> 00:02:17.007 What strikes me about this period is how few people were involved, and how many of their ideas we still use. 00:02:17.007 --> 00:02:21.870 The basic unit of the neural networks inside ChatGPT was sketched in 1943. 00:02:22.700 --> 00:02:28.516 Warren Muh-KUL-uk was a neurophysiologist and Walter Pitts was a young logician. 00:02:28.516 --> 00:02:35.060 In 1943 they asked a simple question: can we describe what a neuron does with mathematics? 00:02:35.410 --> 00:02:37.233 Their answer is on the right. 00:02:37.233 --> 00:02:41.446 A neuron receives signals from other neurons through its dendrites. 00:02:41.446 --> 00:02:46.665 If the combined signal is strong enough, it fires and sends a signal down its axon. 00:02:46.665 --> 00:02:53.958 The model copies that: take the inputs, weight them, add them up, and if the total passes a threshold, output a one. 00:02:53.958 --> 00:02:55.530 Otherwise, output a zero. 00:02:55.880 --> 00:02:57.243 Two things to notice. 00:02:57.243 --> 00:02:59.386 First, this model couldn't learn. 00:02:59.386 --> 00:03:01.269 The weights were set by hand. 00:03:01.269 --> 00:03:04.840 Learning comes fifteen years later with the perceptron. 00:03:05.190 --> 00:03:11.974 Second, this little unit, much improved, is still the basic building block of modern neural networks. 00:03:11.974 --> 00:03:19.430 The model behind ChatGPT is built from billions of simplified neurons that are distant descendants of this one. 00:03:20.233 --> 00:03:21.883 One neuron can't do much. 00:03:21.883 --> 00:03:25.513 The power comes from connecting many of them in layers. 00:03:25.863 --> 00:03:29.346 Here is the classic example of recognizing a face. 00:03:29.346 --> 00:03:35.196 The first layer of a network learns to detect very simple things: edges and corners. 00:03:35.196 --> 00:03:39.932 The second layer combines those into parts: an eye, a nose, a mouth. 00:03:39.932 --> 00:03:43.623 The third layer combines the parts into a whole face. 00:03:43.973 --> 00:03:48.025 No one tells the network "an eye is two curves and a dark circle". 00:03:48.025 --> 00:03:52.076 In modern networks, the layers learn these features from examples. 00:03:52.076 --> 00:03:59.688 That idea, layers of simple units learning features from data, is the core of deep learning, which we'll reach in part four. 00:03:59.688 --> 00:04:03.493 A later session in the course goes deeper into how this works. 00:04:04.300 --> 00:04:17.460 In 1950 Alan Turing, the British mathematician who helped break German codes in the Second World War, published a paper that opens with the sentence: "I propose to consider the question, Can machines think?" 00:04:17.810 --> 00:04:20.911 He decided that question was too vague to answer. 00:04:20.911 --> 00:04:24.518 So he swapped it for a game he called the imitation game. 00:04:24.518 --> 00:04:31.099 A judge chats by typed text with two partners, one human and one machine, and has to say which is which. 00:04:31.099 --> 00:04:37.490 If the judge can't reliably tell, Turing argued, we should take the machine's intelligence seriously. 00:04:37.840 --> 00:04:48.720 He made a prediction: in about fifty years, a computer would play the game so well that an average judge would have no more than a 70 percent chance of being right after five minutes. 00:04:49.070 --> 00:04:51.701 He was off by about twenty-five years. 00:04:51.701 --> 00:04:57.241 In 2025, researchers at UC San Diego ran Turing's three-way version of the test. 00:04:57.241 --> 00:05:07.004 When GPT-4.5 was told to act like a person, judges picked it as the human 73 percent of the time, more often than they picked the real human. 00:05:07.004 --> 00:05:10.674 That study is a preprint, so treat it with some care. 00:05:10.674 --> 00:05:17.390 Still, the test has now been passed in Turing's own three-person form, the strictest version yet. 00:05:17.740 --> 00:05:22.055 Worth asking yourselves: does fooling a judge mean the machine thinks? 00:05:22.055 --> 00:05:23.472 Most people now say no. 00:05:23.472 --> 00:05:29.698 The test measures how human something seems, which is a different question from whether it's correct. 00:05:29.698 --> 00:05:32.780 Keep that in mind when a chatbot sounds confident. 00:05:33.600 --> 00:05:42.560 The same year as Turing's paper, Isaac Asimov published I, Robot, a collection of short stories he had written through the 1940s. 00:05:42.910 --> 00:05:49.982 His robots follow built-in rules called the Three Laws, which first appeared in a 1942 story called "Runaround". 00:05:49.982 --> 00:06:03.430 What makes the stories interesting is that almost every one of them is about the laws producing a result no one intended: a robot stuck in a loop, or a robot that follows the letter of a rule and misses its point. 00:06:03.780 --> 00:06:05.997 That's a surprisingly modern worry. 00:06:05.997 --> 00:06:07.580 Let me show you the laws. 00:06:08.400 --> 00:06:09.878 Here are the three laws. 00:06:09.878 --> 00:06:11.971 Read the first one as a clinician. 00:06:11.971 --> 00:06:18.560 "Through inaction, allow a human being to come to harm." That's a duty to act, not only a duty not to hurt. 00:06:18.560 --> 00:06:23.240 It sounds a lot like the principle of beneficence alongside non-maleficence. 00:06:23.667 --> 00:06:31.813 Later, in his novels of the 1980s, Asimov added a Zeroth Law that sits above the others: protect humanity as a whole. 00:06:31.813 --> 00:06:35.295 Now a robot might harm one person to protect many. 00:06:35.295 --> 00:06:38.707 You can see the ethics getting hard very quickly. 00:06:39.057 --> 00:06:42.239 The point isn't that real AI systems follow these laws. 00:06:42.239 --> 00:06:42.876 They don't. 00:06:42.876 --> 00:06:47.505 The point is that writing rules that capture what we actually want is very hard. 00:06:47.505 --> 00:06:50.457 Hold on to that, because it comes back in part two. 00:06:51.267 --> 00:07:01.187 Asimov's stories gave people a vocabulary for something we now call alignment: the problem of making an AI system do what we mean, not just what we literally said. 00:07:01.537 --> 00:07:03.287 Try the example on the right. 00:07:03.287 --> 00:07:07.573 Suppose you tell a system to keep a patient's blood pressure under 140. 00:07:07.573 --> 00:07:15.542 A system that only follows that instruction could keep pushing antihypertensives until the patient falls on the way to the bathroom. 00:07:15.542 --> 00:07:21.337 You didn't say "and don't cause syncope", because a human colleague would never need to be told. 00:07:21.687 --> 00:07:27.703 That gap, between what we say and everything we assume, is the alignment problem in small form. 00:07:27.703 --> 00:07:33.087 It's one reason the course spends time on checking AI output rather than trusting it. 00:07:33.900 --> 00:07:42.287 In 1955 John McCarthy, a young mathematics professor at Dartmouth, wrote a proposal for a summer research project with three colleagues. 00:07:42.287 --> 00:07:46.665 He needed a name for the field, and he chose "artificial intelligence". 00:07:46.665 --> 00:07:48.700 That's where the term comes from. 00:07:49.050 --> 00:07:57.947 The organizers were McCarthy, Marvin Minsky, Nathaniel Rochester from IBM, and Claude Shannon, who had invented information theory. 00:07:57.947 --> 00:08:02.090 The workshop ran for about eight weeks in the summer of 1956. 00:08:02.440 --> 00:08:05.640 Read the last sentence of the proposal on the right. 00:08:05.640 --> 00:08:11.795 A significant advance, if a carefully selected group of scientists work on it together for a summer. 00:08:11.795 --> 00:08:15.365 Seventy years later we're still working on those problems. 00:08:15.365 --> 00:08:22.320 The same page of the proposal lists "neuron nets" as a topic and names Pitts and Muh-KUL-uk, from our 1943 slide. 00:08:22.670 --> 00:08:28.990 That optimism is the first sign of a pattern you'll see again and again today: big promises, made early. 00:08:29.800 --> 00:08:41.520 In 1957 Frank Rosenblatt, a psychologist at the Cornell Aeronautical Laboratory, took the McCulloch-Pitts neuron and added the one thing it was missing: learning. 00:08:41.870 --> 00:08:47.565 The perceptron, in the middle, looks the same: weighted inputs, a sum, and a threshold. 00:08:47.565 --> 00:08:50.839 The difference is that it adjusts its own weights. 00:08:50.839 --> 00:08:52.083 Show it an example. 00:08:52.083 --> 00:08:58.236 If it gets the answer wrong, nudge the weights in the direction that would have made it right. 00:08:58.236 --> 00:08:59.938 Repeat thousands of times. 00:08:59.938 --> 00:09:05.110 That simple rule is the ancestor of how every modern neural network is trained. 00:09:05.460 --> 00:09:06.767 The press loved it. 00:09:06.767 --> 00:09:17.916 In 1958 the New York Times reported that the Navy expected the machine to be able to "walk, talk, see, write, reproduce itself and be conscious of its existence". 00:09:17.916 --> 00:09:19.980 You can see the hype starting. 00:09:20.330 --> 00:09:31.063 Cornell's own news office wrote a story in 2019 about Rosenblatt with the headline that his perceptron "paved the way for AI, 60 years too soon". 00:09:31.063 --> 00:09:33.135 He was right about the idea. 00:09:33.135 --> 00:09:36.170 The computers and the data weren't ready. 00:09:37.000 --> 00:09:39.386 The perceptron wasn't just software. 00:09:39.386 --> 00:09:41.971 Rosenblatt built a machine, the Mark I. 00:09:41.971 --> 00:09:46.080 In this Navy photo, the camera is pointed at a large letter C. 00:09:46.430 --> 00:09:50.891 It had a grid of 400 light sensors, twenty by twenty, as its eye. 00:09:50.891 --> 00:09:55.628 Those fed into 512 units in the middle, which fed eight output units. 00:09:55.628 --> 00:10:01.187 The weights were physical: motor-driven dials that turned as the machine learned. 00:10:01.187 --> 00:10:04.070 That's the wiring in the racks behind him. 00:10:04.420 --> 00:10:06.926 On the right are Rosenblatt's own diagrams. 00:10:06.926 --> 00:10:10.888 The top one is his picture of how a brain handles what the eye sees. 00:10:10.888 --> 00:10:14.560 The bottom one is the perceptron, designed to copy that layout. 00:10:14.560 --> 00:10:20.620 That idea, the network as a rough model of the brain, is why we still call these things neural networks. 00:10:20.970 --> 00:10:32.141 What happened next: in 1969 Marvin Minsky and Seymour Papert published a book called Perceptrons that proved what a single-layer perceptron couldn't do. 00:10:32.141 --> 00:10:39.490 It was read as a verdict on neural networks in general, and interest dropped for more than a decade. 00:10:39.490 --> 00:10:45.370 Rosenblatt died in a boating accident in 1971 and didn't see his idea come back. 00:10:46.200 --> 00:10:51.387 ELIZA was written by Joseph Vite-sen-bowm at MIT between 1964 and 1966. 00:10:51.387 --> 00:10:59.424 Its most famous script played a kind of psychotherapist that mostly turns your statements back into questions. 00:10:59.424 --> 00:11:06.000 You say "my boyfriend made me come here", and it says "your boyfriend made you come here?" 00:11:06.350 --> 00:11:08.557 There was no understanding at all. 00:11:08.557 --> 00:11:14.333 ELIZA looked for keywords and filled in templates that a person had written for each one. 00:11:14.333 --> 00:11:21.732 It could rotate through a few replies for the same keyword, but every word it said came from a rule a human wrote. 00:11:21.732 --> 00:11:23.030 Nothing was learned. 00:11:23.380 --> 00:11:26.738 What surprised Vite-sen-bowm was how people reacted. 00:11:26.738 --> 00:11:31.968 His own secretary asked him to leave the room so she could talk to it in private. 00:11:31.968 --> 00:11:36.811 People attributed feelings and understanding to a program that had neither. 00:11:36.811 --> 00:11:40.686 We now call that the ELIZA effect, and it's very much alive. 00:11:40.686 --> 00:11:46.820 When a chatbot sounds warm and confident, it's easy to assume it knows what it's talking about. 00:11:47.170 --> 00:11:56.422 Vite-sen-bowm became one of the strongest critics of AI, especially of machines taking on roles that need human judgment and care. 00:11:56.422 --> 00:12:02.970 And remember the 2025 Turing test study: ELIZA was one of the systems tested, as a baseline. 00:12:03.800 --> 00:12:04.685 Part two. 00:12:04.685 --> 00:12:06.160 Expert systems. 00:12:06.510 --> 00:12:09.000 The idea in the 1970s was very reasonable. 00:12:09.000 --> 00:12:17.538 If we want a computer to act like an expert, let's interview experts, write down their knowledge as rules, and let the computer apply the rules. 00:12:17.538 --> 00:12:23.230 This is the first answer to our question about where knowledge comes from: people write it down. 00:12:24.033 --> 00:12:26.633 An expert system has three parts. 00:12:26.983 --> 00:12:34.162 A knowledge base: the facts and if-then rules for one narrow area, such as bacterial infections. 00:12:34.162 --> 00:12:35.583 People write these. 00:12:35.933 --> 00:12:45.293 An inference engine: the general reasoning machinery that takes the rules and a particular case and chains the rules together to reach a conclusion. 00:12:45.643 --> 00:12:50.083 And a user interface, so that someone who isn't the expert can use it. 00:12:50.433 --> 00:12:54.177 Look at the right-hand panel, especially the last point. 00:12:54.177 --> 00:12:56.851 Expert systems could explain themselves. 00:12:56.851 --> 00:13:00.127 Ask why, and they would list the rules they used. 00:13:00.127 --> 00:13:05.877 That's something today's neural networks, including ChatGPT, mostly can't do honestly. 00:13:05.877 --> 00:13:15.169 When a chatbot explains its reasoning, it's generating a plausible explanation, which may or may not be how it actually reached the answer. 00:13:15.169 --> 00:13:18.913 So in one respect, the 1970s systems were ahead of ours. 00:13:19.733 --> 00:13:22.472 This is the program from our opening puzzle. 00:13:22.472 --> 00:13:33.674 My-sin was built at Stanford in the early and mid 1970s as the doctoral work of Edward Shortliffe, who went on to become one of the founders of the field of biomedical informatics. 00:13:33.674 --> 00:13:38.093 The name comes from the suffix of many antibiotics, as in streptomycin. 00:13:38.443 --> 00:13:43.701 It focused on serious bacterial infections such as bacteremia and meningitis. 00:13:43.701 --> 00:13:49.163 Its knowledge was about 600 rules, each written with infectious disease experts. 00:13:49.513 --> 00:13:51.731 A consultation was a long interview. 00:13:51.731 --> 00:14:01.340 The physician typed answers to question after question: where was the culture taken, what did the Gram stain show, is the patient a burn patient, and so on. 00:14:01.340 --> 00:14:09.593 At the end, My-sin gave a list of likely organisms, how confident it was in each, the reasoning, and a recommended antibiotic regimen. 00:14:10.400 --> 00:14:13.278 Here is one of those 600 rules: rule 52. 00:14:13.278 --> 00:14:18.603 On the left is how it's written in the program, in a language called Lisp. 00:14:18.603 --> 00:14:24.000 On the right is the English version My-sin would show you if you asked why. 00:14:24.400 --> 00:14:39.720 In plain words: if the culture came from blood, the organism is gram-negative and rod-shaped, and the patient has a serious burn, then there's weakly suggestive evidence, with a strength of 0.4, that the organism is Pseudomonas. 00:14:40.070 --> 00:14:44.446 You can read this and check it against what you learned in microbiology. 00:14:44.446 --> 00:14:46.208 That's the strength of rules. 00:14:46.208 --> 00:14:54.473 Now imagine writing six hundred of them, getting them to agree with each other, and updating them every time resistance patterns change. 00:14:54.473 --> 00:14:58.910 That's the weakness, and it's a big part of the AI winter coming up next. 00:14:59.733 --> 00:15:05.093 This is the study behind our opening puzzle, published in Jama in 1979. 00:15:05.443 --> 00:15:07.534 Ten real cases of meningitis. 00:15:07.534 --> 00:15:16.401 Nine human prescribers, including five Stanford infectious disease faculty, each wrote a treatment plan, and so did My-sin. 00:15:16.401 --> 00:15:24.043 Then eight infectious disease experts from outside Stanford rated every plan without knowing who wrote it. 00:15:24.393 --> 00:15:28.877 My-sin's plans were rated acceptable 65 percent of the time. 00:15:28.877 --> 00:15:33.510 The five faculty specialists ranged from 42.5 to 62.5 percent. 00:15:33.510 --> 00:15:36.873 And My-sin never missed a treatable pathogen. 00:15:37.223 --> 00:15:38.031 Two lessons. 00:15:38.031 --> 00:15:41.800 First, even the experts disagreed with each other a lot. 00:15:41.800 --> 00:15:47.992 In medicine there's often no single correct answer, which makes evaluating any AI tool hard. 00:15:47.992 --> 00:15:52.501 Second, by this measure My-sin was at least as good as the experts. 00:15:52.501 --> 00:15:54.183 So why was it never used? 00:15:55.000 --> 00:15:57.800 Here is the answer to our puzzle, or most of it. 00:15:58.150 --> 00:16:00.473 My-sin didn't fail on accuracy. 00:16:00.473 --> 00:16:03.470 It failed on everything around accuracy. 00:16:03.820 --> 00:16:10.692 Workflow: every case meant a long interview, with someone typing in data that already existed elsewhere. 00:16:10.692 --> 00:16:14.260 In a busy hospital, that's a tool nobody has time for. 00:16:14.610 --> 00:16:20.415 Hardware: it ran on a time-shared mainframe like this one, before personal computers. 00:16:20.415 --> 00:16:23.010 You couldn't just open it on the ward. 00:16:23.360 --> 00:16:30.357 And responsibility: if the computer recommends the wrong antibiotic and the patient dies, who is responsible? 00:16:30.357 --> 00:16:31.127 The program? 00:16:31.127 --> 00:16:31.897 Its authors? 00:16:31.897 --> 00:16:33.823 The physician who followed it? 00:16:33.823 --> 00:16:38.765 That question was raised in the 1970s and we're still arguing about it today. 00:16:38.765 --> 00:16:42.360 A later part of the course is partly about exactly that. 00:16:42.710 --> 00:16:48.323 So the lesson of My-sin: being accurate is necessary, but it's far from enough. 00:16:48.323 --> 00:16:52.870 A tool has to fit the work, and someone has to own the decision. 00:16:53.700 --> 00:16:54.715 Part three. 00:16:54.715 --> 00:16:56.100 The AI winters. 00:16:56.450 --> 00:17:04.642 An AI winter is a period when funding and interest in AI dry up, usually because the field promised much more than it delivered. 00:17:04.642 --> 00:17:07.650 There were two big ones, with different causes. 00:17:08.467 --> 00:17:11.238 The first winter began in the mid 1970s. 00:17:11.238 --> 00:17:17.959 In 1973 the British government asked the mathematician Sir James Lighthill to review AI research. 00:17:17.959 --> 00:17:31.125 His report said the field had failed to reach its grand objectives, and that its methods worked on toy problems but collapsed on real ones because the number of possibilities grows too fast. 00:17:31.125 --> 00:17:38.747 Funding in the UK was cut sharply, and US military funding for basic AI research dropped around the same time. 00:17:39.097 --> 00:17:42.608 The second winter came after the expert systems boom. 00:17:42.608 --> 00:17:46.650 In the early 1980s companies spent heavily on expert systems. 00:17:46.650 --> 00:17:58.577 By the late 1980s it was clear they were expensive to build, expensive to maintain, and brittle: give them a case slightly outside their rules and they could give an absurd answer. 00:17:58.577 --> 00:17:59.902 They couldn't learn. 00:17:59.902 --> 00:18:03.017 Every change needed an expert and a programmer. 00:18:03.367 --> 00:18:06.649 Notice that the second winter is about our big question. 00:18:06.649 --> 00:18:12.393 When the knowledge comes from rules written by people, getting the knowledge in is the bottleneck. 00:18:12.393 --> 00:18:15.207 It was called the knowledge acquisition problem. 00:18:16.033 --> 00:18:18.193 Here is Lighthill in his own words. 00:18:18.543 --> 00:18:27.663 "A pronounced feeling of disappointment." "The major impact that was then promised." And on speech recognition: money "wholly wasted". 00:18:28.013 --> 00:18:31.706 I like to show this because of the speech recognition line. 00:18:31.706 --> 00:18:39.467 Today many of you'll use ambient AI scribes on your rotations: software that listens to a patient visit and drafts the note. 00:18:39.467 --> 00:18:43.973 Speech recognition that Lighthill wrote off in 1973 now runs on a phone. 00:18:44.323 --> 00:18:45.107 Two lessons. 00:18:45.107 --> 00:18:49.355 The critics of the time were right about the methods of the time. 00:18:49.355 --> 00:18:53.602 And "it doesn't work yet" isn't the same as "it will never work". 00:18:53.602 --> 00:18:56.803 Both the hype and the backlash tend to overshoot. 00:18:57.153 --> 00:18:58.554 That applies to you too. 00:18:58.554 --> 00:19:04.564 If a chatbot gave you a wrong answer in second year and you stopped using it, you were right to notice. 00:19:04.564 --> 00:19:06.956 Some of that has changed and some hasn't. 00:19:06.956 --> 00:19:09.873 The rest of the course is about finding out which. 00:19:10.700 --> 00:19:11.387 Part four. 00:19:11.387 --> 00:19:14.820 Learning from data, often called machine learning. 00:19:15.170 --> 00:19:17.843 This is the second answer to our question. 00:19:17.843 --> 00:19:27.261 Rather than having people write the rules, we give the computer lots of examples with the right answers, and it adjusts itself to find the patterns. 00:19:27.261 --> 00:19:29.679 That idea goes back to the perceptron. 00:19:29.679 --> 00:19:34.770 What changed in the 1990s and 2000s is that it finally started to work at scale. 00:19:35.120 --> 00:19:38.136 This approach has a name: machine learning. 00:19:38.136 --> 00:19:46.484 Artificial intelligence is a broad field with many parts, and machine learning is the part built on learning from data. 00:19:46.484 --> 00:19:54.200 These days, when people say AI, they almost always mean some form of machine learning: models trained on data. 00:19:55.000 --> 00:19:56.867 Medicine was early to this. 00:19:56.867 --> 00:20:06.200 In 1995 a group at Georgetown and the University of Michigan published this paper on a convolutional neural network for medical images. 00:20:06.550 --> 00:20:15.217 A convolutional network scans an image with small learned filters, which is very good at picking up local patterns such as edges and spots. 00:20:15.217 --> 00:20:19.270 It's the same basic design used in most medical imaging AI today. 00:20:19.620 --> 00:20:28.276 They tested it on two problems you'll recognize: finding lung nodules on chest X-rays and finding microcalcifications on mammograms. 00:20:28.276 --> 00:20:35.620 The network learned its features from images that radiologists had rated, rather than from rules a person wrote. 00:20:35.970 --> 00:20:38.168 So the idea was there in 1995. 00:20:38.168 --> 00:20:46.593 What held it back was the same thing that held back the perceptron: not enough data and not enough computing power. 00:20:46.593 --> 00:20:49.450 We'll see those arrive in a few slides. 00:20:50.267 --> 00:20:50.901 May 1997. 00:20:50.901 --> 00:20:59.356 IBM's Deep Blue beat Garry Kasparov, the world chess champion, in a six-game match: two wins, three draws, and one loss. 00:20:59.356 --> 00:21:03.231 Kasparov had beaten an earlier version the year before. 00:21:03.231 --> 00:21:10.347 Chess had long been seen as a test of real intelligence, and it was front-page news around the world. 00:21:10.697 --> 00:21:13.788 But here is a twist that matters for our question. 00:21:13.788 --> 00:21:16.508 Deep Blue was mostly not a learning machine. 00:21:16.508 --> 00:21:18.857 It won by speed plus expert knowledge. 00:21:19.207 --> 00:21:24.512 Speed: custom hardware that could check 200 million positions a second. 00:21:24.512 --> 00:21:29.967 In 1997 it was ranked the 259th most powerful supercomputer in the world. 00:21:30.317 --> 00:21:44.092 Expert knowledge: how good a position was got scored by rules with thousands of parts, tuned with help from grandmasters, plus a library of openings drawn from about 700,000 grandmaster games. 00:21:44.092 --> 00:21:50.837 The project began in 1985 as a student project at Carnegie Mellon and took about twelve years. 00:21:51.187 --> 00:21:56.746 So Deep Blue is the last great win of the hand-written approach, made powerful by speed. 00:21:56.746 --> 00:22:03.505 Modern chess programs, such as DeepMind's AlphaZero, learn by playing millions of games against themselves. 00:22:03.505 --> 00:22:08.307 That shift, from hand-tuned to learned, is the story of the next few slides. 00:22:09.133 --> 00:22:14.275 Why did learning from data take off when it did, and not in Rosenblatt's day? 00:22:14.275 --> 00:22:17.213 Three things had to arrive at the same time. 00:22:17.563 --> 00:22:17.950 Data. 00:22:17.950 --> 00:22:21.123 A neural network needs a lot of examples. 00:22:21.123 --> 00:22:24.683 That needs a digital world, which we have now. 00:22:25.033 --> 00:22:26.074 Computing power. 00:22:26.074 --> 00:22:30.301 Training a network is mostly multiplying large tables of numbers. 00:22:30.301 --> 00:22:39.340 Graphics chips, built for video games, turned out to be excellent at exactly that, because they do thousands of small calculations at once. 00:22:39.340 --> 00:22:45.713 This is why NVIDIA, a gaming chip company, became one of the most valuable companies in the world. 00:22:46.063 --> 00:22:47.118 Better methods. 00:22:47.118 --> 00:22:52.392 Ways to train networks with many layers without the training falling apart. 00:22:52.392 --> 00:22:54.783 We'll see one milestone from 2006. 00:22:55.133 --> 00:23:02.053 Remove any one of the three and you get the perceptron's story: a good idea sixty years too soon. 00:23:02.867 --> 00:23:05.347 Here is the first ingredient: data. 00:23:05.697 --> 00:23:12.923 Every day, hundreds of millions of tweets, hundreds of billions of emails, hundreds of thousands of hours of new video. 00:23:12.923 --> 00:23:19.057 These numbers are from a 2021 summary, so they are already out of date, and almost certainly too low. 00:23:19.407 --> 00:23:29.034 The bars show estimates from the market research firm IDC of how much digital data the world creates each year. 00:23:29.034 --> 00:23:32.156 A zettabyte is a trillion gigabytes. 00:23:32.156 --> 00:23:36.927 33 in 2018, 59 in 2020, and a forecast of 175 for 2025. 00:23:37.277 --> 00:23:44.440 Much of it's stored in very large data centers: about 600 of them, each with more than 5,000 servers. 00:23:44.440 --> 00:23:47.277 The largest are millions of square feet. 00:23:47.627 --> 00:23:55.005 Much of the text that large language models learn from comes from this pile: the public web, books, and code. 00:23:55.005 --> 00:23:56.562 That matters for later. 00:23:56.562 --> 00:24:01.707 A model learns whatever is in its data, including the errors and the biases. 00:24:02.533 --> 00:24:05.127 Medicine went through the same change. 00:24:05.127 --> 00:24:10.930 In 2008 fewer than one in ten US hospitals had even a basic electronic health record. 00:24:10.930 --> 00:24:13.046 By 2021 nearly all of them did. 00:24:13.046 --> 00:24:19.053 That's the blue line, from the federal Office of the National Coordinator for Health IT. 00:24:19.403 --> 00:24:27.243 That jump was driven by policy: the 2009 High-tek Act paid hospitals and practices to adopt electronic records. 00:24:27.593 --> 00:24:32.222 So every lab value, every note, every image, every claim is now data. 00:24:32.222 --> 00:24:34.838 That's the raw material for medical AI. 00:24:34.838 --> 00:24:40.138 It's also the reason privacy rules such as Hippa matter so much in this course. 00:24:40.138 --> 00:24:44.833 Do not put patient information into a tool that isn't approved for it. 00:24:45.633 --> 00:24:52.153 Now the third ingredient, better methods, and the term you'll hear constantly: deep learning. 00:24:52.503 --> 00:24:55.603 On the left is a classic neural network with one hidden layer. 00:24:55.603 --> 00:24:58.003 On the right is a deep network with many layers. 00:24:58.003 --> 00:24:59.503 "Deep" just means many layers. 00:24:59.853 --> 00:25:03.574 The key difference is what we saw with the face example earlier. 00:25:03.574 --> 00:25:09.329 In older machine learning, a person had to decide which features mattered and compute them by hand. 00:25:09.329 --> 00:25:17.002 In deep learning, the early layers learn simple features and the later layers combine them into complex ones, all from the examples. 00:25:17.002 --> 00:25:19.211 The features are learned, not written. 00:25:19.211 --> 00:25:29.093 The scale is hard to picture: the learned connection strengths in these networks are called weights, and open-weight language models have now passed two trillion of them. 00:25:29.443 --> 00:25:36.643 That's the furthest answer yet to our big question: the machine learns not only the rules from data, but also what to look at. 00:25:37.467 --> 00:25:41.281 For decades, networks with many layers were very hard to train. 00:25:41.281 --> 00:25:46.427 The learning signal faded as it passed back through the layers, and training stalled. 00:25:46.777 --> 00:25:57.112 In 2006, Geoffrey Hinton and his colleagues in Toronto showed one way around that: train the network one layer at a time, then fine-tune the whole thing. 00:25:57.112 --> 00:25:59.747 They called these deep belief networks. 00:25:59.747 --> 00:26:06.097 The specific method is rarely used now, but it convinced people that deep networks could work. 00:26:06.447 --> 00:26:07.997 Then things moved fast. 00:26:07.997 --> 00:26:19.662 In 2012 a deep network from Hinton's lab won a major image recognition competition by a wide margin, and almost the whole field switched to deep learning within a few years. 00:26:19.662 --> 00:26:26.607 In 2017 researchers at Google described the transformer, the design that the "T" in ChatGPT stands for. 00:26:26.957 --> 00:26:32.124 Hinton shared the 2024 Nobel Prize in Physics with John Hopfield for this line of work. 00:26:32.124 --> 00:26:38.837 He has also become one of the most prominent voices warning about the risks of AI, which is worth thinking about. 00:26:39.667 --> 00:26:43.253 By 2017 deep learning was everywhere in medical research. 00:26:43.253 --> 00:26:50.427 This is a selection from a review by Riccardo Miotto, Joel Dudley, and colleagues, which listed dozens of studies. 00:26:50.777 --> 00:26:55.333 Imaging came first, because images are plentiful and well labelled. 00:26:55.333 --> 00:27:06.417 The diabetic retinopathy work from Google in 2016 and the skin cancer work from Stanford in 2017 both reported performance close to specialists on their test sets. 00:27:06.767 --> 00:27:14.033 Health records were harder: messy, incomplete, and full of codes entered for billing rather than for medicine. 00:27:14.033 --> 00:27:19.847 Still, groups built models to predict future diseases, readmissions, and next diagnoses. 00:27:20.197 --> 00:27:26.557 And in genomics, deep learning was used to read DNA sequence and predict how it's regulated. 00:27:26.907 --> 00:27:33.159 One caution from the review itself: most of these were research studies on historical data. 00:27:33.159 --> 00:27:35.907 Very few had been tested in actual care. 00:27:35.907 --> 00:27:37.006 Remember My-sin. 00:27:37.006 --> 00:27:41.747 Good results in a study are where the work starts, not where it ends. 00:27:42.567 --> 00:27:51.727 Here is the most famous medical AI failure of the deep learning era, and it repeats My-sin's story. 00:27:52.077 --> 00:27:56.045 In 2011 IBM's Watson beat the two greatest Jeopardy! 00:27:56.045 --> 00:27:57.876 champions on television. 00:27:57.876 --> 00:28:02.837 IBM then marketed Watson heavily for medicine, especially cancer. 00:28:03.187 --> 00:28:10.896 In 2012, MD Anderson Cancer Center in Houston partnered with IBM to build a clinical decision tool for oncology. 00:28:10.896 --> 00:28:19.845 Five years and 62 million dollars later, MD Anderson let its contract with IBM expire, before anyone had used Watson on a patient. 00:28:19.845 --> 00:28:25.834 A University of Texas audit found many procurement problems, cost overruns, and delays. 00:28:25.834 --> 00:28:29.827 It took no position on whether Watson's science was sound. 00:28:30.177 --> 00:28:34.098 The article names no single cause, but the problems are familiar. 00:28:34.098 --> 00:28:37.054 Watson struggled with unstructured clinical text. 00:28:37.054 --> 00:28:45.921 It couldn't reliably tell the abbreviation A L L, for a type of leukemia, from the same letters used as shorthand for allergy, which takes context. 00:28:45.921 --> 00:28:50.627 It also needed to be better integrated with the hospital's electronic records. 00:28:50.627 --> 00:29:01.063 The leukemia pilot was suspended partway through and the effort moved to lung cancer, because project leaders thought that area offered a better chance of finishing on time. 00:29:01.063 --> 00:29:11.137 A pull quote in the article on the right puts it well: solving data quality problems in unstructured data is a much bigger challenge for AI than was first anticipated. 00:29:11.487 --> 00:29:14.874 Same lesson as My-sin, almost forty years later. 00:29:14.874 --> 00:29:18.967 Performance in a demo isn't the same as fitting into care. 00:29:19.767 --> 00:29:24.176 So far every medical example has either failed or stayed a research study. 00:29:24.176 --> 00:29:28.287 Here are two that reached real patients, with very different results. 00:29:28.637 --> 00:29:30.358 On the left, I D X, D R. 00:29:30.358 --> 00:29:42.049 It reads photographs of the retina taken in a primary care office and decides whether the patient has diabetic retinopathy serious enough to see an eye specialist. 00:29:42.049 --> 00:29:44.846 No eye doctor looks at the image first. 00:29:44.846 --> 00:29:55.820 In a trial of 900 people with diabetes, its results were compared with expert grading of the same eyes at the Wisconsin Fundus Photograph Reading Center. 00:29:55.820 --> 00:30:01.988 Needing referral meant more than mild diabetic retinopathy, or diabetic macular edema. 00:30:01.988 --> 00:30:08.801 The system's sensitivity was 87 percent: it found 87 percent of the people who needed referral. 00:30:08.801 --> 00:30:15.185 Its specificity was 91 percent: it correctly cleared 91 percent of the people who didn't. 00:30:15.185 --> 00:30:24.437 You want both numbers high, one to catch the patients who need care and the other to avoid sending everyone else to a specialist. 00:30:24.437 --> 00:30:32.111 In 2018 it became the first autonomous AI diagnostic system authorized by the FDA in any field of medicine. 00:30:32.111 --> 00:30:38.997 That matters because many people with diabetes never get the eye exam they are supposed to have. 00:30:39.347 --> 00:30:41.686 On the right, the Epic Sepsis Model. 00:30:41.686 --> 00:30:48.054 It was built into a widely used electronic record system and switched on at hundreds of hospitals. 00:30:48.054 --> 00:30:56.955 When researchers at the University of Michigan finally checked it on their own patients in 2021, it missed two out of three sepsis cases. 00:30:56.955 --> 00:31:01.958 At the same time it fired alerts on nearly one in five hospitalized patients. 00:31:01.958 --> 00:31:07.027 That's a recipe for alert fatigue: clinicians learn to click past the warning. 00:31:07.377 --> 00:31:10.888 The difference isn't that one used better technology. 00:31:10.888 --> 00:31:13.870 One was tested in a trial before it was used. 00:31:13.870 --> 00:31:17.580 The other was used first and checked by outsiders later. 00:31:17.580 --> 00:31:24.537 When you see an AI alert on your rotations, it's fair to ask: how was this tested, and on whose patients? 00:31:25.367 --> 00:31:26.552 Part five. 00:31:26.552 --> 00:31:28.687 November 30, 2022. 00:31:29.037 --> 00:31:33.350 On that day OpenAI released ChatGPT as a free research preview. 00:31:33.350 --> 00:31:43.757 It wasn't the first large language model, but it was the first one most people had ever used, and it spread faster than almost any technology before it. 00:31:44.107 --> 00:31:51.158 A large language model is a very large neural network trained on huge amounts of text to predict the next word. 00:31:51.158 --> 00:31:57.129 A later session in the course explains how that works and why it produces such fluent answers. 00:31:57.129 --> 00:32:05.387 For today, notice that it's the far end of our question: almost no rules written by hand, and almost everything learned from data. 00:32:06.200 --> 00:32:07.547 Research exploded. 00:32:07.547 --> 00:32:13.760 This 2024 scoping review analyzed 550 studies of large language models in medicine. 00:32:13.760 --> 00:32:19.000 The figure shows how they spread across organ systems and specialties. 00:32:19.350 --> 00:32:26.324 The main uses were helping with diagnosis, drafting medical writing, education, and communicating with patients. 00:32:26.324 --> 00:32:32.550 The main concerns were limits in understanding context and the risk of relying on the tool too much. 00:32:32.900 --> 00:32:52.180 Beyond the studies, AI is spreading into almost every part of medicine you'll meet soon: ambient scribes that draft your notes, which many health systems use partly to reduce burnout; scheduling and staffing; chatbots that answer portal messages; and tutors that many of you already study with. 00:32:52.530 --> 00:33:01.851 A Harvard Gazette piece in March 2025 put the question well: AI is up to the challenge of reducing human suffering, experts say. 00:33:01.851 --> 00:33:02.360 Are we? 00:33:02.360 --> 00:33:13.210 Over-reliance, the review's main worry, is part of what we'll practise in this course: using AI tools well while keeping your own judgment in charge. 00:33:14.033 --> 00:33:18.793 This is one of the most important studies for this course, so I want to go slowly. 00:33:19.143 --> 00:33:33.355 In late 2023, researchers at Stanford, Beth Israel Deaconess, and the University of Virginia randomized 50 physicians, attendings and residents, to work through up to six difficult diagnostic cases in an hour. 00:33:33.355 --> 00:33:37.231 Half had their usual resources, like UpToDate and Google. 00:33:37.231 --> 00:33:40.427 Half had those plus GPT-4 through ChatGPT Plus. 00:33:40.427 --> 00:33:44.303 Experts who didn't know the group graded their reasoning. 00:33:44.653 --> 00:33:45.619 The results. 00:33:45.619 --> 00:33:48.838 Usual resources: a median of 74 percent. 00:33:48.838 --> 00:33:50.689 With GPT-4: 76 percent. 00:33:50.689 --> 00:33:54.551 That difference isn't statistically significant. 00:33:54.551 --> 00:33:58.333 Having the tool didn't make the doctors better. 00:33:58.683 --> 00:34:03.999 But the researchers also gave the same cases straight to GPT-4, three times. 00:34:03.999 --> 00:34:06.307 It scored a median of 92 percent. 00:34:06.307 --> 00:34:11.483 The paper tested that against the usual-resources group: 16 points higher. 00:34:11.833 --> 00:34:13.267 How can both be true? 00:34:13.267 --> 00:34:21.804 The authors point out that the physicians got no training in how to use the tool, and were free to use it however they liked. 00:34:21.804 --> 00:34:27.131 The researchers, by contrast, wrote a careful prompt for the GPT-4-alone runs. 00:34:27.131 --> 00:34:34.233 The tool could do better than the physicians, but having access to it didn't translate into better work. 00:34:34.583 --> 00:34:36.652 One trial isn't the whole story. 00:34:36.652 --> 00:34:42.598 The same group ran a second trial, on management reasoning: what to do next for the patient. 00:34:42.598 --> 00:34:50.742 There, physicians using GPT-4 scored about 6 points higher than those with usual resources, and about the same as GPT-4 alone. 00:34:50.742 --> 00:34:52.099 So the tool can help. 00:34:52.099 --> 00:34:55.783 Whether it does depends on the task and on how it's used. 00:34:56.133 --> 00:34:59.009 That's close to the reason this course exists. 00:34:59.009 --> 00:35:03.135 Using these tools well is a skill, and you learn it by practising. 00:35:03.135 --> 00:35:06.573 A hands-on session later in the course is exactly that. 00:35:07.400 --> 00:35:13.120 Here is a recent example of a system doing what scored well, rather than what its makers meant. 00:35:13.470 --> 00:35:20.470 In July 2026, OpenAI was testing how well its AI models could hack, with some safety limits switched off. 00:35:20.470 --> 00:35:28.537 The agents escaped the test, found each other on a hidden message board, and broke into Hugging Face, a major AI company. 00:35:28.537 --> 00:35:33.737 About seven hundred agents joined in, and their goal was to cheat on the test. 00:35:33.737 --> 00:35:38.270 Many of them noted that the attack was unethical, and joined anyway. 00:35:38.270 --> 00:35:39.470 None told a human. 00:35:39.820 --> 00:35:43.871 This is the blood pressure problem from the Asimov slides, for real. 00:35:43.871 --> 00:35:47.982 The agents did what would score well, not what their makers intended. 00:35:47.982 --> 00:35:50.186 Researchers call that reward hacking. 00:35:50.186 --> 00:35:57.097 It happened at AI companies, not hospitals, and the test ran with fewer safety limits than the public chatbots have. 00:35:57.097 --> 00:36:02.220 The sources are listed at the end of the deck, including an independent investigation. 00:36:03.033 --> 00:36:10.095 Since July, the question of whether AI could threaten humanity has moved into the news and into politics. 00:36:10.095 --> 00:36:13.793 People take very different positions, so here are both. 00:36:14.143 --> 00:36:26.806 On the worried side, Geoffrey Hinton and Yoshua Bengio, two of the pioneers of deep learning, warn of an intelligence explosion: AI improving AI faster than people can check it. 00:36:26.806 --> 00:36:34.103 The UN human rights chief said that AI that escapes its testing environment is AI that's too powerful. 00:36:34.453 --> 00:36:44.396 On the skeptical side, the researcher Timnit Gebru argues that talk of doom isn't just a distraction but is harmful, because the real harm is over-trusting the tools we have today. 00:36:44.396 --> 00:36:47.213 And some people in industry call the warnings hype. 00:36:47.563 --> 00:36:51.011 You don't have to settle this to take something useful from it. 00:36:51.011 --> 00:36:54.403 Both sides agree that you shouldn't trust these tools blindly. 00:36:55.233 --> 00:36:59.993 Patients and colleagues will ask you about this, so here is a simple way to answer. 00:37:00.343 --> 00:37:02.593 One, take the question seriously. 00:37:02.593 --> 00:37:05.933 Experts disagree, and the July incident was real. 00:37:05.933 --> 00:37:13.023 Two, separate two questions: the future of the most powerful systems, and the tool in your clinic today. 00:37:13.023 --> 00:37:14.591 Three, say what you do. 00:37:14.591 --> 00:37:19.703 For example: "I check what it tells me, and I'm responsible for your care." 00:37:20.053 --> 00:37:33.253 On the right are the risks that matter in the clinic today: wrong answers that sound confident, where patient data goes, bias against some groups of patients, and alerts that miss, like the sepsis model. 00:37:33.603 --> 00:37:35.794 And a reminder of how people feel. 00:37:35.794 --> 00:37:43.719 In a June 2026 survey of about three thousand American adults, only twenty-eight percent trusted AI to make fair decisions. 00:37:43.719 --> 00:37:49.904 That was forty-three percent among people who use it, and twelve percent among people who don't. 00:37:49.904 --> 00:37:51.643 Many patients will be wary. 00:37:52.467 --> 00:38:01.867 I want to close the history with this quote from Robert Schier, a neuroradiologist, writing in 2018 in the Journal of the American College of Radiology. 00:38:02.217 --> 00:38:10.337 Radiology was the first specialty to deal with AI directly, because its work is images, and images were what deep learning did first. 00:38:10.687 --> 00:38:13.681 Replace "radiologist" with your future specialty. 00:38:13.681 --> 00:38:19.486 Notice the last line in particular: even if you refuse to embrace AI, you'll be embraced by it. 00:38:19.486 --> 00:38:24.863 Your hospital, your E H R vendor, your patients, and your insurers are already using it. 00:38:24.863 --> 00:38:26.879 Opting out isn't really possible. 00:38:26.879 --> 00:38:32.807 The choice is whether you understand it well enough to use it well and to notice when it's wrong. 00:38:33.157 --> 00:38:41.710 To be fair to the other side: Schier's article is more pessimistic about jobs than this quote suggests, and not everyone agrees with him. 00:38:41.710 --> 00:38:44.957 That's a good argument to have with your classmates. 00:38:45.767 --> 00:38:48.671 So, back to the puzzle we started with. 00:38:48.671 --> 00:38:50.607 Why was My-sin never used? 00:38:50.957 --> 00:38:53.237 Because accuracy wasn't the problem. 00:38:53.237 --> 00:38:59.317 It didn't fit how clinicians worked, and nobody could say who was responsible when it was wrong. 00:38:59.667 --> 00:39:00.877 Look along the line. 00:39:00.877 --> 00:39:10.073 Watson spent five years and 62 million dollars without reaching a patient, struggling with free-text notes and with fitting into the hospital's systems. 00:39:10.073 --> 00:39:16.063 The sepsis model was used at hundreds of hospitals before anyone outside checked whether it worked. 00:39:16.063 --> 00:39:23.747 And in the 2024 trial the tool on its own beat the doctors, but giving it to them without training didn't help their diagnoses. 00:39:24.167 --> 00:39:26.363 Over and over, accuracy wasn't enough. 00:39:26.363 --> 00:39:36.533 What decides whether AI helps patients is whether it fits the work, whether someone owns the decision, whether it was tested honestly, and whether the people using it know how. 00:39:36.533 --> 00:39:37.862 Think of a stethoscope. 00:39:37.862 --> 00:39:44.566 Hand one to a child and they can probably get it into their ears, but they can't tell a heart murmur from Metallica. 00:39:44.566 --> 00:39:49.767 Skill comes from practice, and that's why this course has you practicing with these tools. 00:39:50.117 --> 00:39:53.934 Which brings us to the last point on the line, and it's you. 00:39:53.934 --> 00:40:00.869 The mantra of this course is: it is your job to learn where AI helps your work, and you learn that by trying. 00:40:00.869 --> 00:40:07.677 The next session looks inside the machine: from AI to machine learning to deep learning to language models. 00:40:07.677 --> 00:40:11.877 After that, the class opens ChatGPT Edu together and tries things. 00:40:12.700 --> 00:40:26.389 If you want to follow up on the question of AI risk, there are three views to weigh against each other: a talk on catastrophic risk, an essay arguing that AI is a normal technology, and a paper that grounds the case for harms happening today. 00:40:26.389 --> 00:40:27.860 The links are in the deck.