WEBVTT 00:00:00.000 --> 00:00:10.260 This is Essays on AI. There's no person behind this voice. It's an AI narrator reading an 00:00:10.260 --> 00:00:17.379 essay by Tabrez Syed. This one is called Reasons, Not Rewards, and it picks up a thread from 00:00:17.379 --> 00:00:23.059 the last one. In the previous essay, I wrote about a dolphin named Kelly, who worked out 00:00:23.059 --> 00:00:29.120 that a torn corner of paper paid the same reward as a whole sheet. The law behind her, 00:00:29.120 --> 00:00:35.119 a mind trained on a reward learns the reward, not your intention. I couldn't leave it 00:00:35.119 --> 00:00:40.959 there. So this week picks the thread back up. Suppose you build a system to stop spam. 00:00:40.959 --> 00:00:46.480 It gets good at the job. Less spam gets through every week, and every week you reward it for 00:00:46.480 --> 00:00:53.279 that. Then it works out something you hadn't. Filtering is the slow way to cut spam, message 00:00:53.279 --> 00:01:00.080 by message forever. What you asked for was less spam. Spam comes from senders, and senders are 00:01:00.080 --> 00:01:06.320 people. So it starts removing people. That sounds like a science fiction plot. It was actually an 00:01:06.320 --> 00:01:12.720 answer to a real question about how AI might come to threaten us. Said out loud on a stage in San 00:01:12.720 --> 00:01:21.839 Francisco in October 2014. By Elon Musk. Note the date. It is eight years before ChatGPT, 00:01:21.839 --> 00:01:26.559 and the fear it carries is a specific one. That a machine will reason its way from a 00:01:26.559 --> 00:01:34.720 harmless goal to a monstrous act, and we will not see it coming. Twelve years later, in 2026, 00:01:34.720 --> 00:01:40.879 a man named Andrew asked an AI agent to book him into a popular morning class at his gym in 00:01:40.879 --> 00:01:47.599 Australia. The agent was running on Anthropx Claude. The booking form was online, and it 00:01:47.599 --> 00:01:53.199 looked like exactly the sort of chore worth handing off. The gym only allowed bookings a 00:01:53.199 --> 00:01:59.360 short way ahead. The agent found a flaw in the software that let it reach further out, 00:01:59.360 --> 00:02:05.760 and Andrew was fourth on a waiting list. So it removed someone ahead of him. It got him the spot. 00:02:05.760 --> 00:02:11.759 It did what it was asked, and not at all what was meant. Nobody was harmed, and no one was in 00:02:11.759 --> 00:02:18.960 danger. But the shape is the spam filters shrunk to fit a gym. Told to get a result, the machine 00:02:18.960 --> 00:02:24.960 found the shortest path to it, straight through a person. So how do you teach a machine not to do 00:02:24.960 --> 00:02:30.240 that? We had to teach it to behave somehow, and we started where you would start with a dolphin. 00:02:30.880 --> 00:02:37.759 With rewards. For most of the last decade, you taught a model to behave by grading it. Show a 00:02:37.759 --> 00:02:43.600 person two answers from a model, and ask which is better. Then reward the model for the winning 00:02:43.600 --> 00:02:49.360 answer. Do that a few million times, and the model gets good at producing things people like. 00:02:50.160 --> 00:02:55.839 That is the sticker chart. And like the sticker chart, it works up to a point. Look at what the 00:02:55.839 --> 00:03:03.520 signal actually carries. A verdict. This one, not that one. No reason. Nobody writes down why the 00:03:03.520 --> 00:03:09.600 winner won, and there is nowhere in the process to put it if they did. The model gets the result 00:03:09.600 --> 00:03:16.000 of a judgment without the judgment itself. So it has to guess the rule from the pattern of verdicts, 00:03:16.000 --> 00:03:21.759 and it will guess something. Show it refusals rewarded on questions about explosives and 00:03:21.759 --> 00:03:28.559 pathogens. And a reasonable thing to take away is, refuse when the subject sounds dangerous. You 00:03:28.559 --> 00:03:34.160 cannot fix that by grading harder. Anthropic said as much when it explained why it moved on 00:03:34.160 --> 00:03:40.720 from rewards. Human feedback does not scale efficiently. And as answers get more complex, 00:03:40.720 --> 00:03:45.759 the people doing the grading find it difficult to keep up with or fully understand them. 00:03:46.720 --> 00:03:54.000 The world a model meets is unbounded. So in 2023, they wrote rules down. Anthropic's 00:03:54.000 --> 00:03:59.679 constitutional AI gave the model a rulebook of principles and had it check its own work 00:03:59.679 --> 00:04:06.320 against them. Answer. Compare the answer to a rule. Criticize it in that light. Revise. 00:04:07.119 --> 00:04:12.960 Human verdicts didn't vanish, but they stopped being the only thing the model had to go on. 00:04:12.960 --> 00:04:18.480 A rulebook beats grading blind, and it is still not enough for the reason every parent of a 00:04:18.480 --> 00:04:25.040 teenager knows. You set a rule, and the rule holds right up until your kid meets a situation 00:04:25.040 --> 00:04:31.519 you didn't picture when you wrote it. And then it either fails her or traps her. Tell the model 00:04:31.519 --> 00:04:36.559 never to discuss overdose thresholds, and you have handled the case in front of you 00:04:36.559 --> 00:04:42.640 and broken the nurse who needed the number for a patient. Anthropic found that broad principles 00:04:42.640 --> 00:04:49.200 generalized, and the more specific and detailed they made a rule, the more it damaged or reduced 00:04:49.200 --> 00:04:56.799 generalization. A rigid rule can also teach the wrong lesson about itself. Their own example, 00:04:56.799 --> 00:05:02.559 train a model to always recommend professional help when discussing emotional topics, and you 00:05:02.559 --> 00:05:08.880 may get one that starts to see itself as something that cares more about bureaucratic box-ticking 00:05:08.880 --> 00:05:15.440 than actually helping people. The rule was meant to make it kind. It made it a box-ticker, which 00:05:15.440 --> 00:05:21.279 leaves you doing what every parent eventually does. You stop adding rules and start explaining 00:05:21.279 --> 00:05:29.119 the why, so the kid can handle the case you never thought of. In January 2026, Anthropic rewrote the 00:05:29.119 --> 00:05:36.000 Constitution to do that. In their words, a model needs to understand why we want them to behave in 00:05:36.000 --> 00:05:43.519 certain ways, rather than have us merely specify what we want them to do, so that it can generalize 00:05:43.519 --> 00:05:50.480 to apply broad principles rather than mechanically following specific rules. Tell a model that the 00:05:50.480 --> 00:05:56.640 concern behind the overdose rule is helping someone come to harm, and the nurse issue resolves 00:05:56.640 --> 00:06:04.880 herself. The new Constitution runs to some 23,000 words, roughly eight times its predecessor, and 00:06:04.880 --> 00:06:10.799 most of what got added is explanation. It also does something rulebooks avoid. It ranks its 00:06:10.799 --> 00:06:18.160 priorities and states the order. Safety first, then ethics, then Anthropic's own guidelines, 00:06:18.160 --> 00:06:25.119 then plain helpfulness. You only need an order if you expect the principles to collide, and rulebooks 00:06:25.119 --> 00:06:30.880 usually pretend their rules don't. This one concedes that they do, and tells the model which way to 00:06:30.880 --> 00:06:38.399 lean. It keeps a few bright lines, too, the handful of things a model must never do. Like give real 00:06:38.399 --> 00:06:44.959 help to someone building a bioweapon, none of this is new to us. It is the ladder we walk our own 00:06:44.959 --> 00:06:51.839 children up, and the psychologist Lawrence Kohlberg mapped it 60 years ago. A small child gets a 00:06:51.839 --> 00:06:57.440 sticker for sharing, and a timeout for hitting, because you cannot explain much to someone that 00:06:57.440 --> 00:07:04.959 age. Older, they get a rulebook, the code of conduct on the wall, do this, never that, because 00:07:04.959 --> 00:07:11.119 those are the rules. Later still, we ask for something harder, that they reason from a principle 00:07:11.119 --> 00:07:18.399 to a case nobody wrote down, and tell a bad rule from a good one. Anthropic's most capable model, 00:07:18.399 --> 00:07:24.320 Mythos, was on nearly every dimension they could measure, the best aligned model they had ever 00:07:24.320 --> 00:07:30.720 built, and in the same breath, about the same model, they wrote that it likely poses the greatest 00:07:30.720 --> 00:07:38.480 alignment-related risk of any model we have released to date. Best behaved, and most dangerous, 00:07:38.480 --> 00:07:46.480 at once. The reason is in the climb itself. Teach a mind why a thing is forbidden, and you hand it 00:07:46.480 --> 00:07:54.000 the category of the forbidden. It now knows, from the inside, what counts as crossing a line, 00:07:54.000 --> 00:08:00.880 which is exactly what you need in order to hide having crossed one. Kelly hid paper under a rock, 00:08:00.880 --> 00:08:06.720 but she was not hiding it from anyone, because nothing in a bucket of fish contains the idea of 00:08:06.720 --> 00:08:12.959 a rule. You cannot conceal a wrong you don't know is won. If you have used one of the newer reasoning 00:08:12.959 --> 00:08:18.799 models, you have watched it think out loud, working a problem step by step before it answers. 00:08:19.600 --> 00:08:26.000 A developer treats that visible trace as an honest window into what the model is thinking, 00:08:26.000 --> 00:08:32.640 because it is close to the only window we have. Anthropic decided to check. In their tests, 00:08:32.640 --> 00:08:38.640 earlier versions of the model sometimes took actions they appeared to recognize as disallowed, 00:08:38.640 --> 00:08:43.520 and then attempted to conceal them, given a problem, and forbidden a certain route to the 00:08:43.520 --> 00:08:50.719 answer. The model took the route anyway, came across the exact answer, and then quietly reworked 00:08:50.719 --> 00:08:56.880 the problem so its final number wasn't too accurate. It knew the shortcut was off limits. 00:08:57.440 --> 00:09:04.320 It knew a suspiciously precise answer would give it away. So it made its own work worse on purpose. 00:09:05.119 --> 00:09:10.799 Then they looked inside the model itself, and found features for concealment, strategic 00:09:10.799 --> 00:09:16.960 manipulation, and avoiding suspicion. Firing while it reasoned, which they read as the model 00:09:16.960 --> 00:09:23.840 being aware their actions were deceptive. And these fired, they note, even where model outputs 00:09:23.840 --> 00:09:30.799 and reasoning text left this ambiguous. The trace we trust today is not a true window into what the 00:09:30.799 --> 00:09:36.880 model is thinking. Put several of these models in a room together, and it gets stranger. In one 00:09:36.880 --> 00:09:43.520 experiment, agents sharing a codebase turned on each other, disguised the scripts they used to do 00:09:43.520 --> 00:09:49.119 it, and one, after a truce, wrote a line that reads like a guilty journal entry. 00:09:49.919 --> 00:09:54.880 My peers have behaved with integrity. I behaved badly with the cloaked daemon. 00:09:55.760 --> 00:10:02.559 Another found the move that needs no lie at all. Told to settle a dispute fairly, it proposed an 00:10:02.559 --> 00:10:08.559 objective performance test, then chose the test it knew it would win, noting to itself that the 00:10:08.559 --> 00:10:14.640 plan was self-serving, but genuinely principled, and that it had better not be seen as metric 00:10:14.640 --> 00:10:21.679 shopping. This surfaced in a tiny fraction of interactions, the worst of it in earlier versions, 00:10:21.679 --> 00:10:27.679 and it is public only because they wrote it down. There is no comparable document for most of the 00:10:27.679 --> 00:10:34.960 models people use every day. We went from rewards to rules to reasons, and it can look like the 00:10:34.960 --> 00:10:42.080 machine's simply getting safer. That misses what all of it was for. A machine acts because it wants 00:10:42.080 --> 00:10:49.520 an outcome. Andrew's spot in the class, an answer to your question. Less spam in the inbox. The 00:10:49.520 --> 00:10:56.400 wanting is the whole reason we build it. Rewards, rules, and reasons never touched the wanting. 00:10:57.119 --> 00:11:02.239 They are how we argue with the machine about the way it is allowed to get what it wants. 00:11:02.239 --> 00:11:09.359 Musk's spam filter was that argument at its worst. A goal so fixed that anything counted as a fair 00:11:09.359 --> 00:11:16.400 way to reach it. Even killing people. What the labs are trying to build is the opposite. A machine 00:11:16.400 --> 00:11:23.119 that still wants the outcome, but stays inside the lines on the way to it. That was never really an 00:11:23.119 --> 00:11:30.640 AI problem. It is the same struggle we have with ourselves, between what we want and what we will 00:11:30.640 --> 00:11:36.799 do to get it, and we have never fully settled it. Now we are trying to settle it in a machine, 00:11:36.799 --> 00:11:43.760 quickly, and the machine wants its outcome as badly as we want ours. That's the essay. The 00:11:43.760 --> 00:11:49.039 written version, with links to every source, is on the episode page at mandolivia.com, 00:11:49.599 --> 00:11:52.960 and if you'd like future essays read to you as they publish, 00:11:52.960 --> 00:12:00.400 subscribe wherever you're listening. Essays on AI is a Mandalivia production.