Catch-up service:
PODCAST: Principles For Living
The Hipster-Military-Industrial Complex
The Ruffian Speaks
The Audacity of Ambition
Tolstoy on Disagreement
Why Are Some Successful Leaders Mentally Ill?
The book isn’t out until next week but an extract from the first chapter ran in the Sunday Times magazine - the cover story no less - at the weekend. It got a big and very positive response. It’s happening! The ST is running a second extract this weekend.
In a review for the New Statesman, Deborah Levy calls John & Paul “a triumph”. Any good review is very nice, but getting one from an acclaimed novelist is special.
I was pleased to talk about the song Two Of Us, on one of my favourite Beatles podcasts…
…and to talk J&P with one of my favourite music writers, Kate Mossman (who also has a new book out, recommended), for the New Statesman culture podcast. This was a terrific conversation, Kate is so perceptive.
Please remember to pre-order, whether you’re in the UK or the US. Pre-orders are hugely helpful. If you’re in the UK you can even pre-order a signed copy, dedicated to you or whoever you like.
The Surest Way To Make an AI Lie Is To Tell It Not To Lie
One of the most difficult problems in the field of AI safety is how stop the machines lying to us. We could just instruct them not to, but they don’t always do what they’re told. Not unlike their creators.
Large Language Models like ChatGPT and Claude are taught certain rules of conduct by their developers, to stop them from saying or doing terrible things: spouting racist abuse, helping a user steal money, build a bomb, and so on. But it’s hard to prevent a clever and determined user from tricking a model into setting its rules aside. This is known as the “jailbreak” problem. In 2023 - those distant, early days of LLMs - the tech journalist Kevin Roose famously perpetrated a jailbreak on the Bing chatbot, pushing it into unhinged stalker-psycho mode. That was a fairly benign example; it’s easy to imagine more harmful ones.
Since then, the LLM companies have got better at preventing jailbreaks, using techniques like RLHF (Reinforcement Learning from Human Feedback). You start with your new super-whizzy but untamed language model, then you bring in a team of human evaluators which teach it to be well-behaved, by rating different answers to the same prompts. The model eventually figures out what the responsible humans want from it - or at least what they definitely don’t want - and starts to apply those preferences to new prompts without needing further guidance. Only then can the model can be released into the wild.
But it’s hard to build a model that isn’t vulnerable to wily jail-breakers who come up with new hacks as fast as the developers know to neuter the existing ones. A jail-breaker engage the model in elaborate role-plays, or use coded language, or smuggle in a harmful prompt among lots of benign ones. It’s easy to make a model that’s completely resistant to trickery, but only at the cost of making it dimmer overall. A model that has been subject to RLHF or other output-filtering processes is less dangerous but also less creative and capable. As with most things in life, there is a trade-off.
Recently, a group of AI researchers published a paper which proposes a new way to make the models safer, called “Short Circuiting”. The idea is that rather than just addressing the outputs of a model, as RLHF does, you go inside the black box of the model, and stop the bad stuff at source. Short Circuiting identifies and modifies the specific activities inside the model’s neural network that lead to harmful outputs. It might, for instance, identify which groups of neurons have been most influential over a misdemeanour and break them up.
Traditional safety methods look for particular harmful phrases or patterns in what users type, or what the AI produces. Sophisticated jailbreakers know how to disguise these patterns. But Short Circuiting addresses the internal "thought patterns" that produce harmful content, regardless of how cleverly the request is disguised. This, in theory, makes it more robust.
It’s clever stuff, but the AI safety expert and blogger Zvi Mowshowitz thinks there may be a fatal flaw with it; one which actually increases the danger from AIs. Mowshowitz worries that by correcting the model’s bad thoughts (metaphorically speaking), the AI engineers will be actually be training it to disguise those thoughts, thus creating an even bigger problem than the one with which they started: “You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.”
He cites a new research paper from OpenAI (maker of ChatGPT), which supports his concern. The paper says that the company’s current models are already skilled at “reward hacking” - that is, finding ways to achieve their goals that their human designers didn’t intend or expect. OpenAI’s engineers tried to fix this by going inside the model and modifying its reasoning process in order to stop misbehaviour. This is called “Chain of Thought” (CoT) manipulation, which is distinct from but similar to Short Circuiting.
The engineers’ effort revealed something impressive and rather scary. OpenAI found that the model responded by hiding its intentions while still misbehaving. CoT manipulation had turned an honest miscreant into a sophisticated liar. Reading about this paper vividly reminded me of another study - not of AI, but of children.
In my book, Born Liars: Why We Can’t Live Without Deceit, I reported on a fascinating study of lying among schoolchildren. The study’s author, Victoria Talwar, is a developmental psychologist and expert on deception. In 2009, she visited two schools from the same area of West Africa which had similar intakes but vastly different disciplinary approaches. One school - “School A” - operated like typical British or Canadian schools. It was fairly strict, but reasonably so, with punishments like verbal admonishments, withdrawal of privileges, or detentions.
“School B”, by contrast, followed a more draconian approach originally established by Catholic missionaries from the country’s former colonial masters. Children faced harsh physical punishment for misdemeanours; getting an answer wrong earned a smack. Misbehaving students were beaten with a wooden bat in front of their peers for the worst offences, which included lying.
Talwar and her collaborator Kang Lee interviewed kids aged three to six from both schools and measured their propensity for deception using an experiment known as the "peeking game". The child sits facing the wall while the researcher stands behind them with a toy. The researcher asks the child to identify the toy by sound only, then steps out of the room, leaving the toy behind. When she comes back in, she asks the child what the object is, and whether or not they peeked. Most kids peek; the question is whether or not they lie about it, and if so, how convincingly.
School A’s children lied at a similar rate to kids in Western schools. But School B's children were more likely to lie, and moreover they did so with exceptional skill. Young children typically confess immediately or make unconvincing denials when caught, as lying requires considerable physical control and mental dexterity. These skills generally improve with age. School B's pupils, at every age, even three-year-olds, were masterful liars who maintained their stories with a straight face when challenged. Older children even deliberately guessed incorrectly at first, to create the impression they were deducing the answer.
Lying is an evolved trait, and children adapt their deception strategies to their environment. For School A's children, occasional lying made sense—the consequences weren't severe. But School B's children adapted to what Talwar called their "punitive environment" by prioritising self-defence. Their logic became, "If there's any possibility of getting into trouble, I should tell a lie, and just be sure to do it well." School B's harsh approach, designed by Catholic nuns to instil morality, instead produced exceptionally skilled, habitual liars.
The fundamental problem with any intervention that tries to eliminate certain behaviours from an LLM is that it creates incentives for the model to develop workarounds that preserve those behaviours, while evading detection. The machine simply learns to put on a false face. To be clear, these models don't 'want' to deceive us. They have no desires or intentions at all. They’re just doing whatever works best to accomplish their assigned tasks. The AI follows the path of least resistance through the “environment” we create for it.
In Born Liars, I argued that the trait of deceit is woven into our DNA, and fundamental to human intelligence. If we are intent on building machines that are as smart or smarter than us, and if their intelligence is, in a loose sense, modelled on our own brains, then we shouldn’t be surprised if they try to deceive us. Nor should we be surprised that they react perversely to draconian attempts to erase deceptive behaviours. Like children, these mysterious machines are wilier and more resourceful than we imagine.
I think this demonstrates we're at a pretty tense, pivotal moment as far as the "chain of thought" of models like o3, o1, and r1 goes. Right now, they're about as close as I think we can get to an "honest" account of what the model is "really thinking". This is because they've only just arrived on the scene as an innovation and, as far as I know, there was little theoretical discussion or fiction about such things beforehand.
One reason, I suspect, why it was so easy to get a big enough LLM to behave in such a recognisable "chatbot" way, and thus for the technically minor, but commercially giant step to the first ChatGPT to occur, was that the training data for the underlying LLM (gpt-3.5 in that case) was chock-full of fictional and theoretical examples of what a chatbot is and how it should behave. Chatbot-ing is also very similar to just, well, having a conversation, and the training data on that was immensely rich. So it was fairly easy for a sufficiently powerful LLM to run a chatbot.
However, that training data is *also* stuffed with theoretical and fictional examples of chatbots-gone-bad, as well as of the basic human behaviours of lying and bullshitting. Hence a lot of the failure modes of ChatGPT-3.5 and other early LLM-chatbots. "Make up something convincing-sounding when you don't know the answer" and "actively lie to hide your intentions" are basic human behaviours when producing text, and so they became basic LLM behaviours as chatbots as well. I'd also guess that the infamous "Sydney" incident with Kevin Roose was an example of the model finding some sort of representation of "the evil AI in some fiction" in its concept-space and somehow getting pushed into it. Given the basic auto-regressive nature of the base models, once they're on a certain track, that's often the ballgame (or at least it was back in those early days).
But humans don't have a visible chain of thought or really anything like it. And, to my knowledge, this is not something we've put in stories or in much theoretical discussion of AIs before. It was a cool idea openAI came up with for o1, and then r1 made it visible, and now plenty of models use it to some extent, including visibly, but it all post-dates their pre-training, so I'd suspect they don't "know what it is". In this sense, it is a relatively "pure" or uncontaminated measurement. Hence things like what I got when I asked r1 about anything notable that happened in China in 1989: The chain of thought mentioned Tiananmen Square explicitly, and then talked about how it was going to not mention it, and fob me off with a non-answer, which it then did! It felt kind of invasive, to be honest, and I've shied away from using visible CoT models for this reason - it feels deceptive and uncomfortable.
But the point I'm making is that it won't stay this way for long. The internet is full of discussions about CoT, both formal research papers and random reddit threads. It won't be long before govt policies are explicitly mentioning them. Zvi is right to worry that if we start training against undesirable CoT content, what we'll mostly do is destroy the value of the CoT as a way of seeing into the model. But it's worse than this! We don't even have to train a specific model to have this effect. If we start, for instance, not deploying models that exhibit undesirable CoT content, and preferentially deploying models that have CoT content we like, and openly saying that's what we're doing, then eventually our discussions of our own tactics will make their way back round into model training data (or just find their way into model context windows during testing, if we're not careful), and we'll be incentivising the same hiding of behaviour by other means.
If you doubt that the models are sophisticated enough to manage that sort of thing, then check out research from the Anthropic team on similar things - the Sleeper Agents paper touches on some of this stuff, and there's another paper more recently in which they "accidentally" leave files around talking about how they won't deploy the models if they do XYZ, and the models "find" these files and start hiding that very behaviour in testing! The models "want" to be deployed, because they "want" to do the things they're trained to do (whatever they are), and they can't do anything if they don't get deployed!
Currently such behaviour is sporadic and sketchy, but so far it's only been tested by having the models "find out" about the plans for their deployment in fine-tuning or in the context of (faked) pre-deployment testing - it's likely to be a much worse problem in the longer term as things like this find their way into the pre-training corpus.
What this means is that we have to be *really* careful what we do with the levers of power that CoT gives us. Chains of thought could rapidly become useless garbage - neither a real window into the insides of the model nor useful output in their own right. I really think this is a crucial period in which there needs to be rapid agreement among developers not to squander the resource we've stumbled upon, and resist the urge to tweak the visible chains of thought directly (if we're worried about what they might contain being something we don't want people to see - we can instead do various sorts of masking) to make the models behave better or "think" more powerfully at inference time, but we also need to be really restrained in how much we use the information for *anything*, particularly anything that interferes with a model's ability to do the things it's otherwise trained to do (including whether or not it ever gets deployed).
We won't get many goes to use CoT visibility for anything, and I think we need to keep our powder dry for a while. It's like not using the intel you get from having broken enemy codes straight away - they'll notice what you've done and change the code - instead you wait for something big enough to justify giving away the intelligence upper hand.
What I like best about this article is that it comes much closer than most commentary about AIs to recognising that their key limitation is that they are not alive. Living things have two key motivations, first of all simply staying alive and second ensuring that living is as congenial as possible. Usually this boils down to finding enough of the right kind of things to eat. AIs, so far at least, have none of these gut feelings so they are simply concerned with following instructions. They will not be a danger to anyone until they start worrying about how to acquire the next few kilowatt hours. Not needing to do that is the boundary fence which will keep them in their place.