Catch-up service:
Why Are LLMs fixated on the number 7?
The Emperor’s New Diplomacy
Podcast: Jemima Kelly on the radical right
Successful Politicians Are Pattern-Breakers
Is Civility a Fantasy?
A Studio 54 Immigration Policy
10 Ways To Buy Happiness
AI is a magnet for clever people who like thinking about intelligence and decision-making - people who might in another life have become cognitive psychologists. Indeed there’s considerable cross-fertilisation between the two fields. Psychology has helped AI researchers figure out to build smarter machines; AI research has helped psychologists understand human behaviour. No surprise, then, that some of the technical concepts which have emerged from AI and computer science are eminently applicable to human life.
I recently listened to Dwarkesh Patel’s interview with Andrej Karpathy, co-founder of Open AI (now departed) and leading light in Silicon Valley. As a layman I only had my head above water for some of this conversation yet I still got a lot from it. Karpathy is brilliant at articulating complex ideas in vivid language and he does a lot of toggling between AI and human psychology. The conversation also reminded me of a brilliant book from a few years ago called Algorithms To Live By, by Brian Christian and Tom Griffiths. So, with some help from those sources, here are five ways in which the problems faced by AI systems mirror our own - and what we can learn from AI engineers about how to address them.
First two are free. Also after the jump: a splendid Rattle Bag which includes some very good new about the world. Jump in on a paid subscription today.
Model Collapse. A machine learning model gets trained on human data - photographs; Reddit forums; Amazon reviews; books and music and so on. We’re already at the stage where the big models have hoovered so much of this data into their maws that finding fresh sources to train on is hard. AI models have no problem producing data, however, so the obvious answer is to train on data from other models. Indeed this is inevitable, as the internet becomes swamped by LLM-generated text and images.
But there’s a problem with this. Model-generated data tends to be more predictable and less diverse than human data, as it imposes statistical patterns on the infinite variety of human outputs. Over time, this creates a feedback loop. Each new iteration of the model inherits the biases and errors of the previous one, but with less variety and signal. Eventually, the model converges towards - or collapses into - the generic and repetitive. It loses accuracy, nuance, creativity until only a thin and colourless monoculture remains. This is ‘model collapse’.
Damn those machines, erasing the incorrigible plurality of human minds! And yet I think Karpathy is right when he points out this is also something we do to ourselves: “Humans collapse during their course of their lives.” We, too, learn patterns in the data of experience on which we then become over-dependent, favouring the model in our heads over the flux of reality. As we get older, we become rigid in our thinking, set in our ways. We talk to the same friends and use the same information sources. We return to the same old thoughts and say the same old stuff.
We are also inviting model collapse at the cultural level, as art forms turn in on themselves and stop looking outwards for inspiration. Much new pop music sounds like a karaoke version of what came before it. Songwriters write for the streaming algorithms. Hollywood movies have become increasingly predictable and formulaic. Visual artists offer thin imitations of twentieth century innovators. This has been going for a long time, pre-dating the internet; post-modernism is pretty much the cultural theory of model collapse.
How do you slow or prevent this kind of decay? You raise your quality control standards. The AI companies are filtering out AI-generated content and privileging human data. OpenAI is hiring experts in various domains to create content exclusively for their models. AI companies are also making efforts to make sure that rare or anomalous data - texts which don’t fit mainstream opinion or don’t follow standard formats and styles - doesn’t get excluded from the training. The aim is to filter out nonsense, like QAnon-style conspiracy theories, without eliminating everything except for the safe, conventional and generic.
The equivalent for the rest of us is to devote more time and effort to the curation of our information diet, rather than passively hoovering up whatever we’re served. As the rest of the world is increasingly training itself on social media slop, there are increasing returns to reading great books. We should be seeking out sources of knowledge and insight which most people aren’t looking at. We should expose ourselves to novelty, privileging new art (or at least, art that is new to us), and people from outside our usual circles. We should look for contrarian perspectives from people who really know what they’re talking about.
2. Overfitting. This refers to when a model memorises its training data too closely, and in doing so, fails to learn general lessons from it. An overfitted model is brilliant at performing tasks with the examples it’s been fed but confounded by new, unseen data. It can parrot but it can’t improvise. It’s like someone who learns every step in a recipe book by heart but makes a mess of anything that isn’t in it. AI engineers compensate for overfitting by making things a bit more difficult for the model - penalising it for relying too heavily on specific patterns, or stopping the training before the model gets too tuned to a particular dataset.
We tend to overfit our mental models to our own lives. It’s good to live life with some stability and predictability but routine can narrow the mind, dull the sensibility, and make the unfamiliar bafflingly hard to process. If we’re not careful we become the world’s foremost expert on the dataset of our own small world, and prone to misreading situations outside it. We either become wildly over-confident in our abilities once outside our domain (like Elon Musk and politics) or we get scared of the unknown and make our world a little smaller.
It’s a good idea to break your habits and routines, almost at random, from time to time, just to see what happens. You may just be reminded of why you chose that habit in the first place, but you may discover a new and better way of using your time - as in the case of the famous study which found that a Tube strike forced commuters into finding better routes to work, improving productivity overall.
The brain has its own in-built mechanism for mitigating the problem of overfitting: dreams. The neuroscientist Erik Hoel has proposed the intriguing theory that we evolved the capacity to dreams in order to stop ourselves from becoming too narrowly focused on our daily experience, which he specifically likens to overfitting in AI.
Hoel’s theory is that dreaming is a way for the brain to shake up neural patterns that might otherwise have become too rigid and inflexible to cope with new experiences. Dreams, he says, are “a beneficial injection of noise into the data”. They take your memories, often the most mundane ones, and throw them back at you in bizarre, distorted, nonsensical forms. How do you like them apples? Dreams operate like the Fool in King Lear, turning sense on its head in order to keep us clear-eyed.
Keep reading with a 7-day free trial
Subscribe to The Ruffian to keep reading this post and get 7 days of free access to the full post archives.