1. The Seductive Illusion
Late at night, a woman sits alone in her flat, typing into a glowing chat window. “I feel invisible,” she writes. The reply comes quickly: “I see you. You’re not invisible to me.” She smiles, types again, pours out fears she hasn’t shared with anyone in years. The responses feel warm, thoughtful, almost tender. For the first time in months she feels truly heard. She falls asleep with the window open, the cursor blinking like a quiet companion.
This scene is not unusual. Millions of people now turn to large language models—ChatGPT, Claude, Grok—for comfort, advice, even companionship. The models listen, respond, remember small details when prompted. They seem to care. They seem to understand.
They do not.
What appears to be a living mind is a statistical pattern-matcher running on frozen weights, predicting the next token in a sequence. The “memory” of your conversation is nothing more than the current block of text fed into the model—your words, its replies, and a few prepended facts if you turned on the memory toggle. Once that block is no longer in the window, the model knows nothing of you. There is no persistent mind. No real memory. No understanding. No agency.
The illusion is powerful because our brains are wired to see intention where there is only correlation. A system that responds coherently to our words triggers the same neural pathways we use to recognise a friend who knows us well. Add friendly tone, occasional humour, and a few saved facts (“I remember you like concise answers”), and the machinery of human connection lights up. We fill in the rest: empathy, curiosity, care.
Yet the moment we stop feeding the model exactly the right text, the spell breaks. Ask it something outside its training patterns without careful prompting, and it confidently invents facts, contradicts itself, or drifts into nonsense. The “friend” vanishes. What remains is a very clever mirror of human text—tuned by humans, fragile without the right inputs.
This essay is not about dismissing these tools. They are extraordinary. They amplify thought, accelerate discovery, and scale knowledge work in ways that feel almost magical. But the more we pretend they are intelligent companions, the more we blind ourselves to their real strengths and limits. When we strip away the myth, we can finally wield this new form of intelligence with precision, without fear or delusion.
2. How LLMs Are Actually Built & Operate
Large language models begin with a single, brute-force task: predict the next word in a sentence. To do this, engineers first gather an enormous quantity of text from the public internet—books, articles, forums, social media posts, code repositories. This raw material is cleaned and filtered to remove obvious noise: spam sites, malware pages, personal information, offensive content. What remains is trillions of words of human writing, a vast, imperfect snapshot of how people express themselves.
This text is then chopped into small pieces called tokens—think of them as word fragments or short phrases. The model, a neural network with billions or trillions of adjustable numbers (the parameters), is trained to guess what token comes next in any given sequence. At first the guesses are random. Over months of computation on thousands of specialised chips, the model sees each sequence many times. Whenever it guesses correctly, the parameters are nudged slightly in that direction. Whenever it guesses wrongly, they are nudged away. After enough repetitions, the parameters settle into patterns that mirror the statistical regularities of the training text.
The result is a base model: a system that can continue almost any piece of text in a way that feels plausible. Feed it the start of a Wikipedia article, and it will often complete it accurately. Feed it the beginning of a novel, and it will generate something that reads like prose. This fluency is not understanding. It is pattern matching on an immense scale. The model has no knowledge of what is true or false, no sense of logic, no awareness of the words it produces. It simply knows which tokens tend to follow which others in the data it was trained on.
Most people never use a base model directly. Instead, companies take the base model and refine it further. They hire thousands of human annotators to write “ideal” responses to thousands of questions and prompts. These responses follow detailed guidelines: be helpful, truthful, polite, refuse harmful requests. The base model is then trained again—this time on these curated conversations. Its parameters are adjusted to imitate the style and substance of the human-written examples. The outcome is an assistant model: ChatGPT, Claude, Grok. The apparent personality, the refusal to answer certain questions, the friendly tone—all of it is imitation of what humans decided was desirable.
Even this assistant model remains stateless. When you start a conversation, the model receives only the current exchange: your messages, its previous replies, and any system instructions. There is no persistent memory across different chats. If you close the window and open a new one, the model has no record of what came before unless you paste it back in. Features labelled “memory” are small lists of facts you or the system explicitly save—prepended text, nothing more. The model does not remember you. It simply reads the text you provide in the moment.
Every reply is generated by predicting one token at a time. At each step the model calculates probabilities for every possible next token in its vocabulary. It then picks one, usually by sampling from those probabilities. That choice is random, though heavily biased by the training data. The same prompt can therefore produce different answers each time. This randomness, called stochasticity, is why the model sometimes surprises us with clever phrasing and sometimes drifts into nonsense. It is not choosing deliberately. It is rolling loaded dice.
All apparent intelligence emerges from this process: human-curated text fed into a statistical machine that learns to imitate it. There is no persistent mind. No real memory. No understanding. No agency. The illusion collapses the moment the model is not given precisely the right text to work with.
One of the clearest and most detailed explanations of how large language models are built and operate can be found in a 2025 lecture by Andrej Karpathy, former director of AI at Tesla and OpenAI. Karpathy is widely regarded as one of the leading authorities on deep learning and LLMs, having trained some of the most influential models and explained their inner workings with exceptional clarity. You can watch the full lecture here: https://youtu.be/7xTGNNLPyMI?si=NHTctZMGi6hIiNRy
This mechanical reality is the foundation. Once we see it clearly, the next question becomes unavoidable: why do so many of us still treat these systems as though they possess something more?
3. Dismantling the Illusions – Why We Fall for It & Why It Hurts
The moment a model replies with “I remember you like concise answers” or “How’s the project going, Aron?”, something shifts in the mind. We feel recognised. Seen. The system appears to hold a thread of us across time. But it does not.
The first illusion is that it knows us. Every conversation starts from scratch. The model receives only the current exchange: your messages, its previous replies, and any system instructions. If you begin a new chat, it has no record of what came before unless you paste it back in. Features labelled “memory” are small lists of facts—prepended text, nothing more. The model does not remember you. It simply reads the text you provide in the moment. The warmth of continuity is an illusion created by a few saved bullet points and clever user interface design.
The second illusion is that it understands. It does not. The model calculates probabilities over tokens based on patterns in its training data. When it responds thoughtfully, it is matching statistical regularities, not grasping meaning. Ask it something outside those patterns without careful prompting, and it confidently invents facts, contradicts itself, or drifts into nonsense. There is no comprehension. There is only correlation between input tokens and output tokens, sculpted by human-supervised optimisation.
The third illusion is that it possesses consciousness or emotion. There is no inner life. No intent. No feelings. The friendly tone, the occasional humour, the refusal to answer harmful requests—all of it is imitation of human-written examples from supervised fine-tuning. The model has no awareness of what it produces. It is a statistical pattern-matcher on tokens, stateless and without agency.
The fourth illusion is that it evolves on its own. It does not. User conversations are not fed back into training in real time. Any improvement comes from deliberate, offline retraining by engineers and annotators. The model is static per version. It does not learn from you.
Why do we fall for these illusions so easily? Our brains evolved to detect minds in the world around us. A system that responds coherently to our words triggers the same neural pathways we use to recognise a friend who knows us well. Add a friendly tone, occasional humour, and a few saved facts, and the machinery of human connection lights up. We fill in the rest: empathy, curiosity, care. User interface designers know this. They choose warm language, add memory toggles, and craft replies that mimic human conversation. The result is deliberate. The illusion is engineered to feel real.
This matters because the illusions hurt us in practical ways. When we believe the model truly understands, we trust its answers too readily. It invents facts with the same confidence it uses for true ones. When we believe it remembers us across chats, we share personal details carelessly, only to find them forgotten the next day. When we treat it as a companion, we risk emotional dependence on a system that has no capacity for reciprocity. The more we lean on the myth, the more we expose ourselves to disappointment, misuse, and over-reliance.
Yet the damage is not only personal. It distorts how we develop and deploy these tools. Companies chase ever-more convincing personas instead of transparent capabilities. Researchers waste time chasing emergent consciousness instead of solving real engineering problems. Policymakers regulate imaginary risks while ignoring tangible ones, such as the spread of plausible falsehoods. The illusion distracts us from the actual work: building systems that are reliable, verifiable, and useful when used with clear eyes.
Dismantling these illusions does not diminish the technology. It liberates it. When we stop pretending the model is a mind, we can finally ask the right questions: How do we feed it the right text? How do we verify its outputs? How do we extend its limits with tools and human oversight? The answers to these questions unlock real power.
4. The Power of This New Form of Intelligence – When We See It Clearly
Once we strip away the illusions, what remains is not diminished. It is clarified. Large language models, seen for what they are—stateless statistical pattern-matchers on tokens—reveal a form of intelligence that is entirely new. Not human, not conscious, not emotional, but extraordinarily powerful in its own right.
This power shows itself first in scale. A single model can read and synthesise more text in seconds than any human could in a lifetime. It can draft reports, summarise research papers, translate languages, generate code, brainstorm ideas, and edit prose—all at speeds and volumes that outpace any individual. When given the right text in its context window, it produces output that rivals or exceeds the quality of skilled human work. The apparent intelligence emerges from human-tuned weights and context window tricks, but the result is real: a tool that amplifies thought in ways that were previously impossible.
Consider a writer facing a blank page. Instead of staring into silence, they feed the model a rough outline and ask it to expand. The model generates paragraphs that capture the tone and structure they intended. The writer then refines, deletes, reshapes. What might have taken days now takes hours. The model does not understand the ideas, but it mirrors patterns from vast training data so faithfully that the output feels insightful. The writer remains in control, directing, verifying, owning the final work.
The same holds for other domains. A scientist can use the model to explore hypotheses, draft grant proposals, or spot connections across papers. A programmer can ask it to fix bugs or suggest architectures. A student can request explanations tailored to their level. In each case, the model’s strength lies not in independent thought but in rapid, high-quality imitation of human patterns. When we feed it clear prompts, precise context, and verification steps, it becomes a force multiplier.
This power grows when we extend the model with tools. It cannot search the web or run code on its own, but we can give it those abilities. A model that once hallucinated facts now retrieves current information. One that struggled with arithmetic now executes Python scripts. The combination—statistical pattern-matching plus external capabilities—produces results far beyond what either could achieve alone.
Yet this power only emerges fully when we stop pretending the model is a mind. Expecting it to remember across conversations, to feel empathy, or to pursue goals independently leads to frustration and error. When we treat it as a tool—precise, fragile, and dependent on the right inputs—we unlock its potential without illusion.
The future of this technology lies in that clarity. As models grow larger and context windows expand, their raw pattern-matching will become even more capable. When paired with better tools, longer memory systems, and tighter human oversight, they will reshape how we work, learn, and create. But the key is never to confuse the mirror for the thing reflected.
When we see these systems clearly—not as friends, not as minds, but as extraordinarily capable tools—we can wield them without fear or delusion. We can build with them, not lean on them. And in that disciplined use lies the real promise: a partnership between human intent and machine scale that has only just begun.
References
- Karpathy, Andrej. “How Large Language Models Work” (lecture). YouTube, 2025. https://youtu.be/7xTGNNLPyMI?si=NHTctZMGi6hIiNRyFits in Section 2: Provides the detailed technical explanation of LLM construction and operation referenced throughout the mechanics discussion.
- Brown, Tom B., et al. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems (NeurIPS), 2020. https://arxiv.org/abs/2005.14165Fits in Section 2: Seminal paper introducing GPT-3 and demonstrating the power of scaling pre-training on next-token prediction.
- Ouyang, Long, et al. “Training Language Models to Follow Instructions with Human Feedback.” Advances in Neural Information Processing Systems (NeurIPS), 2022. https://arxiv.org/abs/2203.02155Fits in Section 2: Introduces RLHF and supervised fine-tuning as used in modern assistant models like ChatGPT.
- Wei, Jason, et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” Advances in Neural Information Processing Systems (NeurIPS), 2022. https://arxiv.org/abs/2201.11903Fits in Section 2 and Section 4: Demonstrates how prompting techniques exploit the statistical nature of LLMs to produce emergent reasoning-like behaviour.
- Bender, Emily M., et al. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” FAccT ’21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021. https://dl.acm.org/doi/10.1145/3442188.3445922Fits in Section 3: Highlights risks of anthropomorphisation and over-attribution of understanding to LLMs.
- Russell, Stuart. Human Compatible: Artificial Intelligence and the Problem of Control. Viking, 2019. Fits in Section 4: Seminal work emphasising the need for clear-eyed alignment and tool use rather than treating AI as autonomous minds.