From Autocomplete to Intelligence: How Language Became the Operating System of AI
Introduction
Every week brings a new AI model, a new coding assistant, a new agent framework. The pace can feel overwhelming. Yet beneath the churn lies a single, surprisingly simple fact - the fact this whole revolution turns on:
AI became so powerful because it learned language exceptionally well.
In this article I want to step away from the weekly headlines and ask a more basic question: why is language so tightly bound to intelligence - human intelligence in general, and artificial intelligence in particular?
I have spent the last 27 years building machine-learning systems, and I have come to believe that grasping the link between language and AI is the clearest way to make sense of the current moment.
This article is written for a general reader; no background is assumed. We will move mostly at altitude, dropping down for a closer look at the ideas that matter most - dispelling a few myths along the way and, I hope, reaching the core of what AI really is.
Chapter I - Human Intelligence Runs on Language
About 70% of the neurons in the human brain are devoted to vision. So why don't we think of intelligence as fundamentally visual? And why did the AI revolution erupt in November 2022 with ChatGPT, rather than two years earlier with OpenAI's remarkable DALL·E image generator, which provoked nothing like the same reaction?
The answer is communication. As Yuval Noah Harari has argued, it was the capacity for language - for gossip, for talking about abstract things that aren't physically present - that set Homo sapiens apart from every other species. Vision lets us perceive the world; language lets us share, refine, and accumulate abstract thought.
To understand why language became the operating system of AI, we first need to understand how AI became intelligent at all.
Chapter II - How Machines Acquire Intelligence
Let us begin with the hardest question of all: what is intelligence? What exactly is the human capacity we are trying to give machines? We will be able to answer this properly only after seeing how large language models (LLMs) learn - which is what this chapter is about. We’ll build the term up from its parts, in reverse.
Model
Humans build models to represent the world - in biology, physics, aerodynamics, finance - in order to reduce uncertainty. A good model of financial dynamics lets you forecast revenue or share prices; a good model of ballistics lets you build a missile-defense system like Iron Dome; a good model of traffic lets you build a navigation app like Waze.
You construct a model by observing the world, compressing those observations into a compact representation, and using that representation to predict what comes next. In short, a model is a compressed representation of reality.
Figure 1 shows a two-step process by which both children and machines learn. In Training, a child is shown many trucks and non-trucks, with a supervisor confirming each label - hence the name supervised learning.
From these labeled examples the child forms an internal model of a truck, defined by features such as size and shape. Then in Inference, the child applies that model to an entirely new vehicle and decides whether it is a truck.
Language Model
If a model in general represents the world, a language model represents language. Instead of weather or traffic, it captures the structure of human language itself - a compressed representation of how we write and speak. Given a good one, a machine can understand and generate text automatically and accurately.
Large Language Model
A large language model is, quite literally, a large one - and, as we will see in Chapter III, that size is central to the whole story. The mechanism it relies on is the most surprising fact about LLMs, one we all know from our phones: autocomplete - predicting the next word.
Next-Word Prediction: The Heart of LLMs
Myth #1: LLMs are trained to answer questions.
They are not. LLMs are trained to do something way simpler: predict the next word in a sequence. That is essentially the entire objective. We recognize the behavior from the autocomplete on our phones.
LLMs are later fine-tuned to steer those predictions toward useful answers, but the core objective - and the overwhelming majority of their energy throughout learning - is simply choosing the best next word. That’s all. That’s how LLMs acquire intelligence.
This learning process is outlined in Figure 2. The learning loop, in one sentence: take a sentence, hide a word, ask the model to guess it from the preceding context; measure the loss (the gap between the guess and the true word, the ground truth); and push that error backward to nudge billions of parameters, using backpropagation - a method that dates all the way back to 1986. Repeat across billions of documents and trillions of words. That, in essence, is how LLMs learn.
This approach is called self-supervised learning, because every sentence supplies its own label - the true next word is already there in the text, so no human annotation is needed. It proved so effective that it is sometimes called the miracle of AI: it dissolved machine learning's oldest bottleneck, the scarcity of large-scale labeled data. Feed the machines enough raw text, and they label it for themselves.
LLMs Speak Auto-Regressively
When an LLM writes, it works much as it was trained: it generates one word at a time. Each new generated word is appended to its own growing context, which then informs the next prediction, and so on. This step-by-step, feed-it-back-to-itself process is called auto-regression.
Neural Networks: The Engine
To build an intelligent machine, researchers borrowed from the most capable intelligent machine they knew: the human brain. Ours holds roughly 100 billion neurons, each performing a simple operation - summing weighted inputs from its neighbor neurons and firing once a threshold is crossed.
That is precisely the design of the first trainable artificial neuron, the perceptron, introduced by Frank Rosenblatt¹ about 70 years ago. Scaled up enormously, that same unit still sits inside every modern LLM. The first ChatGPT already had 175 billion of them - nearly twice the neuron count of a human brain. Today's largest models are roughly ten times bigger still.
Deep Learning
Myth #2: LLMs are just statistical machines
A major milestone in the AI revolution already occurred more than a decade before ChatGPT: the rise of deep learning in 2012, marked by the AlexNet results.
Deep learning uses neural networks to learn directly from raw data, and it brought a profound conceptual shift - from statistical models to optimization models. Older machine learning methods estimated explicit statistics; a traditional language model, for instance, stored the observed frequencies of word sequences (n-grams), noting that "new" and "york" tend to co-occur.
Deep learning takes a different route: rather than counting, it learns a rich representation of language by tuning billions of parameters to minimize the loss (Figure 2), over and over, trillions of times. The payoff is that the machine no longer memorizes language - it genuinely understands it.
So, What Is Intelligence?
We can now answer the question we opened with. Having seen how an LLM learns a representation of language rather than storing its statistics, the definition falls out naturally:
Intelligence = Generalization ≠ Memorization
Intelligence is the ability to perform well on data you have never seen before. Not by cataloguing every example, however vast the catalogue, but by building an internal representation of the underlying structure - the semantic relationships among things, whether those things are words, pixels, or a robot's actions - and using it to extrapolate. Intelligence, human or artificial, is not the memorization of all human knowledge; it is a grasp of the rules that generate that knowledge.
Handwritten-digit recognition makes this concrete. The famous MNIST dataset - 70,000 grayscale 128×128 images of handwritten digits, a landmark in deep-learning history - cannot be solved by memorization.
Take the simplest possible case: a tiny image of 128×128 black-or-white pixels, no grays, no color. The number of possible images is 2 raised to the power 128×128 (2128x128) - more than the number of atoms in the universe. Storing them all is hopeless. Instead, a model learns the structure of a digit - how the pixels relate to each other.
And they do it differently from digit to digit. Its internal state is semantic, not statistical. Shown a new digit it has never encountered, it would recognize it.
Chapter III - Why Next-Word Prediction Produces Intelligence
Here is the real puzzle. How does an objective as plain as guessing the next word scale into behavior sophisticated enough to threaten humanity? How can something so nearly trivial give rise to intelligence at all? Answering that will occupy the rest of this article, across six distinct reasons.
LLMs Learn Language Deeper Than Any Human
Although each step predicts just one word, repeating that step trillions of times yields an astonishingly rich model of language. The system acquires grammar and syntax with near-perfect fluency, grasps long-range dependencies and meaning, and absorbs a vast share of human knowledge - because that knowledge lives mostly in text, from Wikipedia outward. It is worth seeing how it represents language internally.
AI Doesn't See Words
We see words; the machine does not. For efficiency, LLMs work in tokens² - language units that may be whole words or fragments of them.
And because it is ultimately a machine, an LLM does not really see even tokens - it sees numbers. Each word is stored as an array of numbers, an embedding, that encodes the word's meaning in context. The idea that words appearing in similar contexts tend to share meaning goes back to Harris's seminal 1954 paper.
Consider "bank": in "I spent hours at the bank" it could be a riverbank or a financial one; in "It was a sunny day, and I spent hours at the bank," the riverbank reading dominates. That shift shows up directly in the embedding - the word's numerical fingerprint.
In a simplified version of embeddings, as can be seen in Figure 7, each word is represented as seven numbers - a vector in seven-dimensional space - with each axis standing for a property such as living being, human, or royalty.
In practice these axes are not neatly labeled; the model discovers them on its own, so they are called latent dimensions. Real models use not seven numbers per word but roughly three thousand. The higher the dimensionality, the more expressive - the more intelligent - the representation.
Figure 8 sketches this latent space in three dimensions, the most we, humans, can picture. Words of similar meaning sit close together; unrelated words lie far apart. This hints at the quiet power of these machines: they reason in spaces of thousands of dimensions, where we are confined to three.
Language Is a Compressed Model of the World
Language is not random text. It is a distillation of thought, knowledge, and literature - humanity's cached thinking. In learning language deeply, an LLM ends up learning the world that produced it.
LLMs compress and encode the DNA of humanity
Recall that models represent the world. Modeling language well turned out to do far more than expected: it modeled not only language but the human world itself that generates this language, deeply and usefully.
The Transformer
If next-word prediction is the objective, the Transformer is the engine that delivers it. Introduced in Google's landmark 2017 paper Attention Is All You Need, it began in natural language processing (NLP) and soon spread to vision.
This GPU-friendly architecture proved so effective that it remains the backbone of every modern LLM - and nine years, on AI's clock, is an age. In high-level³, the Transformer stacks N identical blocks, each combining two sub-blocks: an MLP (Multi Layer Perceptron: a multi-layer stack of those 70-year-old perceptrons) and an attention block.
Attention
Attention solved a problem intrinsic to language: long-range dependency. Any word can reshape the meaning of another far away - a sentence early in a paragraph can flip the sense of one hundreds of words later.
Take: "The chef served the soup after it had cooled down." "It" refers to the soup, not the chef, because "cooled down" tips the balance. Attention lets the model learn which words bear on which others, and how, from the trillions of such relationships it encountered in its training data.
This is exactly what makes text harder than images. An image is static - a patch does not change because of a distant patch. Text is dynamic: a word's meaning bends to its context, however far away. Hence the name Transformer - representations are transformed by the words around them.
Myth #3: Attention was invented as part of the Transformer
Not quite. Attention was introduced earlier, in a 2014 paper, and explored in the work that followed; the 2017 Transformer made it the centerpiece and popularized it. The real breakthrough of that paper was doing away with recurrence (RNNs) and convolutions (CNNs) altogether and replacing them with attention only - hence the title, "Attention Is All You Need."
Scale
The Transformer lets models scale as never before. One sentence yields several training examples; one document, thousands; the internet, trillions. A person might read a few hundred million words in a lifetime; a modern LLM trains on roughly 10 trillion tokens - about ten thousand times more. Reading that much would take a human 50 million years. An LLM does it in a week.
That gap is not a footnote. The ability to scale a trivial objective - predict the next word - together with scaling those ancient perceptrons into the trillions, is the main reason such complex behavior emerges from an unconscious machine. And exploiting data at that scale demands a correspondingly large model.
ChatGPT's 175 billion parameters in 2022 already rivaled the brain; that scale, not the algorithm alone, is why the boom came in 2022 and not in 2017, when Transformers first appeared and models were ten times smaller. Today they are ten times larger (Figure 11).
Intelligence Emerges
Strikingly, many of today's capabilities were neither designed nor anticipated - they simply appeared. The clearest example is reasoning. It began with chain-of-thought prompting: letting a model "think" before it answers. Working through intermediate steps gives it better context, and better, more trustworthy answers. That simple idea of generating “thinking words” before answering grew into the whole family of reasoning models we see today.
AI Meets Software
The next leap came when AI met software - the operating system of modern life - and, in doing so, began to become one. Imagine fitting a brilliant Swiss-army-knife of a mind with an endless supply of arms.
Two threads are unfolding in parallel: AI's skill at writing code, and its smooth, natural ability to plug into software. The first is essentially settled. AI now writes code better and more reliably than ever, and at many companies - the largest among them - engineers increasingly orchestrate coding agents rather than write code by hand. Why did this happen so fast? Because code is a language too - and a simpler one: structured, deterministic, executable, with far fewer long-range dependencies than natural language.
Once LLMs managed to code, they immediately started calling APIs, using tools, searching the web, and chaining together ever more capable workflows. The moment that coupling clicked into place, it set off the fastest burst of innovation we have ever witnessed - fast enough that no one can confidently forecast even the near future.
Chapter IV - Are LLMs the Path to AGI?
For all their power, LLMs have real limits. Three are worth naming.
LLMs Don't Truly Reason
Human reasoning happens in two stages: we form a thought, then put it into words - language is a downstream of thought. For an LLM, language is the thought itself; there is no separate prior step where reasoning occurs and is only then translated to words. LLMs allocate thousands of “thinking tokens” before returning an answer. This has a price, both in cost and latency.
In addition LLM doesn’t possess logic in the human sense. Its "reasoning" is probabilistic - one token after another, drawn from statistical patterns, with no guarantee of consistency. That is why it can, on occasion, produce fluent nonsense.
The Hallucination Problem
Picture an LLM as a tree (Figure 12). Speaking auto-regressively, word by word, each prediction opens a new branch and commits to it - and there is no going back. Every later word depends on that choice. Andrew Ng captured it well: "typing straight through with no backspacing allowed." A hallucination is simply a wrong branch taken.
And the errors compound. A wrong turn can occur at any step, and mistakes accumulate down the tree. Suppose the model is right 99% of the time at each word - wrong just 1%. Over a 100-word passage, the chance of getting the whole thing right is only about 37% ((99%)¹⁰⁰ = 37%)
The Physical World Is Richer Than Language
Language describes reality; it is not the reality itself. We do not merely talk - we live in a physical, three-dimensional, multimodal world. That gap has fueled interest in alternatives to LLMs: world models and other forms of physical AI, championed notably by Yann LeCun, which represent the world as states and predict how actions change those states.
Where LLMs - even multimodal ones - are built for generation and reasoning, these physical AI models are built for planning and control. Some argue that no single model will ever both fold proteins and fold the laundry; instead we may see smaller models, each expert in its own domain.
So, Are LLMs AGI?
It may be the most-asked question in AI. People object that "LLMs don't think like humans" - largely true - or that "they're just parroting their training data." My reply: yes, and so what?
The goal is useful intelligence, not a copy of humans. When humans wanted to move like a cheetah, they built wheels, not legs; when we wanted to fly like birds, we built airplanes, not wings. Machines don't do arithmetic the way we do - they do it faster and more accurately than any human alive. Insisting they imitate us would be a step backward.
So - are LLMs true AGI? In at least one domain, language, they are already masters, probably beyond any human: they summarize, they translate, they hold much of humanity's knowledge, and staying current is a solved problem. They are being wired to external tools, and soon to each other. Most of the world runs on software, and AI seems like taking over it.
Whether we end up calling it AGI may matter less than recognizing how transformative it already is.
Final Thoughts
Human civilization was built on language. For millennia, words made us the rulers of the world. Now, everything made of words is gradually passing into the hands of AI - even, in time, the inner voice in our own heads. It is said that people are the sum of all the words they have absorbed throughout their life. So who are we, if our words come to belong to something else?
This does not diminish human intelligence, but it does force us to rethink it. Rather than fight the tide, or comfort ourselves that machines cannot really think, we would do better to accept that language is no longer ours alone - and to learn to live and work alongside the new kind of species emerging beside us.
References
Papers
Harris, Z. S. (1954). Distributional Structure. Word, 10(2–3), 146–162.
Rosenblatt, F. (1958). The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review, 65(6), 386–408.
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning Representations by Back-Propagating Errors. Nature, 323, 533–536.
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks (AlexNet). NeurIPS.
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473.
Vaswani, A., et al. (2017). Attention Is All You Need. arXiv:1706.03762.
Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS.
Online Materials
Gilad Barkan (2025) | The Emerging Economy of LLMs | Wix Engineering | Link
Richard Feynman (1985) | Can Machines Think? | Link
Geoffrey Hinton (2025) | What Is Understanding? | IASEAI 2025 | Link
Yann LeCun (2025) | Self-Supervised Learning, JEPA, World Models, and the future of AI | Harvard CMSA | Link
Transformer Explainer | Link
Comments
Rosenblat wasn’t actually the first one to introduce the idea of Artificial Neuron (McCulloch & Pitts did already in 1943), but the perceptron was the first practical trainable one, meaning you could learn the weights, rather they are fixed, as was introduced in McCulloch & Pitts theoretical mathematical model.
You can read more here about tokens and how they became the currency of the emerging economy of LLMs.
For detailed x-ray of Transformer go here
This post was written by Gilad Barkan
More of Wix Engineering's updates and insights:
Join our Telegram channel
Visit us on GitHub
Subscribe to our YouTube channel
Comments