What is a large language model?
A large language model is a type of artificial intelligence built to read and write text. ChatGPT, Claude, Gemini and Llama are all built on LLMs. “Large” refers to two things: the amount of text the model learned from (a significant slice of the public internet, books and code) and the number of adjustable settings inside it, called parameters. Modern models have billions of them — GPT-3, released in 2020, had 175 billion.
Despite how conversational they feel, LLMs do not have a database of facts they consult. Everything they “know” is stored indirectly, as patterns in those parameters. Understanding that one point explains most of their strengths and weaknesses.
Step 1: Text is split into tokens
Before a model can process your question, the text is broken into tokens — common words, pieces of words, punctuation and spaces. “Unbelievable” might become “un”, “believ” and “able”. In English, one token averages roughly four characters, or about three-quarters of a word. We cover this in detail in What Is a Token in AI?.
Step 2: Tokens become numbers
Computers only do maths, so each token is converted into a long list of numbers called an embedding. During training, the model learns embeddings that place related meanings close together: “doctor” ends up near “nurse” and “hospital”, far from “volcano”. This is how a model can recognise that two differently worded questions are asking the same thing.
Step 3: The transformer pays “attention”
Almost every modern LLM uses an architecture called the transformer, introduced by Google researchers in the 2017 paper “Attention Is All You Need”. Its key idea is attention: for every token, the model works out how much each other token in the passage should influence it.
Take the sentence “The trophy didn’t fit in the suitcase because it was too big.” Attention is what lets the model link “it” to “trophy” rather than “suitcase”. A transformer stacks dozens of these attention layers, each one building a richer picture of what the text means.
Step 4: Pretraining — learning by guessing the next word
Training starts with a model whose parameters are random, so its guesses are nonsense. It is then shown huge amounts of text, one passage at a time, with the next token hidden. It guesses, compares its guess with the real token, and nudges its parameters slightly so the right answer becomes more likely next time.
Repeat that trillions of times and something remarkable happens. To predict the next word well, the model has to pick up grammar, facts, writing styles, reasoning patterns and even how code works. Nobody programs these skills in directly — they emerge from the prediction task. This stage, called pretraining, is by far the most expensive part, requiring thousands of specialised chips running for weeks or months.
Step 5: Fine-tuning turns a predictor into an assistant
A freshly pretrained model is good at continuing text, not at being helpful. Ask it a question and it might reply with more questions, because that is what it saw on web forums. Two further stages fix this:
- Instruction tuning: the model is trained on thousands of example conversations showing a request and a good response.
- Reinforcement learning from human feedback (RLHF): people rank several model answers from best to worst, and the model is adjusted to produce answers people prefer — clearer, safer and more on-topic.
These stages also teach the model to decline harmful requests and to admit uncertainty, although neither is perfect.
How an LLM writes its answer
When you send a message, the model reads the whole conversation and produces a probability for every possible next token. One token is picked, added to the text, and the process repeats — token by token — until the answer is complete. That is why answers appear word by word on screen.
The model doesn’t always pick the single most likely token. A setting called temperature controls how adventurous it is: low temperature gives predictable, focused answers; higher temperature gives more varied, creative ones. This is also why asking the same question twice can give two different replies.
Why do LLMs make things up?
Because the model is always generating the most plausible continuation, it can produce a confident, fluent answer even when it has no reliable pattern to draw on. The result is a hallucination: an invented statistic, a fake quote or a book that doesn’t exist. Hallucinations are more likely for niche topics, very recent events, precise numbers and citations.
What LLMs are good — and bad — at
| Usually strong | Usually weak |
|---|---|
| Summarising and rewriting text | Exact arithmetic on large numbers |
| Explaining concepts at different levels | Very recent news (after training ends) |
| Drafting emails, posts and outlines | Precise citations and sources |
| Translating between common languages | Counting letters or words reliably |
| Writing and explaining code | Knowing when it is wrong |
How companies make LLMs more reliable
Because a model’s built-in knowledge is frozen at training time, many products connect it to outside information. The most common approach, retrieval-augmented generation (RAG), searches trusted documents first and hands the relevant passages to the model along with your question. Another option is fine-tuning the model on specialist examples. We compare the two in RAG vs. Fine-Tuning.
Key terms at a glance
- Token: a chunk of text — a word, part of a word or punctuation mark.
- Parameters: the internal numbers a model adjusts during training.
- Embedding: a list of numbers representing a token’s meaning.
- Transformer: the neural network design behind modern LLMs.
- Context window: the maximum amount of text the model can consider at once.
- Hallucination: a confident answer that isn’t true.
