By now it has propagated to the mainstream mind that LLMs are essentially "just" next token prediction machines. Context in, first token predicted, second token predicted, repeat until the end of white-collar work is achieved.

Most explanations of how LLMs work however, only cover the training part. 3Blue1Brown will tell you all about backpropagation with animations worthy of a Pixar movie, but the training-perspective drowns out the part where the prompt goes in and tokens come out. The forward pass. Inference.

Furthermore, the inference perspective is frankly better suited for explaining LLM architecture than the training one. And yet I have not found any good explanations of that perspective so … here we are.

In this essay/blog/whatever, I will attempt to convey an intuitive sense of what each part of the LLM does when predicting the next token. You'll understand why memory bandwidth is the actual bottleneck at inference and if things go well, you'll close this tab thinking: that wasn't so complicated after all.

The Full Picture. In Brief.

Enough preamble. Let's predict the next token.

Say, this is our example interaction:

The goal now is to predict the next token. Probably a number. Hopefully "three," because if not, the screenshot will show up on your X feed in about 20 minutes.

Now, if you have read anything about LLMs before (or if you have scrolled LinkedIn recently and still have your sanity) you might know that each token has an embedding. The short explanation is: An embedding is a vector of numbers, capturing the meaning of a token, where similar tokens live closer together in vector space.

The textbook example uses two dimensions because humans can't visualize more than three, and can barely manage three. In reality these vectors have thousands of dimensions, which is less intuitive but considerably more useful.

The embedding of " is" in GPT-3 might be (yes the space is included):

Each token in the interaction above has such an embedding of size 12,288. The goal now is to get from those floating point numbers to a probability distribution over the next token.

The way we will get there is this: take the embedding of the last token (" is") and pass it through a sequence of layers, each adding information without changing the vector's size. Some of those layers gather context from previous tokens; others process the current vector in isolation. Either way, we come out the other side with a vector still of size 12,288, which we'll project back into vocabulary space. Out comes a probability distribution. The token with the highest probability wins (usually).

I get that starting out with the last token's embedding seems weird, but LLMs house so many layers that the output vector bears about as much resemblance to its starting point as today's Elon does to the Elon of 2014.

So if we simplify that process one more time, at the risk of becoming reductionist: map the last token to embedding space, do some very expensive linear algebra which sometimes involves the previous tokens, but always keeps the vector at the same size, and map it back to tokenland to get a probability distribution over the next token. If the final resulting vector is somewhere between "three" and "3" we might predict (30% " three", 25% " 3", 10% " two" ...).

Two Steps. Repeat Until Sentient.

Here is the dirty secret of the transformer architecture: there are basically two things happening, and then they happen again and again and again. GPT-3 has 96 blocks. Each one does the same two things: Neural Network and Attention.

Between the embedding and unembedding steps, GPT-3 runs the same four operations over and over again. Two of those steps are normalization: for every number in your 12,288-dimensional vector, we pretty much just subtract the mean and divide by the standard deviation of all the numbers. The "Feed forward" step is just a regular two layer neural network, mapping a 12,288 vector to a 49,152 (=4*12,288) vector and back to 12,288 dimensions. That leaves the attention step, the only place in the architecture where tokens actually talk to each other, and the part we shall focus on here.

Our 12,288-dimensional embedding of " is" (we'll call "input vector") enters Multi-Head Attention, which runs 96 attention heads in parallel. All of these heads receive our input vector. 96 attention heads in each multi-head attention layer, of which we have 96, makes for 96 × 96 = 9,216 heads in total.

These heads are a large part of the reason why your Claude Code subscription now makes up a substantial part of your monthly expenses. But before we get lost in a debate about whether AI is net-cheaper than developers and how badly we need to get off the AI wagon hurtling toward doom, let's rather skip the doomerism and zoom in on what such an attention head actually does. The process looks like this:

Query

Which nouns are in the previous tokens?

Key Value

<user_start> Special User text starts...

How Question Asks method...

many Quantity Quantifier...

r Letter Letter to count...

's Suffix Plural marker...

are Verb Question verb...

in Prep. Inside relation...

strawberry Noun I'm a strawberry, that fruit thingy

? Punct. Question ends...

<user_end> Special User text ends...

<model_start> Special Model text starts...

The Article Points to noun...

number Noun I'm a number, like a county thing

of Prep. Links count target...

r Letter Letter being counted...

's Suffix Plural marker...

is Verb Answer setup...

We derive a query from our input vector and all tokens can "attend" to that query with their keys.

In the example above "Subject" attends to the query "Are any of you guys subjects?". If a key of a given previous token matches our query, we add its value information to our input vector. For example, the information that a "number"(meaning a specific quantity of something) was just mentioned is added.

Queries, keys, and values are all 128-dimensional vectors, entirely determined by the previous tokens themselves and the model parameters living in our current attention head. If a query and key point in a similar direction, we do some computation on the value vector (128 doesn't go into 12,288) and then add it to the input vector. The more similar the direction, the more is added.

(Saying "key of a previous token" is a bit of a simplification as the deeper the layer, the more each key and value reflects the entire context rather than any single token. I'll call it "key of a previous token" because "a key vector that originated from token X but has since accumulated further information from every token that came before it" would undermine the very understanding it's trying to convey, so I'm keeping the lie)

Attention is simple

Now, the full attention mechanism works as follows:

1. Calculate the query, key and value vector from our input vector
2. Fetch the keys and values from all previous tokens
3. Compare each key against our query
4. The more a key matches the query, the more of its value information is added to our input vector

All of this collapses into one formula, from Vaswani et al. 2017, the paper that introduced this whole kind of architecture. You've seen this equation. It was on that one slide your professor marked "non-examinable".

is our query vector, are the key vectors, is 128 (the head dimension), and are our value vectors.

The softmax part returns a vector, which sums up to 1 and encapsulates how much each of the keys correspond to the query. Concretely:

Query

Are any of you guys part of the prompt?

Key Softmax Value

<user_start> Prompt 0.1 The user prompt begins here.

How Prompt 0.1 The user is asking for a method or count.

many Prompt 0.1 The user wants a numeric answer.

r Prompt 0.1 The letter to count is r.

's Prompt 0.1 This marks the counted letter as plural.

are Prompt 0.1 This turns the prompt into a question.

in Prompt 0.1 The count should happen inside the next word.

strawberry Prompt 0.1 The word to inspect is strawberry.

? Prompt 0.1 The user question ends here.

<user_end> Prompt 0.1 The user prompt ends here.

<model_start> Model output 0.0 Model text starts...

The Model output 0.0 Answer begins...

number Model output 0.0 Quantity being stated...

of Model output 0.0 Links count target...

r Model output 0.0 Restates counted letter...

's Model output 0.0 Plural marker...

is Model output 0.0 Answer setup...

In this case here, the value vectors of the prompt are weighed highly and, after mapping the output of the formula above to 12,288 dimensions, the aggregated information of the value vectors of this head will be added to the input vector.

All 96 heads result in their own query and contribute their tweak to the input vector. Out the other side comes a vector still of size 12,288, enriched with context from the previous tokens. It then passes through two normalizations and one neural network step, before entering the attention stage again in the next block with different queries, different keys, and different values.

All of this requires those key and value vectors to be accessible. The process of storing and fetching key and value vectors is called KV-caching. Fetching from memory is much more efficient than recalculating them every time. The problem is how many numbers there are to fetch. What I described above was one head in one multi-head attention layer. GPT-3 has 96 blocks, each with 96 heads running in parallel, each demanding the key and value vectors of every previous token in the context. Multiply that out if you like. Or skip to the next section, which has already done it for you.

So why is memory bandwidth the bottleneck?

A single H100 (the GPU behind most LLM deployments) stores 80GB of data in High-Bandwidth Memory (HBM). The actual compute units, however, can only do math on data loaded into SRAM, which holds a maximum of 117MB. At some point, the key and value vectors must be loaded from HBM to SRAM. The speed of that transition (called "memory bandwidth") in an H100 is about 3.35 terabytes per second.

This might sound like a lot, but let's do the numbers for GPT-3 and then extrapolate to larger models.

GPT-3 has 96 transformer blocks, each with 96 attention heads. The maximum number of previous tokens on which an LLM can base its next-token prediction is called the context window. To predict the next token at full context, we need to fetch 2 × 96 × 96 × context_size key and value vectors (key and value vectors for all previous tokens for all heads).

With GPT-3's context window of 2,048 tokens and a head dimension of 128, that's 2 × 96 × 96 × 2,048 × 128 ≈ 4.8 billion numbers. In FP16 that's 9.6 gigabytes of data fetched to predict a single token! Considering that context windows are now 100× larger and models have additionally grown in size, you can see that memory bandwidth becomes the bottleneck quicker than you can say "Spare me AI overlords".

As you can see, the KV-cache scales linearly with context length. Compute grows too, but memory bandwidth usually hurts first: more context means more KV-cache data to fetch.

We could compare the H100 to a Ferrari on the autobahn and the KV cache to a traffic jam. However, I'm just as sick of bad AI metaphors as you are, and most of them are misleading anyway. So I'd rather just save your time and end this essay right here.

One more thing. Remember to clear your context windows.

For any open questions, send me a mail or DM :)

Context

Me

Viel Spaß heute :)

Them

Dankee
Sind jetzt im größten Club der City ☺️

Responses

Actual Julius Reference

Oha wild, wie isses?

Qwen3 32B Finetuned

Aww nicee
Bin schon gespannt auf deinen Report

Qwen3 14B Finetuned

Jaa nicee

Qwen3 4B Finetuned

Jaa
Und?

Qwen3 32B Base Prompted

Juhu! Hoffe, es läuft gut!

Qwen3 32B Base

Das klingt ja mega! ✨
Im größten Club der Insel – ab ins Rampenlicht, ihr seid die Stars des Abends!

Looking at the Qwen3 32B reply ("Aww nicee / Bin schon gespannt auf deinen Report"), it already captures multiple of my specific writing patterns, such as:

• Repeating letters at the end of words
• Mixing German and English
• Dropping pronouns (using "Bin schon" instead of "Ich bin schon")

Overview

There are two main ways to get a model to adapt to a specific writing style:

1. System Instructions
2. Finetuning

System instructions are great because they are model-independent, cheap, and easy to implement.

They do, however, limit the style specifications to general guidelines and lose power as the desired style becomes more fine-grained and situation-dependent. This detailed level of style modification is where we get to use finetuning instead.

Finetuning's main advantage is that it allows small models, which have a harder time generalizing from system instructions anyways, to adopt a highly specific style.

I finetuned Qwen3-32B/14B/4B and compared these replies against the 32B base model, the base model with a system prompt about my specific style, and my actual responses to get a feel for how these approaches compare.

Data

180k tokens worth of my replies + context were extracted, parsed, redacted, and cleaned from WhatsApp messages sent after 2023, with the consent of the chat partner.

Training

I rented an Nvidia L4 and A100 80GB and used QLoRA with 4-bit quantization.

Basically, the original model remains frozen in 4-bit precision while only smaller adapter weight matrices are trained and added to the actual model matrices. This enables finetuning on less powerful GPUs with less memory.

Evaluation

For evaluation, I held out 103 items, each consisting of a few back-and-forth messages and the actual message I sent next. The model's task was to predict this message.

To evaluate this, I ranked all the responses (including my actual message) by how much they sounded like me. First place means it sounded a lot like me (or was my actual message), while sixth place means it sounded the least like me.

Here are the results:

I only finetuned the unprompted model. Combining prompting and finetuning could improve performance further.

Examples

Context

Them

There you go

Me

Ahh vielen Dank. Sehr nice 👍

Them

Dann siehts eher schlecht aus haha. Muss es wohl bezahlt werden
😣

Ranked responses

1.

Actual Julius Reference

😂 😂
Glaube ist auch ohne extra cash erschwingbar

2.

Qwen3 32B Finetuned

Jaa, hab nochmal nachgeschaut
Da gibt es aber ne kleine Chance dass ich das bis Montag schaffe
Aber wenn nicht dann müssen wirs bezahlen

3.

Qwen3 14B Finetuned

🤔

4.

Qwen3 4B Finetuned

🤔

5.

Prompted Baseline

Jap, scheint so. Naja, zahlen halt. Irgendwie. 🤔

6.

Qwen3 32B Base

Kein Problem, gerne gesehen! 😄
Wenn's um so viel Geld geht – hoffentlich ist's was, wofür's sich lohnt! Oder zumindest was man irgendwann wieder zurückbekommt...
✌️ 😄

The base model notably gives very generic assistant-like answers

Context

Me

Du bist aber auch ne Nachteule, oder?

Them

Nein hahaha eigentlich nicht
Ich passe mich nur den Begebenheiten an

Ranked responses

1.

Actual Julius Reference

Jaja Lauras schuld

2.

Qwen3 32B Finetuned

Was machst du denn so nachts?

3.

Qwen3 4B Finetuned

Jaa
Aber du bist ja so nachttypisch
Und die Party war krass
Was machst du jetzt

4.

Qwen3 14B Finetuned

Aber wie?
Sind ja keine so große Partyleute

5.

Prompted Baseline

Haha, klar, die berühmte Ausrede
Schlaf gut!

6.

Qwen3 32B Base

Ach, verstehe! Also eine „Gelegenheitsnachteule“
Dann viel Erfolg beim Anpassen – und vielleicht wird’s irgendwann zur Gewohnheit
Gute Nacht nochmal, bis morgen! ✨

Messages that mention information not present in the context often give away the actual Julius message.

Context

Them

4:00 brooooo 😭 😭 😭

Ranked responses

1.

Actual Julius Reference

💀

2.

Qwen3 4B Finetuned

Haha, dass is ja bisschen wild

3.

Qwen3 14B Finetuned

Haha ja

4.

Qwen3 32B v2 Finetuned

Haha was is los?

5.

Prompted Baseline

was ist los? 🥺

6.

Qwen3 32B Base

Hey, 4 AM is a rough time to be feeling down. What’s going on? If you want to talk about it, I’m here. Sometimes just getting things out can help lighten the load. If not, know that it’s okay to feel this way—everyone has moments like this. You’re not alone.