By now it has propagated to the mainstream mind that LLMs are essentially "just" next token prediction machines. Context in, first token predicted, second token predicted, repeat until the end of white-collar work is achieved.

Most explanations of how LLMs work however, only cover the training part. 3Blue1Brown will tell you all about backpropagation with animations worthy of a Pixar movie, but the training-perspective drowns out the part where the prompt goes in and tokens come out. The forward pass. Inference.

Furthermore, the inference perspective is frankly better suited for explaining LLM architecture than the training one. And yet I have not found any good explanations of that perspective so … here we are.

In this essay/blog/whatever, I will attempt to convey an intuitive sense of what each part of the LLM does when predicting the next token. You'll understand why memory bandwidth is the actual bottleneck at inference and if things go well, you'll close this tab thinking: that wasn't so complicated after all.

The Full Picture. In Brief.

Enough preamble. Let's predict the next token.

Say, this is our example interaction:

The goal now is to predict the next token. Probably a number. Hopefully "three," because if not, the screenshot will show up on your X feed in about 20 minutes.

Now, if you have read anything about LLMs before (or if you have scrolled LinkedIn recently and still have your sanity) you might know that each token has an embedding. The short explanation is: An embedding is a vector of numbers, capturing the meaning of a token, where similar tokens live closer together in vector space.

The textbook example uses two dimensions because humans can't visualize more than three, and can barely manage three. In reality these vectors have thousands of dimensions, which is less intuitive but considerably more useful.

The embedding of " is" in GPT-3 might be (yes the space is included):

Each token in the interaction above has such an embedding of size 12,288. The goal now is to get from those floating point numbers to a probability distribution over the next token.

The way we will get there is this: take the embedding of the last token (" is") and pass it through a sequence of layers, each adding information without changing the vector's size. Some of those layers gather context from previous tokens; others process the current vector in isolation. Either way, we come out the other side with a vector still of size 12,288, which we'll project back into vocabulary space. Out comes a probability distribution. The token with the highest probability wins (usually).

I get that starting out with the last token's embedding seems weird, but LLMs house so many layers that the output vector bears about as much resemblance to its starting point as today's Elon does to the Elon of 2014.

So if we simplify that process one more time, at the risk of becoming reductionist: map the last token to embedding space, do some very expensive linear algebra which sometimes involves the previous tokens, but always keeps the vector at the same size, and map it back to tokenland to get a probability distribution over the next token. If the final resulting vector is somewhere between "three" and "3" we might predict (30% " three", 25% " 3", 10% " two" ...).

Two Steps. Repeat Until Sentient.

Here is the dirty secret of the transformer architecture: there are basically two things happening, and then they happen again and again and again. GPT-3 has 96 blocks. Each one does the same two things: Neural Network and Attention.

Between the embedding and unembedding steps, GPT-3 runs the same four operations over and over again. Two of those steps are normalization: for every number in your 12,288-dimensional vector, we pretty much just subtract the mean and divide by the standard deviation of all the numbers. The "Feed forward" step is just a regular two layer neural network, mapping a 12,288 vector to a 49,152 (=4*12,288) vector and back to 12,288 dimensions. That leaves the attention step, the only place in the architecture where tokens actually talk to each other, and the part we shall focus on here.

Our 12,288-dimensional embedding of " is" (we'll call "input vector") enters Multi-Head Attention, which runs 96 attention heads in parallel. All of these heads receive our input vector. 96 attention heads in each multi-head attention layer, of which we have 96, makes for 96 × 96 = 9,216 heads in total.

These heads are a large part of the reason why your Claude Code subscription now makes up a substantial part of your monthly expenses. But before we get lost in a debate about whether AI is net-cheaper than developers and how badly we need to get off the AI wagon hurtling toward doom, let's rather skip the doomerism and zoom in on what such an attention head actually does. The process looks like this:

We derive a query from our input vector and all tokens can "attend" to that query with their keys.

In the example above "Subject" attends to the query "Are any of you guys subjects?". If a key of a given previous token matches our query, we add its value information to our input vector. For example, the information that a "number"(meaning a specific quantity of something) was just mentioned is added.

Queries, keys, and values are all 128-dimensional vectors, entirely determined by the previous tokens themselves and the model parameters living in our current attention head. If a query and key point in a similar direction, we do some computation on the value vector (128 doesn't go into 12,288) and then add it to the input vector. The more similar the direction, the more is added.

(Saying "key of a previous token" is a bit of a simplification as the deeper the layer, the more each key and value reflects the entire context rather than any single token. I'll call it "key of a previous token" because "a key vector that originated from token X but has since accumulated further information from every token that came before it" would undermine the very understanding it's trying to convey, so I'm keeping the lie)

Attention is simple

Now, the full attention mechanism works as follows:

1. Calculate the query, key and value vector from our input vector
2. Fetch the keys and values from all previous tokens
3. Compare each key against our query
4. The more a key matches the query, the more of its value information is added to our input vector

All of this collapses into one formula, from Vaswani et al. 2017, the paper that introduced this whole kind of architecture. You've seen this equation. It was on that one slide your professor marked "non-examinable".

is our query vector, are the key vectors, is 128 (the head dimension), and are our value vectors.

The softmax part returns a vector, which sums up to 1 and encapsulates how much each of the keys correspond to the query. Concretely:

In this case here, the value vectors of the prompt are weighed highly and, after mapping the output of the formula above to 12,288 dimensions, the aggregated information of the value vectors of this head will be added to the input vector.

All 96 heads result in their own query and contribute their tweak to the input vector. Out the other side comes a vector still of size 12,288, enriched with context from the previous tokens. It then passes through two normalizations and one neural network step, before entering the attention stage again in the next block with different queries, different keys, and different values.

All of this requires those key and value vectors to be accessible. The process of storing and fetching key and value vectors is called KV-caching. Fetching from memory is much more efficient than recalculating them every time. The problem is how many numbers there are to fetch. What I described above was one head in one multi-head attention layer. GPT-3 has 96 blocks, each with 96 heads running in parallel, each demanding the key and value vectors of every previous token in the context. Multiply that out if you like. Or skip to the next section, which has already done it for you.

So why is memory bandwidth the bottleneck?

A single H100 (the GPU behind most LLM deployments) stores 80GB of data in High-Bandwidth Memory (HBM). The actual compute units, however, can only do math on data loaded into SRAM, which holds a maximum of 117MB. At some point, the key and value vectors must be loaded from HBM to SRAM. The speed of that transition (called "memory bandwidth") in an H100 is about 3.35 terabytes per second.

This might sound like a lot, but let's do the numbers for GPT-3 and then extrapolate to larger models.

GPT-3 has 96 transformer blocks, each with 96 attention heads. The maximum number of previous tokens on which an LLM can base its next-token prediction is called the context window. To predict the next token at full context, we need to fetch 2 × 96 × 96 × context_size key and value vectors (key and value vectors for all previous tokens for all heads).

With GPT-3's context window of 2,048 tokens and a head dimension of 128, that's 2 × 96 × 96 × 2,048 × 128 ≈ 4.8 billion numbers. In FP16 that's 9.6 gigabytes of data fetched to predict a single token! Considering that context windows are now 100× larger and models have additionally grown in size, you can see that memory bandwidth becomes the bottleneck quicker than you can say "Spare me AI overlords".

As you can see, the KV-cache scales linearly with context length. Compute grows too, but memory bandwidth usually hurts first: more context means more KV-cache data to fetch.

We could compare the H100 to a Ferrari on the autobahn and the KV cache to a traffic jam. However, I'm just as sick of bad AI metaphors as you are, and most of them are misleading anyway. So I'd rather just save your time and end this essay right here.

One more thing. Remember to clear your context windows.

For any open questions, send me a mail or DM :)

Looking at the Qwen3 32B reply ("Aww nicee / Bin schon gespannt auf deinen Report"), it already captures multiple of my specific writing patterns, such as:

• Repeating letters at the end of words
• Mixing German and English
• Dropping pronouns (using "Bin schon" instead of "Ich bin schon")

Overview

There are two main ways to get a model to adapt to a specific writing style:

1. System Instructions
2. Finetuning

System instructions are great because they are model-independent, cheap, and easy to implement.

They do, however, limit the style specifications to general guidelines and lose power as the desired style becomes more fine-grained and situation-dependent. This detailed level of style modification is where we get to use finetuning instead.

Finetuning's main advantage is that it allows small models, which have a harder time generalizing from system instructions anyways, to adopt a highly specific style.

I finetuned Qwen3-32B/14B/4B and compared these replies against the 32B base model, the base model with a system prompt about my specific style, and my actual responses to get a feel for how these approaches compare.

Data

180k tokens worth of my replies + context were extracted, parsed, redacted, and cleaned from WhatsApp messages sent after 2023, with the consent of the chat partner.

Training

I rented an Nvidia L4 and A100 80GB and used QLoRA with 4-bit quantization.

Basically, the original model remains frozen in 4-bit precision while only smaller adapter weight matrices are trained and added to the actual model matrices. This enables finetuning on less powerful GPUs with less memory.

Evaluation

For evaluation, I held out 103 items, each consisting of a few back-and-forth messages and the actual message I sent next. The model's task was to predict this message.

To evaluate this, I ranked all the responses (including my actual message) by how much they sounded like me. First place means it sounded a lot like me (or was my actual message), while sixth place means it sounded the least like me.

Here are the results:

I only finetuned the unprompted model. Combining prompting and finetuning could improve performance further.

Examples

The base model notably gives very generic assistant-like answers

Messages that mention information not present in the context often give away the actual Julius message.

If there is one thing I consistently find myself drawn to, it is finding metrics and analyzing how they might be optimized. So naturally, on my current trip to Thailand, while I could be diving among beautiful reefs or getting massaged by sparsely clothed ladyboys, I find myself going down another rabbit hole.

So, let’s skip the intro (as always) and let me make an argument.

We start by asking the following question:

How can we actually minimize animal suffering through our dietary choices?

The obvious answer is that the spectrum of animal suffering probably runs from a carnivore diet to veganism, and that we should tend towards veganism. In our moral analysis, we usually rate an omnivorous diet as not so moral, vegetarianism as pretty good, and veganism as fully moral.

This picture, however, is only a rule of thumb that simplifies our moral analysis. If we truly want to minimize animal suffering rather than merely clustering people into different moral categories, we have to ask: How much suffering do different dietary choices actually inflict on animals?

This does not just depend on which of the aforementioned labels one assigns to oneself, but rather on which specific animal products one consumes. And this is where the animal suffering actually inflicted might diverge from our colloquially defined moral categories.

For example, consider someone who eats 300 g of beef a week and decides to become vegetarian to reduce animal suffering. To retain some lean protein in their diet, they switch to eating 10 eggs a week. Now, keeping the rest of their diet equal, let’s compare the animal suffering associated with these two products.

A normal laying hen usually lays about 506 eggs in a lifetime. About 94% of the hen's economic burden goes to eggs.^1,2,3

So, 10 eggs account for 10 / 506 × 0.94 = 0.0186 hen-equivalents under economic allocation. Assuming a lifespan of about 600 days, this amounts to 0.0186 × 600 = 11.1 hen-days. In other words, producing 10 eggs represents roughly 11 days of a hen’s life and the suffering experienced during that time.

Now the same for beef. Beef accounts for 92.9% of a cow's economic burden, and each animal yields about 333 kg of carcass weight.^4,3 That means 0.3 kg / 333 kg × 0.929 = 0.000837 cow-equivalents, and 0.000837 × 700 days = 0.586 cow-days = 14.1 cow-hours required for the steak.

We must also consider that cattle likely have a greater capacity for pain than chickens. However, considering the amount of each animal’s life required to produce these foods, cattle would have to carry at least 19 times the moral weight of a chicken for the eggs to be the more moral choice.

And we have not even taken the following into account:

• The calculation excludes male chicks killed as part of egg production.⁵
• Producing the eggs accounts for the slaughter of 22.2 times as many animals as producing the beef.
• Laying hens usually have a much lower quality of life than cattle raised for beef.⁵

Even though quantifying a chicken’s life through its economic value might seem cold and overly analytical, we must keep in mind that this economic value is precisely why the hen was brought into existence. Animals kept for animal products exist because there is economic demand for those products. This kind of cold quantification is our best means of estimating the actual suffering we inflict on animals and a less analytical approach would be blind to these considerations. By replacing beef with eggs, this person likely causes much more animal suffering despite going vegetarian.

This is not to criticize vegetarians or the movement as a whole. As a general rule, being vegetarian certainly causes much less animal suffering than being omnivorous, and the conversion of so many people to vegetarianism in the 21st century is perhaps one of the most morally positive trends this century has seen. It is simply to say that there is a lot of untapped potential to prevent suffering, and we must make informed decisions on that basis.

So, how can we effectively minimize animal suffering?

To estimate how much suffering different animal products cause, we have to consider the following factors:

• What is the animal’s capacity for pain?
• What quality of life does the animal experience?
• How long must the animal live to produce one unit of the product?

We must also recognize that different farming systems have different standards of animal well-being and that our intuitions about some of these factors may differ. For example, people may disagree about the relative sentience of salmon and chickens.

Even though a comprehensive science of quantifying animal suffering is still a long way off, there have already been attempts to rank animal products by the suffering they cause. In “How Much Direct Suffering Is Caused by Various Animal Foods?”, Tomasik⁵ considers many of the factors mentioned above and develops the metric “equivalent days of suffering caused per kg demanded.” This metric combines the animal’s capacity for pain, its average quality of life, and the length of time it must live to produce one kilogram of the product. Using his default assumptions, the ranking is as follows, with the equivalent days per kilogram listed next to each category (higher numbers indicate more suffering):

1. Salmon (120)
   Farmed salmon live for roughly 650 days and yield relatively little meat per animal, resulting in a high number of suffering-days per kilogram.
2. Eggs (64)
   Eggs rank highly because laying hens often experience very poor living conditions over relatively long lives.
3. Chicken (41)
   Chickens raised for meat experience many of the same welfare problems as laying hens, although they generally have shorter lifespans and somewhat better living conditions.
4. Pork (7.5)
   Pork ranks above beef because pigs likely have a high capacity for pain and generally experience worse living conditions than cattle.
5. Beef (1.8)
   Cattle generally experience better living conditions than most other farmed animals. Some authors even argue that their lives are positive overall and therefore worth living.⁶
6. Milk (0.11)
   Milk ranks so low because a dairy cow produces a large quantity of milk over her lifetime, distributing the associated suffering across many kilograms of the product.

Even if one disagrees with the exact ranking, adjusting it according to one’s own judgments about animals’ capacity for pain and quality of life can provide a practical basis for choosing products that minimize suffering.

The most effective way to minimize animal suffering is, of course, veganism. Many people, however, struggle to maintain a vegan diet over the long term. This raises the following question: Is there a diet that could reduce 95% of animal suffering while retaining the long-term adherence that vegetarianism seems to enable?

Yes, there is: lacto-vegetarianism.

Simply put, lacto-vegetarianism is veganism plus dairy products. To understand how much suffering this could prevent, consider the following comparison:

In 2025, roughly 10 whole-chicken equivalents and 252 eggs were produced for the average German.^7,8,5 Many of those chickens had trouble supporting their own weight and lived in a space smaller than a DIN A4 sheet.^9,10 This represents an enormous amount of suffering.

In contrast, if one were to consume one liter of milk a day, it would take roughly 80 years to consume the estimated lifetime output of a single dairy cow.^11,5 If the dairy products are carefully selected, that cow most likely had a life that could generally be considered worth living.⁶ This is what an effective reduction in animal suffering looks like.

Lacto-vegetarianism also has other potential advantages:

• Many people find dairy harder to give up than other animal products, such as eggs.
• Dairy consumption prevents nutritional deficiencies common in vegans.¹²
• Dairy provides abundant protein. This includes whey, which is a by-product of cheese production and may therefore carry a smaller share of the moral burden.

Hopes and Outlook

Lacto-vegetarianism feels like vegetarianism, but from an animal-suffering perspective, it is much closer to veganism. It offers an easy way to pursue a goal that many people already share: minimizing suffering while retaining taste, protein, and nutrition.

I fear, however, that the hyperanalytical reasoning needed to reach this conclusion is uncommon among the general population, making a broader lacto-vegetarian movement unlikely. Additionally, much work remains to be done before we have a complete science of quantifying animal suffering.

How much suffering is caused by different farming systems? How much more do caged hens suffer than free-range hens? How does the suffering experienced by shrimp compare with that experienced by salmon? These questions will have to be answered another day, probably by people more qualified than me.

I genuinely hope this field of research is pursued further and becomes more mainstream. For now, however, I need to get back to diving and ladyboy massages.