R
↗
Before we can begin evaluating and auditing AI systems, we have to understand them from first principles. On Feb 12, 2026, Andrej Karpathy (co-founder at OpenAI; helped build Tesla Autopilot) released a 200-line pure-Python program implementing the fundamental ideas behind GPT. I've taken his post and turned it into a lab with exercises and visuals to help us understand the concepts deeply rather than skim them. Karpathy's post is already well written — the goal is to augment it. The Python here is also rewritten in a slightly less compressed style: ~2XX lines instead of 200, but a bit easier to read. As always, feel free to work with the people at your table. You've got this. Original post: karpathy.ai/microgpt.html · companion video on autograd: The spelled-out intro to neural networks and backpropagation (2.5 hr) Try it · generate names from the trained microgpt microgpt is tiny (just 4,192 parameters) but it's still a real neural language model. Karpathy trained one on 32,033 first names (the makemore dataset). The weights are loaded right here in your browser, and the same forward pass you'll dissect later in the lab runs every time you press Send. Type a single letter and press Enter. Temperature 0.70 low = conservative · high = wild Take it with you: ↓ model.json (weights) Where to find it GitHub gist with the full source code: microgpt.py Also available on this web page: karpathy.ai/microgpt.html Also available as a Google Colab notebook — you can run it without installing anything The following is a guide that steps an interested reader through the code. Dataset The fuel of large language models is a stream of text data, optionally separated into a set of documents. In production-grade applications, each document would be an internet web page — but for microgpt, we use a simpler example of 32,000 names, one per line: # Let there be an input dataset `docs`: list[str] of documents (e.g. a dataset of names) if not os.path.exists('input.txt'): import urllib.request names_url = 'https://raw.githubusercontent.com/karpathy/makemore/refs/heads/master/names.txt' urllib.request.urlretrieve(names_url, 'input.txt') docs = [l.strip() for l in open('input.txt').read().strip().split('\n') if l.strip()] random.shuffle(docs) print(f"num docs: {len(docs)}") The dataset looks like this. Each name is a document: emma olivia ava isabella sophia charlotte mia amelia harper ... (~32,000 names follow) The goal of the model is to learn the patterns in the data and then generate similar new documents that share the statistical patterns within. As a preview, by the end of the script our model will generate ("hallucinate"!) new, plausible-sounding names. Skipping ahead, we'll get: sample 1: kamon sample 8: anna sample 15: earan sample 2: ann sample 9: areli sample 16: lenne sample 3: karai sample 10: kaina sample 17: kana sample 4: jaire sample 11: konna sample 18: lara sample 5: vialan sample 12: keylen sample 19: alela sample 6: karia sample 13: liole sample 20: anton sample 7: yeran sample 14: alerin It doesn't look like much, but from the perspective of a model like ChatGPT, your conversation with it is just a funny-looking "document". When you initialize the document with your prompt, the model's response from its perspective is just a statistical document completion. Tokenizer Under the hood, neural networks work with numbers, not characters, so we need a way to convert text into a sequence of integer token ids and back. Production tokenizers like tiktoken (used by GPT-4) operate on chunks of characters for efficiency, but the simplest possible tokenizer just assigns one integer to each unique character in the dataset: # Let there be a Tokenizer to translate strings to discrete symbols and back uchars = sorted(set(''.join(docs))) # unique characters become token ids 0..n-1 BOS = len(uchars) # token id for Beginning of Sequence vocab_size = len(uchars) + 1 # total tokens, +1 for BOS print(f"vocab size: {vocab_size}") We collect all unique characters across the dataset (which are just the lowercase letters a–z), sort them, and each letter gets an id by its index. The integer values themselves carry no meaning — each token is just a discrete symbol. Instead of 0, 1, 2 they could be different emoji. We also create one special token, BOS (Beginning of Sequence), which acts as a delimiter: it tells the model "a new document starts/ends here". Later during training, each document gets wrapped with BOS on both sides: [BOS, e, m, m, a, BOS]. The model learns that BOS initiates a new name, and that another BOS ends it. So we have a vocabulary of 27 (26 lowercase letters + BOS). Tokenizer playground Type a name. It gets wrapped in BOS and converted to integer ids. This is exactly what the model sees as input. Name: Try it The character "a" is the first alphabet letter, so it has id 0. What's the id of "z"? Of "BOS"? If your full name has 9 letters, how many tokens does the model see when you train on it? Show answer"z" is id 25 (last of a–z, indices 0..25). BOS is 26 (length of uchars = 26 alphabet letters). A 9-letter name produces 9 + 2 = 11 tokens: BOS, the 9 letters, then BOS again. From a neuron to a network Before we open up gpt() and stare at multi-head attention, let's build up the underlying object — the neuron — and stack neurons into a network. The end goal of this section: by the time we hit the architecture diagram, every box in it will feel like an obvious composition of things we already understand. Here's roughly where we're going. Don't worry about the details — file the picture mentally, then we'll build to it. (You can already play with this — drag the input sliders and watch the activations propagate.) x2.0 b-1.0 Output a = 2.0 + (−1.0) = 1.0 Step 1 The simplest "neuron" One input x, one bias b, and an output a = x + b. That's it — just an adder. No learning yet, no bend in the output. It's a useful starting object because every more complex neuron is just this one with more parts bolted on. def neuron(x, b): return x + b Try it If x = 3 and b = -1, what does the neuron output? What if I want this neuron to always output 0 no matter the input? What b would I need (and would it work for every x)? Show answerOutput is 3 + (−1) = 2. To force the output to 0 we'd need b = −x, which depends on x — a single bias can't do it. That's why we'll add a weight next: it lets the neuron scale its input before the bias. x2.0 w1.5 b-1.0 Step by step z = 2.0 × 1.5 = 3.0; a = z + (−1.0) = 2.0 Step 2 Add a weight Multiply the input by a learned weight w before adding the bias: a = x*w + b. Now the neuron has two knobs. With both w and b the neuron can shift and scale — it can learn any affine [affine = scale the input, then shift it] response. This is the canonical "linear neuron". def neuron(x, w, b): return x * w + b x2.0 w1.5 b-1.0 Active. z = 3.0 > 0, so a = z = 3.0 Step 3 Add a nonlinearity (ReLU) Stacking linear neurons on top of linear neurons just gives you another linear function. To learn interesting things, we need a nonlinearity. ReLU is the simplest: $f(z) = \max(0, z)$. It passes positive values through and zeros out negative ones. def relu(z): return max(0, z) def neuron(x, w, b): z = x * w + b a = relu(z) return a Try it With w = 2 and b = -3, plug in x = 1 and x = 4. What does the neuron output in each case? At what value of x does the ReLU "turn on" — i.e., where does the output stop being zero? Show answerx = 1 → z = 1·2 − 3 = −1 → a = max(0, −1) = 0. x = 4 → z = 5 → a = 5. The ReLU turns on at z = 0, i.e. when x = 3/2 = 1.5. The neuron has learned a soft threshold. Step by step — Step 4 Many inputs in, one output out Real neurons take a vector of inputs. Each input x_i has its own weight w_i; the neuron sums them up, adds bias, and applies ReLU: $$ a = \mathrm{ReLU}\!\left(\sum_{i=1}^{n} x_i w_i + b\right) $$ def neuron(x, w, b): # x and w are lists of length n z = sum(xi * wi for xi, wi in zip(x, w)) + b return max(0, z) The inner sum is a dot product — the fundamental operation of neural networks. In microgpt, linear(x, w) does this dot product once per row of w. (Karpathy's version drops the bias b — modern Transformers often do.) Python aside · what does zip() do? Python's built-in zip() walks through two (or more) lists in lockstep and hands back tuples of matching elements — one tuple per "column" — stopping when the shortest list runs out. So for xi, wi in zip(x, w) gives us the i-th input and the i-th weight together on each loop iteration, ready to multiply. x 0.5 −0.3 1.2 w 0.4 0.7 −0.1 ↓ zip(x, w) ↓ → (0.5, 0.4) (−0.3, 0.7) (1.2, −0.1) The dot product is then just "sum the products of each pair": $0.5{\cdot}0.4 + (-0.3){\cdot}0.7 + 1.2{\cdot}(-0.1) = 0.20 - 0.21 - 0.12 = -0.13$. The same pattern shows up everywhere in microgpt — adding token + position embeddings (zip(tok_emb, pos_emb)), residual sums (zip(x, x_residual)), every matrix-vector multiply inside linear(). Anywhere you see two same-length lists walked together, zip is the glue. Forward pass In a neural network, the forward pass is the trip from inputs to a prediction. You hand the network some numbers, they flow through every layer — getting multiplied by weights, summed with biases, occasionally bent by a nonlinearity — and out the other end falls a single answer. The forward pass doesn't change the network at all; it just runs it. Every weight stays exactly where it was; only the activations move. It's worth pausing on this before we get to backprop, because backprop is just the forward pass run in reverse. If we can't picture the forward pass clearly, the backwards version will feel like magic. Below is a deliberately tiny network so you can wiggle every knob and watch the output respond. Three inputs x₁, x₂, x₃ feed into two hidden ReLU neurons that join at a single ReLU output a. The three weights and three biases (w₁, w₂, w₃, b₁, b₂, b₃) are yours to play with. As you change them, the prediction surface on the right re-draws — it plots a as a height over the (x₁, x₂) plane, with x₃ swept by its slider. The forward pass is that mapping from input space to output. A 2-layer network — and its prediction surface Drag the weight/bias sliders and watch the surface change shape. Sweep x₃ with its slider to lift / fold the surface. Because every neuron has a ReLU, the surface is piecewise linear — each ReLU contributes a sharp fold. Click and drag the surface to rotate. w₁+0.80 w₂−0.60 w₃+0.50 b₁+0.20 b₂+0.00 b₃+0.00 x₃+0.00 Forward pass: h₁ = ReLU(w₁·x₁ + w₂·x₂ + b₁) h₂ = ReLU(w₃·x₃ + b₂) a = ReLU(h₁ + h₂ + b₃) Loss vs target: — best: — loss over time → each slider tweak adds a new point The same thing, in code Here's the network we've been playing with, written out as a small class hierarchy: Neuron → Layer → MLP. This is essentially how Karpathy's micrograd packages neural networks. The Neuron.__call__ method is doing exactly what the circles in the diagram do — weighted sum of inputs, plus bias, through a ReLU. import random class Neuron: def __init__(self, nin): self.w = [random.uniform(-1, 1) for _ in range(nin)] self.b = random.uniform(-1, 1) def __call__(self, x): # forward pass: a = ReLU(w · x + b) z = sum(wi * xi for wi, xi in zip(self.w, x)) + self.b return max(0, z) class Layer: def __init__(self, nin, nout): self.neurons = [Neuron(nin) for _ in range(nout)] def __call__(self, x): return [n(x) for n in self.neurons] class MLP: def __init__(self, nin, nouts): sizes = [nin] + nouts self.layers = [Layer(sizes[i], sizes[i+1]) for i in range(len(nouts))] def __call__(self, x): for layer in self.layers: x = layer(x) return x # 3 inputs → 2 hidden neurons → 1 output (one forward pass) x = [1.0, 0.5, -0.3] mlp = MLP(3, [2, 1]) print(mlp(x)) # e.g. [0.42] How this maps to the viz The MLP(3, [2, 1]) above is slightly more general than the network in the diagram. In a standard MLP every input feeds every hidden neuron, so the first layer alone would have 2 × (3 weights + 1 bias) = 8 parameters. The interactive diagram uses a deliberately restricted variant — h₁ sees only x₁, x₂, and h₂ sees only x₃ — so we end up with just 3 weights and 3 biases. That's small enough that the prediction surface stays readable as you wiggle the sliders. The Neuron / Layer / MLP scaffolding is identical either way. Predict the outputs Here's a small batch of inputs. Using the MLP class above, write code that produces predictions for each one: xs = [ [ 2.0, 3.0, -1.0], [ 3.0, -1.0, 0.5], [ 0.5, 1.0, 1.0], [ 1.0, 1.0, -1.0], ] ys_target = [1.0, -1.0, -1.0, 1.0] # what we WISH the network said ypred = ? # ← your job Show answerOne line: ypred = [mlp(x) for x in xs]. With random weights you'll get whatever the freshly-initialized model says — almost certainly nothing like ys_target.Bonus observation: our network's output is wrapped in a ReLU, so ypred[i] ≥ 0 for every input. That means we can never match a target of −1.0 no matter what the weights are. To handle negative targets we'd need a different output activation (or none). This is a real design choice in real models — the output activation has to match the kind of answer you want. What is loss? Once we have predictions, the obvious question is: how wrong are we? The standard way to turn that question into a single number is a loss function. The simplest one — mean squared error (MSE) — just averages the squared gap between each prediction and its target: $$ L = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)^2 $$ A few properties worth internalizing: Loss is always ≥ 0 — squared gaps can't be negative. Loss = 0 means perfect predictions — every ŷᵢ exactly hits its target yᵢ. Big gaps cost much more than small ones — because they're squared. A model that's off by 2 on one example loses 4× more than one that's off by 1. Loss is the only thing the optimizer cares about — every weight in the model will be nudged in whichever direction makes this single number smaller. This is the whole game of training: find weights that minimize the loss. Now try it yourself Scroll back to the interactive diagram and click 🎯 Train against a target surface. A hidden target network is generated, its surface is overlaid as a dark wireframe, and the live loss appears as both a number and a bar. The little chart underneath records every loss reading — as you nudge sliders, you can literally watch the line go down (or up — easy to make it worse). See if you can get the loss below 0.02 by hand. It's harder than it looks — and that's the whole motivation for the gradient-based training we'll build in the next section. The weights and biases in our code are still plain Python floats, so we can run the model and measure the loss but we can't yet ask "which weight should I nudge, and by how much, to reduce the loss?". To answer that, we need gradients — and that's exactly what the next section is about. Autograd Training a neural network requires gradients: for each parameter in the model, we need to know "if I nudge this number up a little, does the loss go up or down, and by how much?". The computation graph has many inputs (the model parameters and input tokens) but funnels down to a single scalar output: the loss. Backpropagation starts at that single output and works backwards through the graph, computing the gradient of the loss with respect to every input. It relies on the chain rule from calculus. In production, libraries like PyTorch handle this automatically. Here, we implement it from scratch in a single class called Value. Companion lecture This is the most mathematically intense part of microgpt. Karpathy has a 2.5-hour video that builds the whole thing live: The spelled-out intro to neural networks and backpropagation. The walk-through below condenses the key points. Building Value piece by piece The same Lego mindset works here: start with a wrapper, add operators, then add the graph bookkeeping that makes backprop possible. Try it live: A Value object — built up in three stages Type numbers and toggle the stages to see what Value remembers at each version of the class. Stage 3 is what microgpt actually uses. a b Value, in four steps Step 1 — A scalar that prints itself. __repr__ is the dunder Python calls when you print an object. class Value: def __init__(self, data): self.data = data def __repr__(self): return f"Value(data={self.data})" a = Value(-6.0) b = Value(7.0) print(a) # Value(data=-6.0) print(b) # Value(data=7.0) Step 2 — Teach Value arithmetic. __add__ and __mul__ are the dunders Python calls when you write a + b or a * b. We return a fresh Value. class Value: def __init__(self, data): self.data = data def __repr__(self): return f"Value(data={self.data})" def __add__(self, other): return Value(self.data + other.data) def __mul__(self, other): return Value(self.data * other.data) a = Value(-6.0); b = Value(7.0); c = Value(10.0) d = a * b + c print(d) # Value(data=-32.0) Step 3 — Neural networks are computation graphs, so a node needs to remember what produced it. Each operation records its inputs as _children. class Value: def __init__(self, data, children=()): self.data = data self._children = children # the values that produced this one def __add__(self, other): return Value(self.data + other.data, (self, other)) def __mul__(self, other): return Value(self.data * other.data, (self, other)) a = Value(2.0) b = Value(3.0) c = a * b # c knows its children are (a, b) L = c + a # L knows its children are (c, a) Step 4 — Each operation also records its local derivative. backward() walks the graph in reverse topological order, applying the chain rule and accumulating gradients. class Value: __slots__ = ('data', 'grad', '_children', '_local_grads') def __init__(self, data, children=(), local_grads=()): self.data = data # forward-pass scalar self.grad = 0 # dL/d(this), filled in backward pass self._children = children # inputs to this node self._local_grads = local_grads # d(this)/d(child) for each child def __add__(self, other): other = other if isinstance(other, Value) else Value(other) return Value(self.data + other.data, (self, other), (1, 1)) def __mul__(self, other): other = other if isinstance(other, Value) else Value(other) return Value(self.data * other.data, (self, other), (other.data, self.data)) def __pow__(self, other): return Value(self.data**other, (self,), (other * self.data**(other-1),)) def log(self): return Value(math.log(self.data), (self,), (1/self.data,)) def exp(self): return Value(math.exp(self.data), (self,), (math.exp(self.data),)) def relu(self): return Value(max(0, self.data), (self,), (float(self.data > 0),)) def __neg__(self): return self * -1 def __radd__(self, other): return self + other def __sub__(self, other): return self + (-other) def __rsub__(self, other): return other + (-self) def __rmul__(self, other): return self * other def __truediv__(self, other): return self * other**-1 def __rtruediv__(self, other): return other * self**-1 def backward(self): # 1) Build reverse-topological order via DFS topo, visited = [], set() def build_topo(v): if v not in visited: visited.add(v) for child in v._children: build_topo(child) topo.append(v) build_topo(self) # 2) Seed the loss gradient, then propagate self.grad = 1 for v in reversed(topo): for child, local_grad in zip(v._children, v._local_grads): child.grad += local_grad * v.grad Briefly, a Value wraps a single scalar number (.data) and tracks how it was computed. Think of each operation as a little Lego block: it takes some inputs, produces an output (the forward pass), and it knows how its output would change with respect to each of its inputs (the local gradient). That's all the information autograd needs from each block. Everything else is just the chain rule, stringing the blocks together. Every time you do math with Value objects (add, multiply, etc.), the result is a new Value that remembers its inputs (_children) and the local derivative of that operation (_local_grads). For example, __mul__ records that $\frac{\partial(a\cdot b)}{\partial a}=b$ and $\frac{\partial(a\cdot b)}{\partial b}=a$. The full set of Lego blocks: OperationForwardLocal gradients a + b$a+b$$\partial/\partial a = 1,\; \partial/\partial b = 1$ a * b$a \cdot b$$\partial/\partial a = b,\; \partial/\partial b = a$ a ** n$a^n$$\partial/\partial a = n\,a^{n-1}$ log(a)$\ln a$$\partial/\partial a = 1/a$ exp(a)$e^a$$\partial/\partial a = e^a$ relu(a)$\max(0,a)$$\mathbf{1}_{a>0}$ The backward() method walks this graph in reverse topological order (starting from the loss, ending at the parameters), applying the chain rule at each step. If the loss is $L$ and a node $v$ has a child $c$ with local gradient $\frac{\partial v}{\partial c}$, then: $$\frac{\partial L}{\partial c} \mathrel{+}= \frac{\partial v}{\partial c}\cdot\frac{\partial L}{\partial v}$$ This looks scary if you're not comfortable with calculus, but it's literally just multiplying two numbers in an intuitive way: "If a car travels twice as fast as a bicycle, and the bicycle is four times as fast as a walking man, then the car travels 2×4 = 8 times as fast as the man." The chain rule is the same idea — you multiply the rates of change along the path. We kick things off by setting self.grad = 1 at the loss node, because $\frac{\partial L}{\partial L}=1$. From there, the chain rule just multiplies local gradients along every path back to the parameters. Note the += (accumulation, not assignment). When a value is used in multiple places in the graph (i.e. the graph branches), gradients flow back along each branch independently and must be summed. This is the multivariable chain rule: if $c$ contributes to $L$ through multiple paths, the total derivative is the sum of contributions from each path. After backward() completes, every Value in the graph has a .grad containing $\frac{\partial L}{\partial v}$, which tells us how the final loss would change if we nudged that value. Watch backprop happen Backprop is easier to internalize if you build it up. Below are four cases in increasing complexity — start with what a single + does to a gradient, then a single ×, then both with a branch, then a full training-style pipeline (input, prediction, loss). Each tab is its own little graph; step through it one click at a time. Backprop in four shapes Each case shows: forward construction → topological sort → backward pass propagating gradients. The local derivative on each edge is the only thing each operation has to know. Press Next step to begin building the computation graph. Topo list: (empty) step 0 / 14 Chain rule worked out Math appears here as you step through. By hand — same algorithm, different shape Here's a small neuron computing a = ReLU(x·w + b). The forward values are filled in. Try to compute the gradients with respect to x, w, and b by hand assuming ∂L/∂a = 1. Then click "Run backward" to check. Doing this once by hand is the single best way to internalize what backward() is doing. x2.0 w1.5 b-1.0 Forward pass shown. Press Run backward to propagate gradients (assuming ∂L/∂a = 1). This is exactly what PyTorch's .backward() gives you: import torch a = torch.tensor(2.0, requires_grad=True) b = torch.tensor(3.0, requires_grad=True) c = a * b L = c + a L.backward() print(a.grad) # tensor(4.) print(b.grad) # tensor(2.) This is the same algorithm that PyTorch's loss.backward() runs, just on scalars instead of tensors (arrays of scalars) — algorithmically identical, significantly smaller and simpler, but a lot less efficient. Let's spell out what backward() gives us. Autograd calculated that if L = a*b + a, with a=2 and b=3, then a.grad = 4.0. This is telling us about the local influence of a on L: if you wiggle a, in what direction is L changing? The derivative of L w.r.t. a is 4.0, meaning that if we increase a by a tiny amount (say 0.001), L would increase by about 4× that (0.004). Similarly, b.grad = 2.0 means the same nudge to b would increase L by about 2× that. These gradients tell us the direction (positive or negative) and the steepness (magnitude) of each input's influence on the final output (the loss). This lets us iteratively nudge the parameters of our neural network to lower the loss, and hence improve its predictions. Gradient descent · w and b chase the target Case 4 above computed the gradients for one step: ∂loss/∂b = −6 and (chaining through x) ∂loss/∂w = 2·err·x = −18. Now we repeat the step. x = 3 and the target y = 10 are fixed; each step nudges the two parameters against their gradient — w ← w − lr·∂loss/∂w — and you watch the prediction ŷ climb toward the target while the loss shrinks. The nudge is just the gradient multiplied by the learning rate. Click Next step a few times. x = 3 target y = 10 learning rate = 0.02 Computation graph — node values and local gradients (∂) recompute every step: loss = err² step 0 / 4 Architecture The model architecture is a stateless function: it takes a token, a position, the parameters, and the cached keys/values from previous positions, and returns logits (scores) over what token the model thinks should come next in the sequence. We follow GPT-2 with minor simplifications: RMSNorm instead of LayerNorm, no biases, and ReLU instead of GeLU. We'll step through the model one block at a time. Each sub-section below covers one piece — first the intuition, then any small helper functions it needs, then the relevant code, the actual parameter matrices, and finally a small interactive widget showing what we've built up so far. Model config at a glance HyperparameterToy (this walkthrough)Real microgptGPT-2 small (ref) vocab_size4 (BOS, a, b, c)27 (a–z + BOS)50,257 n_embd · d_model216768 n_head1412 head_dim2464 block_size (context len)4161,024 n_layer1112 parameters (rough)~704,192~124M All three columns run the same code. The widgets below render the toy column; the "Scaling up" callouts in each subsection map the toy back to the real-microgpt and GPT-2 columns. A running toy example · d_model = 2 To make each step concrete, we'll track a single token through the whole block using a deliberately tiny model. The vector at each stage will only have two numbers, so you can do every multiplication by hand and watch what changes. Setup. Pretend the vocabulary is just 4 tokens — BOS=0, 'a'=1, 'b'=2, 'c'=3 — and the embedding width is d_model = 2, with n_head = 1 (so head_dim = 2) and block_size = 4. We're partway through generating: the model has already seen BOS at position 0 and 'a' at position 1, and now it's processing 'b' at position 2. We want it to predict what comes at position 3. Each subsection below pulls in the toy weights it needs, walks the numbers forward, and the resulting vector becomes the input to the next subsection. By the end of Output, we'll have one concrete probability over the 4-token vocab. Embeddings The neural network can't process a raw token id like 2 directly. It only works with vectors (lists of numbers). So we associate a learned vector with each possible token, and feed that in as its neural signature. The token id and position id each look up a row from their respective embedding tables (wte and wpe). These two vectors are added together, giving the model a representation that encodes both what the token is and where it is in the sequence. Modern LLMs usually skip the position embedding and use relative-based positioning schemes like RoPE. Concrete example: say our current token is 'b', which the tokenizer mapped to id 2, sitting at position 2. The lookup wte[2] gives a length-2 vector — that's the x the network actually sees. Click a different letter below and you'll watch a different row of wte get pulled in and flow all the way through the three views (and the numeric tour at the bottom of the section). The block so far — Embeddings only Pick the current token. Its row of wte becomes x; wpe[pos=2] gets added; that vector flows through every downstream view. The fine-grained sliders at the bottom of the section still work for off-vocabulary values. sequence so far → pos 0BOS · pos 1'a' · pos 2'b'current → wpe[2] = [−0.10, 0.10] current token → token_id = 2 → x = wte[2] = [−0.30, 0.40] Same step, drawn as neurons Parameter matrices Two learned tables — one row per token, one row per position. Hover any cell to see its value. The pattern is just random Gaussian initialisation (std = 0.08); training reshapes these into something meaningful. wte · token embedding toy d = 2 · (4 × 2) — real microgpt is (27 × 16) hover a cell wpe · position embedding toy d = 2 · (4 × 2) — real microgpt is (16 × 16) hover a cell Helper used here · rmsnorm Once we've added the token and position vectors, we normalize. rmsnorm (Root Mean Square Normalization) rescales a vector so its values have unit root-mean-square. This keeps activations from growing or shrinking as they flow through the network, stabilizing training. It's a simpler variant of the LayerNorm used in the original GPT-2. def rmsnorm(x): ms = sum(xi * xi for xi in x) / len(x) scale = (ms + 1e-5) ** -0.5 return [xi * scale for xi in x] Code in gpt() tok_emb = state_dict['wte'][token_id] # length 16 pos_emb = state_dict['wpe'][pos_id] # length 16 x = [t + p for t, p in zip(tok_emb, pos_emb)] x = rmsnorm(x) Toy walkthrough · embeddings at d = 2 Our token is 'b' (id 2) at position 2. Pick tiny wte and wpe tables to look up from: # wte: 4 rows (one per token), each a length-2 vector wte = [[ 0.20, 0.30], # BOS [ 0.50, -0.10], # 'a' [-0.30, 0.40], # 'b' [ 0.10, 0.20]] # 'c' # wpe: 4 rows (one per position), each a length-2 vector wpe = [[ 0.10, -0.05], # pos 0 [ 0.05, 0.15], # pos 1 [-0.10, 0.10], # pos 2 [ 0.15, 0.00]] # pos 3 token_id, pos_id = 2, 2 tok_emb = wte[token_id] # → [-0.30, 0.40] pos_emb = wpe[pos_id] # → [-0.10, 0.10] x = [t + p for t, p in zip(tok_emb, pos_emb)] # → [-0.40, 0.50] x = rmsnorm(x) # → [-0.88, 1.10] Doing the RMSNorm by hand. Mean-squ
