From Character Soup to Instruction-Following: Building a GPT from Scratch on a 16GB Laptop

I wanted to actually understand how ChatGPT-style models work — not read about them, build one. So I started with an empty folder on my M1 Pro MacBook (16 GB RAM) and a rule: write every piece myself, in readable PyTorch, and only scale up when I understood the previous step.

What follows is the whole journey — five stages, the loss numbers at each, the wall I hit at 16 GB, and the honest lesson about what a laptop can and can't do. Everything is ~450 lines of commented Python. The architecture is the same idea behind ChatGPT; it's just tiny.

The one idea everything is built on

A language model plays exactly one game: given some text, predict the next token. The "correct answer" is just the token that actually came next in the training text — no human labels required. Do this billions of times and, as a side effect, the model is forced to learn grammar, facts, and style. That's it. Everything below is that idea, scaled.

Stage 1: The simplest thing that works — a character-level GPT

The smallest real version: one token per character. Vocabulary is just the ~65 unique characters in the training file. I trained a tiny transformer (~818K parameters) on 1 MB of Shakespeare.

step     1 | train loss 4.23 | val loss 4.23
step  5000 | train loss 1.46 | val loss 1.65   # ~3 minutes on the GPU

Result:

ROMEO:
Our parer the saw it think, thou must thing these,
From well his brange bound rest to me pare

It learned the shape of a play — character names, colons, line breaks, Shakespearean cadence — purely from predicting one character at a time. But "brange" isn't a word. A character model has to learn English spelling from scratch, and on 1 MB it can't. That's the ceiling of stage 1, and it points straight at stage 2.

Stage 2: Real words — subword tokenization

The single biggest lever on output quality is the tokenizer. Instead of characters, real models use subwords — fragments like the, ing, understand. I switched to GPT-2's tokenizer (via tiktoken, ~50,257 tokens). Now the model picks from real word-pieces, so it almost never invents non-words.

That one change jumped the model from 818K to 30M parameters — a 37× leap. Here's the surprising part I didn't expect:

64% of the parameters are just the vocabulary lookup table (50,257 tokens × the embedding width). The actual "thinking" layers were a minority of the model.

That reshaped how I thought about scaling: to make a small model smarter, you grow the transformer layers, not the vocab.

Stage 3: Real data — and a disaster called <unk>

Shakespeare is 1 MB of archaic English. To understand modern English, I needed modern English — a lot of it. I first tried WikiText-2 (11 MB), and the model produced fluent Wikipedia-flavored text... until I gave it a name:

> Hitoshi Sakimoto
<unk> <unk> and <unk> . The novel was <unk> University of <unk> ...

The culprit wasn't the model — it was the data. WikiText replaces rare words (especially names) with a literal <unk> placeholder; there were 54,625 of them. The model faithfully learned "a name is followed by <unk>" and spiraled. Garbage in, garbage out, demonstrated perfectly.

The fix: stream a slice of FineWeb-Edu — Common Crawl (the actual web) filtered for quality. I pulled 500 MB of clean, natural English (no <unk>, no artifacts) without downloading the full 1.3 TB, because streaming lets you stop at a byte budget.

A key realization about data at this scale: you don't train on all of it. The trainer samples random chunks, so progress is measured in steps and falling validation loss, not epochs. A 500 MB pool you only sample a fraction of is healthier than a small one you grind through 50 times — it can't memorize, so it's forced to learn general patterns.

Stage 4: Bigger — and the 16 GB wall

I scaled up to 51M parameters (8 transformer layers, 512-wide, 256-token context) and trained on the FineWeb-Edu data. This is where the laptop pushed back.

Apple Silicon has unified memory — the GPU shares the same 16 GB as the OS, my editor, and my browser. Batch 16 at 256-context fit overnight when the machine was idle, but OOM'd the moment I had apps open. The memory hog is the loss computation over the 50,257-word vocabulary at every position, so cost scales with batch × sequence_length. The fixes were classic small-hardware engineering: smaller batches, capped sequence length, and training at a shorter context.

Two features earned their keep here:

• Best-val checkpointing — save the model whenever validation loss improves, so a multi-hour run is safe to interrupt and testable mid-run.
• Resumable training — I could stop the base run, go do something else, and continue from where it left off. (Fun detail: a checkpoint with optimizer state was 585 MB vs. 195 MB for weights alone — a preview of a lesson below.)

The base model bottomed out around validation loss 4.8. A useful way to read that: e^loss ≈ how many tokens it's effectively choosing between. It went from ~54,000 (random, at the start) down to ~123. Samples were now fluent and on-topic, if not yet coherent across a paragraph:

> The human brain
The human brain was discovered in the late 1960s ...
The study was published in the journal Nature Neuroscience.

Real entities, clean sentences — but it can't hold a thought for long. That's the honest ceiling of 51M params on 500 MB.

Stage 5: Teaching it to follow instructions

A base model only continues text. It won't answer "Explain why the sky is blue" — it'll just continue the sentence. Making it respond is a separate training stage: instruction tuning.

I fine-tuned the base on Alpaca (~52,000 instruction→response pairs), reformatted into a fixed template:

### Instruction:
Give me three tips for studying.

### Response:
<the answer><|endoftext|>

The crucial trick is loss masking: you compute the loss only on the response tokens, not the instruction — so the model learns to answer, not to echo the prompt. And you watch the behavior flip:

The format and behavior flipped completely. The content stayed shallow — it knows the shape of an answer but can't reliably retrieve "Paris." That's the distinction that finally clicked for me:

Instruction tuning teaches behavior. The base model provides knowledge. You can't prompt-engineer your way to knowledge that isn't in the weights.

The lessons

• Tokenization is the biggest quality lever for a small model. Characters → subwords was night and day.
• Data quality dominates. The <unk> spiral was a data bug, not a model bug.
• On a small model, most parameters are the vocabulary, not the reasoning. Scale the layers to get smarter.
• Unified memory is a real constraint. Memory scales with batch × sequence × vocab; that's your budget.
• Training needs ~9× the memory of inference — because it stores, per parameter, a gradient, two optimizer stats, and a high-precision copy, plus activations for backprop. (That 195 MB → 585 MB checkpoint jump is this difference.)
• You measure progress in steps and val loss, not epochs, once the data is bigger than you can memorize.

The ceiling, and what's next

A laptop caps you at roughly the 50–150M range for from-scratch training. To go further — a ~1B model that's genuinely coherent, or fine-tuning a real 7–70B open model into a useful assistant — you need more memory and a real CUDA GPU. But the beautiful thing is that none of the concepts change. Tokenization, the training loop, next-token prediction, checkpointing, instruction tuning with loss masking — it's all identical, just scaled. The device = "mps" becomes device = "cuda", the data grows, and the libraries change; the mental model you built from scratch stays exactly the same.

Which was the whole point. I didn't end up with a useful chatbot — I ended up understanding, from the ground up, how one is made.

The numbers, end to end

Built entirely in PyTorch on an M1 Pro (16 GB), ~450 lines of commented code. Every stage is a separate git commit.