Training a small LLM from scratch

Problem

Large language models (LLMs) are often decoder-only Transformer networks trained to predict the next word. This project trains a very small character-level version from random weights using Tiny Shakespeare, a roughly 1.1-million-character collection of Shakespeare dialogues in tinyshakespeare.txt(taken from karpathy’s github karpathy/char-rnn).

L=−1BT∑b=1B∑t=1Tlog⁡pθ(xt+1(b)∣x1(b),…,xt(b)) L = -\frac{1}{BT} \sum_{b=1}^{B} \sum_{t=1}^{T} \log p_\theta\left(x^{(b)}_{t+1} \mid x^{(b)}_1, \dots, x^{(b)}_t\right)

This is an end-to-end language-model training problem: preparing text, building a vocabulary, training, evaluating on held-out text, and generating text. The transformer block is one part of it. The goal is to learn basic letter patterns, spaces, common words and some dialogue-like structure, not to generate fluent Shakespeare. The computation has the same broad structure as LLM training, only much smaller.

How to check the results are correct:

How the algorithm/baseline code works

small_llm.cc is a single C++17 file using only the standard library. ALl code for layers, backward pass, optimizer etc. will be written by hand as plain serial loops.

Data preparation

  1. Read tinyshakespeare.txt as raw bytes and map each distinct byte to an integer id in byte order; do not discard line breaks or punctuation.
  2. Use the first 90% of the text for training and the last 10% for validation. Windows from the two parts never cross the boundary.
  3. Sample random contiguous windows from the training part. The vocabulary contains the characters in the file, but model weights are learned only from training windows.

Model

For a window of TT characters (default 32) with model width dd:

  1. Embedding: each character becomes a learned vector of size dd, plus a learned vector for its position.
  2. One causal self attention layer (single head):

Attention(Q,K,V)=softmax(QKTd+M)V \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left( \frac{Q K^T}{\sqrt{d}} + M \right) V

where QQ, KK, VV are linear projections of the input and the mask MM sets future positions to −∞-\infty, so position tt only uses positions 0,…,t0, \dots, t. The result goes through an output projection and is added to the input (residual connection). 3. Feed forward layer: a linear layer to size 2d2d, tanh, a linear layer back to size dd, added to its input (residual connection). 4. Output layer: a linear layer to VV scores, then softmax gives next-character probabilities.

There is no layer norm and only one block, which keeps the backward pass short enough to read.

Training loop

for step = 1 .. steps:
    for b = 1 .. batch:
        pick a random start position in the training text
        take T input characters and their T next-character targets
        forward pass: compute and cache every layer's activations
        backward pass: add dL/dw for every parameter w into its gradient
    Adam update of every parameter (learning rate 0.003), then zero the gradients
estimate training and held-out loss on 32 evenly spaced windows in each section
generate 240 characters: prompt with "ROMEO:\n" and repeatedly sample the next character

Building and running

Build on Miyabi from code/ after loading the course compiler environment:

clang++ -O3 -Wall -std=c++17 small_llm.cc -o small_llm
./small_llm 1200 8 32 32 42

Run on a compute node. The program reads the corpus at run time.

Problem size

Argument Meaning Default Allowed
1 training steps 1200 at least 1
2 windows per batch BB 8 1 to 1024
3 model width dd 32 4 to 256
4 context length TT 32 4 to 128

Where is parallelism

What cannot be parallelized directly: