Making LLM Inference from Scratch

Problem

I study text generation (inference) of a small large language model (LLM), implemented from scratch.

The model is a tiny Llama-2 style transformer trained on TinyStories. I use the stories110M checkpoint, which has about 110 million parameters. The task is: given a short prompt, generate the next tokens one by one to produce a story.

This is a realistic application because LLM inference is widely used in real systems. It is also a good target for this course because:

So it is a good baseline for later multicore/GPU/SIMD optimization and performance analysis.

How the baseline code works

The baseline program is llama2_infer.cpp in this directory.

Main steps:

  1. Load model checkpoint
  2. Load tokenizer
  3. Run transformer forward pass token-by-token
  4. Sample next token
  5. Repeat until stop
  6. Multiple independent stories with --samples N

Build and run

Download files:

wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories110M.bin
wget https://github.com/karpathy/llama2.c/raw/master/tokenizer.bin

Compile:

clang++ -O3 -Wall -std=c++17 llama2_infer.cpp -o llama2_infer

Run example:

./llama2_infer --checkpoint stories110M.bin --tokenizer tokenizer.bin \
  --prompt "Once upon a time" --steps 1024 --samples 20 --seed 123 --temperature 1.0

Observed serial baseline (one run):

Increasing --samples increases total runtime approximately proportionally, so --samples is a practical way to set serial runtime to a few minutes.

How to verify results

I use two simple checks:

  1. Human readability check
  2. Deterministic check with --temperature 0

Where time is spent and where parallelism exists

Most time is spent in the transformer compute for each token, especially:

Parallelization opportunities:

What is hard to parallelize:

This mix (large inner parallel work + outer sequential dependency) is exactly why this problem is interesting for performance analysis.