I want a serial baseline for “LLM training from scratch” that can run in a few minutes and can later be parallelized on multicore CPU and GPU.
The task is next-character prediction on text. Given a sequence of
bytes x[0], x[1], ..., x[L-1], the model predicts
x[1], x[2], ..., x[L]. Training minimizes cross-entropy
loss.
I use byte-level tokens (vocabulary size 256), so no external
tokenizer or dictionary is needed. The input text is read from a local
file by command-line option (for example
../harrypotter.txt), or a built-in tiny fallback text.
The baseline should:
File: code/app.cc
Model structure (small GPT-style block):
E[token] -> R^DWq,
Wk, Wv)0..tWoWout to logits over 256 byte
tokensTraining:
batch_size random subsequences per stepprint_every steps:
train_loss, valid_losstrain_ppl = exp(train_loss),
valid_ppl = exp(valid_loss)Main command-line parameters:
--data PATH--steps N--seq-len N--d-model N--batch-size N--lr X--seed NThis makes runtime tunable from seconds to several minutes by
adjusting steps, seq-len,
d-model, and batch-size.
This baseline is intentionally serial, but it has many parallel opportunities:
(t, j) pair,Potential mapping later:
Likely performance limits to analyze later include memory bandwidth, reduction overhead (softmax/dot products), and arithmetic intensity.