Making a Character-Level MLP Language Model from Scratch

Problem

I want to train a tiny character-level language model in C++ from scratch, without any external libraries. The model predicts the next character from the previous fixed-length context of characters.

Why this is a good target for this course:

How the baseline code works

File: char_mlp_lm.cc

The baseline is a serial, deterministic C++17 implementation of a one-hidden-layer MLP language model:

  1. Data generation
  2. Model
  3. Training (serial SGD)
  4. Correctness / sanity checks
  5. Timing

Main runtime knobs (command line):

Where is parallelism

This baseline has substantial parallel work that can be exploited later:

  1. Across mini-batch samples
  2. Inside matrix-vector/matrix-matrix style operations
  3. Gradient accumulation and reduction
  4. Memory locality / cache optimization
  5. Limits to speedup

Results on Miyabi

The serial baseline was compiled with clang++ -O3 -Wall -std=c++17 and run on Miyabi through the lecture-mig queue on October 30, 2026 (job 3738279). This run used the CPU serial implementation; submitting to this queue does not mean the program used the GPU.

Run command (from the code directory):

./char_mlp_lm --steps 4000 --eval_every 200 --sample_every 1000 --batch 64 --context 16 --embed 32 --hidden 256 --train_tokens 800000 --val_batches 100 --lr 0.05 --seed 42 --grad_check 1

Observed results:

The decreasing loss indicates improved next-character prediction on the synthetic corpus. However, the generated text at step 4000 was still not fluent English. These results do not demonstrate general English language ability. The difference between validation performance and free-running text generation needs further investigation.

This provides a serial timing baseline for later parallelization. The parallelization opportunities and possible bottlenecks discussed above are hypotheses to investigate; no parallel speedup or detailed profiling has been measured yet.