A router has settings such as a Wi-Fi channel, a DNS server, and a guest network. This program learns to turn a short, fixed-format English request into a sequence of configuration commands. For example:
Request: set guest wifi on, set channel 11, set dns 1.1.1.1,
firewall strict, tx power 70, max clients 16
Target: set wifi.guest=1; set wifi.channel=11; set wan.dns=1.1.1.1;
set firewall.mode=strict; set wifi.tx_power=70;
set wifi.max_clients=16; commit;
These commands are a synthetic teaching format which looks similar to the real FRR/vtysh commands but cannot operate on the real routers. The goal is to study the computation and performance of a small language model.
The program creates its own training and test data. It does not use an external dataset.
Each example contains a request and its expected command. The program chooses one value for each setting:
| Setting | Possible values | Command rule |
|---|---|---|
| Guest Wi-Fi | on, off | Convert on to 1 and off to 0 |
| Wi-Fi channel | 1, 6, 11 | Copy the selected number |
| DNS server | 1.1.1.1, 8.8.8.8, 9.9.9.9, 8.8.4.4 | Copy the selected address |
| Firewall mode | normal, strict | Copy the selected mode |
| Transmit power | 40, 70, 100 | Copy the selected number |
| Maximum clients | 8, 16, 32, 64 | Copy the selected number |
The commands always appear in the same order and end with
commit;. These are made-up commands for this experiment,
not verified commands for a real router.
Request:
set guest wifi on, set channel 6, set dns 1.1.1.1, firewall strict, tx power 70, max clients 16
Expected command:
set wifi.guest=1; set wifi.channel=6; set wan.dns=1.1.1.1; set firewall.mode=strict; set wifi.tx_power=70; set wifi.max_clients=16; commit;
The complete example is stored in this format:
[INTENT] request [CMD] expected command
The code creates the expected command using the rules above. The model then tries to learn this conversion by predicting one command character at a time.
There are 576 possible combinations:
2 × 3 × 4 × 2 × 3 × 4 = 576
The program separates these into 460 training combinations and 116 test combinations. The same combination cannot belong to both groups.
Samples are randomly drawn from each group, and repetitions are
allowed. Therefore, train_samples=12000 means 12,000
samples drawn from the training group, not 12,000 different
configurations.
Each example is stored as [INTENT] request [CMD] command
followed by a newline. A character is one token. The default context
window contains 256 characters, enough for the longest complete example
in this task. A shorter window can remove a requested setting before the
model has generated it.
The C++17 program uses only the standard library. It has one attention head, one block, and a small feed-forward network (FFN). Model width is 48, FFN width is 96, and vocabulary size is 45 including padding. There are 35,229 parameters at context 256, or 24,477 at context 32.
The main calculation steps are:
Only earlier characters are inputs. This model calculates only the last query’s attention, not a full square attention matrix, and has no LayerNorm.
Training supplies the correct earlier characters (teacher forcing). Loss is calculated on the command and final newline. Backpropagation finds gradients, gradient clipping limits large changes, and serial SGD updates the weights. Invalid numerical values in gradients or output normalization cause an error. One step selects one sample and updates weights for each command character; it is not an epoch.
The following result is from a serial run on Miyabi. The settings were:
| Setting | Value |
|---|---|
| Training samples | 12,000 |
| Test samples | 116 |
| Training steps | 1,000 |
| Context length | 256 |
| Model width / FFN width | 48 / 96 |
| Learning rate | 0.005 |
Gradient clipping limit (clip_norm) |
5.0 |
| Random seed | 42 |
| Maximum samples for exact command evaluation | 116 |
All reported measurements from this run are listed below.
| Measurement | Result |
|---|---|
Training time (training_time_ms) |
166,173.283 ms (166.173283 s) |
| Character-level weight updates | 140,431 |
| Updates with gradient clipping | 77,629 |
| Training character updates per second | 845.088 |
| Test command teacher-forced loss | 1.286 |
| Test command next-character accuracy | 0.506 (50.6%) |
| Exact command match rate | 0.000 (0/116) |
| Teacher-forced evaluation time | 7,247.528 ms |
| Generation time | 11,332.144 ms |
Training time includes context construction, forward/backward
calculations, clipping, updates, and logging; it excludes compilation,
data creation, testing, and generation. Three printed request/answer
examples allowed manual checking. Generated commands repeated
set set set ....
Training loss fell from about 2.81 for the first 100 steps to 1.38 for the last 100 steps. Some character patterns appear to be learned, but 50.6% accuracy uses the correct previous characters. During generation, the model uses its own outputs, so errors may vary from time to time.
I do not yet know whether model structure, training settings, or an implementation issue is responsible. Testing whether it can learn one example would help. The model generated some character patterns when given the correct preceding characters, but it failed to generate complete commands on its own.
c++ -O3 -std=c++17 app.cc -o app
./app --train-samples 12000 --test-samples 116 --steps 1000 \
--ctx 256 --d-model 48 --d-ff 96 --lr 0.005 --clip-norm 5 \
--seed 42 --print-every 100 --eval-exact-n 116| Option | Description |
|---|---|
--train-samples |
Number of training samples |
--test-samples |
Number of test samples |
--steps |
Number of training sample selections |
--ctx |
Context length |
--d-model |
Model dimension |
--d-ff |
Feed-forward hidden dimension |
--lr |
Learning rate |
--clip-norm |
Maximum global gradient norm |
--seed |
Random seed |
--print-every |
Logging interval in training steps |
--eval-exact-n |
Maximum number of samples for exact-match evaluation |
Increasing --steps gives more training. Increasing model
widths or context length changes the model and usually increases
computation. For a speed comparison, keep these settings fixed.
Parallelism opportunities (the baseline is fully serial)
Training samples are not independent: each character changes weights used by the next character. Mini-batches could provide more parallel work but would change the update method. Generated characters also depend on earlier outputs.
Key/value calculations grow with context length and the square of model width. The program also recalculates overlapping contexts. These may be costly, but profiling is needed. SIMD and better memory access may help; tiny parallel tasks may add overhead instead.
Parallelization will not automatically fix command generation though.