slm-125m-base

Llama-style decoder · runs 100% in your browser · weights from HuggingFace

Completion Playground

temperature 0.80

max new tokens 96

top-p 0.95

repetition penalty 1.30

The completion will appear here — the seed prompt in grey, the model's continuation highlighted.

This is a base completer, not a chat model. It continues text in the legal/financial register it was trained on — it does not answer questions or follow instructions. At 125M params it speaks the register fluently but is not a reliable source of facts. Runs entirely in your browser — the first visit downloads ~139 MB once, then it's cached and works offline.

Model Card / Spec Sheet

125M

params

2.19B

train tokens

11.0 ppl

val perplexity

Architecture

Tokenizer

Training

Curves

Family	Llama-style decoder maps 1:1 to transformers.LlamaConfig
Parameters	≈125.8M tied embeddings → LM head adds 0
Hidden size (d_model)	768
Layers	12
Attention heads	12 head dim 64
KV heads	12 (MHA) num_kv == num_heads → export-safe
MLP	SwiGLU intermediate 3072 (4×), silu gate
Normalization	RMSNorm eps 1e-5, pre-norm
Positional	RoPE θ (rope_theta) = 10,000
Context length	1,024 tokens
Embeddings	Tied input embed == output projection
Attention bias	False
Serving dtype	int8 (q8) ONNX bf16 trained → quantized for browser

# param budget (tied) embed 16,384 × 768 = 12.58M attn/layer 4 × 768² = 2.36M mlp/layer 3 × 768 × 3072 = 7.08M × 12 layers = 113.2M total ≈ 125.8M

Deliberately vanilla geometry: MHA (not GQA) and a 1:1 LlamaConfig mapping so the model exports cleanly to ONNX (and GGUF) with zero custom-converter work — which is exactly what lets it run in-browser here via Transformers.js.