How to build a small language model from scratch: a 2026 guide

A small language model, usually somewhere between a few hundred million and about ten billion parameters, can run on one GPU or even a CPU server, inside your own network, at a fixed monthly cost. For a narrow job such as reading invoices, shipping documents, contracts or support tickets, a well-trained small model can match a large API model on that job at a fraction of the running cost. This guide walks through how such a model is built in 2026, step by step, with the choices that matter and the ones that do not. Where it helps, we use our own in-house model, AiGanak-SLM-1B, as a worked example.

First: do you need to train from scratch?

Usually not. Hugging Face, who train the open SmolLM models, open their training playbook with exactly this warning: many failed projects trained a model nobody needed. There are three ways to get a small model that knows your domain, and they differ in cost by orders of magnitude.

A practical rule: spend a few days testing a current open model with good prompts and a small fine-tune before committing to anything bigger. If it gets close, fine-tuning wins. If it fails in ways that more examples cannot fix (it splits your identifiers into nonsense tokens, loses track of long documents, or the licence or data provenance is a problem), that is the case for going further.

The 2026 small-model baseline to beat

Whatever you build, it has to beat what you could download today. The open small models are much stronger than they were a year ago:

• Qwen3.5 small models (March 2026): 0.8B, 2B, 4B and 9B dense models under Apache 2.0, with a 262K-token context and native vision. Artificial Analysis rated the 9B the most capable model under 10B parameters at release, while also measuring hallucination rates above 80% for the 4B and 9B on its knowledge test. Small models still need grounding in your documents.
• Gemma 4 (31 March 2026): Google's on-device sizes E2B and E4B, alongside larger 26B and 31B models, with multi-token-prediction versions added in April.
• SmolLM3 (3B): less capable than the newest releases, but Hugging Face published the full recipe, data mixture and training settings, which makes it the best public reference for anyone building their own.
• Others worth knowing: Nanbeige 4.1 3B (a Llama 3.2-style design whose gains came mostly from post-training) and Cohere's Tiny Aya 3.35B (strong multilingual coverage, but a non-commercial licence).

Check every licence before you build on a model. "Open weights" ranges from Apache 2.0 to research-only.

Step 1: data

Every serious training report says the same thing: data curation is the biggest lever, ahead of architecture. Four decisions matter.

How much. The 2022 Chinchilla paper found the compute-optimal ratio to be about 20 training tokens per parameter. Small models are now trained far beyond that, because extra training is paid once while a smaller model is cheaper every time it runs. SmolLM3, a 3B model, was trained on about 11 trillion tokens, roughly 3,700 per parameter. For a domain model you will rarely have that much of your own text, which is why most projects mix a large, cleaned public web corpus with a smaller, heavily weighted domain set.

What mix, and when. Modern runs change the mixture over time, ending on the highest-quality data. SmolLM3's three stages show the pattern:

A domain model follows the same idea: general text early to learn language, then a rising share of your own documents and synthetic task data towards the end.

Cleaning. Deduplicate (near-duplicate web pages and templated documents waste tokens and teach repetition), filter for quality, strip personal data you have no right to train on, and keep an exact record of every source. Data provenance is often the reason a company wants its own model in the first place.

Synthetic data. A larger model can write examples, explanations and question-answer pairs for your domain, and a larger model's answers can be used to train the small one (distillation). SmolLM3's reasoning data was generated with Qwen3-32B. Check the larger model's licence allows this, and review samples by hand: a small model copies its teacher's mistakes faithfully.

Step 2: tokenizer

The tokenizer decides how text is cut into the pieces the model reads. It is easy to overlook and expensive to get wrong, because it cannot be changed cheaply after pretraining.

• Reuse a strong one by default. SmolLM3 used the Llama 3.2 tokenizer almost unchanged. A well-tested tokenizer with a vocabulary of around 100,000 to 150,000 tokens handles most languages and code well.
• Extend it for your domain if your identifiers fragment. Check how your documents tokenize. If a container number, tax ID or product code becomes eight meaningless pieces, the model has to learn to reassemble it every time, which costs accuracy and context length. Adding domain tokens fixes this.
• Mind the embedding cost. In a 1B model the embedding table is a large share of all parameters. Tying the input and output embeddings, as SmolLM3 does, saves that space with little loss.

Step 3: architecture

Start from a proven design and change only what you have tested. The baseline that most 2026 small models share:

• A decoder-only transformer in the Llama style, with RMSNorm and SwiGLU feed-forward layers.
• Grouped-query attention (GQA): several query heads share each key-value head. Quality matches full multi-head attention while the memory for long contexts shrinks several times.
• Document masking: when several documents are packed into one training sequence, tokens cannot attend across document boundaries.
• Positional encoding for long context: rotary embeddings (RoPE) with a raised base frequency, extended at the end of training with YaRN. SmolLM3 also drops RoPE from every fourth layer (a "NoPE" hybrid) to help long-context behaviour.
• For models under about a billion parameters, several studies favour deeper, thinner networks over wide, shallow ones.

The newest large open models in 2026 add mixture-of-experts layers, sliding-window attention and hybrids that replace most attention layers with linear-attention variants such as Gated DeltaNet (used in the Qwen3.5 large models). These pay off mostly at large sizes or very long contexts. For a first domain model, they add risk without a clear gain.

Step 4: pretraining and its cost

The training settings themselves are well understood:

• Optimizer: AdamW remains the default. Muon is the main challenger: Moonshot AI reported about twice the compute efficiency of AdamW when training their Moonlight model. It is worth an ablation, not a blind switch.
• Learning-rate schedule: warmup-stable-decay (WSD), which keeps the rate flat for most of training and decays it at the end. You can stop, add data and resume without redoing the schedule.
• Ablations first: test each change on a small proxy model or a short run, one variable at a time. For SmolLM3, ablations and debugging used about 161,000 GPU hours, more than half of the roughly 276,000 the main run used.

How much compute? Training needs roughly 6 × parameters × tokens floating-point operations. An H100 delivers about 989 trillion operations a second in BF16 at peak; real training typically reaches 30% to 40% of that. At 30%:

These figures cover one clean pretraining run only. Budget at least as much again for ablations, failed runs, long-context extension and post-training, plus the engineering time, which is usually the biggest cost. Hardware choice is wider than it was: AMD's MI300X with ROCm now trains standard PyTorch models well, and its 192 GB of memory per GPU is generous for long-context work.

Step 5: mid-training and post-training

A pretrained model predicts text. Turning it into something that follows instructions takes several more stages:

• Long-context extension: a short extra phase on long documents with a raised RoPE base. SmolLM3 used 100 billion tokens to go from 4K to 64K, then YaRN to reach 128K at inference.
• Mid-training: a focused phase on task-shaped data, such as reasoning traces or your domain's question-answer pairs.
• Supervised fine-tuning (SFT): instruction-and-answer examples, with the loss computed only on the answers.
• Preference optimization: DPO or a variant such as APO teaches the model which of two answers is better. It is much simpler to run than reinforcement learning with a separate reward model.
• Model merging: averaging the weights of checkpoints from different stages often recovers abilities a later stage damaged. SmolLM3 used it to restore long-context performance lost during alignment.

Step 6: quantization and deployment

A model is only useful where it can run. Quantization stores the weights at lower precision: at 4-bit, a 1B model's weights take about 0.5 GB instead of 2 GB at 16-bit, plus memory for the context (the KV cache). There are two ways to get there:

• Post-training quantization: compress a finished model. Quick, and usually fine at 8-bit, but small models can lose noticeable accuracy at 4-bit.
• Quantization-aware training (QAT): simulate low precision during training so the model learns to tolerate it. It costs some extra training but keeps far more accuracy at 4-bit. We use it for AiGanak-SLM-1B.

Before you deploy, compare the running cost with simply calling an API. Self-hosting has a fixed monthly cost whatever the volume, so it only wins above a certain number of requests. Our free self-host vs API calculator works out that break-even volume for your traffic, using current API prices.

Step 7: evaluation

Public benchmarks tell you whether training worked. They do not tell you whether the model does your job. Build your own test set before you train:

• A held-out domain test set: 200 to 1,000 real examples with correct answers, never used in training, checked by someone who knows the domain.
• A baseline: run the same test on the open models above and on the API model you use today. Your model has to beat the open models to justify itself, and get close enough to the API model to justify the switch.
• Long-context tests such as RULER if you rely on long documents, and a hallucination check: ask about facts that are not in the document and count how often the model invents an answer.
• Use standard harnesses such as lighteval or lm-evaluation-harness for public benchmarks, and validate your setup against published scores before you trust it.

What we learned building AiGanak-SLM-1B

AiGanak-SLM-1B is our in-house model: about 1 billion parameters, decoder-only, with a 131,072-token context window, built first for logistics and supply-chain documents such as manifests, tax identifiers, coordinates and timestamps. It is in Phase 2 of training and not yet released; we will publish benchmarks when evaluation is complete. Four choices from that work apply to most domain models:

• Domain tokens pay off. We added specialised BPE tokens for manifest and tax identifiers. After expanding the vocabulary, the model re-converged within 400 training steps, a small price for identifiers that stay whole.
• Long context needs to be designed in. We use YaRN-scaled rotary embeddings with a base of 1,000,000 so reasoning stays stable across very long audit trails and document sets.
• Plan for 4-bit from the start. A straight-through estimator in the training loop prepares the model for near-lossless 4-bit inference on ordinary local hardware.
• Tell the model what matters. A structured attention-steering mask prioritises tokens such as IDs, coordinates and timestamps, aimed at reducing invented values in messy real-world documents.

Companies where privacy, air-gapped deployment or local-first AI matter can request early access to the model.

A realistic project plan

A custom small model is best built in stages, each with a decision point, so you never spend months before seeing whether it works:

1. Proof of concept (2 to 4 weeks): collect a sample of your documents, build the test set, measure open models and your current API model, then fine-tune or continue pretraining a strong open model. The result is an evaluation report that says whether a custom model will beat what you have.
2. MVP (1 to 3 months): scale the data, train the model (fine-tuned, continued or from scratch, depending on what the proof of concept showed), quantize it and put it in front of a pilot team.
3. Production: private deployment on your servers or in your cloud account, with monitoring, a retraining routine and a running-cost report.

This is how we run custom SLM development for clients, using the training pipeline, data tooling and evaluation methods built for AiGanak-SLM-1B. If you are weighing a custom model against your current API bill, start with the numbers: the free LLM Bill Analyzer shows what your API traffic costs today, and the break-even calculator shows when your own model would pay off.