Most teams meet small language models through one decision: a large model is too slow, too expensive, or not allowed to see the data, so they need something smaller. That decision opens a design space that API-only systems never face. How big should the model be? Which architecture fits a phone's memory? Should it be distilled, pruned, or trained from scratch? How many bits can it lose before quality drops? Which engine serves it, and how do you prove it is good enough?

Small Language Model System Design answers those questions as a system designer would: with explicit requirements, computed numbers, and trade-offs you can defend. Each chapter defines its core concepts, explains every subtopic in depth, draws the architecture as a diagram, compares the options in tables, and ends with a lab that runs the idea on real code.

The numbers in the book come from scripts, not from memory. Parameter counts, KV cache sizes, memory plans, scaling-law fits, quantization error, serving capacity, and evaluation confidence intervals are all printed by code in the companion repository, so you can change an assumption and see how the design changes.

What is inside

Part I: Foundations

1. The SLM Landscape and When Small Wins

2. Transformer Mathematics for Small Models

3. Tokenization and Vocabulary Design

4. SLM Architecture Design Choices

5. Scaling Laws and Compute Budgeting

Part II: Building SLMs

6. Pretraining Data Curation and Synthetic Data

7. Pretraining an SLM From Scratch

8. Knowledge Distillation

9. Pruning and Structured Compression 10. Long Context for Small Models

Part III: Post-Training

11. Supervised Fine-Tuning

12. Preference Optimization and Alignment

13. Reasoning in Small Models

14. Parameter-Efficient Fine-Tuning

15. Tool Use, Structured Output, and Agentic SLMs

Part IV: Efficiency and Deployment

16. Quantization

17. Inference Optimization and Serving 18. On-Device and Edge Deployment

Part V: Quality, Safety, and Production 19. Evaluation and Benchmarking

20. Safety, Security, Domain Specialization, and MLOps

Part VI: Bonus 21. System Design Interview Preparation: twenty SLM design cases, each answered with a six-step framework and a three-layer architecture diagram

Back matter: a capstone project (a 1B domain model under 1 GB that beats a general 7B model on its domain), a learning path, a glossary of about 85 terms, an index, and APA 7 references.