Task-Specific LLM Distillation (7B → 1.5B)

01 / OVERVIEW

The idea behind it.

A knowledge-distillation system that trades general capability for deployment efficiency on one narrow task. The premise: a 7B model answering backend debugging questions is overkill at inference time — it costs memory, latency, and money per call, and struggles on modest hardware. Instead, a 7B teacher generates a large corpus of structured debugging answers, and a 1.5B student is fine-tuned with LoRA adapters to reproduce that behavior within the narrow domain. The student emits a fixed JSON schema — root cause, ordered debugging steps, recommended fix — across Docker, Kubernetes, Python, FastAPI, Redis, SQL, AWS, CI/CD, and networking failures. Training runs locally on Apple Silicon through MLX, so no cloud compute is involved. NOTE: this entry documents the system design and training pipeline. The model is still being trained, and no accuracy, latency, or memory figures are claimed yet — those will be published here once the held-out evaluation has actually been run.

02 / FEATURES

What it does.

• 01Teacher–student distillation pipeline targeting a single narrow domain
• 02Fixed JSON output schema: root cause, ordered debugging steps, recommended fix
• 03Synthetic dataset generated by a locally-hosted teacher model — no external API
• 04Coverage across nine failure domains: Docker, Kubernetes, Python, FastAPI, Redis, SQL, AWS, CI/CD, networking
• 05LoRA adapters over frozen student weights rather than full-parameter fine-tuning
• 06Local training on Apple Silicon via MLX, using unified memory instead of cloud GPUs
• 07Held-out evaluation harness comparing teacher and student on schema validity, root-cause agreement, latency, and memory footprint

03 / CHALLENGES

The problems worth solving.

• 01Keeping teacher-generated output schema-valid enough to train on without hand-cleaning thousands of examples
• 02Teacher error propagation — the student inherits the teacher's wrong answers as ground truth
• 03Avoiding homogeneity in a synthetically generated dataset, where prompts tend to collapse toward similar phrasings
• 04Defining an objective accuracy metric for free-text debugging advice, where many phrasings are equally correct
• 05Fitting the training run inside laptop unified memory alongside the model and optimizer state
• 06Choosing LoRA rank and alpha that specialize behavior without erasing the student's general reasoning

04 / LEARNINGS

What I took away.

• 01How knowledge distillation transfers narrow capability from a large model into a small one
• 02Why LoRA trains low-rank update matrices against frozen weights, and the memory math that makes it viable on a laptop
• 03The real inference cost argument: memory footprint, per-call latency, and hardware floor — not just accuracy
• 04When a specialized small model is the better engineering choice than a general large one
• 05That evaluation design, not training, is the hard part of a distillation project — a benchmark table is only worth as much as the methodology behind it
• 06Local training workflows on Apple Silicon with MLX and unified memory