Course contents
Project 6: Go Local, Then Fine-Tune a Specialist
Answer two questions with numbers: can a local model handle triage, and can a small fine-tuned model match the cloud model on it? Use the Project 5 evals to decide what the desk should run.
Triage is the desk's highest-volume step and its simplest decision, which makes it the natural candidate for a cheaper, private model. This project tests that idea properly: a baseline, a local prompted model, a fine-tuned small model, the same evals for all three, and a written decision. A negative result (“the fine-tune didn't beat the prompted model”) is a perfectly good outcome if the numbers show it.
| Stage | What you produce |
|---|---|
| Baseline | Your current cloud model on the triage evals: accuracy, p95 latency, cost per 1,000 tickets |
| Local, prompted | Gemma 4 through Ollama (or LM Studio) with the same prompt and schema, on the same evals |
| Training data | 800 to 1,500 labeled tickets, a 10% held-out split, and a hand-reviewed sample |
| Fine-tune | A LoRA adapter on a small model such as Gemma 4 E4B, with train and eval loss logged |
| Compare and decide | All three models in one table, and a paragraph saying what triage should use |
format field; validate the result with Pydantic either way, exactly as you do for cloud models.Baseline the cloud and local models
Run the Project 5 triage cases against your cloud model and against
gemma4through Ollama with the same prompt and schema. Record accuracy, p95 latency and cost per 1,000 tickets for each (for local, note the hardware instead of a token price).VerifyTwo rows of your comparison table are filled in from real runs.Build the training set
Generate about 1,200 varied synthetic tickets with a frontier model, covering every label and your known hard cases. Label them with your best prompted model, hand-review 100, fix what's wrong, and split 90/10. Write them as chat-format JSONL, one example per line (the sample below is wrapped for reading). Keep every Project 5 eval case out of the training data.
json{"messages": [ {"role": "system", "content": "You triage Acme support emails. Reply with JSON only."}, {"role": "user", "content": "<email>My parcel never arrived and it's been two weeks.</email>"}, {"role": "assistant", "content": "{\"category\": \"shipping\", \"urgency\": \"high\", \"order_id\": null, \"needs_human\": false}"} ]}Verifytrain.jsonlandeval.jsonlexist, no eval case appears in either, and your review notes list what you fixed.Train a LoRA adapter
Run the TRL and PEFT script from lesson aa-16 on a GPU with enough memory, or use Unsloth's free Colab notebook for Gemma 4 and swap in your data. Start with r = 16, a learning rate of 2e-4 and two epochs, and watch eval loss after each epoch.
VerifyYour log shows eval loss after every epoch, and the run you keep is the one where eval loss was still falling or flat.Merge and serve it locally
Merge the adapter into the base weights with PEFT (
merge_and_unload(), thensave_pretrained). Ollama imports Safetensors weights for supported architectures withFROM <directory>in a Modelfile; otherwise convert to GGUF first (Unsloth and llama.cpp both export it) and useFROM ./model.gguf. Thenollama create acme-triage-ft -f Modelfile.Verifyollama run acme-triage-ftanswers a sample ticket with valid triage JSON.Evaluate on unseen cases
Run the Project 5 triage cases and your held-out split through the fine-tuned model, and add its row to the table.
VerifyAll three rows are filled in, from runs on cases none of the models trained on.Decide and wire it in
Write the decision in a paragraph with the numbers. If a local model wins, point Project 1's model config at the local OpenAI-compatible endpoint and keep the cloud model as a fallback behind the same interface.
VerifyThe decision is in your repo, and switching triage between local and cloud is a single config change.
Plain-English definitions from the AI Coding Dictionary.