LLM inference and fine-tuning

Lab: QLoRA fine-tune on the caption catalogue. PLO: PLO2 / PLO3. Points: 7.

Overview

The catalogue has 5,000 assets without captions. The unit could prompt a large model every time, retrieve similar captions and let a frozen model copy their shape, or fine-tune a small model on its own catalogue. The one big idea for this session: inference is limited by memory bandwidth, and fine-tuning buys behaviour while retrieval buys knowledge — the choice is about which one the task needs.

The lab makes the trade concrete: a 4-billion-parameter model, a 1,000-row synthetic caption catalogue, and a LoRA adapter trained on a Mac, a free T4 or the faculty RTX Pro 6000. The base model writes fluent marketing copy; the tuned model learns the unit's one-sentence, factual voice.

Lesson

Why inference is memory-bound

A generated token requires reading the model's weights and the KV cache. At batch 1, the arithmetic is small compared with the bytes moved, so the rate is set by memory bandwidth, not by floating-point throughput: roughly tokens/s ≈ bandwidth ÷ bytes moved per token. That is why a 4-bit model can be several times faster than the same model in FP16 on the same chip, and why batching changes the picture — more tokens per weight read raises arithmetic intensity until compute becomes the limit.

The media unit feels this as latency: a caption request that waits 20 seconds is a different product from one that waits 2.

Precision and quantisation

Precision is bytes per parameter. FP16 stores 2 bytes, FP8 stores 1 and FP4 stores 0.5, so the weights' memory bill divides by the same factor. Quantization is the act of rounding weights into that smaller format; it trades a little quality for a lot of memory and bandwidth.

The KV cache does not always shrink with the weights, and it grows with context length and batch size. Long media prompts and multiple captions per call can make the cache, not the weights, the surprise. The lab's 4-bit model fits in about 3 GB of weights but peaks near 4.4 GB on the Mac once the cache and activations are counted, and near 5.8 GB on CUDA during training.

LoRA and QLoRA

Full fine-tuning updates every weight and needs memory for gradients and optimizer state several times the model size. LoRA freezes the base and trains small low-rank matrices — the adapter — so the trainable share is a fraction of a percent. In this lab the CUDA path reports 32.46 million trainable parameters of 4.57 billion (0.71%); the MLX path uses rank 8 rather than 16 and trains roughly half that.

QLoRA combines a 4-bit base, LoRA adapters in higher precision, double quantisation and paged optimizers. That is what makes a 4-billion-parameter fine-tune fit a free 16 GB T4. The adapter is small and portable between runs of the same model, but it is not interchangeable across model families or runtimes: the MLX and CUDA adapters are different files.

Fine-tuning buys behaviour; retrieval buys knowledge

Need Fine-tuning Retrieval Prompting
A consistent style or format Strong Weak Fragile
Facts that change weekly Poor — retrain Strong Depends on the prompt
Citations and access control Weak Strong Weak
Small, one-off task Overkill Overkill Good
Private data stays local Possible Possible Depends on the service
Update cost A training run A document update A prompt edit

A useful decision rule: fine-tune when the gap is behaviour (voice, structure, refusal, classification), retrieve when the gap is knowledge (fresh facts, documents, permissions), and prompt when the task is small enough that neither earns its cost. The research question asks the student to defend that rule with numbers from the lab.

The lab: 4B parameters, 1,000 rows

The catalogue is 800 training rows and 200 validation rows, generated with seed 42. The base and tuned models are scored with token-F1 on held-out rows: a teaching metric that rewards factual overlap and punishes fluent filler, not a judge of style.

The measured MLX path loads the 4-bit model in about 40 seconds, trains 100 iterations in about 7 minutes and peaks at 4.42 GB. The CUDA path is measured on both GPUs: the faculty RTX Pro 6000 trains 100 steps in 219.8 s and peaks at 5.76 GB; a free Kaggle T4 trains the same 100 steps in 855.4 s and peaks at 5.78 GB. The device changes the wall-clock time, not the learning: token-F1 rises from 0.5444 to 0.7746 on the RTX and from 0.5449 to 0.7749 on the T4.

What the numbers do and do not show

Token-F1 can rise while the captions become less readable, and it can miss a factual error that the metric's word overlap happens to reward. The samples matter: read three base and three tuned captions and say what changed. A model that learns the unit's voice but repeats a wrong licence is not a successful fine-tune.

The economics

Training is a one-off cost; inference is the recurring one. A 4-bit adapter makes both smaller. The next session compares the Mac, the T4 and the RTX Pro 6000 on the same task, so this session records the time and memory that make that comparison possible.

Check yourself

  1. Why can a 4-bit model generate faster than the same model in FP16, and what limits the gain?
  2. What does LoRA freeze, and why does that make the fine-tune fit a 16 GB card?
  3. A client asks for captions in a new brand voice, and the product list changes every week. Which do you fine-tune, and which do you retrieve?
สรุปภาษาไทย แคตตาล็อกมี 5,000 แอสเซ็ตที่ยังไม่มีคำบรรยาย หน่วยสื่อจะเรียกโมเดลใหญ่ทุกครั้ง ค้นคำบรรยายที่คล้ายกันแล้วให้โมเดลที่แช่แข็งลอกรูปแบบ หรือปรับโมเดลเล็กบนแคตตาล็อกของตัวเอง แนวคิดหลักคือการอนุมานถูกจำกัดด้วยแบนด์วิดท์หน่วยความจำ และการปรับโมเดลซื้อพฤติกรรม ในขณะที่การค้นคืนซื้อความรู้ ทางเลือกจึงขึ้นกับว่างานต้องการสิ่งใด การสร้างโทเคนหนึ่งตัวต้องอ่านน้ำหนักและ KV cache ที่ batch 1 การคำนวณน้อยกว่าไบต์ที่เคลื่อนที่ อัตราจึงขึ้นกับแบนด์วิดท์ ความแม่นยำคือไบต์ต่อพารามิเตอร์ FP16 สองไบต์ FP8 หนึ่งไบต์ FP4 ครึ่งไบต์ LoRA แช่แข็งโมเดลฐานและฝึกเมทริกซ์ขนาดเล็ก ส่วน QLoRA ใช้โมเดล 4 บิตกับ LoRA จึงพอดีกับการ์ด 16 จิกะไบต์ การปรับโมเดลเหมาะกับสไตล์และรูปแบบ การค้นคืนเหมาะกับข้อเท็จจริงที่เปลี่ยนบ่อย
中文总结 目录里有 5,000 项尚无字幕的资产。媒体单元可以每次都调用大模型,检索相似字幕再让冻结的模型照抄格式,或者在自己的目录上微调一个小模型。核心概念是:推理受内存带宽限制;微调购买的是行为,而检索购买的是知识,因此选择取决于任务需要什么。生成一个 token 必须读取权重和批大小为 1 时的 KV cache,计算量小于移动的字节数,所以速率取决于带宽。精度就是每个参数的字节数:FP16 两字节、FP8 一字节、FP4 半字节。LoRA 冻结基础模型并训练小矩阵,QLoRA 则用 4 位模型加 LoRA,因此能放进 16 GB 的显卡。微调适合风格和格式,检索适合经常变化的事实。

Interactive: the precision stack

The scene shows the same 4-billion-parameter model at FP16, FP8 and FP4. Switch precision and watch the weights, KV cache and activations change height. The totals are teaching estimates for a 4B model; the lab records the measured peak.

FP4 · 0.5 bytes/param · weights 2.0 GB + KV 0.5 GB + activations 0.5 GB ≈ 3.0 GB. Inference is memory-bound: tokens/s ≈ bandwidth ÷ bytes moved.

The stack in words. Weights dominate the memory bill: 8 GB at FP16, 4 GB at FP8 and 2 GB at FP4 for a 4B model. The KV cache adds 2, 1 and 0.5 GB, and activations add about 1, 0.7 and 0.5 GB. The totals are about 11, 5.7 and 3 GB. Fewer bytes per parameter means fewer bytes moved per token, which is why quantization raises throughput until compute or the cache takes over.

Mode Bytes/param Weights KV cache Activations Approx. total
FP16 2.0 8.0 GB 2.0 GB 1.0 GB 11.0 GB
FP8 1.0 4.0 GB 1.0 GB 0.7 GB 5.7 GB
FP4 0.5 2.0 GB 0.5 GB 0.5 GB 3.0 GB

The table is the same information as the scene; the lab measures the real peak, which includes framework overhead and the adapter.

Prerequisite: A phone-verified Kaggle account, or an Apple-silicon Mac with the kit's S9 dependency group; internet access for Kaggle's install cell. Set it up in Session 0: Accounts, Tools, The kit.

Lab — QLoRA fine-tune on the caption catalogue

Time: 120 minutes — 15 setup, 60 fine-tune, 25 evaluate, 10 evidence, 10 teardown. On the Mac the full run is about 15 minutes; on a free Kaggle T4 the measured full run is about 18 minutes end to end (14.3 min training + ~2 min load/eval) at 5.78 GB peak VRAM; on the faculty RTX Pro 6000 Blackwell it is about 4.5 minutes (3.7 min training + ~40 s load/eval) at 5.76 GB. Start the smoke run first so the full run is not the first time you meet the runtime.

Prerequisite: S8; Python 3.14; basic ML vocabulary. The data is bundled at data/caption_catalogue.jsonl — 1,000 synthetic rows, 800 train / 200 validation, course-owned, seed 42. The provenance, licence and fetch instructions are in data/README.md; there is no runtime download.

You need to know: at batch 1, inference is memory-bound; precision divides the weights' memory bill; LoRA freezes the base and trains a small adapter; QLoRA adds a 4-bit base and memory-saving optimizer tricks.

Safe-data acknowledgement: I have read the lab safe-data policy. I will use only the course-supplied synthetic media and will not upload real personal media to any cloud service.

Task

Fine-tune a 4-bit QLoRA adapter on the synthetic caption catalogue, compare the base and tuned captions with token-F1 on held-out rows, and explain when this beats prompting or retrieval.

Choose your runtime

Runtime Notebook Time Cost
This Apple Silicon Mac notebook.ipynb (mlx-vlm) about 15 min $0
Kaggle T4 (free) notebook-cuda.ipynb (transformers, peft, bitsandbytes) ~18 min (measured: 855.4 s train, 5.78 GB VRAM) $0 (30 GPU-hours/week)
Faculty RTX Pro 6000 notebook-cuda.ipynb ~4.5 min (measured: 219.8 s train, 5.76 GB VRAM) $0 (faculty card)
RunPod RTX Pro 6000 (fallback) notebook-cuda.ipynb ~4.5 min + setup about $1.69/hr, under $3.40

Steps

1. Set up and run the smoke cell

Mac. Open notebook.ipynb. The install cell (the single documented pip install exception) checks imports first: in the course environment the MLX stack is already there, so it prints already installed — nothing to do and installs nothing. If you built the environment by hand it installs mlx==0.32.4, mlx-metal==0.32.4 and mlx-vlm[train]==0.7.12 once; restart the kernel after that run. For the smoke run, start the kernel with the two documented overrides — export them in the terminal you launch JupyterLab from, or edit the S9_ITERS and S9_EVAL_ROWS defaults in the configuration cell:

export S9_ITERS=2 S9_EVAL_ROWS=2   # smoke run; unset both (or use 100 and 10) for the full run

Then run the cells from the import cell down.

Kaggle. Create a private notebook and attach the catalogue as a private dataset. The notebook searches /kaggle/input for caption_catalogue.jsonl, so the dataset name and Kaggle's mount layout do not matter; to be explicit, set COURSE_DATA to the folder that contains the file. Set Accelerator → GPU T4 ×2 (the notebook puts the model on one 16 GB GPU). Open notebook-cuda.ipynb, turn Internet on for the install cell, run it, restart the kernel, and set ITERS=10, EVAL_ROWS=5 for the smoke run.

Faculty RTX. Use the same CUDA notebook; the training stack may already be installed. Confirm the GPU with nvidia-smi before starting.

Console/UI checks:

# Kaggle: Settings → Accelerator shows GPU T4 x2; Internet is ON for the install cell.
# Faculty RTX / RunPod:
nvidia-smi
python3 -c "import torch; print(torch.cuda.get_device_name(0))"

Record the GPU name, the smoke time and the peak memory. If the smoke run's loss does not fall, stop and diagnose before the full run.

2. Run the full fine-tune

Restore the full settings (ITERS=100, EVAL_ROWS=20 on CUDA; 100 and 10 on MLX) and run from the data cell down. The notebook prints the loss every ten steps and writes the result JSON.

  • Mac: results/mlx_results.json.
  • CUDA: results/cuda_results.json.

Record the training time, evaluation time and peak memory. The expected MLX peak is about 4–5 GB; the measured CUDA peak is 5.8 GB on both the T4 and the RTX card, well under the T4's 15.6 GB. The measured 100-step T4 run takes 855 s; if your session is slower, set ITERS=50. If it runs out of memory, keep MAX_SEQ_LEN=512 and batch 1, and rely on gradient accumulation.

3. Evaluate and read the samples

The notebook computes token-F1 for the base and the tuned model on the same held-out rows. Build the base-vs-tuned table:

Model Token-F1 Seconds Peak memory
Base
Tuned

Read three sample captions from the JSON. Say what the base got wrong and what the tuned model changed. Token-F1 is a teaching metric; it does not judge style, and it can miss a factual error.

4. Explain the choice

Write 150–250 words: when would this fine-tune beat retrieval over a larger frozen model, and when would retrieval win? Use your numbers — training time, adapter size, inference speed, update cost — and name one thing the unit would need to redo when the product list changes. This is the lab half of the research question.

5. Save the evidence and tear down

Copy the result JSON and the three samples out of the notebook before the session ends. On Kaggle, stop the session; the output is not a durable store. On RunPod, terminate the pod and its network volume; the container disk clears on stop but a network volume persists.

Evidence bundle

  • The result JSON (mlx_results.json or cuda_results.json) with runtime, model, time, peak memory and base/tuned F1.
  • The base-vs-tuned table and three sample captions.
  • A three-sentence what this proves note: what the base got wrong, what the tuned model changed, and why LoRA is cheaper than full training.
  • A 150–250 word fine-tune-versus-retrieval argument with your numbers.
  • A teardown log: session stopped or pod and volume terminated, adapter and outputs copied or deleted, GPU confirmed idle.

The bundle is scored 3/2/1/0.

Failure and recovery

The deliberate failure is the memory wall: on a 16 GB T4, raise MAX_SEQ_LEN to 1024 (or batch_size above 1) and watch it run out of memory. Recover by returning to MAX_SEQ_LEN=512, batch 1, gradient accumulation and ITERS=100 (or 50 if time is short). On the Mac, the same experiment with a larger sequence or batch shows the unified-memory limit; keep grad_checkpoint=True and the 4-bit model. The lesson is that the memory budget, not the parameter count alone, sets the feasible configuration.

If the loss stalls or the model diverges: confirm the chat template strips thinking, check that train_on_completions is on, lower the learning rate, and compare a base sample to a tuned sample rather than trusting one scalar.

Cost ceiling

$0 on this Mac, the free Kaggle T4 and the faculty RTX Pro 6000. A RunPod fallback stays under $3.40 at $1.69/hr for two hours; set S9_HOURLY_COST in the CUDA notebook to record the line. There is no persistent storage in the lab beyond the result JSON.

Teardown

  • Mac: delete the adapter and the results/ folder after copying the JSON; confirm no process is using the GPU.
  • Kaggle: stop the session; download the result JSON and samples first.
  • Faculty RTX: stop the kernel; remove the adapter and outputs from the shared workspace.
  • RunPod: terminate the pod and the network volume; confirm the billing page shows no running instance.

Fallback

If the runtime cannot complete the fine-tune, use the precomputed prototype results in prototypes/ai-inference-finetuning/results/ (t4_smoke.json, t4_full.json, rtx_pro_6000_*.json) and run the evaluation on the base model only; the evidence shape is unchanged but the note must say which row is measured and which is cited. If the 4B model does not fit, use Qwen3.5-2B and record the change. The T4 smoke run is the gate and has been verified; if a student's session fails, use the Mac MLX path or the faculty RTX card.

Notebooks

Downloads for this session

Each document name opens a rendered view you can read in the browser. The download link beside it gives you the raw Markdown working file: the copy you edit, run and submit. The page above and the lab steps carry the teaching.

lab.md download lab.md
The numbered lab brief: the steps, the commands and the evidence you submit.
refs.md download refs.md
The reference list behind this session's claims.
PLO.md download PLO.md
This session's scoring rows and descriptors.
download notebook.ipynb
The runnable notebook that produces the lab's numbers.

Research question

For a small media team with domain data, when does a small fine-tuned model beat retrieval over a larger frozen model?

Submit an 800–1,200 word answer with 5–8 sources at the next session. The rubric and the exemplar guidance show the expected shape.

References — Session 9

  1. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2106.09685
  2. Makes fine-tuning a small model affordable by training low-rank adapters instead of all weights — the method the lab uses. Adapter capacity and rank selection for multimodal media tasks are still empirical. The LoRA adapter diagram needs a redraw or permission.
  3. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems 36. https://arxiv.org/abs/2305.14314
  4. Shows how 4-bit quantisation plus LoRA fits a useful model into a free T4, which is the lab's constraint. The quality cost of 4-bit fine-tuning on small domain datasets is task-dependent and under-reported. The memory diagram is a redraw candidate.
  5. Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 33. https://arxiv.org/abs/2005.11401
  6. The alternative to fine-tuning: keep the model frozen and supply documents, which changes cost, freshness and governance. RAG versus fine-tuning for media-domain knowledge has no settled cost model. The RAG pipeline can be rebuilt as a node-graph scene.
  7. Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361. https://arxiv.org/abs/2001.08361
  8. Explains why model size, data and compute trade against each other — the numbers behind the week's memory-bandwidth discussion. Scaling laws for small, domain-specific models and inference-optimised architectures are less clear. The loss curves can be redrawn from the reported data.

Teaching note. Narrate Hu et al. (2022) for 5–8 minutes, assign the abstract of Dettmers et al. (2023) as reading, and close by restating the open problem as the weekly question. Figures are third-party; redraw by default.

PLO — Session 9: LLM inference and fine-tuning

CLO: By the end of this session, the student can explain why inference is memory-bound, fine-tune a 4-bit QLoRA adapter on a small catalogue, and defend a choice between fine-tuning, retrieval and prompting with measured time and token-F1.

PLO Role Research answer Lab evidence Points Evidence Rubric
PLO2 primary 4 3 7 An 800–1,200-word fine-tune-versus-retrieval argument with 5–8 sources; the results JSON, base-vs-tuned table, three samples, the cost note and the teardown Research 4/3/2/1; Lab 3/2/1/0
PLO3 secondary 0 0 0 Lecture context only; not scored —
Total 4 3 7

Total: 7 points

Four-level descriptors

Research (4/3/2/1). 4 = a decision rule that separates behaviour from knowledge, applied to the unit's captioning task with measured time, memory and token-F1; cost, latency, update cadence and privacy compared; a counterargument (retrieval is cheaper to update) weighed. 3 = a sound rule with the lab numbers and some comparison. 2 = describes fine-tuning and RAG without choosing or using measurements. 1 = a summary with fewer than five sources. 0 = nothing submitted or misconduct.

Lab (3/2/1/0). 3 = the smoke and full runs complete, the result JSON records runtime, time, peak memory and base/tuned F1, the samples are read and explained, the fine-tune-versus-retrieval argument uses the numbers, and teardown is complete. 2 = mostly complete, one check thin. 1 = a partial run or an unrecorded measurement. 0 = nothing submitted, unsafe data, or no teardown.

The full course arithmetic is in production/baseline.md; this file is the weekly instance.