# Lab — QLoRA fine-tune on the caption catalogue

**Time:** 120 minutes — 15 setup, 60 fine-tune, 25 evaluate, 10 evidence, 10 teardown. On the Mac the full run is about 15 minutes; on a free Kaggle T4 the measured full run is about **18 minutes** end to end (14.3 min training + ~2 min load/eval) at **5.78 GB** peak VRAM; on the faculty RTX Pro 6000 Blackwell it is about **4.5 minutes** (3.7 min training + ~40 s load/eval) at **5.76 GB**. Start the smoke run first so the full run is not the first time you meet the runtime.

**Prerequisite:** S8; Python 3.14; basic ML vocabulary. The data is bundled at `data/caption_catalogue.jsonl` — 1,000 synthetic rows, 800 train / 200 validation, course-owned, seed 42. The provenance, licence and fetch instructions are in [`data/README.md`](data/README.md); there is no runtime download.

**You need to know:** at batch 1, inference is memory-bound; precision divides the weights' memory bill; LoRA freezes the base and trains a small adapter; QLoRA adds a 4-bit base and memory-saving optimizer tricks.

**Safe-data acknowledgement:** I have read the lab safe-data policy. I will use only the course-supplied synthetic media and will not upload real personal media to any cloud service.

## Task

Fine-tune a 4-bit QLoRA adapter on the synthetic caption catalogue, compare the base and tuned captions with token-F1 on held-out rows, and explain when this beats prompting or retrieval.

## Choose your runtime

| Runtime | Notebook | Time | Cost |
|---|---|---|---|
| This Apple Silicon Mac | `notebook.ipynb` (`mlx-vlm`) | about 15 min | $0 |
| Kaggle T4 (free) | `notebook-cuda.ipynb` (`transformers`, `peft`, `bitsandbytes`) | ~18 min (measured: 855.4 s train, 5.78 GB VRAM) | $0 (30 GPU-hours/week) |
| Faculty RTX Pro 6000 | `notebook-cuda.ipynb` | ~4.5 min (measured: 219.8 s train, 5.76 GB VRAM) | $0 (faculty card) |
| RunPod RTX Pro 6000 (fallback) | `notebook-cuda.ipynb` | ~4.5 min + setup | about $1.69/hr, under $3.40 |

## Steps

### 1. Set up and run the smoke cell

**Mac.** Open `notebook.ipynb`. The install cell (the single documented `pip install` exception) checks imports first: in the course environment the MLX stack is already there, so it prints *already installed — nothing to do* and installs nothing. If you built the environment by hand it installs `mlx==0.32.4`, `mlx-metal==0.32.4` and `mlx-vlm[train]==0.7.12` once; restart the kernel after that run. For the smoke run, start the kernel with the two documented overrides — export them in the terminal you launch JupyterLab from, or edit the `S9_ITERS` and `S9_EVAL_ROWS` defaults in the configuration cell:

```bash
export S9_ITERS=2 S9_EVAL_ROWS=2   # smoke run; unset both (or use 100 and 10) for the full run
```

Then run the cells from the import cell down.

**Kaggle.** Create a private notebook and attach the catalogue as a **private dataset**. The notebook searches `/kaggle/input` for `caption_catalogue.jsonl`, so the dataset name and Kaggle's mount layout do not matter; to be explicit, set `COURSE_DATA` to the folder that contains the file. Set **Accelerator → GPU T4 ×2** (the notebook puts the model on one 16 GB GPU). Open `notebook-cuda.ipynb`, turn **Internet** on for the install cell, run it, restart the kernel, and set `ITERS=10`, `EVAL_ROWS=5` for the smoke run.

**Faculty RTX.** Use the same CUDA notebook; the training stack may already be installed. Confirm the GPU with `nvidia-smi` before starting.

Console/UI checks:

```bash
# Kaggle: Settings → Accelerator shows GPU T4 x2; Internet is ON for the install cell.
# Faculty RTX / RunPod:
nvidia-smi
python3 -c "import torch; print(torch.cuda.get_device_name(0))"
```

Record the GPU name, the smoke time and the peak memory. If the smoke run's loss does not fall, stop and diagnose before the full run.

### 2. Run the full fine-tune

Restore the full settings (`ITERS=100`, `EVAL_ROWS=20` on CUDA; `100` and `10` on MLX) and run from the data cell down. The notebook prints the loss every ten steps and writes the result JSON.

- **Mac:** `results/mlx_results.json`.
- **CUDA:** `results/cuda_results.json`.

Record the training time, evaluation time and peak memory. The expected MLX peak is about 4–5 GB; the measured CUDA peak is 5.8 GB on both the T4 and the RTX card, well under the T4's 15.6 GB. The measured 100-step T4 run takes 855 s; if your session is slower, set `ITERS=50`. If it runs out of memory, keep `MAX_SEQ_LEN=512` and batch 1, and rely on gradient accumulation.

### 3. Evaluate and read the samples

The notebook computes token-F1 for the base and the tuned model on the same held-out rows. Build the base-vs-tuned table:

| Model | Token-F1 | Seconds | Peak memory |
|---|---:|---:|---:|
| Base | | | |
| Tuned | | | |

Read three sample captions from the JSON. Say what the base got wrong and what the tuned model changed. Token-F1 is a teaching metric; it does not judge style, and it can miss a factual error.

### 4. Explain the choice

Write 150–250 words: when would this fine-tune beat retrieval over a larger frozen model, and when would retrieval win? Use your numbers — training time, adapter size, inference speed, update cost — and name one thing the unit would need to redo when the product list changes. This is the lab half of the research question.

### 5. Save the evidence and tear down

Copy the result JSON and the three samples out of the notebook before the session ends. On Kaggle, stop the session; the output is not a durable store. On RunPod, terminate the pod **and its network volume**; the container disk clears on stop but a network volume persists.

## Evidence bundle

- The result JSON (`mlx_results.json` or `cuda_results.json`) with runtime, model, time, peak memory and base/tuned F1.
- The base-vs-tuned table and three sample captions.
- A three-sentence *what this proves* note: what the base got wrong, what the tuned model changed, and why LoRA is cheaper than full training.
- A 150–250 word fine-tune-versus-retrieval argument with your numbers.
- A teardown log: session stopped or pod and volume terminated, adapter and outputs copied or deleted, GPU confirmed idle.

The bundle is scored 3/2/1/0.

## Failure and recovery

The deliberate failure is the **memory wall**: on a 16 GB T4, raise `MAX_SEQ_LEN` to 1024 (or `batch_size` above 1) and watch it run out of memory. Recover by returning to `MAX_SEQ_LEN=512`, batch 1, gradient accumulation and `ITERS=100` (or 50 if time is short). On the Mac, the same experiment with a larger sequence or batch shows the unified-memory limit; keep `grad_checkpoint=True` and the 4-bit model. The lesson is that the memory budget, not the parameter count alone, sets the feasible configuration.

If the loss stalls or the model diverges: confirm the chat template strips ` thinking`, check that `train_on_completions` is on, lower the learning rate, and compare a base sample to a tuned sample rather than trusting one scalar.

## Cost ceiling

$0 on this Mac, the free Kaggle T4 and the faculty RTX Pro 6000. A RunPod fallback stays under $3.40 at $1.69/hr for two hours; set `S9_HOURLY_COST` in the CUDA notebook to record the line. There is no persistent storage in the lab beyond the result JSON.

## Teardown

- **Mac:** delete the adapter and the `results/` folder after copying the JSON; confirm no process is using the GPU.
- **Kaggle:** stop the session; download the result JSON and samples first.
- **Faculty RTX:** stop the kernel; remove the adapter and outputs from the shared workspace.
- **RunPod:** terminate the pod **and the network volume**; confirm the billing page shows no running instance.

## Fallback

If the runtime cannot complete the fine-tune, use the precomputed prototype results in `prototypes/ai-inference-finetuning/results/` (`t4_smoke.json`, `t4_full.json`, `rtx_pro_6000_*.json`) and run the evaluation on the base model only; the evidence shape is unchanged but the note must say which row is measured and which is cited. If the 4B model does not fit, use Qwen3.5-2B and record the change. The T4 smoke run is the gate and has been verified; if a student's session fails, use the Mac MLX path or the faculty RTX card.

## Notebooks

- [Apple Silicon / MLX notebook](notebook.ipynb)
- [Kaggle T4 / RTX Pro 6000 notebook](notebook-cuda.ipynb)
