# Lab — Latency, sizing and the edge

**Time:** 120 minutes — 10 setup, 80 core, 20 evidence, 10 teardown.

**Prerequisite:** the S1 account and project, the `course-s1-<id>` SSH key and the root domain you registered in S1 self-study. The Cloudflare zone must be **Active** before this lab, because step 3 puts `s2.<root>` behind the edge.

**You need to know:** latency is time to the first byte; bandwidth is bytes per second; a faster link does not shorten the wait before the first byte. A p95 is the value 95% of requests finish within, so it describes the tail rather than the average. DigitalOcean has no region in Thailand, so the origin is in Singapore (SGP1) and Cloudflare's edge carries the request from Chiang Mai. A Basic Droplet shares its vCPU with other customers and has no CPU-credit balance; benchmark it to learn what it delivers. Cloudflare caches a `.jpg` by default and does not cache `.html` by default. SSL mode **Full** encrypts the edge-to-origin leg but does not validate the origin certificate.

**Safe-data acknowledgement:** I have read the lab safe-data policy. I will use only the course-supplied synthetic media and will not upload real personal media to any cloud service.

## Task

Put one 1 GB Rocky Linux 10 Droplet in SGP1 behind a proxied `s2.<root>` record with SSL **Full**. Measure the same JPEG from the origin and through the edge, watch it go `MISS → HIT` while the HTML control stays uncached, benchmark the CPU and disk, exhaust the instance's memory, resize it to 2 vCPU / 2 GB without growing the disk, and rerun.

Work from the downloaded week folder. Commands marked **Laptop** run on your machine; commands marked **Droplet** run after you SSH in. Panel paths sit beside the CLI step where both exist. A step with no CLI path says so.

## Steps

### 1. Pre-flight and token regeneration

Start with the zone and the account, then regenerate the S1 token. The S1 token cannot be edited, so the resize later would be refused with a 403. That refusal is the lesson: create a new token with the extra scopes and delete the old one.

**Panel:** Cloudflare dashboard → your zone → check **Active**. DigitalOcean Control Panel → **API** → **Tokens & Keys** → **Generate New Token**.

Create `course-s2-<id>` with the S1 scopes plus two more:

| Scope group | Scopes |
|---|---|
| Droplet | `droplet:read`, `droplet:create`, `droplet:delete`, **`droplet:update`** |
| SSH key | `ssh_key:read`, `ssh_key:create` |
| Firewall | `firewall:read`, `firewall:create`, `firewall:update`, `firewall:delete` |
| Project | `project:read`, `project:create`, `project:assign_resource` |
| Tag | `tag:read`, `tag:create` |
| Account and reads | `account:read`, `regions:read`, `sizes:read`, `image:read`, `billing:read` |

Set the expiry to 90 days and copy the secret once. If `doctl` does not recognise the new token, add it as a context; the macOS config path differs from Linux.

**Laptop:**

```bash
export DIGITALOCEAN_ACCESS_TOKEN=<new-token>
doctl account get
```

`doctl account get` proves the token works and returns the account status. The token value is a credential: never paste it into a file that enters the repository, and keep the config at `chmod 600`.

**Laptop:**

```bash
dig NS <root> +short
doctl projects list --format ID,Name
doctl compute ssh-key list --format FingerPrint,Name
```

`dig NS` must return exactly two Cloudflare nameservers. `doctl projects list` gives the `<project-uuid>` to attach the Droplet to; `doctl compute ssh-key list` gives the `<fingerprint>` of the `course-s1-<id>` key you created in S1.

**Panel:** delete the old `course-s1-<id>` token. The new token keeps the same least privilege plus the resize scopes.

### 2. Create the week-2 firewall

S1's firewall is attached by the tag `week:1`; this lab owns a separate rule set, so a changed laptop IP does not depend on the old rule.

**Laptop:**

```bash
doctl compute tag create "week:2"
```

A cloud firewall can only reference a tag that already exists, so create the tag first. Step 3's Droplet carries it, and the firewall applies as soon as the tag is attached.

**Panel:** Manage → Tags → **Create Tag** `week:2`, then Networking → Firewalls → **Create Firewall** — name `course-<id>-s2`, inbound TCP 22 from **your current IP only** and TCP 443 from anywhere, outbound all, applied to the tag `week:2`.

**Laptop:**

```bash
MYIP=$(curl -s https://api.ipify.org)
doctl compute firewall create --name "course-<id>-s2" \
  --inbound-rules "protocol:tcp,ports:22,address:$MYIP/32 protocol:tcp,ports:443,address:0.0.0.0/0" \
  --outbound-rules "protocol:tcp,ports:0,address:0.0.0.0/0 protocol:udp,ports:0,address:0.0.0.0/0 protocol:icmp,ports:0,address:0.0.0.0/0" \
  --tag-names "week:2"
```

`MYIP` resolves your public address, so SSH is open only to you. Port 443 is open to everyone because Cloudflare's edge connects from many addresses, not one. The output is the `<firewall-id>`.

**Laptop:**

```bash
doctl compute firewall get <firewall-id> -o json > data/firewall-rules.json
```

Save the rule set as evidence: the inbound SSH source and the 443 rule are what the marker checks.

### 3. Create the Droplet

**Panel:** Droplets → **Create Droplet** — `course-<id>-s2`, region **SGP1**, image **Rocky Linux 10 x64**, plan **Basic / 1 vCPU / 1 GB / 25 GiB**, the `course-s1-<id>` SSH key, tags `course:42-05-0112`, `student:<id>`, `week:2`, project `course-<id>`.

**Laptop:**

```bash
doctl compute droplet create "course-<id>-s2" \
  --region sgp1 --image rockylinux-10-x64 --size s-1vcpu-1gb \
  --ssh-keys <fingerprint> \
  --tag-names "course:42-05-0112,student:<id>,week:2" \
  --project-id <project-uuid> \
  --wait
```

`--region sgp1` is DigitalOcean's nearest region to Thailand; there is no Thailand region. `--wait` returns when the Droplet is active. The command prints the `<droplet-id>`.

**Laptop:**

```bash
doctl compute droplet list -o json > data/droplet-before.json
doctl compute droplet get <droplet-id> --format PublicIPv4 --no-header
```

The JSON is the "before" inventory. The second command prints the `<droplet-ip>` you use for SSH and for the origin leg of the latency test.

### 4. Add the proxied record

**Panel:** Cloudflare dashboard → **DNS** → **Records** → **Add record** — type **A**, name `s2`, IPv4 address `<droplet-ip>`, proxy status **Proxied**, TTL **Auto**. There is no CLI path in this lab; the API-token path is an extension in Session 4.

**Laptop:**

```bash
dig +short s2.<root>
```

The answer must be Cloudflare anycast addresses, not `<droplet-ip>`. If it returns the Droplet address, the record is not proxied: edit it and switch the orange cloud on. Propagation is usually seconds.

**API-token extension (reference runner only).** The panel path is the student path. The reference runner has no browser, so it creates the same record with a scoped Cloudflare token and records the API call in its command log.

```bash
curl -s -X POST -H "Authorization: Bearer <cloudflare-token>" -H 'Content-Type: application/json' \
  -d '{"type":"A","name":"s2","content":"<droplet-ip>","ttl":1,"proxied":true}' \
  "https://api.cloudflare.com/client/v4/zones/<zone-id>/dns_records"
```

The response contains the `<record-id>` used by the teardown. The token needs `Zone:DNS:Edit`; it never appears in the command log.

### 5. Build the HTTPS origin

**Laptop:**

```bash
ssh -i ~/.ssh/do_s1 root@<droplet-ip>
```

Rocky's cloud image accepts the key only; the login is `root`.

**Droplet:**

```bash
dnf install -y python3 openssl
```

Install the interpreter and the certificate tool. The cloud image usually carries both; the command is idempotent.

**Droplet:**

```bash
systemctl is-active firewalld && { firewall-cmd --permanent --add-service=https; firewall-cmd --reload; } || true
```

If Rocky's host firewall is active, open 443 inside the VM as well. The DigitalOcean cloud firewall is the outer filter; this is the inner one. A host with firewalld blocking 443 returns 522 at the edge.

**Droplet:**

```bash
mkdir -p /srv/s2 && cd /srv/s2
cat > index.html <<'HTML'
<!doctype html>
<html lang="en">
<head><meta charset="utf-8"><title>S2 origin</title></head>
<body><h1>S2 origin</h1></body>
</html>
HTML
```

Create the served directory and the **HTML control**. It has no cache-friendly extension, so Cloudflare will not cache it by default; it is the comparison for the JPEG.

**Droplet:**

```bash
openssl req -x509 -newkey rsa:2048 -nodes -keyout key.pem -out cert.pem \
  -days 2 -subj "/CN=s2.<root>"
```

Generate a self-signed certificate for the hostname. Cloudflare's **Full** mode accepts it because Full encrypts but does not validate; Full (strict) would reject it. The certificate is temporary by design.

**Laptop** (open a second terminal, stay in the week folder):

```bash
scp -i ~/.ssh/do_s1 data/thumb-sample.jpg root@<droplet-ip>:/srv/s2/thumb-<id>.jpg
scp -i ~/.ssh/do_s1 serve.py root@<droplet-ip>:/srv/s2/serve.py
```

Upload the synthetic JPEG under a unique name and the origin server. The JPEG is the cacheable object; the unique name keeps your run out of another student's edge copy.

**Droplet:**

```bash
cd /srv/s2
nohup python3 serve.py --port 443 --dir /srv/s2 --cert cert.pem --key key.pem > serve.log 2>&1 &
sleep 2
curl -sk https://127.0.0.1/thumb-<id>.jpg -o /dev/null -w '%{http_code}\n'
```

Start the HTTPS server on 443. The `curl` uses `-k` because the certificate is self-signed; it must print `200`. A `000` means the server is not running; read `serve.log`.

**Droplet:**

```bash
nproc; lscpu | grep -E 'Model name|^CPU\(s\)'; free -h; df -h /
```

Record the "before" system facts: one vCPU, about 1 GiB of memory, a 25 GiB disk. These lines belong in the benchmark evidence.

### 6. Set SSL Full and observe the cache

**Panel:** Cloudflare dashboard → **SSL/TLS** → **Overview** → select **Full**. There is no CLI path in this lab.

**API-token extension (reference runner only).** The runner sets and later restores the same setting through the zone API:

```bash
curl -s -X PATCH -H "Authorization: Bearer <cloudflare-token>" -H 'Content-Type: application/json' \
  -d '{"value":"full"}' "https://api.cloudflare.com/client/v4/zones/<zone-id>/settings/ssl"
```

The token needs `Zone:Zone Settings:Edit`. The runner records the previous mode and restores it at teardown, because a shared demo zone may serve other hostnames.

**Laptop:**

```bash
curl -sD - -o /dev/null "https://s2.<root>/thumb-<id>.jpg" | grep -i cf-cache-status
```

The first request arrives at the nearest colo, misses, fetches the origin and stores the JPEG. Expect `cf-cache-status: MISS` and HTTP 200. A 522 means the edge could not reach the origin: check the Droplet's server and its host firewall.

**Laptop:**

```bash
curl -sD - -o /dev/null "https://s2.<root>/thumb-<id>.jpg" | grep -i cf-cache-status
```

The second request is served from the colo. Expect `cf-cache-status: HIT`. The default edge TTL for a JPEG is two hours.

**Laptop:**

```bash
curl -sD - -o /dev/null "https://s2.<root>/index.html" | grep -i cf-cache-status
```

The HTML control has no `HIT`: Cloudflare does not cache `.html` by default, so this request reaches the origin every time. A missing header is the expected result, not an error.

**Laptop:**

```bash
curl -s "https://s2.<root>/cdn-cgi/trace" | grep -E 'colo|loc'
```

`/cdn-cgi/trace` is answered by Cloudflare itself. `colo=` names the PoP that served you — `CNX`, `BKK` or `SIN` in this region — and `loc=` the country Cloudflare inferred. Cloudflare chooses by routing, not by geography, so record what you actually got.

### 7. Measure both paths

**Laptop:**

```bash
python3 latency.py --origin "https://<droplet-ip>/thumb-<id>.jpg" \
  --edge "https://s2.<root>/thumb-<id>.jpg" \
  --samples 10 --insecure-origin --warm-edge > data/latency_measured.csv
head -8 data/latency_measured.csv
```

The origin path connects directly to Singapore and uses `--insecure-origin` because of the self-signed certificate. The edge path connects to the colo. Each row splits the request into DNS, TCP connect, TLS and time to first byte; `total_ms` is the wall time. Ten samples per path are the minimum for a p95.

`--warm-edge` adds a third block, `cloudflare-edge-warm`: ten requests on one kept-alive connection. A browser reuses a TLS connection after the first request, so this block is the viewer's steady state. The `cloudflare-edge` block is the cold start, and the difference is the handshake and connection cost that a cache cannot remove. On the reference network the cold edge p95 sat near 70 ms and the warm p95 near 15 ms; both are below the 145–150 ms origin path. The warm block is the one that can meet the 30 ms target; the cold block pays the handshake that a cache cannot remove.

### 8. Benchmark the instance

**Droplet:**

```bash
cd /srv/s2
python3 bench.py --seconds 5 --workers 1
python3 bench.py --seconds 5 --workers 2
```

The first row is one worker on one vCPU; the second is two workers. On `s-1vcpu-1gb` the two workers share the advertised vCPU, so the parallel rate usually stays close to the single rate; a shared host can also burst above it. After the resize the second vCPU may or may not add a clean multiplier. Record what you get and explain it; the resize's hard proof is `nproc`, `free -h` and the memory recovery, not a CPU ratio. Copy both CSV rows into `data/benchmark_measured.csv`.

**Droplet:**

```bash
dd if=/dev/zero of=/root/testfile bs=1M count=1024 oflag=direct
dd if=/root/testfile of=/dev/null bs=1M iflag=direct
rm -f /root/testfile
```

Write and read a 1 GiB file with direct I/O. `oflag=direct` and `iflag=direct` bypass the page cache, so the MB/s figures describe the disk instead of RAM. Without them the read measures memory and looks unrealistically fast. Record the MB/s lines: this is the disk baseline the resize must leave unchanged.

### 9. Exhaust the memory

**Droplet:**

```bash
free -h
swapon --show
```

Confirm the plan has about 1 GiB and no swap. With no swap, the ceiling is hard.

**Droplet:**

```bash
python3 - <<'PY'
try:
    block = bytearray(1500 * 1024 * 1024)
    print("allocated", len(block))
except MemoryError:
    print("MemoryError")
PY
echo "exit=$?"
dmesg | tail -5
```

Ask for 1.5 GiB on a 1 GiB machine. Python raises `MemoryError` or the kernel kills the process; either result is the deliberate failure. `dmesg` shows the out-of-memory line when the kernel ends the process. Save the output.

### 10. Resize without growing the disk

**Panel:** Droplets → `course-<id>-s2` → **Resize** → **CPU and RAM only** → `s-2vcpu-2gb`. Do **not** tick "Resize disk". There is also a CLI path.

**Laptop:**

```bash
doctl compute droplet-action resize <droplet-id> --size s-2vcpu-2gb --resize-disk=false --wait
```

`--resize-disk=false` keeps the 25 GiB disk and makes the change reversible; a disk can only grow, so growing it is a one-way decision owned by the storage session. The action powers the Droplet off and on. If it comes back off, start it.

**Laptop:**

```bash
doctl compute droplet-action power-on <droplet-id> --wait
```

Run this only if the resize left the Droplet powered off. `--wait` returns when the action finishes.

**Laptop:**

```bash
ssh -i ~/.ssh/do_s1 root@<droplet-ip>
```

Reconnect. The public IP and the disk are unchanged; only vCPU and RAM grew.

### 11. Rerun and prove the recovery

**Droplet:**

```bash
nproc; free -h; df -h /
python3 - <<'PY'
block = bytearray(1200 * 1024 * 1024)
print("allocated", len(block))
PY
```

A 1.2 GiB allocation now succeeds. It is deliberately below the 1.5 GiB that failed: a 2 GB plan reports about 1.7 GiB usable after the kernel's own memory, and the running system already holds a few hundred megabytes, so 1.5 GiB can still be refused. The proof is that the ceiling moved and the disk size is still 25 GiB; it is not a claim that every allocation up to 2 GB fits.

**Droplet:**

```bash
cd /srv/s2
nohup python3 serve.py --port 443 --dir /srv/s2 --cert cert.pem --key key.pem >> serve.log 2>&1 &
sleep 2
curl -sk https://127.0.0.1/thumb-<id>.jpg -o /dev/null -w '%{http_code}\n'
```

The resize powered the Droplet off, so the in-memory server from step 5 stopped. Restart it before rerunning the measurements; the 200 proves it is serving again.

**Droplet:**

```bash
cd /srv/s2
python3 bench.py --seconds 5 --workers 1
python3 bench.py --seconds 5 --workers 2
dd if=/dev/zero of=/root/testfile bs=1M count=1024 oflag=direct
dd if=/root/testfile of=/dev/null bs=1M iflag=direct
rm -f /root/testfile
```

Rerun the CPU and disk measurements. Single-thread throughput is roughly flat; two-worker throughput should now approach twice the single-worker rate because the second vCPU is usable. Disk throughput stays close.

### 12. Fill the notebook and assemble the evidence

Put your latency rows in `data/latency_measured.csv` and your benchmark rows in `data/benchmark_measured.csv`, then open [the lab notebook](notebook.ipynb). The notebook computes p50/p95, marks both paths against the 30 ms target, reads the cache result, checks the resize, and writes `latency_results.json`. It runs offline on the bundled samples if you use the fallback.

### 13. Tear down

**Laptop:**

```bash
doctl compute droplet delete <droplet-id> --force
doctl compute firewall delete <firewall-id> --force
```

Delete both resources. A deleted Droplet stops billing; a detached volume would not.

**Laptop:**

```bash
doctl compute droplet list -o json > data/droplet-after.json
doctl compute firewall list -o json > data/firewall-after.json
doctl balance get
```

Both lists must be `[]`, and the balance proves the account returned to its starting state. **Panel:** DNS → delete the `s2.<root>` record. Keep the project, SSH key, token, zone and spend alert for later sessions.

**API-token extension (reference runner only).** The runner deletes the record and restores the previous SSL mode:

```bash
curl -s -X DELETE -H "Authorization: Bearer <cloudflare-token>" \
  "https://api.cloudflare.com/client/v4/zones/<zone-id>/dns_records/<record-id>"
curl -s -X PATCH -H "Authorization: Bearer <cloudflare-token>" -H 'Content-Type: application/json' \
  -d '{"value":"strict"}' "https://api.cloudflare.com/client/v4/zones/<zone-id>/settings/ssl"
```

`strict` is the value a shared demo zone normally carries. A student zone can keep `full` or go `strict` once it has a trusted origin certificate in Session 4.

## Evidence bundle

Ten items, scored 3/2/1/0:

1. `data/latency_measured.csv` and the p95 chart from the notebook.
2. The `/cdn-cgi/trace` output showing the serving colo.
3. The benchmark table with `lscpu`, `free -h` and `df -h`, before and after the resize.
4. The cache headers for the JPEG (MISS then HIT) and the uncached HTML control.
5. The memory-failure log and the post-resize success.
6. A redacted screenshot of the proxied record and the SSL mode; no token, account ID or email visible.
7. The sanitized command log.
8. The empty droplet and firewall lists plus `doctl balance get`.
9. `latency_results.json` from the notebook.
10. A 150–250-word *what this proves* note.

## Cost ceiling

Under **$0.10**. The run is one `s-1vcpu-1gb` hour ($0.00893) and one `s-2vcpu-2gb` hour ($0.02679) at most; the measurement itself takes minutes. The domain is paid once in S1 and is not part of this ceiling.

## Teardown list

- Droplet `course-<id>-s2` deleted.
- Firewall `course-<id>-s2` deleted.
- DNS record `s2.<root>` removed.
- No volume, snapshot or reserved IP created.
- Project, SSH key, scoped token, Cloudflare zone and spend alert kept.

## Failure and recovery

| Symptom | Likely cause | Fix |
|---|---|---|
| `dig +short s2.<root>` returns the Droplet IP | Record is not proxied | Edit the A record and enable the proxy |
| `curl https://s2.<root>/...` returns 522 | Edge cannot reach the origin | Check `serve.py` on 443, the Droplet's firewalld, and the cloud firewall's 443 rule |
| The resize returns 403 | The token lacks `droplet:update` | Regenerate the token with the S2 scopes; this is the step-1 lesson |
| The second JPEG still says MISS | The cache key changes, or the colo changed | Use the same URL; read `cf-ray` to see the colo; the edge may not have stored it yet |
| `MemoryError` never appears | The allocation touched swap or a larger plan | Confirm `free -h` shows ~1 GiB and `swapon --show` is empty |
| The two-worker rate does not rise | One vCPU plan, or the workload is serial | Recheck `nproc`; use Amdahl's law in the note |

**Fallback chain.** Inactive zone: use the instructor demo hostname for the edge and cache items, and say so in the note; the origin-direct measurement is unaffected. `doctl` blocked: use the Control Panel checklist and record the panel paths. Outbound probes blocked: use the bundled sample CSV and the published measurements. Cloudflare unavailable: measure the origin only and label the missing edge item. The evidence shape does not change; every substituted item is named.

## Notebook

Open [the lab notebook](notebook.ipynb); it loads the bundled samples, accepts your measurements, asserts the shape, and writes `latency_results.json`. The reference run executes it headlessly on Python 3.14.2 with pandas 3.1.1, numpy 2.5.4 and matplotlib 3.11.2; it exits 0 and every assertion passes.
