Skip to content
Practical lab

Website Operations Lab

Six findings, one hour. Order them, pick a read-only diagnostic before any change, and learn what a backup log saying SUCCESS seven nights running proves.

Fifteen minutes, four decisions. Monday's health report for a small WordPress site has six findings and you have an hour. Order them by consequence, certainty and time; choose a read-only diagnostic before any change; and read a backup log that says SUCCESS seven nights running.

Declared sample data

The report and the log excerpts are fixtures written for this lab. No real host, certificate authority or plugin is being described. The report template is a public download.

Evidence

weekly-health-report.md (generated 06:00 today)
# Harbourline Physio — weekly health report — 2026-09-14

| Check | Result | Evidence |
|---|---|---|
| Uptime (7 d) | 100% | monitor: 2,016/2,016 checks 200 |
| TLS certificate | expires in 4 days | issuer R11; auto-renew: enabled |
| Backup (nightly) | 7/7 SUCCESS | see backup.log |
| Contact form | last test 47 days ago | last marker received 2026-07-29 |
| Plugin updates | 6 pending (1 flagged security) | forms plugin 3.4.1 → 3.4.2 (security) |
| 404s (7 d) | 312 (up from 9) | 298 of them: /blog/feed |
/var/log/letsencrypt/letsencrypt.log — last renewal attempts (excerpt)
2026-09-05 03:14:02 Attempting to renew cert (harbourline.example)
2026-09-05 03:14:05 Challenge failed for domain harbourline.example
2026-09-05 03:14:05 http-01 challenge: Fetching http://harbourline.example/.well-known/acme-challenge/…: Timeout during connect (likely firewall problem)
2026-09-09 03:14:03 Attempting to renew cert (harbourline.example)
2026-09-09 03:14:06 Challenge failed for domain harbourline.example
2026-09-13 03:14:04 Attempting to renew cert (harbourline.example)
2026-09-13 03:14:07 Challenge failed for domain harbourline.example
backup.log (last 2 nights)
2026-09-13 02:00:02 job started
2026-09-13 02:00:41 files: 2,114 uploaded (148.1 MB)
2026-09-13 02:00:41 database: skipped (mysqldump: command not found)
2026-09-13 02:00:41 STATUS: SUCCESS
2026-09-14 02:00:03 job started
2026-09-14 02:00:40 files: 2,114 uploaded (148.1 MB)
2026-09-14 02:00:40 database: skipped (mysqldump: command not found)
2026-09-14 02:00:40 STATUS: SUCCESS

Steps

Step 1 — What comes first?

Two findings are urgent. Decide the order and be able to say why.

Evidence: weekly-health-report.md (generated 06:00 today), /var/log/letsencrypt/letsencrypt.log — last renewal attempts (excerpt)

First, in the hour you have:

  1. The certificate. It expires in four days, auto-renew is 'enabled' but the log shows three consecutive failed renewals — so in four days the site goes down for every visitor, on a known date. The security update is second, today.
  2. The security update to the forms plugin. Security always comes first.
  3. The 404 spike — 312 errors is the biggest number on the page.
  4. The contact form — 47 days untested.
Answer and explanation

The certificate. It expires in four days, auto-renew is 'enabled' but the log shows three consecutive failed renewals — so in four days the site goes down for every visitor, on a known date. The security update is second, today.

Priority is consequence × certainty × time. An expired certificate is a total outage for every visitor, and the renewal log makes it near-certain: three failures, four days left, 'enabled' meaning nothing. The security patch is genuinely urgent too and gets done in the same hour — but a vulnerability is a possibility and the expiry is a date.

Step 2 — What is the next safe diagnostic for the certificate?

Observe before you change. The log says the http-01 challenge times out.

Evidence: /var/log/letsencrypt/letsencrypt.log — last renewal attempts (excerpt)

Next step:

  1. From outside the server, fetch http://harbourline.example/.well-known/acme-challenge/test and note what happens (timeout, redirect to https, 404). Then a dry run: certbot renew --dry-run. Nothing changes; both tell you exactly what is blocking the challenge.
  2. Run certbot renew --force-renewal now and see if it works.
  3. Switch the site to HTTP until the certificate is sorted out.
  4. Reboot the server; renewals often fix themselves.
Answer and explanation

From outside the server, fetch http://harbourline.example/.well-known/acme-challenge/test and note what happens (timeout, redirect to https, 404). Then a dry run: certbot renew --dry-run. Nothing changes; both tell you exactly what is blocking the challenge.

The log already names the symptom (timeout on the http-01 fetch, 'likely firewall'). The cheapest next fact is whether port 80 answers from outside at all. A dry run repeats the real challenge without touching the certificate. Both are read-only, both take a minute, and the answer tells you which change to make — usually a firewall rule or a redirect that fires before the challenge path.

Step 3 — What does '7/7 SUCCESS' mean?

Read the log the report links to.

Evidence: backup.log (last 2 nights)

The backup line establishes that:

  1. The files are being copied and the database is not — mysqldump is missing, so every 'SUCCESS' is a backup of the theme and uploads with no posts, pages, bookings or settings. Nothing is known about restorability until one is restored.
  2. Backups are healthy: seven successes, 148 MB each.
  3. The site can be restored to any of the last seven nights.
Answer and explanation

The files are being copied and the database is not — mysqldump is missing, so every 'SUCCESS' is a backup of the theme and uploads with no posts, pages, bookings or settings. Nothing is known about restorability until one is restored.

The line 'database: skipped' is inside a job that reports SUCCESS. On WordPress the database is the site; the files are furniture. This is the second lab on this site where a green backup line hides the fault, and it is not a coincidence — 'the job ran' is the most common thing a backup check actually measures. Fix: install mysqldump (or the platform's export), then restore one backup somewhere and open it.

Step 4 — The contact form, untested for 47 days

The receptionist arrives in forty minutes.

Evidence: weekly-health-report.md (generated 06:00 today)

What is the check?

  1. Submit the form from a phone with a marker (a date-stamped string) and confirm the marked message arrives in the inbox it is supposed to reach. Record the time both ways.
  2. Fetch the page and confirm it returns 200 with the form in the HTML.
  3. Update the forms plugin first; the update is a security fix, and it will fix the form if anything is wrong.
Answer and explanation

Submit the form from a phone with a marker (a date-stamped string) and confirm the marked message arrives in the inbox it is supposed to reach. Record the time both ways.

A form check is a delivery check. The page returning 200 and the plugin being current both leave the actual question open: does a message sent now arrive where a human reads it? A marked submission answers it, takes two minutes, and — if it fails — is the most important finding of the week.

The corrected version

After: what happened in the hour
07:12 external fetch of /.well-known/acme-challenge/test: 301 → https (a blanket redirect rule fires before the challenge path)
07:14 certbot renew --dry-run: same failure, confirmed cause
07:20 redirect rule given an exception for /.well-known/acme-challenge/ (one line, reverted if needed)
07:22 certbot renew --dry-run: success · 07:23 certbot renew: certificate renewed, expires in 90 days
07:30 contact form: marker OPS-2026-09-14-0730 sent from phone; received in reception inbox 07:30:48
07:35 forms plugin 3.4.1 → 3.4.2 on staging, form re-tested, then production; re-tested (marker …-0741)
07:50 mysqldump installed; manual backup with database; restore to scratch host opened: posts and settings present
08:00 /blog/feed → /feed redirect added; 404s to be re-read next week
Report updated with evidence for each line.

Every change in that hour was preceded by a read-only look that made the change small and reversible. That is what the maintenance guide's weekly checklist is for, and what the report template's Evidence column enforces.

Walkthrough

Four screenshots of this lab, taken from the published page: the start, a deliberately wrong answer, the matching answer, and the corrected version. Every screen is described in the image text.

  1. Step 1 of 4 of the Website Operations Lab: the weekly health report table and the certificate renewal log, with the question of what comes first in the hour.
    1. Step 1: six findings, one hour, order them.
  2. The wrong option 'the security update first' is checked; the red feedback explains why a certificate expiring in four days with three failed renewals is a scheduled outage.
    2. A wrong answer, explained by consequence, certainty and time.
  3. The correct option — the certificate first — is checked, with the green explanation and the Next step button.
    3. The matching answer unlocks the next step.
  4. The end of the lab: what happened in the hour, from the external fetch of the challenge path to the restored backup, the score, Save to My Project and the links onward.
    4. Every change preceded by a read-only look.