Backups you have actually restored: running a quarterly drill

Almost every site we take over has backups. Fewer than half have backups that anyone has restored. When we run the first restore drill for a new client, something goes wrong more often than not: the database dump is incomplete, the uploads bucket was never included, the encryption key lives only on the server being backed up, or the restore takes nine hours instead of the one hour everyone assumed. None of those problems are visible until you try. That is why we treat the restore, not the backup, as the thing to test.

Start with two numbers

Before designing a drill, agree on what the business needs. Two figures define it:

• Recovery point objective (RPO). How much data you can afford to lose, measured in time. A brochure site might accept 24 hours. A store taking orders all day probably wants 15 minutes or less.
• Recovery time objective (RTO). How long you can afford to be down while restoring. Four hours is common for small businesses; revenue-critical platforms often need under one.

These numbers drive everything else. A 15-minute RPO rules out nightly dumps and requires point-in-time recovery or continuous log shipping. A one-hour RTO rules out restoring a 400 GB database from cold storage over a slow link. Write the numbers down and get the business owner to sign off; they are a commitment, and the drill is how you prove you can keep it.

Know what a full restore includes

A restorable system is more than a database dump. For a typical web application, the inventory looks like this:

1. The primary database, with enough transaction logs to hit your RPO.
2. User-uploaded files and generated media, usually in object storage.
3. Application configuration and secrets, stored in a vault, not on the server.
4. Infrastructure definitions, so servers and networking can be recreated.
5. DNS records, TLS certificates and any third-party webhook configuration pointing at the site.
6. Search indexes and caches, or at least the procedure to rebuild them.

Items five and six are the ones most often forgotten. A restored application with an empty search index will look broken to customers, and rebuilding an index for a large catalog can take longer than the database restore itself.

The question is never whether you have backups. It is how long it takes a tired engineer to turn them back into a working site at two in the morning.

Running the drill

We run drills quarterly, scheduled like any other maintenance, with one engineer executing and another observing and taking notes. The steps are the same each time.

Restore into isolation. Never restore on top of production. Spin up a separate environment in a separate network, ideally in a different region or account, which also tests your ability to recover if the primary account is compromised.

Follow the runbook exactly. The point is to test the written procedure, not the engineer's memory. If the runbook says to run a command and that command no longer exists, that is a finding. Update the runbook as part of the drill.

Time every step. Record when the drill starts, when each component is restored, and when the application is serving requests. Compare the total against your RTO.

Verify the data. Check that the newest record in the restored database is within your RPO of the backup time. Compare row counts on key tables against production. Open a sample of uploaded files. Log in as a test user and complete a core journey such as placing an order.

SELECT MAX(created_at) AS newest_order,
       COUNT(*)        AS order_count
FROM orders;
-- compare against production at the snapshot timestamp

Tear down and write up. Destroy the restored environment, which contains real data, and record the results: total time, gaps found, and tickets created.

What drills usually uncover

Across the drills we have run for clients, the same issues come up repeatedly:

• Backups that silently stopped months ago after a credential rotation or a disk filled up, with no alert on failure.
• Encryption keys for the backups stored only on the production server, which makes the backups useless if that server is lost.
• Restore times three to five times longer than expected, usually because of network throughput from archive storage or index rebuilds.
• Missing pieces: a second database for a reporting tool, a cron-generated file directory, or a separate media bucket added by a plugin.
• Runbooks that reference servers, tools or people that are no longer around.

For one e-commerce client, the first drill took just under seven hours against a stated RTO of two. Moving the backups to a warmer storage tier, switching to point-in-time recovery for the database and scripting the search reindex brought the second drill to 72 minutes. Nothing about the backup tooling changed; the difference came from actually rehearsing.

Monitoring between drills

Quarterly drills catch structural problems, but you also need daily signals. At a minimum:

• An alert when a scheduled backup does not complete, not just when it fails loudly.
• An alert when backup size changes by more than a set percentage from the previous run, which catches truncated dumps.
• Immutable or write-once storage for at least one copy, so that ransomware or a compromised credential cannot delete your history.
• A retention policy that matches your compliance needs, typically daily for 30 days and monthly for a year.

These checks fit naturally into an uptime monitoring setup, and our backup and recovery engagements include the first drill and the runbook so your team can repeat it without us.

Find out how fast you can recover

If you are not sure your backups would survive a real restore, that uncertainty is the answer. Tell us about your stack and we will send a fixed-price proposal for a first drill within 24 hours. Book it through our contact page.