Backups that actually restore: how we design and test them

Every site we onboard has backups, according to its owner. In practice, around a third of the time the backups turn out to be incomplete, stored on the same server as the site, or silently failing for months. A backup is only as good as the last time you restored from it. The difference between a two-hour recovery and a two-week rebuild almost always comes down to decisions made long before the incident, so this is how we design backups for the sites and applications we run.

Start with recovery objectives, not tools

Before picking software, agree on two numbers with the business:

• Recovery point objective (RPO): how much data you can afford to lose. For a brochure site that changes weekly, 24 hours is fine. For a store taking orders every few minutes, losing a day of orders is not acceptable, and you need point-in-time database recovery measured in minutes.
• Recovery time objective (RTO): how long the site can be down. Four hours might be acceptable for a marketing site; a revenue-critical application might need under 30 minutes, which usually means a warm standby rather than a backup restore.

These numbers drive everything else, including cost. Tightening RPO from 24 hours to 15 minutes can triple storage and tooling costs, so it should be a conscious choice.

What a sound backup design includes

Coverage

A complete backup has more parts than people expect. We list them explicitly for every project:

1. The database, dumped consistently rather than copied as raw files from a running server.
2. User uploads and media, which are often the largest and most irreplaceable part.
3. Application code, ideally reproducible from a Git repository rather than backed up as files.
4. Configuration and secrets, stored separately and encrypted.
5. Infrastructure definitions, such as server configuration or infrastructure-as-code templates.
6. DNS records and third-party settings, which people forget until they need to rebuild.

Location and isolation

We follow a version of the 3-2-1 rule: three copies, on two different types of storage, with one off-site. For web properties, the critical addition is isolation. Backups should live in a separate account with separate credentials, so that an attacker who compromises the hosting account, or an engineer who deletes the wrong resource, cannot also delete the backups. Where the storage provider supports it, we enable object lock or immutability for at least the most recent 14 days.

Retention

A common default we use: daily backups kept for 30 days, weekly for 12 weeks, and monthly for 12 months. The long tail matters because compromises and data corruption are often discovered weeks after they happen. If your only backups are the last seven days, you may have nothing clean to restore.

If you have never restored a backup, you do not have backups. You have files you hope are backups.

Restore drills, every quarter

Testing is the step almost everyone skips, and it is the one that matters most. Each quarter we pick a backup at random, restore it to an isolated staging environment and check it properly:

• Time the restore from the moment someone starts to the moment the site works. Compare it against the RTO.
• Verify content by checking a handful of recent records: the latest orders, the newest posts, a recently uploaded file.
• Test functionality, not just the home page. Log in, submit a form, run a test checkout.
• Write down every manual step. Anything that required tribal knowledge goes into the runbook.

The first drill on a new client almost always finds something. Recent examples: a backup job that excluded uploads because of a size limit set years ago, database dumps that were silently truncated at 2 GB, and a restore that worked but took nine hours because nobody had documented the search index rebuild. Each of these would have turned an incident into a crisis.

Monitoring the backups themselves

Backup jobs fail quietly. We monitor three things and alert on any of them:

• A job did not run, or did not report success, within its expected window.
• The backup size changed sharply. A 40 percent drop usually means something is missing; a large jump can indicate an attacker staging data or a runaway log table.
• The newest restorable backup is older than the RPO.

These alerts go to the same on-call channel as uptime alerts. A backup failure is not an emergency, but it should be fixed within a working day.

Trade-offs and costs

For a typical content site, a robust setup with off-site, immutable storage and a year of retention costs a few dollars to a few tens of dollars per month in storage. The real cost is engineering time for drills, around two to four hours per quarter per site once the runbook is mature. For larger applications, managed database point-in-time recovery is usually worth paying for rather than building your own.

What we avoid is relying solely on a hosting provider's built-in backups. They are useful as one layer, but they share an account and a failure domain with the site, and restore options are often limited to the whole server.

Our backup and recovery service sets this up end to end, including the runbook and the first drill. For custom applications running on cloud infrastructure, we usually pair it with infrastructure as code, so rebuilding the environment is as repeatable as restoring the data. Ongoing drills are part of our monthly care plans.

Find out if your backups would survive a bad day

We can run a restore drill against your current backups and tell you, in writing, how long recovery would take and what would be missing. Ask for a fixed-price quote for the drill or a full backup redesign.