blog

How do you know your backups actually work? Try getting something back

Ask most business owners if they have backups and they'll say yes. Ask when they last got something back from one and it goes a bit quiet.

That second question is the one that matters. A backup only proves itself when you restore from it, and the worst time to find out it doesn't work is the morning you need it.

The plain answer

You know your backups work when you've recently restored something from them, opened it, and it was what you expected. Nothing else counts. A green tick in a dashboard tells you a job ran. It doesn't tell you the right things were in it, or that you can get them out again.

The National Cyber Security Centre says the same thing in gentler words. Its small business guide tells you to "regularly test that the backup is working" and to make sure you know how to restore one. Its backup page adds that you should check the backup "contains all your important data".

The ten-minute test

Do this one this week. It's small on purpose.

Pick one real file. Something that changed recently, like this week's invoice spreadsheet or a document someone was working on yesterday.

Restore it to a different place. Never over the top of the live copy. If the backup turns out to be old or broken, you don't want it replacing the good version. Most tools let you choose where things go. With restic, for example, the restore docs show a single file going into a separate folder using --target and --include.

Open it. Not just "the file is there". Open it and check it's the version from last night, not from March.

Note how long it took. Including the time spent remembering where the backups are and what the password is. That bit is often longer than the restore.

If any of that went wrong, you've learnt something useful for ten minutes' work.

The fuller rehearsal

A single file shows the backup isn't empty. It doesn't show you could rebuild the business from it. A couple of times a year, it's worth pretending the server has died and bringing it back somewhere else.

For a website, that means restoring the files and the database onto a spare server or a cheap temporary one, pointing a test address at it, and clicking around. Can you log in? Are last week's orders there? Do the images load? Then throw the test server away.

While you're doing it, write down every step. The notes are worth as much as the test, because the person doing the real restore might not be you, and they'll be doing it under pressure.

Things that tend to catch people out

None of these are exotic. They're just easy to miss until you try a restore.

The files are backed up but the database isn't. Copying a website's folders doesn't capture what's inside a running database. That usually needs a separate dump, and it needs to be in the backup too.

The job quietly stopped. A full disk, an expired password or a changed path, and the backups stop. If nothing alerts you, you find out months later. Check the date of the newest backup, not just that some exist.

The password lives on the thing that died. Good backup tools encrypt everything, which is great until the only copy of the key is on the server you're trying to replace. Keep it somewhere separate, and make sure more than one person can get to it.

The settings weren't included. Some systems need configuration and secret files alongside the data. GitLab is a good example: its backup leaves out a secrets file you need for a full restore, which I covered in the GitLab post.

It takes far longer than you'd think. Downloading a few hundred gigabytes from cloud storage on an office line isn't quick.

There's only one copy. If the backup sits on a drive permanently plugged into the same machine, whatever hits the machine can hit the backup. The NCSC says a backup device "should not stay connected" when it's not in use. Ours go to a separate box every night, and a copy leaves the building as well.

Checking the backup itself

Some tools can also check their own backups for damage. restic has a check command, and its documentation suggests running it regularly. Adding --read-data-subset lets you check a slice of the stored data each time rather than downloading the lot.

Worth scheduling, but it tells you the backup is intact, not that it's the backup you needed. If you're still choosing a tool, I've compared restic, Borg and UrBackup.

What to do this week

  1. Restore one recent file to a different folder and open it.
  2. Check the date of your newest backup.
  3. Make sure the backup password is written down somewhere other than the server.
  4. Put a fuller rehearsal in the diary for the new year.

If the backups belong to a bespoke app rather than a standard website, the restore steps are usually particular to that app. That's the sort of thing the team at Dev Partners include in their support and maintenance work, where backups get tested by actually restoring them.