Do Your Backups, People

One day in 1998, an animator at Pixar watched Woody disappear. First his hat vanished from the file system, then his boots, then the rest of him. Somebody had run the Unix command /bin/rm -r -f * on the server that held Toy Story 2, and the command was doing exactly what it was built to do: delete everything, recursively, without asking.

By the time the machine was unplugged, about 90 percent of the film was gone. Two years of work by dozens of people.

No problem, backups exist for exactly this. Except Pixar's backup system had been failing quietly for about a month, and nobody had noticed. The most recent usable copy was old enough to be useless.

Saved by maternity leave

The movie survived for one reason. Galyn Susman, the film's supervising technical director, had recently had a baby and was working from home. Her workstation at home held a copy of the film, refreshed regularly so she could keep working between feedings.

Her computer was driven back to the studio, plugged into the network, and the file tree copied over. Most of the film came back. Toy Story 2 went on to earn about half a billion dollars at the box office, and the studio got a lesson it has retold ever since: a backup is only a backup if someone checks that it works.

Paper burns too

Losing everything is not a digital invention. On December 15, 1836, the building that housed the US Patent Office caught fire, and with it burned about 10,000 patent documents and 7,000 patent models, the entire record of American invention up to that day. The government had kept no copies.

The restore came from the users. In 1837, Congress passed an act asking inventors to send in their own copies of the burned patents so the office could rebuild its records. About 2,845 patents came back that way. They are known today as the X-patents. The other seven thousand or so are still lost.

The bigger fire came in 1973. Shortly after midnight on July 12, the National Personnel Records Center in St. Louis started burning, and it burned for 22 hours. The sixth floor held around 22 million US military personnel files; an estimated 16 to 18 million of them were destroyed, including about 80 percent of all Army personnel records from 1912 to 1960. There were no duplicates and no microfilm. Veterans applying for benefits found that the government's proof of their service no longer existed, and archivists in St. Louis still reconstruct records from scattered sources today, more than 50 years later.

The cloud loses things

The cloud was supposed to end this. In October 2009 it produced one of its first famous failures instead: the servers behind the T-Mobile Sidekick, run by Microsoft's subsidiary Danger, lost the personal data of an estimated 800,000 users. Contacts, photos, calendars. The Sidekick stored almost everything server-side, so for many users there was no local copy to fall back on. T-Mobile's compensation for a wiped digital life: a $100 voucher.

MySpace topped it in scale. During a server migration, the platform corrupted the files of every song uploaded between 2003 and 2015, around 50 million tracks from 14 million artists. The loss happened quietly and only became public in 2019, after users noticed the music player was gone. The company's statement: there is no way to recover the data.

And in 2017, GitLab showed how thin the line is even when you think you've prepared. A tired administrator, meaning to clean up a database replica, ran a delete on the wrong server and took a 300-gigabyte production database down to 4.5 gigabytes. The company had five backup and replication mechanisms. In the postmortem's own words, none of them were working reliably. What saved GitLab was a manual snapshot an engineer happened to have made six hours earlier. The recovery took about 18 hours, and the company livestreamed it on YouTube while thousands watched.

A backup you can't read

There is a second way to lose data: keep it perfectly safe in a format nothing can open.

In 1986, the BBC celebrated the 900th anniversary of the Domesday Book, England's great land survey of 1086, by building a digital version of Britain. About a million people contributed photos, maps and local descriptions, all stored on laserdiscs that required one specific setup to read: an Acorn BBC Master computer wired to a special Philips laserdisc player.

By 2002, sixteen years later, working players had become so rare that the discs were nearly unreadable, and it took a university emulation project to rescue the content. The original Domesday Book, ink on parchment, was more than 900 years old and still perfectly legible.

Three, two, one

The standard answer to all of this fits in three numbers. The 3-2-1 rule: keep three copies of anything that matters, on two different kinds of storage, with one copy somewhere else. It was popularized not by an IT department but by a photographer, Peter Krogh, in a 2005 book about managing digital photos. US government security agencies now recommend the same rule.

There is even a holiday for it. World Backup Day is March 31, deliberately placed one day before April Fools'. Its founder started it in 2011 after reading a post by someone who had lost a hard drive and wished someone had reminded them.

Consider yourself reminded. The pattern in all these stories is the same: the loss was silent, the backup was assumed, and the copy that saved the day, when there was one, existed half by accident. Woody survived because of maternity leave. The X-patents survived in inventors' desk drawers. Your files deserve something a little more deliberate.

The full story of the Toy Story 2 rescue is told firsthand by Pixar co-founder Ed Catmull in Creativity, Inc. * (affiliate link), his book about how the studio works.

The timeline

• 1086

  The Domesday Book is written on parchment. It is still readable today.

• 1836

  The US Patent Office burns down. About 10,000 patents are lost; the government kept no copies.

• 1837

  Congress asks inventors to send in their own copies. 2,845 patents are restored as "X-patents."

• 1973

  Fire destroys 16 to 18 million military personnel files in St. Louis. No duplicates existed.

• 1986

  The BBC Domesday Project stores a digital snapshot of Britain on laserdiscs.

• 1998

  A stray delete command erases about 90 percent of Toy Story 2. The backups had been failing for a month.

• 2002

  The Domesday laserdiscs are nearly unreadable; an emulation project rescues them. The 1086 original is fine.

• 2005

  Photographer Peter Krogh popularizes the 3-2-1 backup rule.

• 2009

  The Sidekick outage wipes personal data of around 800,000 users. Compensation: a $100 voucher.

• 2011

  World Backup Day is founded: March 31, one day before April Fools'.

• 2017

  GitLab deletes a production database. Five backup mechanisms, none working; a six-hour-old manual snapshot saves it.

• 2019

  MySpace confirms it lost every song uploaded from 2003 to 2015, around 50 million tracks.

*Affiliate link. As an Amazon Associate I earn from qualifying purchases.