Babarchive: checksum and verify directory tree archives

babarchive

Babarchive is a system to manage babarchives, checksummed directory trees that can be validated. It is designed to preserve digital archives of static content that are intended to last for decades or more. Its goal is to detect random corruption in data at rest, or in errors introduced during data copies.

Pre-built versions for RPM (Fedora, CentOS, REHL): see https://copr.fedorainfracloud.org/coprs/johnh/babarchive/

• babarchive-1.8.tar.gz [SRPM] (2024-02-16); current release

  Handle files with spaces in them, and remove the test suite from archives to check.

• babarchive-1.7.tar.gz [SRPM] (2023-11-16)

  additional release engineering to update the spec since f32 for f39

• babarchive-1.6.tar.gz [SRPM] (2020-06-24)

  babarchive_check_one avoids false errors on variants of shasum.fsdb

• babarchive-1.5.tar.gz [SRPM] (2020-06-22)

  babarchive_check_all can now optionally send mail on error.

• babarchive-1.4.tar.gz [SRPM] (2016-01-30)

  Improve systemd support.

• babarchive-1.3.tar.gz [SRPM] (2016-01-30)

  Add systemd support.

• babarchive-1.2.tar.gz [SRPM] (2016-01-26)

  Fix backwards compatibility for sha1sum archives; fix man page install.

• babarchive-1.1.tar.gz [SRPM] (2016-01-26)

  Clean packaging.

• babarchive-1.0.tar.gz [SRPM] (2016-01-26)

  A complete release (manual pages, runs by default).

• babarchive-0.8.tar.gz [SRPM] (2016-01-25)

  First public release.

Philosophy

The overall idea: when saving data, you don’t get what you deserve, you get what you verify.

Babarchive provides end-to-end checks on the validity of a directory tree.

We assume:

1. Data will live on many platforms over its life.
2. Data should stay on-line
3. We will check it regularly to make sure it hasn’t changed
4. You have separate backups to recover from errors. (Babarchive only detects problems.)
5. You will run babarchive_check_all regularly on your file server.

It detects the following problems or events:

1. Bit-flips inside files
2. Removal of files
3. Removal of babarchives to a system (not necessarily a problem)
4. Addition of babarchives to a system (probably not a problem)

It saves two copies of checksums so partial corruption can partially recovered (by hand), and standard tools can be used to assist in partial recovery or verification.

Babarchive has been in use since 1998 for archives of media files and academic datasets. It has detected two silent bit-flips on disks over that time period. As of 2016 it currently protects more than 100 TB of data at USC/ISI.

Manual pages

For more information, please see the babarchive manual pages:

• overview: babarchive(8)
• verification: babarchive_check_all(1), babarchive_check_one(1)
• preparation: babarchive_prep_directories(1), babarchive_prep_one(1), babarchive_prep_topshasum(1)

Packaged Versions

We provide packages for some operating systems:

• Fedora, REHL, and CentOS: see our package in COPR

We would like to support more OSes. If you can help, please contact us.